How do you measure something invisible, like inclusion? How do you pave the way for accountability and change, inside organizations, both big and small? Stela Lupushor from ReFrame.Work and Paolo Guadiano from Aleria talk to us about the next level of workforce analytics that goes beyond diversity indicators, and how the latest research shows how looking at exclusion dynamics is one of the keys to unlocking new quantifiable ways to measure inclusion.
FANNY: The hard things to measure are often the most important things to measure: happiness, culture, maturity, commitment, motivation, passion, quality of life, creativity, value of life, sustainability, teamwork, collaboration, morale, engagement.
How do you make the seemingly immeasurable become measurable? How do we find evidence and specificity to be able to measure inclusion in a team, an organization, or a community?
MINDY: And if diversity indicators companies are currently using aren’t enough, what are better ways to measure how inclusive your culture is? How can you more tangibly measure change? How do you create accountability without fear in the workplace?
In this episode, we speak to Stela and Paolo, trailblazers in the inclusion metrics and cultural transformation domain, to get you a glimpse of the latest ideas in this space.
FANNY: Stela Lupushor manages a team at Reframe.Work that helps companies rethink how to create inclusive workplaces through the use of technology, human-centered design, people analytics, and future thinking. She is the founder of an amazing community, a nonprofit expanding the work horizon for women over 45, and she leads workplace transformation at companies like Fidelity, TIAA, IBM, and PwC driven by technology, analytics, and HR.
MINDY: Paolo Gaudiano is an Adjunct Associate Professor at the NYU, Stern School of Business, Executive Director of the Center for Quantitative Studies of Diversity and Inclusion at the City College of New York, Founder and CEO of Aleria, a “diversity tech” company that helps organizations plan, execute and measure the impact of diversity & inclusion initiatives, and a Forbes contributor on “the Complexity of Diversity.” Paolo’s work is centered around not only how people and businesses think about diversity, but also what they do about it.
FANNY: Let’s start with why inclusion is so difficult to measure. Why aren’t companies measuring inclusion? And if inclusion is so hard to measure, what kinds of proxies are being used to measure it instead?
MINDY: I really loved how Paolo put it. When inclusion is there, it’s completely invisible. But when it’s not there, it’s one of the first things you notice.
PAOLO: This is very exciting, and I’d love to talk about this topic. And in particular, the question that you just asked is very meaningful because through our work, we realized that inclusion is invisible. And I like to use an analogy with health care, which is that when you introduce me, you can say this following his healthy and if you had seen me well, but now we’re only talking on audio. But if you’d see me a few months ago, you would have noticed that I had a big brace on my shoulder. You might have asked me what happened. And the fact is that we don’t notice when we’re healthy, we notice when we’re unhealthy and in a similar way. We don’t notice when we’re included, we notice when we’re being excluded.
That is a couple of profound implications. The first one is that it is incredibly difficult for those people who are the most included to recognize that exclusion exists and to even know that that happens in the organization very much as a healthy person tends not to know a lot about diseases. And then the second implication that helped us to develop some of the work that we’re doing is to recognize that when you go to a doctor’s office, they don’t just ask you, how healthy are you? On a scale of one to 10 they give you this questionnaire is where they ask you about symptoms. And the reason they do that is because symptoms tend to be representative of certain diseases in a similar fashion. When people talk about inclusion, there are ways and we’ve developed a way to look at specific categories of exclusion that represent clusters of situations that make people feel excluded in an organization.
And what I see referenced in the second part of your question is that very often people, when they measure inclusion, they stop a very high level questions like How included do you feel? How satisfied are you? And we feel that that’s really not irrelevant. It would be just like asking somebody else for you. So we were very excited about it and found a way by drawing on the health analogy to recognize that inclusion isn’t feasible, but that it can’t be measured, but looking at situations that make one feel excluded.
FANNY: Stela also brought up a really great point. Inclusion is in the eye of the beholder. Inclusion isn’t something we can observe from the outside. It’s what is felt by the individual on the inside. It’s a lived experience.
STELA: It is also very personal and it shows up in so many ways, in some cases is driven by the culture in which you grew up, the values that you personally hold as an individual in the biases because we all have biases, whether we know or are aware of them. That’s a different story. The expectations we have from that workplace and the environment in which we want to be included from proxies perspective, obviously, a lot of companies are still at the kind of early stages of the measurement system or evolution. Many start with just plain diversity numbers, how many people do have and they revert to the traditional numbers or categories that you are expected to report to the government on race, gender that are on disability. And increasingly, we see a few more statistical groups that we pay attention to, such as self disclosing what your sexual orientation or sexual preferences, whether you have different nationalities, the languages you speak. So there are a lot more nuance that can help you inform whether a first of all, your organization is diverse enough, and then you can begin to infer whether they feel included.
Based on the survey results engagement survey results, you typically have a question of whether you feel you belong increasingly in analytics world, though more mature organizations are starting to look at organizational network analysis. So this is either actively looking at the running a survey and actively asking people who they work with, who they interact with, who they go to find out, answers, what they need to etc.. Additionally, there is a positive way of looking at the network through their meeting invites who gets invited to what meetings through the email exchanges through the pings. So while may not necessarily be 100 percent accurate data set, it allows you to really start looking at how the communication flows. Who is being invited versus not? Is there a team that, more likely than not, the person gets excluded from being of the attending meetings and then not a manager to involve that person? So there’s increasingly more power that you can give to the data to inform and make decisions differently. But again, it’s really difficult to infer the sense of. Individual belonging and inclusion based on the data alone.
MINDY: When it comes to making someone feel excluded at the workplace, there will be times it might be completely unintentional. When we asked Paulo if there were any examples in his assessments, that most leaders might not think of, when it came to moments where people felt excluded at work, he said that two things that came up multiple times when people were asked about times they felt excluded at work, and they were (1) conversations with references to sports and (2) being left out of meetings. That’s why it’s so important for all of us to build a habit of being curious about the people around us, to really get to know them, to connect with them.
FANNY: That’s why it’s so important for companies to also have ways to catch themselves and develop inclusive awareness. Exclusion is like a chronic disease. You can get so used to living with a chronic disease – used to living with that pain or living without inclusion.
But there are symptoms that you can identify to help you see when there is a problem, and Paulo’s work on the impact of exclusion dynamics and how to make measuring inclusion more tangible and quantitative touches on this so perfectly.
PAOLO: So we’d be happy to comment because in a way, actually, we have to be very careful because inclusion. I agree that generally it is viewed as a kind of a soft, fuzzy thing. But in reality, what I am very happy about is that we developed a unique way to measure it in a much more tangible way. And it boils down to the fact that inclusion I’ve actually heard this said about racism. Racism is not about attitudes, it’s about behaviors inclusion in general. So every form of exclusion is not about attitudes, it’s about behaviors. And while yes, it ends up impacting the way that you feel, the fact is that it comes down to what happens to you day in and day out in your work. And what we found is that one way to quantify that and bring it to a level that is at least as good a diversity. In fact, I would argue that it’s a much better way of measuring what a company’s organization and diversity is to ask people to these interactive workshops, specifically to share in a very confidential, completely anonymous way situations in which they felt excluded. And then we ask them to tell us what was the source of that exclusion? Was it a policy? Was it your leadership? Is it your management? Was it your peers? Was it your clients? And we also ask them, Is this something that happens on a daily basis? Is something that happens on a very infrequent basis.
When you do that, you get incredibly detailed information, and it’s something that you can measure frequently like diversity, which only changes over many years because you brought up a very good example for, for instance, effect Stella was talking about meetings. One of the biggest categories that we see as being problematic in almost every company we talk to is organizations or people who feel within organizations feel disrespected because they either they were not included in meetings in which there should have been included or they’re included. But then they’re the ones that are always asked to get the coffee or to take the notes or when they speak up. Their answers are not heard until the white guy sitting next to them raises his hand. And all of a sudden it’s like, Oh my God, what a great idea. So it is in fact possible to get away from the soft, fuzzy definitions of inclusion, making it much more complicated. And when we’ll talk about dry, which is really ninety nine point nine percent D, it’s actually possible to bring that eye to the same level as the D and really measure it in a tangible way.
PAOLO: A lot of the work came out of literally 30 years of developing computer simulations that capture the behavior of individuals to then understand what happens in the entire organization. And it’s a recognition that a lot of times we look at statistical data. But the problem is that statistics really hide individual experiences. By definition, that’s what they’re designed to do. We developed a series of simulations that are a bit like those of you that play video games like The Sims or SIM City. I used to love to play this game called Roller Coaster Tycoon, where you’re literally controlling the behavior of individual people to see if you can achieve certain high level objectives like make a successful amusement park, right? So what I did is I developed the simulation years ago when I started to work in this. That takes a very simple idea. You have a company that has four layers of people from entry level managers and executives and two kinds of people, men and women. Not that I don’t want to focus just on gender binary gender, but there was for a simplified concept. And what we realized was that if you take a company and you assume very basic ways in which they’re promoting people, firing people, et cetera, you find that under most circumstances the company will stay very gender balanced if it starts gender balance. But if you introduce a form of exclusion that favors men during the promotion process, not that that ever happens in your companies, right? Then we were asking the question what would happen? And when we did that, we saw something astounding. We saw that the companies that came out, the sheet that came out was very similar.
In fact, we could match it to data about gender imbalance in real companies where you get a predominance of men at the top and then fewer and fewer men as you go towards the bottom. And you know, paradoxically, they have more women in the in the entry levels, even though you’re hiring the same number of men and women. And what became a wake up call for me was that one day I was showing this to a company that said, Wow, this is so fascinating. Can you build this for my company? And what I realized was that just knowing the diversity makeup of the company was completely useless. Why? Because the problem is not that you have a man and a woman or a person with a disability or a member of the LGBTQ community. It’s understanding on a day to day basis. I’m a certain person with certain characteristics, and I’m surrounded by people who are behaving in a certain way toward me. What’s going to be my experience and how will that impact my productivity? And that’s when we realized that what we needed to ask question with the questions that we needed to ask companies was not how many black people, how many women, how many gay people. Well, we had to ask them was, what are the experiences that these people have? And that’s what we realized that, hey, we’re basically measuring inclusion. So the simulation was profoundly impactful because it showed us that we needed to understand behaviors, we needed to understand what makes people feel included and excluded, and that diversity alone is insufficient to help us understand whether a company is going to perform better or not and whether people will be more inclusive.
MINDY: Paulo makes such a great point about, how we’ve gotten into a habit, of looking at an organization’s health, through the lense of statistical data, but statistics, by definition, are designed to hide individual experiences. And Stela really helps us put this in perspective. There are a lot of subtleties to how you build situational awareness, around inclusion at your organization, in full color. That traditional broad strokes approach, by itself, doesn’t work when it comes to creating inclusive workplaces.
Stela says diversity is really only a lagging indicator for inclusion, and how nothing really replaces the value, of making room for human connection. Especially as managers, nothing beats caring enough to ask somebody, how they’re feeling or what they think, as part of this journey to do better by the people we work with. She also talks about how putting both humans and technology at the center of DEI efforts, is really the key to developing the level of self-awareness and organizational-awareness you need for cultural transformation, at scale, to give you that strong balance of both active and passive ways to keep an eye on those signals and classifiers to be able to identify exclusion as and when it’s happening.
STELA: I think the tricky part was measurements. Once you start measuring things and it’s explicitly communicated and people are aware it creates this Hawthorne effect that make people behave in a way you either expect them to behave or they they may not necessarily take the the behaviour or give you the data through the channels you expect them to. So for example, if you’re going to measure the number of meetings, people get invited them, the manager knows maybe they will now over. Over ere on the side of inviting everybody, however, the decisions that are being made is made outside the meetings that they invite. So the important part is number one, if you measure and you communicate, take actions right and those actions have to be communicated publicly about what you’re you’re doing to alleviate the exclusion behavior or rectify some of the the impact those had. Additionally, you have to really create a culture of psychological safety and make people feel comfortable to say I. I felt excluded in this situation, and I know I’m not going to be penalized for erasing that problem. The easiest solution is to help managers showcase those behaviors because they will role model for others to behave the same way. And just ask simply what would make you feel included today or have this week? Have you had any moments that you felt that you weren’t part of the team or you didn’t get the chance to contribute or you felt excluded? Sometimes a simple permission to give you the answer and know that somebody is going to care and do something about it will have a huge, huge impact on the sense of that particular individual.
STELA: The diversity segment may not necessarily give you all the answers. They do, however, enable you to see more of a lagging indicator because if you are, if you’re ending up with a very top heavy male population. More likely than not, you need to look at the earlier intervention points where you make decisions about who gets promoted, who gets rewarded and who gets put on high potential development programs. Who gets invited to even some of these senior leadership and have mentors or has sponsors in the organization.
I’ll give a little example from from my analytics past, I think the data has power only when there is an interested leader to do something about the outcomes and pay attention to. So you have to start there. And if the organization is only able to say, Well, we’ll do this annually and this is where we’ll start, well, I would take that over not doing anything ever. Mm hmm. On the other extreme, I think we are getting into an era of where data is everywhere and organizations are getting more sophisticated. By investing in this space, they are putting some of these tools to work.
In one of my past companies, we build a sentiment analysis tool and we were looking at all the chatter that the employees were making online on the internal social network, and all of that was made was the public awareness. So none of it was identified. It was all anonymized at the point of storage and were able to infer conclusions on who talks about X, Y and Z. And is that a positive or negative expression? And we can see globally how some of the topics are starting to emerge, whether we have hot spots that we need to start intervening. And then, of course, that led to a different clusters of functions inside the organization to get interested and say, I would like to know how people in sales functions feel.
And do we see a negative spike of potentially leading to a spike in attrition? So more specific questions that we’re the businesses cared about, and that data was refreshed every five minutes and it was kind of near real time that you could keep the finger on the pulse of the organization. Again, that is a passive data that you can detect certain signals and then you have to supplement that with kind of an active probing. And we have a company wide poll on a weekly basis. So once we see certain trends but may point to a negative sentiment, then you can say, why is this happening? And then it can have a meaningful trying to get to the bottom of those issues. So you really need to have both active and passive way of looking at the data to make certain inferences. And then, of course, if it’s a bigger issue, systemic issue that is bubbling to the top, you really need to have certain interventions and focus groups and conversations to get to the bottom of what’s driving some of the dissatisfaction.
FANNY: And how should what a company measures evolve as they step through those phases of making their workplace more inclusive? What is the most practical cadence for a company to take these measurements?
PAOLO: It’s really a great question, and I completely agree with everything that Stella was making, and I love this kind of the two different perspectives that we bring to the table, right? You can tell that she understands H.R. processes and some of those things much better than I do, right? I come at it from an unusual kind of oblique angle. I would say, I’m going to answer the second part of the question first.
I have found that there is no such thing as a universal answer to virtually any question that you ask about practices or processes or cadence. Right. So something that may work for one company may not work for a very different company. One of the things that I would recommend is that when when you think about inclusion or diversity, it doesn’t really matter.
There’s going to be there are going to be some issues that are more prevalent and we certainly see that when we measure inclusion. We provide one of the probably fact the very first results that we give to companies is a rank ordering of which of our nine categories of inclusion are most prevalent in the comments that we get from people. And there is almost always one that stands out. Typically, it’s respect or what we call workplace interactions, which is literally those one on one. So it’s the interestingly, it’s the day to day activities. So our recommendation of organizations is don’t get overwhelmed. Start with the low hanging fruit, start with, you know, here is a problem that is impacting and we can we can do a demographic breakdown and we can say, Look, this particular issue is very pervasive in your organization, and it impacts virtually virtually every form of identity in your organization.
So start here. Make sure that you come up with a way of addressing the specific problems that you see. And then go on to the next one and the next one and the next one.
The cadence, the one thing that I’m finding that I really love, although I don’t want to contradict what I said earlier. But people came to us initially saying, Can we do this again in a year? And that’s because that’s the way people think about it, right? Doing surveys is a pain. Nobody wants to do those engagement surveys. So what do you do? You do it every 12 months and even then you only get maybe 15 percent participation. One of the things that we’re showing people is that if you find that there is a problem with the aspect of inclusion, for example, managers do not know how to include the right people in meetings, and then they always ask the same people to take notes or to bring the coffee. That’s something that you can educate the managers on immediately, and you can then literally measure the impact two weeks later, a month later. And so the answer the cadence really depends on what we find. Some things can be done very quickly can be measured again very quickly. Others really require a longer amount of time. And certainly when you’re working with diversity metrics, that’s where you run into a big problem that you can’t really see changes in diversity in a matter of weeks. You really have to wait a very long time.
FANNY: So how do you get people to give you that information? There are so many surveys and so many times that people are asked for information. How do you get people to give you the data that you need to make sure leaders can track their progress toward making micro- and macro- changes to reduce experiences of exclusion?
PAOLO: Currently, what we do is we do not do surveys. We do interactive workshops that are very engaging, and during the workshop, we do a live activity using a website that we developed where people literally share experiences in a very anonymous way. And then we kind of go through them and talk about them. As we do that, we’ve captured all the data that we need and we can then crunch that and create reports. We’re currently working on an app that will be in the hands of users, will basically third party information where we will encourage users on a daily basis or very regularly whenever they feel something that has made them or they have an experience that has made them feel excluded. To let us know through the app and the app serves two purposes. One is to provide them guidance so that if something happened, they can look up information about who do I talk to in that company and what are some of the remedies that I have? And then secondly, it’s giving us real time information so that presumably if there are incidents coming up during meetings and you have meetings all the time after you do an initiative, you should immediately see a decline as long as people are using that app. So that’s a dream in the future is to make this a real time continuous measurement.
MINDY: There’s no silver bullet or one-size-fits-all approach to inclusion. Every organization’s context, culture, and climate, is different, and one of the best things any organization can do, is to recognize that paving the way for inclusion, is both an art and a science.
You need to have both, the heart, and the technology, to create pathways for change, because at its core, inclusion is a journey we have to promise each other, to walk on together, to create a foundation for future generations, and you can’t have one without the other.
We need to start measuring, the invisible – and there are people like Paulo and his team, working on tools, to make it easier to measure the invisible.
But how do we hold leaders accountable for change, and how can we create accountability without fear?
PAOLO: the fact remains to me, the value of the simulations is that as long as we think of accountability towards diversity and inclusion as a punishment, as a matter of doing the right thing, then there’s bound to be that sense of guilt. And there is bound to be that pressure from the bosses or the leaders to balance what they see as the well-being of the company versus the well-being of the individuals. We need to break that barrier. We need to realize that a manager that systematically misbehaves towards women, towards people of color, toward people with disabilities is creating a material damage to the organization that is worth way more than the potential revenues that, on paper, that managers helping to generate.
STELA: It’s such a great point about connecting to the mighty dollar. It goes from macro view. Yes, policies matter. And if you stop looking at what gets rewarded for what gets punished in the company, you will very quickly see a relationship on the behaviors of exhibited by leaders. So I think if you really want to make an impact, you have to have accountability of the leadership level for what happens to people, right? So if it’s the only thing in the scorecards that the leaders get measured at the end of the year is the revenue they brought in. I suspect there probably will be some of the people issues, but if you have some sort of a people measurement in there, the likelihood is that they will be held accountable and then they will role model.
MINDY: We talked about quantitative ways, we can measure instances, of exclusion and progress, at very large organizations, where their sheer size easily lends itself, to being able to collect, this type of data, but what about, smaller organizations? Let’s say less than 100 employees, where there isn’t the same level of anonymity or psychological safety, if you want to collect, this type of data. What are reliable ways, to measure and monitor exclusion and inclusion, when you can’t go, that quantitative route?
PAOLO: We realized early on that when we were pushed by smaller companies, it’s like, Well, how do we get enough data to preserve the, let’s say, privacy and sort of a sense of safety, psychological safety of the individuals and to provide enough quantitative data to be meaningful? And what we found is that it was actually something kind of interesting. We now have kind of a parallel nonprofit track where we work a lot with very, very early stage startups. And we’ve realized two things. One of them is that just the process of introducing a different way of thinking about inclusion and thinking about inclusion in terms of the exclusive behaviors and how they impact different people differentially is incredibly valuable and where you lose the benefit of the numerical strength. The smaller companies, you tend to have less formal processes and more of an opportunity for people to want to listen. Because let’s face it, when you’re a 20 or 30 person organization, you cannot afford to lose even one person that could be five percent of your company. So we’ve seen an opportunity for leaders of smaller organizations to go through a similar exercise and workshop that we do. But then, rather than thinking of the numerical, the exact numerical quantities to think about it as a learning opportunity, as an opportunity to engage in conversations with the rest of the company and to encourage people to be open and to create pathways for those people to share situations which they felt excluded.
STELA: I completely agree.I feel when you have a small organization, the network density, it’s a lot tighter. So just by the sheer force of introducing the notion or even showing that you care, things will start changing because in this case, it’s more about doing as opposed to learning about it. You already see where the possible negative areas might be. You probably can do easily with a focus group or some sort of island table in a simple conversation or a simple survey. But a lot of it can be just by sheer having a social event and bringing people together say, Hey, we want to be inclusive and we want to learn more about the social aspect of you. And you know, what are your hobbies and what are the projects that you are working outside the work environment or even giving people opportunities to give back and do some volunteering opportunities together? That in itself, what I do is start solving the problem to people to solve include as opposed to try to measure. I think measurements are effective when you really have a huge, huge scope and you need to kind of simplify the message for your leadership or be able to justify investments when you have a small team that’s go to the doing.
FANNY: A common theme we see in how we can better measure inclusion is how we can better measure and facilitate connection both as a means to more effectively gain situational awareness around the state of inclusion inside an organization, but also as a remedy to some of the barriers to inclusion. In many ways, the key to creating more inclusive cultures inside organizations is to first consider how we can help weave a more connected cultural fabric throughout this process.
MINDY: These human connections, are like the building blocks, for inclusion. When you have connection, it creates room, for trust, for communication, for more psychological safety, for transparency, for faith, and good will. Connection too, is, invisible, but you can measure it more tangibly, through interactions. Putting in the effort, to regularly connect, at a human level, in the workplace, goes a long way, because once that connection, is lost, what you’ve paved, the way for … is lots of paper cuts.
STELA: I usually think of this problem is in terms of paper cuts like first time it happens, you don’t pay attention. You know, it’s a moment of exclusion. You just ignore it because it’s not something that registers with you. Second time, you kind of start looking like what happened here on and on and on until it becomes one big festering wound where you say, I’m done, I’m out of here, I’m going to quit. I just can’t deal with this anymore. Or you go and take it to the to the lawyer the opportunity.
I think the more important message here is not to wait until it’s too late. it’s a good practice to have interviews. but not wait until the person decides to leave to ask Why did you leave or have exit interviews? Ask Why? What makes you stay here? Because usually in those conversations, you’ll get a lot of interesting insights much sooner than when you need to make interventions.
And then at the macro micro level, I think. The onus is on each one of us when I think about how I show up and how sometimes I feel excluded. The first question I ask myself is What have I done to either contribute and combat that right is that I’m not responding to certain emails just because the norm of the team is to respond immediately. That may put me in a position of less attention from my manager. Is it that I may not, I don’t know, contribute to the social life of the team? So what is it that individually I can do to make myself more inclusive? That’s number one. Number two, what do I do in the moment when I see exclusion happen? Do I stand up for it? Do I? The way pointed to people that I keep a mirror in front of others. So I think it’s a really integrated challenge, but we all have a part to play in it to to influence it in a positive way.
FANNY: Stela makes a really great point. It’s important for leadership to pave the infrastructure needed to make progress towards inclusion, through tools, technology, policies, and workplace rituals, but it also cannot happen unless we all play our part to make an effort to connect with each other at a human level, make our perspectives seen when we witness exclusion or see opportunities for progress, and ensure our voices are heard through the roles and functions we take part in.
Both technology and humans at all levels of an organization need to work together in this journey to create more inclusive cultures. Let’s walk this journey of measuring the invisible together.
MINDY: Thank you for joining Fanny and I, in this conversation with Paulo and Stela, about new ideas, around how we can measure, inclusion, and pave the way, for accountability and change, inside organizations, both big and small.
