Data is not objective. Power and perspective matter, a lot.
Heather Krause, founder of We All Count, presents a concrete approach to embedding equity in data projects that is immediately relevant to Metro DNA’s Equity Principle + Commitment concerning the Collection, Use, and Dissemination of Data.
This post is a summary of a presentation delivered to the GEO (Grantmakers for Effective Organizations) Community earlier this year and used as part of a program evaluation course in the Nonprofit Pathway Certificate program at Red Rocks Community College. You can also follow Heather Krause on Twitter @datassist.
What we have said on the topic
While data is critical to the work of Metro DNA, we acknowledge that historically the collection of data has not always benefitted, and has sometimes harmed, those from whom the data was collected.
Recognizing the complex history of the use and collection of data and information within marginalized communities, Metro DNA commits to responsible and ethical use of all data and information gathered for collaborative projects. The data will be used to increase equitable access and participation in nature and the outdoors and collected in ways that uplift people with marginalized identities.
What Heather has to say, based on her education and experience
Embedding equity and rooting out bias in data projects should and very much can be a part of each step in any program or project evaluation, quantitative research project, or exploration of data.
Funding
Each project begins with funding. The collection, use, and dissemination of data is expensive and there are lots of opportunities for power to influence results. Developing a funding web can help clarify who is giving and receiving money, influence (decision making power), and data with respect to a particular project. In the example below, what is missing?
In this, quite common, model of data collection there is no citizen compensation or power, even though these individuals are the lifeblood of the project; they are providing the data. Corrective measures could include paying people for their time and creating opportunities for shared ownership of the data and its interpretation and use.
Motivation
Next, consider motivation. Why are you doing the project in the first place? The definition and framing of any project create opportunities to consciously and unconsciously embed power and perspectives. A clearly stated motivation statement can create benefits for both the providers and users of data.
The motivation statement can and should relate to and contribute to a program, project, or organization’s theory of change and logic model. In other words, it should relate to what work is being done, for whom, why, and with what assumptions and understanding underpinning the actions being taken and evaluated for their effectiveness.
Project Design
Project design then proceeds in a three-step process. This process moves us from “why” to “how” we’re approaching a data project.
Envisioning the project without resource limitations and from multiple cultural perspectives, what Heather calls the “Blue Sky” phase can lead to more innovative solutions.
A great example is Native Land Digital, a project that “strives to create and foster conversations about the history of colonialism, Indigenous ways of knowing, and settler-Indigenous relations.” Their map and Territory Acknowledgement Guide offers up an alternative frame for asking and answering geographically explicit questions using pre-colonial geographic boundaries defined from a multitude of cultural perspectives. We have the capacity – right now – to explore data through more than just the lens of the nation, state, county, city, and census tract if we choose to do so.
Next, how we define our research questions should be considered. Where are we placing the responsibility for change? How else could this research question be phrased?
If the data project asks this question, a student’s response may be healthy, but the environment in which they are living may still not be healthy (e.g., bullying is still rampant). Likewise, we often look at “equity” using an image like this. How else could we envision both the current reality and desired future?
What if we depict the individuals as equal (the same size) and the system as unequal?
Then, not only is our illustration of the situation closer to reality, the differences and barriers are institutional and structural as opposed to individual…
… and the change we seek to affect is systemic rather than individual. Again, neither approach is “wrong”, but the framing of the project and the phrasing of the research question or questions has a fundamental impact on the process and results.
Third, and only once you know what you want to ask and why, you design methodologies to collect and analyze the data. Often randomized control trials (RCTs) are considered the “gold standard” for quantitative research, but the standard for what and for whom? RCTs give an unbiased estimate of the average treatment effect (intervention) on a study population. RCTs do not show (in fact, they hide) within-population differences.
This example, from a data project evaluating the effectiveness of a program in rural Bangladesh to increase women’s income, shows that the same data looked at one way show overall success, but when looked at another way, shows significant differences in results for different ethnic groups. With a more detailed methodology, the data show that the program intervention tripled income inequality gap in this community.
Data Collection & Sourcing
Next, comes data collection and sourcing. Are you collecting new or using existing data? If using existing data, do you know who has measured it and how they have define social constructs (demographics like gender, sexual orientation, race, ethnicity, etc.) or other categories into which you will group data? Finding or creating detailed Data Biographies allows us to compare “apples to apples” across datasets and populations. Without documentation and understanding of when, about what, for whom, why, how, and where data were collected, we can draw spurious and potentially damaging or misleading conclusions.
Analysis
Our statistical choices are deeply embedded with power dynamics and world views that directly influence results. You don’t need to be a statistician to embed equity into your work, but you do need to understand and think about:
Denominators. Who is the population being engaged and studied? Whose perspective or point of view is being considered? For example, if you are interested in determining the average size of a classroom, are you asking teachers or students?
The answer depends on who is being asked the question. The answer is mathematically objective, but how the question is asked is completely subjective.
Elements of a Model. What variables are being considered? How are they being analyzed independently and/or cumulatively? For example, if you are interested in who is most likely to have a low birthweight baby, how you consider ethnicity, community and state of residence, individual characteristics, and other variables of concern matters.
Again, there is no one right answer, but what variables the analyst choses to include and ignore influences the results.
Interpretation
Now it’s time to interpret your results. Interpretation is not the same as analysis and it is not the same as communication. What do your results mean? What might you do with your results? Your goals and your perspective influence how you see data and how you use data, making it easy to overstate or understate your case. For example, in a study looking at vulnerability to mental health challenges, interpreting the data from one perspective would lean one to focus resources on students who are black (racial equity lens), or male (gender equity lens), or not poor (economic equity lens). Interpreting the data looking across racial, gender, and economic perspectives, however, yields a different conclusion and leads to different interventions on behalf of the most vulnerable population in this particular sample: poor, white males.
Also, data results are generally NOT generalizable or transferrable from one community or study population to another. This is one important driver behind the Denver Urban Field Station; urban social-ecological studies that have occurred on the East and West coasts do not tend to conform to ecological and social patterns and processes that characterize Denver, Metro Denver, and the Intermountain West.
Communication & Distribution.
Finally, you are ready to share your results and conclusions! There are still choices you can make that will support or further undermine equity.
For example, data visualization “best practices” are not culturally universal; they are based on white, educated, western perspectives and designed to “speak to” individuals who have learned to see and interpret information, shapes, and color in specific ways. It is important to know and work with your audience in order to deliver information that is relevant to them in a way that is understandable to them.
It is equally, if not more, important to check your assumptions and interpretations at this stage. Communicating incompletely or inaccurately what a data project does and does not say, conclude, or mean can have life and death implications for those who may be the target of policy or practice interventions based on the data.
For example, when ProPublic took a closer look at the COMPAS recidivism algorithm, they found that while COMPAS claimed to offer a prediction of how likely an individual is to reoffend, what it really predicts is how likely it is that an individual might come in contact with the police again, be arrested again, and not have the bail money for release. This is a chilling example of incomplete, inaccurate, and racially biased data interpretation, communication, and use.
In conclusion
I certainly have a better understanding of Metro DNA’s Equity Principle + Commitment concerning the Collection, Use, and Dissemination of Data. I hope you do to. Thank you for joining us in exploring how these concepts can guide our approach to the Regional Conservation Assessment and deepen our commitment to equity in pursuit of a Regional Vision for People + Nature.
