Asking the right data questions

Delta Analytics is a 501(c)3 non-profit based out of San Francisco serving social sector organizations around the world with pro-bono data consulting services since 2014.

This article is the second in a series of resources by Delta Analytics on how social sector organizations that are getting started or reevaluating their data efforts can more effectively integrate data with their organization and mission. The first article, “Data Strategy for Nonprofits: Why” dove into the prevailing reasons why data is becoming increasingly important for nonprofits to effectively use and maximize their impact. This second article dives into how a nonprofit should go about asking the right data questions and identifying where they have a current gap.

The purpose of our series of articles is to provide practical, actionable insights that nonprofits and data experts could start using to help nonprofits make better decisions with data. Along with delving into some of the concepts from an abstract point of view, we also want to provide relatable examples and effective toolkits. For any questions or would like to find out more about Delta’s services, feel fill out an inquiry here.

Identifying the questions

Any effective data strategy must begin by identifying the most important question you want to answer. These are linked to the Theory of Change, a framework that identifies the outcomes that your organization has the best chance of effecting. The question should be designed to be measurable, clear, and actionable. For example, the following organizations have these key questions:

• Doctors Without Borders: Are we delivering adequate medical aid to people affected by crises (e.g., conflicts, epidemics, disasters) or exclusion from healthcare?
• American Red Cross: How many lives have been saved from the blood donated by our donors?
• Feeding America: How many people have we been able to provide food for this year?

For BUILD, the question to answer is are we improving the academic and professional outcomes of our students? BUILD, one of Delta’s 2017 grant recipients, is an organization that offers students from under-resourced communities a four year entrepreneurship experience that helps students graduate high school and prepare for better future professional success. For the rest of this article, we will be using BUILD as an example to illustrate the somewhat abstract concepts around data strategy.

The Delta BUILD Team

The answers to key data questions should be resolved by a “North Star metric” or the single, defining outcome metric that best captures the core value that your nonprofit delivers. This is not to say that this is the answer to all the questions that one may have or that there are no other items that one should measure. For BUILD, the “North Star metrics” are on-time high school graduation.

It is particularly important to make sure that the critical data questions you are asking are addressing the social outcome and not program output of your organization. For example, just answering the question of how many people go through your program, how many volunteers you recruit / retain, are not sufficient because they focus only on outputs — but not social impact. You would still need to assess whether those outputs are truly feeding into the social outcomes you want to accomplish through your organization.

The Logic Model

One tool that can be used for distinguishing output vs. outcome is to use a logic model. The logic model helps determine how the inputs of your organization translates to the final social outcomes you’re hoping to achieve. It focuses on how inputs and activities translate to outputs and eventually, social impact Below is an example of the Logic Model for BUILD.

It is important to identify the data questions and “North Star metric” that is most pressing because there are both time and resource constraints. Or else, nonprofits can run the risk of chasing non-critical metrics. Very frequently, the largest constraint for an organization is time, as staff member has a limited amount of bandwidth that s/he could dedicate to each problem.

The process for obtaining data with the right level of quality and granularity is time-consuming. For BUILD, the majority of the program data came from the National College Access Network (NCAN) and was collected over time in Salesforce, and refined over the course of 6 years.

Defining the data availability and gaps

Once you have landed on a set of “North Star metrics” that you want to track and report to show your organization’s social outcome, you will need to brainstorm the steps to finding, collecting, and understanding those data points.

The best place to start is to tap into existing internal and external sources. As data management is a time-intensive process, it is beneficial to tap into what already exists.

• Internal resources: Existing historical datasets could be repurposed for new insights.
• External resources: Governmental organizations, nonprofits, and research institutions have free, accessible datasources that span all different sectors (e.g., agriculture, healthcare, education).

Here are a list of helpful, open-source datasets:

Data.gov

10 Great Nonprofit Research Resources

Forbes — 35 Brilliant and Free Datasources

Springboard — Free Public Datasets

Closing any “data gaps”

If you find that you need to collect more data, you should clearly define the requirements of obtaining that data. When creating a new data project, here are some following items to consider:

• Whether you want to collect qualitative or quantitative data
• The method for collecting (e.g., surveys, using other reports)
• The timeframe for the data
• Sample size
• Owners of the data
• Data sensitivity
• Data storage and reporting method (e.g., Salesforce)
• Potential pitfalls or biases in the data (e.g., sample bias, confirmation bias)

For these, we will go into more detail in following posts.

For BUILD, the next project they want to undertake is to test out their set of “Spark Skills” — skills designed for 21st century success, including collaboration, communication, grit, innovation, problem solving, and self-management. Currently, BUILD gauges the success of “Spark Skills” based on perception surveys from students, mentors, and teachers (e.g., 93% of students report that the programming allows them to learn about problem-solving, grit, and innovation). To expand upon this next year, BUILD is prioritizing a series of more in-depth pilots to better measure and test student growth and proficiency in Spark Skills.

Conclusion

It is well understanding that data is important. But getting started with asking the right question is harder, and often requires working backwards from the most valued outcomes of the organization or identifying “North Star metrics.”