In some places, Christians are not welcome, nor are Westerners. Questions about values may or may not be acceptable. What are we to do in those places? Should we abandon the idea of measuring impact entirely? Each organization will have to come up with its own answer to that question. There is a lot of gray area, but there do seem to be a few guiding principles.

There Is No Silver Bullet!

Every restricted-access context is dangerous in its own way. An action that is acceptable in Cambodia may be suspicious in Indonesia and fatal in Yemen, and vice versa. It is not only that some places are more dangerous than others (though there are definite levels of danger), it is that they are differently dangerous. There are hundreds of different ways that enterprising individuals have managed to establish and maintain an identity in these locations. I don’t believe there is any possible spiritual metric that would be appropriate in all locations. Since having an impact is more important than proving that you have an impact, put the safety of the mission first. Even if your organization has created a standard indicator set, measure the possibilities and risks carefully, and be willing to put aside those indicators that aren’t appropriate.

Measure Your Team

There will be situations where it is impossible to take any meaningful data from beneficiaries without generating unacceptable levels of mistrust. That doesn’t mean that you can’t measure anything. Measuring the process of the your team does not ensure spiritual impact. But collecting data about your team’s process and priorities over time improves the chances of having a spiritual impact. This qualitative data collection should help strengthen your relationships and their faith. I am reminded of Paul’s letters to Timothy: checking in, reinforcing priorities, advising on difficult situations, and offering encouragement. As I wrote previously, the act of discipleship, done well, provides copious information on how things are being done. We just need to be disciplined about collecting it.

Use Local Community Leaders

Change in behavior is always a result of a change in values, regardless of whether those values were explicitly taught. A gift, in most cultures is expected to produce a reaction of some kind. With careful consideration, it is possible to make plausible connections between program objectives and feelings, values, and spiritual ideas. Talking with respected leaders and elders, it is often possible to get a sense of which of these related ideas (if any) might be observed. If ab appropriate cultural connection is made, the community leaders will even help with data collection.

Non-Cognitive Skills

Some contexts are not anti-Western, just anti-Christian. In those contexts, it is worth looking into the secular / scientific work on “non-cognitive skills” being conducted in the field of economics. This research is being spearheaded by the Christian economist community. These instruments might not give the data you want, but they are much more likely to be accepted as non-evangelistic.

Conclusion

There is nothing easy about spiritual metrics in a restricted-access context, and the contexts are too diverse to permit simple generalizations. In these contexts, faithfulness is far more important than effectiveness, and so we do the best we can.

The principles of measuring spiritual impact outline some potential ways forward. However, it bears repeating that “tearing down is easier than building up”. The problems with measuring spiritual impact are obvious. Possible solutions are much more tenuous. These too I will outline as principles that will need to be adapted to a local context.

The Anti-Hypocritic Oath

Collect no data that does not benefit the local body of believers.

This is obviously a play on, and strengthening of, the Hippocratic Oath: “Do no harm”. In light of the many dangers associated with measuring spiritual impact, an organization should be able to state positively the benefits to the local body of believers. Two different actions would be hypocritical in this context. First, to prioritize data over the spiritual impact that is the purported goal. Second, to require metrics from projects that they don’t (and wouldn’t) collect from their own ministries or families. This one, admittedly high, standard puts the good of the project squarely at the forefront.

Disciple!

Process-based metrics are embedded in good discipleship.

The biggest source of information that an organization has is in its regular contacts with leaders and beneficiaries. This is where change is happening! Are we emphasizing the process and following up? Good disciple makers are. Wise disciple makers are collecting that process-oriented data systematically. I can’t stress this enough. Three to five simple questions that you discuss during every meeting provide a wealth of valuable data that is relatively easy to verify over time. Questions like – “Who are you praying for now?” “Were you able to share about your faith with anyone this week?” These kinds of question fit the next dictum precisely.

Measure things that are intrinsically valuable and costly to fake.

There are very few things in this world that cannot be corrupted by faulty motivations. Fasting, prayer, giving, attending worship all can and will be corrupted by faulty motivation. However, there are things that are intrinsically valuable, (almost) regardless of motivation, and these are worth measuring. Especially when the cost of faking the metric is greater than the perceived non-spiritual benefits of faking.

For example, measuring the ability of an oral Bible storying group to retell a story accurately and self-correct is a great measure. Both the required memorization of Scripture and the skill of self-correction require large investments of time and effort. Both are also valuable skills. This combination of intrinsic value and high cost make the metric likely to: 1) benefit the group and 2) be indicative of real spiritual impact. Measuring a group’s ability to apply what they have learned in ways that were not taught is another example. Both could be “faked”, but the group would have to really process the material to do so. Net gain in a way that is different than church attendance, which is, in most circumstances, passive and not costly. A record of serving the poor in a tangible way is another example of a “fake-able” metric with and inherent value and a high cost in time and effort.

Develop Indicators Locally

Spiritual impact is seen in changes in the capacities and behaviors of community members. Since the ultimate goal is the growth of local churches, it makes sense to involve local leaders from the beginning. The history of missiology is littered with examples of missionaries focusing on changing things that were considered irrelevant locally, thereby greatly reducing the impact of the church until local leaders took charge. For example, John V. Taylor writes that the central concern of Ugandan pastors and converts in the early church in Uganda were primarily concerned with slavery, poverty, and learning humility. Missionaries were focused on polygamy, leading to a gospel that felt irrelevant and a quenching of the work of the Holy Spirit.

Lesslie Newbigin said it this way, summing up decades of experience and worship:

Who has the right to decide the ethical content of conversion at any time or place – the evangelist or the convert? The place where the virus of legalism gets into the work of evangelism is the place where the evangelist presumes that he or she knows in advance and can tell the potential convert what the ethical content of conversion will be.

The Open Secret

It seems best then to work together with local pastors to think through the 2 to 3 breakthroughs that they want to see in the next 5 years and what those changes will look like. Then the responsibility for identifying progress is local.

Supplementary – Measure Knowledge

Changes in knowledge is a relatively easy thing to test. It is not itself a great measure of spiritual impact, but it is often a necessary precursor to spiritual impact. Understanding that the Bible says that God loves the poor comes before the more difficult task of believing that fact or living in accordance wth it. Replacing misinformation with truth and and ensuring that the truth underlying belief / behavior changes and then testing for comprehension is perfectly valid and valuable. Unfortunately, too many organizations act as if increase in comprehension is equivalent to increase in faith, evidently forgetting that demons understand many things (James 2).

Supplementary – Ask about Motivations

“Why do people go to church?”

“Why do you (or why don’t you) go to church?”

These are easy questions to ask and provide answers that are impossible to verify. However, you don’t need to know if something is true for it to be valuable. Sometimes it is enough to have insight into what people want you to think. These kinds of questions respond well to triangulation – depth through focus groups and then breadth through questionnaires.

Conclusion

It is clear that there is still a lot of work to be done here. It seems that there are a few metrics that will be generalizable. But developing a measurement strategy in a location seems to require careful consultation with the local body or risk undermining the spiritual impact it wishes to measure. Of course, all bets are off in a restricted-access context. We will examine those issues next time.

Metrics to measure spiritual growth is the holy grail of integral mission, at least among organizations that receive funding from Western \ evangelical donors. Ever since development agencies began trying to develop holistic programs by incorporating spiritual aspects, they have been scrambling for ways to assess the success (or failure) of those initiatives. Last time, we investigated the forces behind the search for spiritual metrics. Today, we are going to look at four fundamental principles and their associated problems. Next time, we will look at ways to move forward.

The Observer Effect

Arguably the most fascinating and disturbing finding of the 20^th century was that photons behave differently if their behavior is observed in the “two slit experiment”. It is much less surprising to assert that the “observer effect” applies to humans as well. From early childhood until old age, people behave differently when their behavior is being observed. Peer pressure is a real thing.

How does this affect measuring impact? Here we can borrow from experience with measuring social impact. The observer effect is relatively insignificant when it comes to tangible outcomes. Bridges, schools, wells, and utilizing new farming practices leave tangible results, which are verifiable. But intangible or ephemeral results are much harder to capture. Measuring school attendance (ephemeral) or changes in attitudes about gender (intangible) are much more challenging and require measuring tangible substitutes (called proxies) and then trying to infer the actual impact to the intangible outcome.

This introduces big problems because of the observer effect. The community is being offered powerful incentives to demonstrate (or report) positive change on issues of interest to the NGO. “Does the male head of the household help with the housework?” “Of course he does, please keep helping my family.”

This is the principle that motivates the “rice Christian” to profess faith in Jesus. As long as the interested organization pays attention and provides rice, the rice Christian demonstrates Christian behaviors. But these behaviors are not indicative of real spiritual impact, are they? The observer effect is a crucial consideration when thinking about measuring spiritual impact, by definition intangible.

The Unknowable Heart

I mentioned earlier that “a comprehensive measure of social impact is practically impossible” Given the complexity of the problem and current limitations in our science and technology, that kind of analysis isn’t feasible. But it may become feasible at some point in the future.

Directly measuring spiritual impact is not practically impossible. It is impossible. Impossible theologically, philosophically, theoretically. It has always been impossible and will always be impossible. Scripture elegantly captured our conundrum more than 3,000 years before the term “metrics” was invented.

1 Samuel 16:7 The Lord does not look at the things people look at. People look at the outward appearance, but the Lord looks at the heart.

As humans, we are limited to seeing outward things. We can see actions, hear words, but only God Almighty can see the human heart, which is the only place that spiritual impact happens. In fact, we are told in Scripture repeatedly that we can’t even fully understand our own hearts!

Jeremiah 17:9 The heart is deceitful above all things and beyond cure. Who can understand it?

People can tell us what they believe, but we are bound by the observer effect. Are they just telling us what we want to hear? The perceptive Bible student will point out James 2 (faith and works) and Matt 7 (good tree good fruit) both directly associate spiritual and behavioral change. This observation gives us some hope, but also leads directly to our third issue.

The Pharisee Problem

The idea that we can see behavior (or life situation for that matter) and assess spirituality is not novel. The Pharisees were particularly good at determining which behaviors indicated a healthy relationship with God. The problem we need to face now is how Jesus reacted to those Pharisees. “You white-washed tombs” comes immediately to mind. More subtly, Jesus very intentionally undermined their system – refusing to wash his hands, reaping on the Sabbath, healing on the Sabbath, hanging out with the undesirables. He also condemned the good actions of the Pharisees because they were done for the wrong reason.

The unknowable heart forces us to use behavior as a proxy, but the Pharisees clearly show that this approach is fraught with danger. What metric can we think of that the Pharisees didn’t already use? How will we deal with the ever-present danger of legalism? We know that we can see something from behavior because that is also scriptural, but what exactly and how precisely are open questions.

How will this kind of assessment affect the community? The organization doing the assessing? We can’t overlook the fact that choosing to evaluate spiritual impact will itself have an impact. That’s just the observer effect writ large.

Measurement Is Not Neutral

There is one final issue that merits investigation. The culture of the development organization is biased towards measurement, as the underlying culture is. The West has a love affair with data! We measure our steps, heart rate, exercise patterns, weight, calories, mood, even blood sugar. It is a paradise for data nerds. But measurement, like every other human action, is not value-neutral. Everyone understands that at some deep, possibly subconscious level. For no one keeps careful track of the monetary value of gifts from their family. It would be “good” data, that might tell you the relative affection of your family members over time. But we know that the act of measuring that data would harm our relationships and insult our family. Measuring something always has a cost, both relational and monetary, and that cost bears careful consideration.

After considering the cost, we must also consider our own motivation. When the Lord commanded a census in Numbers, it was good. But God was deeply offended when David took it upon himself to conduct a census in 1 Chronicles 21, and the judgment was grievous.

Why are we searching so desperately for spiritual metrics? Is it to attract more funding? Are we worried about God’s provision? Do we feel like better data would lead to better spiritual outcomes? Is spiritual development a management problem at heart? I stand with Lesslie Newbigin, the great 20^th century missiologist, and assert that it isn’t. I deal with this issue at length elsewhere, but more important is that we all put our motivations under the microscope.

Next Steps

The goal of this discussion was to consider the very real limitations and difficulties confronted by any attempt to measure spiritual impact, as well as the costs and potential pitfalls. I hope that it has tempered some of the irrational optimism that comes along with the technocratic mindset. With realistic expectations, we can now consider how we can move forward.

Clarity. It’s the biggest difference between corporations and non-profit organizations. Not the profit motive or market forces or greed. The key difference is clarity.

Success for a corporation is profitability over time, unambiguously displayed on the “bottom line” of the P&L statement. Success for a non-profit organization is achieving its mission and vision, or making a meaningful contribution towards that goal. Successful corporations have meaningful and inspiring mission statements, whose purpose is to help those corporations achieve greater profitability, greater success. Non-profit organizations pay close attention to their bottom line as well (revenues vs. expenses, the non-profit equivalent of P&L) because managing that bottom line is an essential element of achieving their mission and vision.

At precisely this point, we see the impact of clarity. Financial accounting is fiendishly complicated for a large transnational corporation. But profit is measured according to strict standards and verified by third parties with a vested interest in ensuring that the rules of accounting are rigorously and consistently applied. Profit is recorded in currency, and the profitability of one company can be compared to any other company in the world.

But how does one quantify social impact? The most complicated accounting scheme is orders of magnitude simpler than measuring social impact. And comparing across sectors? It is not even imaginable. How does one compare the worth of providing a quality education to 100 urban American youth with decreasing the size of the Texas-size island of trash drifting around the Pacific by 50% with teaching peacebuilding in South Sudan (which may or may not prevent a bloody civil war)? Generic units of social good (the non-profit equivalent of currency) are impossible. How then can we measure success?

Make no mistake, there are a powerful set of forces pushing for measurements of impact, both internal and external. Well-intentioned professionals, often forgoing more lucrative opportunities in order to make a difference, want to know that their work is making a difference in society. They want to understand that change so that they can make it deeper, broader, and more sustainable. Non-profits desire to allocate resources to provide the maximum benefit, while minimizing risk and uncertainty. Is addition to the internal forces, there are powerful external forces at work. Charitable donations function as a market, as do grants. Both current and prospective donors demand proof that their funds will be used effectively.

For a variety of good reasons, non-profits are attempting to measure their social impact just as their for-profit cousins measure profit. But the problem is much more difficult. Humans are unique and unpredictable. Give one man $10 and he buys food for his family. Give the same man $10 a week later and he goes on a daylong drinking spree. There are too many interdependencies, too many unintended consequences. The problems are numerous and well-documented. So a comprehensive measure of social impact is practically impossible. But non-profits can capture certain tangible, well-defined impacts, e.g., the increase in profit from improved agriculture techniques or the decrease in maternal mortality after the construction of a new hospital. Measuring these types of outcome is difficult, time consuming, and expensive, but it is possible.

The field of measuring social impact is still in its infancy. It is urgently needed, and vast resources are being invested in its development, but the difficulties are real and undeniable. “I have been struck again and again by how important measurement is to improving the human condition (Bill Gates 2013).” It is the pressing issue in development.

In this landscape, we find the non-profit dedicated to integral mission, the proposition that social and spiritual development can and should go hand in hand. All of the same forces are pushing IM organizations to measure their impact. The only difficulty is that their impact is not only social, but spiritual. If anything, there is even more urgency in measuring spiritual impact. After all, “For what is a man profited, if he shall gain the whole world, and lose his own soul?” Unfortunately, the difficulties surrounding spiritual impact are far greater than those around social impact. The next few posts will attempt to address the issues and possibilities inherent in attempting to measure spiritual impact.