Getting Real About Real-Time Evaluation
Evaluators explain a type of program evaluation — and how to make it work for your nonprofit.
Article Highlights:
Exploring the differences between RTE and traditional evaluation.
Have you ever had an evaluation conducted for a program, and then waited months—or even years—for the findings to come out? When you finally got the evaluation report, were you annoyed because you had already changed the program significantly?
You’re not alone! Traditional evaluations emphasize proving whether or not a program has worked. This requires a rigorous study design (with things like interventions and control groups), and findings are typically issued after all data are collected and thoroughly analyzed. That approach can mean a long time frame that makes the results useful for proving something to a funder, but less useful to a nonprofit earnestly trying to improve its services.
Real-time evaluation (RTE)
Fortunately, the field of evaluation has evolved and other forms of evaluation have emerged that emphasize improving programs. Such evaluations can go by these names:
- Formative
- Utilization-focused
- Participatory
- Empowerment
- Developmental
These terms differ in meaning, but they all describe evaluation approaches with one interest in common—producing information that can be used to help make programs work better.
Real-time evaluation (RTE) is the latest form of this evaluation type, and one that is growing in popularity among funders. RTE emphasizes timely evaluation feedback to strengthen program design and implementation. However, RTE isn’t always “real-time” in the sense that information is available immediately. Some types of data — like interviews and focus groups — take more time to process and analyze. To account for this, real-time evaluation reporting can be timed to occur after major program milestones or at key decision points. And in contrast to traditional narrative reports, RTE findings are commonly presented more informally through memos, slide decks, and even conversations.
How is RTE different from traditional evaluation?
Here’s a real-life example that illustrates how RTE differs from traditional formative and outcome evaluations. Juma Ventures (Juma) is a Bay Area nonprofit that serves youth through employment in social enterprises, college preparation, and financial asset building. Last year, Juma started a new program called CollegeSet that combines financial capability training, match-savings for college, and coaching to apply for student financial aid.
A traditional outcome evaluation might look at the success of individual students in the program: were CollegeSet students more successful in achieving academic and career success than they would be if they did not have this program? A traditional formative evaluation, on the other hand, might examine the effectiveness of the different service components with the overall goal of providing information that Juma could use to strengthen the next iteration of the program.
But Juma didn’t want to wait a year or even six months to learn what worked and what didn’t – they wanted to make program adaptations in real time. So we designed a real-time evaluation approach to help Juma do this.
For example, CollegeSet was being implemented by seven different partner agencies. Juma wanted to use evaluation to learn how to enhance these collaborative partnerships and how to ensure that the partners were receiving the support they needed to implement the program. Early in the evaluation, we interviewed Juma staff to hear their perspectives on strengths and challenges of this collaborative approach, and to identify areas where they would like more feedback and input. We used these perspectives to guide our interviews with partner organizations. As a result, the evaluation was able to capture valuable data to help staff craft stronger partnerships and inform the selection process for the next round of partners. Following each data collection milestone, we talked with Juma staff about what we were learning and how this information could be used to further refine the program.
Is a real-time evaluation right for you?
The table below outlines some of the key questions to ask in deciding which type of evaluation is right for you.
|
|
Real-time evaluation
|
Traditional evaluations
|
|---|---|---|
|
What do you want out of evaluation?
|
In-the-moment feedback at critical decision points
|
In-depth analysis in a detailed report, with the clarity of hindsight.
|
|
What types of deliverables do you prefer?
|
Frequent in-person meetings and data summaries.
|
Full report at a defined end point and potentially at mid-point.
|
|
What is the end goal?
|
Getting the program to work as efficiently as possible, as soon as possible.
|
Learning what worked and what didn’t, and using that information to inform the next iteration of the program.
|
|
How much does it cost?
|
May be more costly due to multiple rounds of data analysis and meetings. Since evaluation activities may evolve to meet changing information needs, costs are not always as predictable.
|
Costs are generally more predictable because you know what activities will be conducted at the evaluation outset.
|
|
What are the trade-offs?
|
The analysis will not be as rigorous because in-the-moment feedback cannot achieve the same clarity as hindsight.
|
The analysis will not be available until midway through or after a program’s end. However, with the additional time available, a higher degree of rigor is possible.
|
If you’re contemplating a real-time evaluation, other important considerations are whether your organization has the capacity to respond to real-time feedback and picking the right evaluator. Evaluators who have an understanding of organizational development issues and experience working in the community may be a better fit than academic experts. The American Evaluation Association maintains a searchable directory of practicing evaluators, many of whom have experience evaluating community programs.
What if my funder wants a traditional evaluation?
For funders interested in a “proof” approach, a summative or outcome evaluation will be more satisfying to them. But some funders are beginning to recognize the value of RTE for nonprofits working in advocacy and systems change efforts – see, for example, the Packard Foundation’s emphasis on RTE. If this might align with the interests of one of your funders consider passing along this report.
Finally, RTE is still a relatively new form of evaluation and the literature regarding best practices and lessons learned is still emerging. We’re interested in learning from folks reading this whether they have experience participating in real-time evaluations. If yes, to what extent did the RTE realize the goal of providing useful information for your work?
See also:
- Succeeding With — Or Maybe Inspite of — Evidence-Based Practices, by Clare Nolan
You might also like:
- A Nonprofit Partnership: How One Board Member Connected Two Organizations and Boosted Both
- Innovative Leadership — Culture Doesn’t Have to Eat Strategy: Tending to Human Factors During Strategic Planning
- The Ultimate Guide to Nonprofit Fundraising in 2025
- Insider Newsletters: An Easy Way to Keep Your Board in the Loop and Engaged
- Five Years and Growing: How One Nonprofit Built a Sustainable, Collaborative Mission
You made it to the end! Please share this article!
Let’s help other nonprofit leaders succeed! Consider sharing this article with your friends and colleagues via email or social media.
About the Author
Clare Nolan is vice president of Harder+Company Community Research, a California consulting firm that specializes in research and strategic planning for the nonprofit, philanthropic and public sectors.
Fontane Lo is an evaluator at Harder+Company Community Research, a California consulting and research firm working with the nonprofit, philanthropic, and public sectors. Fontane crews with Absolute Dragons, a Bay Area dragon boat team.
Articles on Blue Avocado do not provide legal representation or legal advice and should not be used as a substitute for advice or legal counsel. Blue Avocado provides space for the nonprofit sector to express new ideas. The opinions and views expressed in this article are solely those of the authors. They do not purport to reflect or imply the opinions or views of Blue Avocado, its publisher, or affiliated organizations. Blue Avocado, its publisher, and affiliated organizations are not liable for website visitors’ use of the content on Blue Avocado nor for visitors’ decisions about using the Blue Avocado website.
It’s great to see Clare still going wonderful work in this field. Some service areas are inherently difficult to evaluate, such as mental health services or substance abuse services. Clients have deep seated and long term issues that can’t be quickly resolved. What kinds of real time evaluation outcome data would you suggest?