Three Questions to Ask Before Hiring a Program Evaluator

Three Questions to Ask Before Hiring a Program Evaluator

During a webinar that I recently hosted a participant asked if it was better to have an internal or external evaluator to conduct an evaluation plan.

This is a common question among nonprofit leaders, whether their program is new or established because at some point, they need to evaluate your program, whether it be for your team’s internal work or because a funder requires them to do so.

The short answer to the question:

It depends. 🙂

While there’s no hard and fast answer, here are three initial questions that will help you know whether you should hire an external evaluator.

  1. What are your evaluation questions? (What are the things that your stakeholders need and want to know about your program?)
  2. How long is the evaluation planned to take? (A few months, a year, more than a year?)
  3. Does your staff have the time and the necessary skills (i.e., the capacity) to perform the evaluation? (Given your evaluation questions and the length of the evaluation, is your current team able to take on the work?)

Once you have the answers to those questions, you’ll have a good sense of whether you can conduct the evaluation in house or if you will need support.

Let’s see how this applies to a nonprofit who is evaluating Queens, an after school program for Black high school girls that is in its pilot phase. The program manager (who has some experience with conducting program evaluations) knows that the success of Queens is contingent on 1) ensuring they’ve reached their target audience and 2) students are satisfied with the program. If they don’t get nail those two things, the program won’t work as intended. So, the program manager crafted a few evaluation questions:

      1. How many participants were present at the first session?
      2. How did participants hear about the program?
      3. How many participants completed the entire three-month program?
      4. How satisfied were the participants with the program?

In this scenario, the team decides that they don’t need an external evaluator and the program manager will also serve as the internal evaluator. Here are their reasons for that decision:

    • The evaluation questions that the program manager has crafted can be answered using their existing systems (attendance records, pre and post surveys, etc.)
    • The evaluation will be done over the course of three months and can be easily incorporated into the program manager’s role and responsibilities.
    • The program manager has familiarity with conducting an evaluation and knows how to use data to interpret the results and make strategic decisions.

Ok, now let’s fast forward a few years and Queens is still going strong… thanks in large part to the decision to incorporate evaluation from the beginning.  😉

They’re ready to learn about their short-term program outcomes so they can understand what has changed in the lives of program participants. The original program manager is still in her role and has crafted the following evaluation questions:

    1. To what extent did our program change participants’ knowledge, attitude, or skills?
    2. Have participants benefitted from their participation in the program? If so, how?
    3. Did certain participants benefit more than others?

This time around, the Queens team decides to hire an external evaluator because:

  1. While their program manager has experience with evaluation, she doesn’t have the necessary technical skills to determine appropriate indicators and tools that will give them the data they need to answer their evaluation questions.
  2. The team estimates that the evaluation will take about a year to complete and no one has the capacity to fully take on the evaluation work, given their other responsibilities.

Now comes the fun part, which is finding the right evaluator. There are a host of questions that you should ask when hiring an evaluator. One of them should be:

Are we seeking someone who will add or build evaluation capacity?

In the case of adding capacity, the evaluator will do the evaluation for you. While they work with you to make sure everyone is aligned on the evaluation questions and methods and share the results of the evaluation, the evaluator does all of the actual evaluation work. On the other hand, building your team’s evaluation capacity means that the evaluator works with you throughout the evaluation process giving you the knowledge and skills to apply evaluation methods and approaches to your work, now and in the future.

One thing to keep in mind is that whether you have an internal or external evaluator, your team will need to be involved in the evaluation from start to finish. The evaluation cannot and should not be handed off completely to the evaluator. The entire purpose of an evaluation is to help your organization seek and use data to continuously assess and improve your work.  And that only happens if all stakeholders are involved throughout the process.

Before you can perform an evaluation, you need to have a program logic model. I advocate for logic models so much because they serve many purposes, one of which is to help you create an evaluation plan. In the absence of a logic model, creating an evaluation plan is 10x harder because you don’t have the benefit of a roadmap that shows the outcomes and impact you planned for.

If you need help creating or modifying an existing program logic model, I encourage you to enroll in the Set Your Nonprofit Program Up for Success In Two Hours: A Logic Model Masterclass!

Avoid This Common Mistake When Planning Your Nonprofit Programming

Avoid This Common Mistake When Planning Your Nonprofit Programming

While people currently know me as Dr. Bell, 12 years ago I was Ms. Bell to my classroom of 1st-grade students in Houston, Texas. I became a teacher because of my belief that all children, regardless of where they are born deserve a high-quality education. I entered the profession with a ton of passion, but lacking in the knowledge of exactly what I needed to do to make sure my students had a strong foundation to make sure they were successful after they left my classroom. All I knew was that I wanted to help my students!

As one might expect, as I prepared my daily lessons, I made several missteps. And a major mistake that I made was that I focused on creating activities that were fun. For example, oh, it’s Thanksgiving! It’ll be great if they use their hands and paint to make cute little turkeys! One day, one of my mentors pulled me to the side and explained to me that it was ok for my students to do fun and cute activities, but only if they were connected to the larger goals that I had for them. Was that turkey activity going to make sure they were ready for 2nd grade when they left my classroom? If not, I needed to remove it entirely or think critically about how to modify if so it helped them reach their goals.

That lesson has always stuck with me and it’s a major reason why I do the work that I do with nonprofits today. As I work with leaders, I notice a common thread: they are passionate about serving. And that passion oftentimes leads them to create a number of activities within a program that are intended to help their communities. However, what is lacking is a clear picture of how each of those activities connects to the outcome of the organization and with the data to back it up. For example, if you have a mentoring program that aims to increase the number of girls who enter STEM careers, you need to be sure that each activity that mentors and mentees engage in leads to that outcome. Your organization has limited resources, so it’s important to be sure every action is contributing to your larger outcome.

If you need help with connecting your program activities with your outcomes, feel free to reach out to me. I’m happy to help!

Avoid Making This Common Mistake When Communicating Your Nonprofit’s Results

Avoid Making This Common Mistake When Communicating Your Nonprofit’s Results

“What are your results?” and “What is your impact?” are questions that nonprofit leaders have probably been asked and answered a hundred times. After all, that is the purpose of nonprofits: to contribute to the solution of societal problems. Gone are the days when nonprofits were simply trusted that they were doing what they said they would do. Now, the requirement is to actually show your results. 

A common mistake that I’ve seen nonprofits leaders make when communicating their results or impact is that they place too much emphasis on one type of data.

Either they report quantitative data such as the number of program participants, demographic data, volunteer hours, dollars raised, etc. Or they only share qualitative data such as poignant stories of individuals whose lives have been improved after working with their organization.

Quantitative data addresses questions about “who, what, and how many?” This is especially useful when you need/want to provide a “just the facts” overview of your program. On the other hand qualitative data addresses the “why?” behind the data. Qualitative data allows you to capture the rich experience of participants and provides you with insight that cannot be gleaned from quantitative data.

A compelling story from program participants of how your program changed their life will draw your audience in, but it should be coupled with quantitative data about all participants. Conversely, avoid relying solely on quantitative data. People need to feel the heart and soul behind your work and qualitative data allows them to do that more easily.

Here’s an example of how this might play out. Let’s say a nonprofit  has a mission to address food insecurity in your community by providing free lunch and dinner each day. In the past month, due to budget constraints, the team has opted for a more cost effective food supplier. However, the team has also noticed that since they started with the new vendor, there has been a 30% decrease in the number of people showing up for a meal each week. This 30% decrease is certainly a cause for concern.

However, without qualitative data to explain why the decrease has happened, the team could only make educated guesses. That is until a volunteer has the idea to talk with families to understand what may have changed for them in the past month. And, after a few conversations a clear pattern emerges: people feel that the quality of the food has reduced greatly. While this organization isn’t sure where those 30% are now going, but you do know they likely aren’t showing up because the food isn’t that great. Now that they are equipped with qualitative and quantitative data, they are better equipped to make an appeal for additional funding.

Their request might look something like this, “ For the past year, our organization has helped ensure that each month, approximately 500 families in our community have at least two meals per day. However, in recent months, funding cuts have meant that we needed to switch to a more cost effective food supplier. In the past month, since switching to the new supplier, we’ve seen a 30% reduction in the number of families coming in for meals. After talking with families, we learned that one thing had changed in the past month: they felt the quality of the food had decreased substantially. This means that instead of serving 500 families each month, we’re only serving 350. Over the course of a year this means that 1,800 families are potentially going hungry. We’re asking for your financial support to help us switch back to our previous supplier so we can help make sure our community continues to be fed.” 

As the example above demonstrates, it’s important for nonprofits to share quantitative and qualitative data. Depending on your program goals, one data type may be better than the other, but both are needed to present a complete picture of your program. 

What type of data does your non-profit collect? How can I help you determine what data will communicate your story most effectively?

The (Second) Most Important Action Your Nonprofit Needs to Take to Ensure an Effective Response to COVID-19

The (Second) Most Important Action Your Nonprofit Needs to Take to Ensure an Effective Response to COVID-19

Last week, I wrote about the After Action Review, which is an approach to making sure your COVID-19 response is effective. Pandemic or not, I strongly believe that every non-profit should conduct an AAR after every task or project because evaluation of the response is equally important as the response itself.

For many organizations, a natural reaction is to act quickly when a need arises. And, oftentimes, that is exactly right, especially as your work has likely been turned upside down as you moved quickly to ensure health and safety. However, as things are starting to settle a bit, I encourage you to use the Before Action Review. A Before Action Review helps help future responses to COVID-19 is effective because it forces your team to get clear on our outcomes before the work even begins.

The primary difference between the After Action Review and the Before Action Review is that (as the name suggests), a BAR is done before a program or project happens and the purpose is to think through the intended results of the program/project and how to apply lessons learned in order to ensure the intended results are accomplished.

Who should participate in a BAR?

In order to ensure all voices are heard and everyone is aligned on what the intended results are, every team member who will be involved in the project should attend the BAR. In addition, you may consider inviting teammates who have done similar work, as they may have insight into lessons learned that can be applied to your project.

When should teams conduct a BAR? 

The timing depends on the needs of your organization. Your team may need to start a project next week while other projects are scheduled to begin two months away. Ideally, you would conduct a BAR before a formal project plan is completed. You don’t want to run the risk of leaving out key steps that would come to light in the Before Action Review.

What time and resources do we need for a BAR?

The time required to conduct an BAR varies and depends on the project’s scope and could last anywhere from 1-2 hours to 30 minutes. Whether in person or virtual, you’ll need a place to capture your team’s reflections, whether that be on chart paper (be sure to take a photo when you’re done), in a Word/Google document or even in PowerPoint slides.

A Before Action Review is centered around five primary questions. 

  1. What are the intended results?
  2. What will that look like?
  3. What challenges might we encounter?
  4. What have we learned from similar situations?
  5. What will make us successful this time?

And don’t forget to schedule the After Action Review!

  1. When will we do an After Action Review (AAR)?

Need a template for your Before Action Review? Click here!

Has your organization ever used a BAR review? How did it work for you? Do you have questions about using the BAR? Send me a message and I’m happy to help!

The Number One Thing Your Nonprofit Needs to do to Ensure an Effective COVID-19 Response

The Number One Thing Your Nonprofit Needs to do to Ensure an Effective COVID-19 Response

As your non-profit organization moves quickly to respond to COVID-19, the first thought, naturally, is to focus on execution. However, it is equally critical to know if what you are doing is working, should be modified, or discontinued altogether. 

A powerful and practical tool to accomplish this is the After Action Review, which is a structured approach for reflecting on a task or project that a group has completed to identify strengths, weaknesses, and areas for improvement and can be done in as little as 15 minutes.

As your non-profit responds to ever changing needs, it’s important to capture what you have learned along the way. This will save you time in the future so that you don’t waste time reinventing the wheel or doing something that is simply ineffective. It also provides you an opportunity to celebrate your successes and leverage them in the future. 

An AAR is centered on four questions: 

  • What was expected to happen?
  • What actually happened?
  • What can we learn from this?
  • What should we do next time

Who should participate in an AAR?

This AAR is for all teams who want to maximize learning from their work (ranging from one-time events to long-term projects).  Regardless of project outcomes, there are always successes to document and lessons to learn. In order to ensure all voices are heard, every team member involved in the project should attend the AAR.

When should teams conduct an AAR? 

Ideally, your team should conduct an AAR shortly after a project or program ends. The same approach can also be used with less structure or formality midway through a project if the work isn’t progressing as the team would like.

What time and resources do we need for an AAR?

Ideally, formal AARs are conducted with a facilitator who is not part of the team, while informal AARs can be led by a member of the team.  The time required to conduct an AAR varies. A formal review may take 1-2 hours. Informal AARs may be conducted in whatever time your team can allot. A conversation as short as 15 minutes might identify barriers to your progress and strategies to overcome them.

Whether in person or virtual, you’ll need a place to capture your team’s reflections, whether that be on chart paper (be sure to take a photo when you’re done), in a Word document, or even a Google document. Here is a downloadable template you can use! 

I hope the After Action Review proves to be useful for your team as you evaluate your work, whether you’re responding to COVID-19 or otherwise.

Who Should Conduct Our Program Evaluation?

Who Should Conduct Our Program Evaluation?

Conducting  program evaluation for your non-profit can be a daunting (but not impossible!) process. Today’s post is part two in a series where I walk you through the steps of conducting a program evaluation for your non-profit.

In the first post, I discussed investing stakeholders in your non-profit’s program evaluation. Now, let’s discuss the second step: deciding who should conduct your non-profit’s program evaluation. The program evaluation team is responsible for carrying out the remaining steps in the program evaluation process.

Team Options

There are a number of different team configurations, but here are three common options.

  • An external evaluator (i.e., an individual or consulting firm)  leads the evaluation work while program staff provide support.
  • An in-house evaluator leads the work, with support from other program staff and an external consultant.
  • An in-house evaluator leads the team and is supported by program staff.

Which Team is Right for You?

There’s no right answer to this question, but there are two main factors that you’ll need to keep in mind as you make this decision:

  1. Your funder’s requirements. Some funding agencies require non-profits to hire an external evaluator. This person is intended to serve as an objective third party and as such, should not be should not be affiliated with your non-profit.
  2. Your staff’s capacity and capabilities. Specifically, you will need to have a clear understanding of your staff’s skills and experience in planning an evaluation, designing data collection tools and procedures, and collecting and analyzing data.

Whatever team you select, it’s important to remember that while a specific team is responsible for program evaluation, all staff members are partners in the work. Staff members’ knowledge and experience working with program participants and the community are critical for a successful evaluation that benefits program staff and participants, the community, and funders.