Evaluating Educational Programs

What is the purpose of educational program evaluation models?

"Evaluation is the process of determining the merit, worth, and value of things, and evaluations are the products of that process" (Reiser & Dempsey, 2012, p. 97).

Living in our constantly changing world where new ideas about learning and innovative programs are popping up and pushing the boundaries of educations, it is becoming increasingly necessary to have a method of evaluating the worth of these innovations. According to Reiser and Dempsey (2012) educational program evaluation in the 1950s-1960s focused on traditional instruction and the "main purpose of the evaluation was to determine the value or worth of the innovation that was being developed" (p. 96). As a result of a failed curriculum reform in the 1960s, educational psychologists determined that educational program evaluation needed to be re-conceptualized and the terms formative and summative evaluation were first coined (Reiser & Dempsey, 2012, p. 96).

Just as teachers must assess students during the process of learning the skill (formative) and then at the end of the learning process (summative), educational programs can be evaluated or judged based upon how they are working while the program is in the process of implementation (formative) and the outcomes after implementation/completion of the project (summative).

In the following sections, a brief overview of two models for program evaluation are presented.

Using the Logic Model to Assess Programs

While researching educational evaluation models, one of the models that frequently showed up in search results was the Logic model. According to the University of Wisconsin-Extension, logic models date back to the 1970s and "the first publication that used the term 'logic model' is usually cited as Evaluation: Promise and Performance by Joseph S. Wholey (1979). Bennett's hierarchy, The Seven Levels of Evidence (1976)...is an early forerunner of today's logic model" (University of Wisconsin-Extension, 2003, p. 10). 

A simple logic model is comprised of three basic parts: inputs, outputs, and outcomes. In the simple model, the inputs represent the investment or resources that go into creating the program in order to achieve the desired outputs. The outputs represent the activities done to reach the target audience. The outcomes represent the short, medium, and long term changes or benefits. An example of an extended simple logic model is below:

Source: University of Wisconsin-Extension, 2003, p. 17

The following example shows how the simple logic model has been used to evaluate a program to minimize child abuse:

Source: University of Wisconsin-Extension, 2003, p. 19

A complete or full logic model is comprised of six parts: situation, inputs, outputs, outcomes, assumptions, and external factors. The situation is "the problem or issue that the program is to address within a setting or situation" (University of Wisconsin-Extension, 2003, p. 24). Part of understanding the situation is defining the problem. UW Extension recommends asking the following questions to fully understand the situation:

University of Wisconsin-Extension, 2003, p. 24

Assumptions are our underlying beliefs about how the program will work and about the people involved in the process. Based on prior knowledge we may ASSUME that a certain output or outcome will occur. External factors are things that may affect the output and/or outcomes of the project that are non necessarily able to be controlled. For example, if a school program is dependent upon certain staff members and those staff members quit necessitating the hiring and training of new staff partway through the implementation process, this may affect the results of the program.

A sample of a complete/full logic model is below:

University of Wisconsin-Extension, 2003, p. 23

So, how can we use this to evaluate an individual classroom unit? I've frequently mentioned a units I've done in the past on stop motion and videography. In terms of a classroom, the situation will not necessarily address a problem, but will most likely address a skill to be taught. Let's take a look at how a camera shots and angles unit could be evaluated using this model:

Step One - Situation:

• The students need to learn camera shots and angles and they need to understand the psychological response of various camera shots. 

Step Two - Inputs:

• The inputs or invest include the technology and equipment used, the time spent to develop the lessons and teach them, research into shots and angles, the teacher, and other materials.

Step Three - Outputs:

• The output includes activities which include live demonstrations, videos, quiz games, student websites for shots and angles, and student video projects. The students and teacher are the participants.

Step Four - Outcomes:

• Short Term: The students learn shots and angles and are able to identify them and the represented emotional subcontext.
• Middle Term: The students are able to create a video which uses the camera shots and angles taught. The students are able to choose a shot/angle that will convey an emotional response appropriate to the setting/scene. 
• Long Term: The students are able to analyze shots and angles when watching movies or television outside of the classroom. The students are able to think critically about why a filmmaker chose to use a certain camera shot in a particular scene. The student will be inspired to learn more about videography. 

Step Five - Assumptions:

• Most humans react in a particular way to a specific camera shot/angle. Based on varying cultures and upbringing this may not be the case. It's a similar issue with colors used in movies. If It's Purple, Someone's Gonna Die is a popular book that analyzes color use in film from a psychological perspective. However, it should be noted that not all cultures have the same reactions or feelings about specific colors. What a color represents on screen in one culture may not be the same as in another. In the same way, how one culture or group of people reacts to a camera angle may not be the same as another culture. I've included a preview from Google Books of the color analysis book below for those who are curious. 

Step Six - External Factors:

• Filming spaces being occupied, broken equipment, student absences, other classes occupying filming spaces around the school, fire drills, and weather that makes filming outside not possible. 

Additional resources:

• There are a large amount of materials on logic models available online. The University of Wisconsin-Extension website provides examples, templates, training guides and other resources at the following link: http://fyi.uwex.edu/programdevelopment/logic-models/. 
• The training guide I used for this blog entry is available in .pdf format here: Enhancing Program Performance with Logic Models. 
• An additional training guide published by the U.S. Dept. of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance is located here: Logic Models for Program Design, Implementation, and Evaluation: Workshop Toolkit
• A webinar produced by the U.S. Dept. of Education Institute of Education Sciences from Youtube: 

Kaufman's Organizational Elements Model (OEM)

Kaufman's OEM model is based on Kirkpatrick's training evaluation model. The model is comprised of five steps: inputs, processes, products, outputs and outcomes. "The OEM identifies what an organization uses (Inputs), does (Processes), produces (Products), and delivers (Outputs) and the consequences (Outcomes) for external clients and society" (Kaufman et al., 2003, p. 64). The focus in this model is on results. OEM separates categorizes results as occurring at the "Mega level (societal), Macro level (organizational), or Micro level (operational)" (Kaufman et al., 2003, p. 64).

A breakdown of the three levels of results as applicable to a few different situations is presented in Kaufman's (2003) book Strategic Planning for Success: Aligning People, Performance, and Payoffs:

(P. 65)

One thing I really like about this model is that it asks evaluators to look at what is happening versus what should be happening. This essentially allows the evaluator to examen performance gaps in order to improve the program. This table by Lisa Llewellyn presented on the HPT (Human Performance Technologists) Toolkit provides a breakdown of the model:

Source: Lisa Llewellyn, HPT Toolkit

The following is an example of how this might be used in education:

Source: Ryan Watkins, Is Distance Learning Right for Your Organization

Applying my shots and angles lesson to the example above, I needed instructional materials, equipment and students (input), the unit had required a lot of planning (process) as I had no course book, a curriculum and materials were developed (products), the students performed their "job" by creating videos (outputs), and the students were able to use their skills to analyze videos later in life aka they became self-sufficient at achieving the task (outcomes).

Additional Resources:

• An overview of using OEM by Ryan Watkins:  Is Distance Learning Right for Your Organization
• A Google Books preview of Strategic Planning For Success: Aligning People, Performance, and Payoffs: 

• A webinar with the theories main founder, Dr. Roger Kaufman, facilitated by Boise State University, on HPT and Applying the OEM is below. The website here provides a transcript of the video.

Investment Return in Evaluation

When evaluating the efficacy of a project or program, I believe it is equally as important to consider the potential return on the investment and the management throughout the process. If we implement a program that is fabulously successful and our students suceed, BUT the program was so cost-intensive that it cannot be sustained long term OR the school faces major budget fallout, was it really the most effective program? Was there a better way to do it that would result in less investment fallout? In terms of management, how effective were the managers at carrying out their tasks? When the project ends, will they still be there to continue implementation?

This sort of situation reminds me of the TIPPS grant and resulting programs that were a part of my school campus during my first and second year teaching. My school had been awarded a TIPPS grant during the year before I began teaching. As a result of that grant, they implemented new computer programs on campus which required the campus to upgrade the computers. The campus also purchased iPads for the classrooms. With all of the extra technology, a special technology coach was hired. The school was also able to hire social workers and community liaisons. On the surface, the technology was awesome! It was great having the extra staff. Our scores rose by just a bit. That was the good side.

However, even with the seeming success of the programs, teacher and administrator turnover was excessively high. This resulted in a lack of consistency in the implementation of the programs in terms of administrative practices and classroom procedures. Additionally, once the program ended, all of the extra staff who were funded by the TIPPS grant program were released from employment with the school. The following school year, without guidance, the programs were less faithfully implemented. In fact, the iPads that we had spent so much money on ended up collecting dust in a corner of the library until the second semester. This happened for the next two years running. Without a social worker on campus, many of the outreach sources for students and parents were gone. Evaluation models should consider the impact or longterm outcome of what happens when the program is ended in terms of investment and human capital.

Human Performance Investment and Performance Support

Reiser and Dempsey's (2012) reference to Gilbert's 2007 principles and theorems regarding performance stood out to me:

"If you pit the individual against the environment, the environment will ultimately win" (p. 137).

On my campus in the past, many teachers stressed that fact that the hostile environment caused by a particular principal was the cause of many classroom problems at the school. In a sense we were all fighting against the environment that we were placed in. I think there's a difference in how teachers succeed between having supportive leadership in an environment conducive to happiness in the workplace and working in a hostile environment.

"Hard work, great knowledge, and strong motivation without valued accomplishment is unworthy performance. A system that rewards people for their behavior...without accomplishment encourages incompetence. A system that rewards accomplishment without accounting for behavior incites waste" (p. 137).

This is essentially saying that teachers should be motivated a judged on both their work AND demonstrated outcome. This brings to my mind the example of a previous teacher of the year at my school. That particular teacher was very popular with the staff and students. She definitely met the first criteria of being hard working, having great knowledge, and strong motivation. However, the majority of her students did not pass the STAAR test. In fact, she had one of the worst passing rates for her subject in the school. As teachers we want to make our classes fun and engaging, but we can't forget that our students are there to learn. It is not just a popularity contest. With whatever fun activities we are using to engage the students, they must still attain knowledge.

I think it is vitally important for administrators to support the educators on their campuses. Without a supportive work environment, teachers will not reach their full potential. Stifling teachers making them feel stressed, unsafe, or incompetent at work is going to effect their ability to help their students reach their full potential. If a teacher is motivated to do better, but there is no one to turn to for help, will things change? At my former school, one thing many teachers noted was the lack of available professional development. A possible solution to this problem would have been for the school to implement professional development via an online learning management system like Moodle or Google Classrooms. With these programs, teachers would be able to complete professional development from anywhere at their own pace and on their own timeframe. They'd also be able to go back and reference the materials.



Kaufman, R., Oakley-Brown, H., Watkins, R., & Leigh, D. (2003). Strategic planning for success: aligning people, performance, and payoffs. San Francisco: John Wiley & Sons.

Reiser, R. A., & Dempsey, J. V. (2012). Trends and issues in instructional design and technology (3rd ed.). Boston: Pearson.

University of Wisconsin-Extension. (2003, February). Enhancing program performance with logic models [PDF]. University of Wisconsin-Extension.