Evaluation

The standard
The rationale
Annotated bibliography

The standard

The rationale

Evaluation design is determined by the purpose for the evaluation-to improve something or to judge its worth-and by the audience for the evaluation's findings. The evaluation process begins in the planning stages and is based on clarity of thought regarding outcomes, the adult learning processes that will be used, and the evidence that is required to guide decision making. It asks and answers significant questions, gathers both quantitative and qualitative information from various sources, and provides specific recommendations for future action.

If staff development is to improve student learning, many levels of change are required, each with its own particular evaluation challenges. Unfortunately, a great deal of staff development evaluation begins and ends with the assessment of participants' immediate reactions to workshops and courses. While this information may be helpful to staff development planners, good evaluation design also gathers additional information. Beyond the (1) initial collection of data on participants' reactions, evaluation must focus on (2) teachers' acquisition of new knowledge and skills, (3) how that learning affects teaching, and in turn (4) how those changes in practice affect student learning. In addition, evaluators may also be asked to provide evidence of (5) how staff development has affected school culture and other organizational structures.

Staff development leaders must also recognize that different audiences require different evidence. Because the vast majority of decisions about staff development are made in district offices and at school improvement team meetings, the urgent pressure that many school leaders feel to improve student learning means that they are interested in knowing now if staff development as it is practiced with their teachers and administrators is making a difference. They are not willing to wait several months for the district to receive the results of its standardized testing. Likewise, teachers want to know if staff development is making their work more effective and efficient, particularly whether improvements in student learning justify the often difficult changes they are being asked to make.

School board members and state legislators, however, want to know if their increased investment in staff development is paying off in improvements on state measures. While state and local policy makers may prefer evidence derived from more rigorous evaluation designs, it is important to remember that they may also be influenced by anecdotes and other informal assessments they hear from teachers or principals at meetings or in other settings.

Staff development evaluation must take into consideration each group's needs with regard to evaluation data. It must ensure the process is in place to collect the needed data and that the audience has the prerequisite knowledge and skills to interpret and use the information.

Annotated bibliography

Calhoun, E. (1994). How to use action research in the self-renewing school. Alexandria, VA: ASCD.
Order from ASCD's web site: http://www.ascd.org

Emily F. Calhoun presents her practical definition of action research for organization improvement: "Let's study what's happening at our school (through the collection and utilization of data) and decide how to make it a better place." She outlines a model for a quick start to action research. She then says that action research (1) uses student data to inform us about success, (2) must be focused on student learning as a collective mission, (3) can develop the school as a learning community, (4) can build organizational capacity to solve problems, and (5) can be a form of personal as well as professional development.

Collins, D. (1997). Achieving your vision of professional development. Tallahassee, FL: The Regional Educational Laboratory at SERVE.
Download in PDF form at http://www.serve.org/publications/htayv.htm.

This "how-to" resource guide offers many tips to help at each stage of building an effective professional development system. The guide reports the Southwest Educational Development Laboratory's six strategies for implementation: developing a vision, creating a context for change, planning, investing resources, providing continual assistance, and assessing and monitoring progress. It also includes summaries of the 1997-98 winners of the U.S. Department of Education's National Awards Program for Model Professional Development and five examples of model schools.

Cushman, K. (1996). Looking collaboratively at student work: An essential toolkit. Horace, 13(2), 1-12.
Read at http://www.essentialschools.org/cs/resources/view/ces_res/57

The author describes a strategy used by teachers in Essential Schools. The teachers come together to examine student work and use specific protocols to focus their discussion on the qualities of the work and what they can learn from it about their students and themselves. Cushman describes a "tuning protocol" that creates a ritual of presentation and response and provides structure for conversations among teachers. Cushman reports that teachers say the examination of student work has had far-reaching impact on their practices.

Guskey, T. (1998). The age of our accountability. Journal of Staff Development, 19(4), 36-44.
Read at http://www.nsdc.org/library/publications/jsd/guskey194.cfm

Thomas Guskey discusses the importance of evaluation of staff development efforts. He claims that educators can no longer operate with the assumptions that all staff development is good and that more staff development is better. He proposes questions that should be asked to determine how effective staff development is in enhancing student learning.

Guskey, T. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press Inc.
Order from the NSDC Bookstore.

In this book, Guskey helps readers to effectively assess professional development, understand the dynamic nature of professional development, and identify what contributes to improved student learning. The evaluation processes and tools recommended by Guskey make it clear that if staff development is to improve student learning, many levels of change are required, each with its own particular evaluation challenges.

Guskey, T. & Sparks, D. (1996). Exploring the relationship between staff development and improvements in student learning. Journal of Staff Development, 17(4), 34-38.
Read at http://www.nsdc.org/library/publications/jsd/f_gusky.cfm

Thomas Guskey and Dennis Sparks argue that documenting the connections between staff development and improved student learning is becoming crucial. They propose a model for understanding this connection and suggest that student learning outcomes should provide the starting point for all school improvement and staff development efforts.

Hall, G. & Hord, S. (2001). Implementing change: Patterns, principles, and potholes. Boston. MA: Allyn & Bacon.
Order from Allyn & Bacon's web site: http://www.ablongman.com

This new text that focuses on the Concerns-Based Adoption Model contains four primary sections: the context for implementing change, tools and techniques for change facilitators (includes Stages of Concern, Levels of Use, Innovation Configurations), the imperative for leadership in change, and constructing and understanding the realities of change (includes a focus on organizational culture, climate, and context).

Hord, S. (1994). Staff development and change process: Cut from the same cloth. Issues…about Change, 4(2). Austin, TX: Southwest Educational Development Laboratory.
Read at http://www.sedl.org/change/issues/issues42.html

In this paper, Shirley Hord describes the Joyce and Showers staff development model and relates it to a change model derived from school improvement studies. Noting the fit of the two models, Hord suggests successful strategies for a comprehensive approach to changing teachers' practices which include developing and articulating a vision, planning and providing resources, investing in training, monitoring progress, providing continuous assistance, and creating a context conducive to change.

Joyce, B. & Calhoun, E. (1996). Learning experiences in school renewal: An exploration of five successful programs. Eugene, OR: ERIC Clearinghouse on Educational Management. Order from the ERIC Clearinghouse on Educational Management web site: http://www.eric.uoregon.edu

The authors present five case studies of programs used to build improved learning communities. Each of the five programs presented focuses on unique components of school renewal. Technical and social aspects of school renewal are examined, and the goal of building a learning community for the whole school remains a central theme throughout. The programs include the use of staff development as a tool for school improvement, the effective use of governance structures, the use of an initiative to create a culture of readers and writers, the use of staff development to increase the capacity of inner city schools, and the use of action research as a tool for school improvement.

Killion, J. (1999). What works in the middle: Results-based staff development. Oxford, OH: National Staff Development Council.
Download in PDF form at http://www.nsdc.org/midbook/index.cfm
Order hard copy from the NSDC Bookstore.

As project director for Results-Based Staff Development for the Middle Grades, Joellen Killion discusses processes and resources for selecting, designing, and evaluating staff development to improve student achievement. The guide describes 26 successful staff development programs in language arts, mathematics, science, social studies, and interdisciplinary programs that were studied and evaluated to ascertain their impact on student learning. It offers guidelines for selecting and/or designing initiatives to improve student performance.

Kirkpatrick, D. (1994). Evaluating training programs. San Francisco, CA: Berrett-Koehler Publishers.
Order from Berrett-Koehler's web site: http://www.bkpub.com

Donald Kirkpatrick provides a rationale for the evaluation of programs and outlines four levels of evaluation: reaction, learning, behavior, and results. Evaluation at the level of reaction measures how those who participate in the program feel about it. Outcomes refer to the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of participation. Kirkpatrick defines the behavior change as to the extent to which on-the-job performance has changed because the participants attended the training program. Finally, he defines the results level as the "bottom line" improvements that occurred because of participants' attendance in the program.

Peterson, P., McCarthey, S., & Elmore, R. (1996). Learning from school restructuring. American Educational Research Journal, 33, 119-153.
Order from AERA by emailing subscriptions@aera.net

The authors analyze successful restructuring experiments from three elementary schools. Their study found that (1) teaching and learning are mainly a function of the teacher's beliefs, understandings, and behaviors within the context of specific classroom problems; (2) changing classroom practice is primarily a problem of continuous learning resulting in improved practice for teachers, not a problem of school organization; (3) school structures can provide opportunities for learning, but structures by themselves do not cause learning to occur; and (4) where teachers have a shared vision, teaching practice and student learning are successfully connected.

Schmoker, M. (1996). Results: The key to continuous school improvement. Alexandria, VA: Association of Supervision and Curriculum Development.
Order from ASCD's web site: http://www.ascd.org

Mike Schmoker sees tangible, measurable results as the goals to successful school improvement. He explores the conditions under which dramatic results may be achieved and the theory behind them. Creating opportunities for meaningful teamwork, setting clear and measurable goals, and regularly collecting and analyzing data are stressed as the means to improvement. Examples are given to illustrate successful applications by schools from around the country. Schmoker concludes with the note that, "Schools improve when purpose and effort unite. One key is leadership that recognizes its most vital function: to keep everyone's eyes on the prize of improved student learning."

Sparks, D. & Hirsh, S. (1997). A new vision for staff development. Alexandria, VA: ASCD and National Staff Development Council.
Order from the NSDC Bookstore.

Dennis Sparks and Stephanie Hirsh describe three powerful ideas altering the shape of schools and staff development: results-driven education, systems-thinking, and constructivism. Major shifts in staff development resulting from these three ideas include movement from individual development to individual and organizational development; from fragmented, piecemeal improvement efforts to staff development driven by a clear, coherent plan; from a focus on adult needs and satisfaction to a focus on student needs and learning outcomes; from training conducted away from the job to multiple forms of job-embedded learning; and from staff development as a "frill" to staff development as indispensable. Sparks and Hirsh elaborate on these shifts and provide examples from around the country.

Stiggins, R. (2001). Student-involved classroom assessment (3rd ed.). Upper Saddle River, NJ: Prentice Hall.
Order from Prentice Hall's web site: http://www.vig.prenhall.com/catalog/academic/product/1,4096,0130225371,00.html

Richard Stiggins describes how to create high quality classroom assessments and use them to build student confidence and maximize student achievement. He emphasizes what teachers need to know to manage day-to-day classroom assessment effectively and efficiently and he focuses on student well-being and potential for self-assessment. He offers practical guidelines on how to use various assessment methods and how to match them with achievement targets. He offers time- and energy-saving ideas for teachers, and he connects the concepts in the book with traditional notions of validity and reliability.

U.S. Department of Education Professional Development Team. (1994). Building bridges: The mission and principles of professional development. Washington, DC: U.S. Department of Education.
Read at http://www.ed.gov/G2K/bridge.html

The Professional Development Team used available research to create a set of principles for staff development. According to their study, high quality professional development: (1) focuses on teachers as central to student learning; (2) focuses on individual, collegial, and organizational improvement; (3) respects and nurtures the intellectual and leadership capacity of individuals within the school community; (4) reflects best available research and practice in teaching, learning, and leadership; (5) enables teachers to develop further expertise in subject content, teaching strategies, and technology; (6) promotes continuous inquiry and improvement; (7) involves collaborative planning; (8) requires substantial time and other resources; (9) is driven by a coherent long-term plan; and (10) is assessed by its impact on teacher effectiveness and student learning.

Wenglinsky, H. (2000). How teaching matters: Bringing the classroom back into discussions of teacher quality. Princeton, NY: Milken Family Foundation and Educational Testing Service.
Download in PDF form at http://www.ets.org/research/pc/teamat.pdf

Harold Wenglinsky's study explores the influence of classroom practices, professional development, and teacher input on student achievement. The study uses data from the National Assessment of Educational Progress based on a national sample of students and their schools. The study finds that while teacher inputs, professional development, and classroom practices all influence student achievement, the greatest role is played by classroom practices, followed by professional development that is specifically tailored to those classroom practices most conducive to the high academic performance of students.