Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

PPOL 810: Policy and Program Evaluation

Fall 2023

COURSE INFORMATION

Class Days: Wednesdays

Class Times:  12:00 – 3:00 pm

Room: Walker Building 103

Instructor: Emily Vargo, MPA, ELR MA, EdD candidate

Email: [email protected]

Office Hours: By appointment via Zoom: Wednesday 9:00 am -12:00 pm or Wednesdays after class for in-person

Course Overview

Course description

This course provides students an overview of public policy and program evaluation as a scientifically based means of assessing whether such programs and policies are effective after they have been adopted and implemented. The scientific logic underlying evaluation research is discussed before turning to conceptualizing public policies and programs as testable hypotheses. After then reviewing measurement theory and its application to public policies and programs, the course discusses the inferential validity criteria used to assess a variety of research designs. A major portion of the course will entail an in-depth discussion of several different research designs, including their logic, implementation, strengths, and weaknesses. These will include discussions of pre-experimental, experimental, correlational, interrupted time series, regression discontinuity, comparison group, case study, and nested research designs. Ethical and other practical problems of constructing evaluation research in the field are examined. Finally, the reporting of evaluation research results along with utilization problems associated with evaluation reports are discussed. The goals of the course include enabling students to both critically interpret evaluation research reports and to design, conduct, and report evaluation studies of public policies and programs.

Overall, this course is aimed at giving students the skills to create a study design using qualitative research methodology approaches coupled with quantitative approaches to help triangulate data to create a high validly study.

Course objectives

By the end of the course, students should be able to:

1. Explain, compare and exemplify the different causal effects that can be estimated in a program evaluation.

2. Utilized both qualitative and quantitative research design methodology.

3. Estimate the main research designs used in policy analysis to estimate causal effect of a public program.

4. Differentiate the distinct attributes of each of the main research designs used in policy analysis to estimate causal effect of a public program.

5. Distinguish under what conditions each of the main research designs used in policy analysis to estimate causal effect of a public program can be implemented.

6. Assess whether a given research design is appropriate to estimate the causal effect of interest of a particular public program.

7. Design fundamental research methodology using grounded theory and other qualitative and quantitative research methods.

8. Able to use and design SWOT analysis, randomized control trials, cost benefit analysis, mind maps, stakeholder analysis, social network analysis, regression models, and the logic models for program analysis.

9. Should be able to design and execute a theory-based program analysis, report and presentation.

10. PhD students will learn how to triangulate qualitative data. MPP will be introduced to this concept.

Communication

Email is the preferred method of communication. All email messages must originate from your Penn State email account or Canvas site. Please use a professional salutation, proper spelling and grammar, and patience in waiting for a response and I am the DGS. I do not work on weekends or outside of the 8:00 to 5:00 workday. Please begin your subject with PPOL 810 when composing emails to me (e.g. “PPOL 810: Question About Case Study).

Enrollment Information

Drop/Add

Please visit the Penn State Registrar  website for all add/drop dates https://www.registrar.psu.edu/academic-calendars/2023-24.cfm

Course Materials

Required materials

Newcomer, K. E., Hatry, H. P., & Wholey, J. S. (2015). Handbook of Practical Program Evaluation (4th ed., pp. 492–505). Jossey-Bass, A Wiley Imprint. https://www.researchgate.net/publication/301738442_Conducting_Semi-Structured_Interviews

PhD Students should get

Babbie, E. R. (2021). Practice of social research (15th ed.). Cengage Learning.

Other required readings will be posted on the class website in Canvas.

Academic Support

Canvas

Some materials needed for this course will be provided online through Canvas. If you have any questions about obtaining or activating your Penn State Access Account, please contact the Penn State Central IT (http://it.psu.edu/get-support).

Library Resources & E-Reserves

Many of the University Libraries' resources can be utilized from a distance. Through the Libraries' website, you can

· access magazine, journal, and newspaper articles online using library databases;

· borrow materials and have them delivered to your doorstep—or even your desktop;

· get research help via email, chat, or phone using the Ask a Librarian service (https://libraries.psu.edu/ask);

· access Electronic Reserves (E-Reserves); and much more.

You can view the Online Students' Library Guide (http://guides.libraries.psu.edu/onlinestudentlibraryguide) for more information.

Academic Success Resources

The website below includes Penn State tutoring resources, enrollment processes, student engagement and student success programs and student disability resources and more!

https://success.psu.edu/resources

*Also please work collaboratively within the course to help each other succeed!

Course Structure and Conduct

Graduate Level Learning

I am fully committed to help you succeed in this course. Life is complicated and graduate school is designed to be challenging. Please talk to me if you are feeling behind or having trouble understanding the course material. I will make whatever accommodations I can to help you finish assignments, understand the course material and do well on your final project.

Course Assessment and Grading

Attendance

I expect you to show up to class on time at 12 pm ready to fully participate in this class. If you know ahead of time you will miss a class, inform the instructor beforehand. Attendance on presentation days (see class schedule) is particularly important. If you know you will be absent on any of these days, inform the instructor as soon as possible to reduce the impact on your grade.

Assignments

The graded components in this course include de following assignments: (1) Reading Group Summary (participation); (4) Homework Assignments; (1) Policy Evaluation Report; and (2) Presentations. A description of each assignment is presented below.

Assignments are due at the times specified in Canvas. Due dates are subject to revision as the instructor sees fit—any changes will be to the students’ advantage (i.e., assignment deadlines may be extended but not shortened).

Late work is generally not accepted. If you are concerned about completing an assignment in the allotted time, please speak with the instructor prior to the deadline. Accommodations will be made on a case-by-case basis.

Reading Group Presentation and Summary Assignment

Participation will be based on reading group summary submission (15 points) and classroom engagement (15 points) this makes up 30 points of the total 200 in the class. As for the reading group summary presentation, each student reading group will lead off one class with a 9-minute summary (3 minutes per chapter, not to exceed 10 minutes) presentation of their assigned week's readings. Summaries should relate to the chapter readings for that week. This exercise is designed to help you read and synthesize information and be able to present it in a condensed and concentrated format. This is often referred to as an “elevator pitch” in the field of government relations. This short presentation will be worth 10 points, please review the rubric in Canvas.

Students are expected to participate in every class period. Students will be graded on the quality of participation and contribution throughout the semester. Students should provide insightful dialogue based on the readings. Students may discuss personal experiences as it relates to the readings, however, the majority of the discussion should be grounded in the text. We will also draw upon current events and pop culture as it relates to the class. Students are expected to take a proactive stance with respect to class contribution. While the instructor may sometimes use cold calling, students should not expect this to be sufficient for opportunities to make substantive contributions.

Homework

There will be 4 homework assignments to allow you to practice concepts from each of the course units/parts and to hone your analysis skills (20 points each). These assignments will be designed to help you build your final report and presentation. For example, the first homework assignment will be the identification of your program evaluation topic.

Assignments will be graded in accordance with a rubric for each assignment. The rubrics are displayed in Canvas under each assignment.

Program Evaluation Report

The final assessment of this course consists of carrying out an impact evaluation (40 points).

For this task, students will choose a topic that fits within the definition of a program and perform an analysis of the program. This final report will consist of the four homework assignments that were assigned over the course of this semester with provided feedback on those assignments. This final report will include data visualization and matrices showing the students' fundamental knowledge of the methodology approach and overall program evaluation analysis. A rubric and temple will be provided.

The final report should be no less than 5 pages and no more than 10, typed double spaced, 12-point font, 1” margins. The reference list, charts, tables, appendices, and other non-text supplemental materials do not count toward the minimum page requirement. The report must be submitted on Canvas, no later than Friday, 11:59PM on the final week of classes. No late submissions will be accepted. Detailed explanation of the requirements of the policy evaluation report are available on Canvas.

Presentations

There will be two presentations throughout the semester, where students will be required to present. In these presentations, students should cover the relevant reading topics research methodology, and analysis.

In the first presentation, student should focus on learning how to synthesize material that they are reading and presenting on within the class in a timely manner, aka the evaluator pitch.

The second presentation should focus mostly on the analysis of the research design used to evaluate the program of choice by the student. Please practice your presentation. There will be point deductions for exceeding or going significantly below the time limit. The presentations will be a recorded submission on Canvas.

Assignments and Point Break Down

Assignment	Due Date	Percent of Final Grade
Reading Group Pres/Elevator Pitch	When group is assigned	10% (10 points)
Homework (4)	Weeks 3, 6, 9, and 12	20% (20 points for each assignment 80 points total)
Weekly Reading Group Summaries		15% (1 point per submission 15 points total 15 weeks of class)
Participation		15% (1 point per class, 15 points total)
Final Report		20% (40 points total)
Presentations		20% (40 points total)
	Total Percentage	100% (200 points)

Grading Scale

Course Schedule

[A] refers to Angrist, Joshua D. and Jörn-Steffen Pischke. Mastering Metrics

[G] refers to Gertler, Paul J et al. Impact Evaluation in Practice

[C] refers to articles or materials available on Canvas

22

Week	Date	Topic	Readings to complete before class	Assignments
1	8/23	Introductions and course overview · What is Program Evaluation? · Why is it important? · Methodology approach and research structure · Logic modeling	Chapters 1 – 3 (Newcomer et al., 2015) With (Babbie, 2021) highlights	None Add/Drop week everyone gets points for summaries and participation
2	8/30	Introductory to exploratory evaluation and research Performance measurements Comparison group design	Chapters 4 – 6 (Newcomer et al., 2015) Grounded Theory Readings Data Visualization	Reading Group Summary
3		Randomized Controlled Trials Conducting Case Studies Recruitment and Retention of Study Participants	Chapters 7 – 9 (Newcomer et al., 2015)	Homework Assignment 1 Reading Group Summary
4	9/6	Designing, Managing, and Analyzing Multi-site Evaluations Evaluating Community Change Programs Cultural Responsive Evaluations Social Network Analysis	Chapters 10 – 12 (Newcomer et al., 2015)	Reading Group Summary Group 1 Elevator Pitch
5	9/13	Here we will be talking about part one of the textbook and how it supports building a program evaluation.	Recap Part 1	Reading Group Summary
6	9/20	Using agency records Using surveys Role-playing	Chapters 13 – 15 (Newcomer et al., 2015) Qualtrics introduction IRB requirements for human subject studies and ethics behind research designs	Homework Assignment 2 Reading Group Summary
7	9/27	Using ratings by trained observers collecting data in the field using the Internet	Chapters 16– 18 (Newcomer et al., 2015) IRB requirements for a human subject studies	Reading Group Summary Group 2 Elevator Pitch
8	10/4	Conducting semi structured interviews focus group interviewing using stories and evaluation	Chapters 19– 21 (Newcomer et al., 2015)	Reading Group Summary Group 3 Elevator Pitch
9	10/11	Here we will talk about gathering data and how to analyze it and supporting our evaluation out comes.	Recap Part 2 NVivo - thematic analysis	Homework Assignment 3 Reading Group Summary
10	10/18	Qualitative data analysis Using statistics in evaluation Cost effectiveness and cost benefit analysis	Chapters 22 – 24 (Newcomer et al., 2015)	Reading Group Summary Group 4 Elevator Pitch
11	10/25	Meta-analysis, systematic review, and evaluation synthesis	Chapters 25 – Recap Part 3 (Newcomer et al., 2015)	Reading Group Summary
12	11/1	Pitfalls or limitations in evaluation Providing recommendations comma suggestions comma and options for improvement Writing for impact	Chapters 26 – 27 (Newcomer et al., 2015)	Reading Group Summary Group 5 Elevator Pitch
13	11/8	Contracting for evaluation products and services Using evaluation in government	Chapters 28 – 30 (Newcomer et al., 2015)	Reading Group Summary Homework Assignment 4
14	11/15	Evaluation challenges, issues, trends	Chapters 31 (Newcomer et al., 2015)	Reading Group Summary
15	11/22	Thanksgiving Break	No Class
16	11/29	Here we will review part four of the textbook and start to discuss final project requirements ideas and designs	Recap Part 4	Reading Group Summary
17	12/6