APD1292H Instrument Design and Analysis Summer 2024

2. Course Description

Various quality indicators, assessment tools, and survey instruments are widely used in professional, academic, clinical, and research settings. They are crucial for informing theories and advancing practice. Poor instrument quality and misuse have profound impact on research, policy, practice, and most importantly, the lives of people. This course is designed to introduce core concepts and principles required for high-quality instrument design and evaluation. It is intended to provide a sound foundation for developing instruments for one’s own purpose or evaluating instruments developed by others. The course brings together theoretical foundations and measurement approaches. Building on the key concepts of measurement, students will have the opportunity to gain hands-on experiences with instrument design, analysis, and interpretations. There are four learning modules, each of which consists of two parts: 1) knowledge building and 2) skill building through hands-on data application.

3. Course Delivery Modes

This course was designed to be delivered in an Online Synchronous mode. Classes will be held on Tuesdays and Thursdays ( 18:00 – 21:00).

A. Quercus

Quercus is the primary online learning space for the course. All course activities such as class announcements, course contents, assignment uploads, and feedback sharing, will be through Quercus. The Zoom link for class meetings is available on Quercus.

To access Quercus:

.    Go tohttps://q.utoronto.ca

.    Use your UTORid and password to log in

.    Once you've logged in, you'll be in your "Dashboard"

.    You should see a "Course Card" for each of the courses you are or enrolled in

.    Click on a "Course Card" to access the course.

.    If you do not see your courses listed, tryactivating your UTORidand/orverifying your UTORid

B. Zoom

Online synchronous classes will be held through Zoom. Click the invitation link posted on Quercus. Below are some useful tutorials for our zoom meetings:

Joining a zoom meeting

Main tool bar during a zoom meeting(meeting info, audio/video, screen share, breakout rooms, etc.)

In-meeting chat and reaction

More tutorials videos here for other functions:https://learn-zoom.us/show-me

C. Succeeding and Excelling in Online Learning

Please seeoiseonline.org, a resource put together by members of the OISE community to help students succeed and excel in online learning.

Also, resources available from the School of Graduate Studies:

.    Guide to Working from Home for Graduate/Postdoctoral Researchers:

https://studentlife.utoronto.ca/wp-content/uploads/Strategies-for-Online-Learning.pdf

.    SGS Graduate Wellness Portal:

https://www.sgs.utoronto.ca/gradhub/resources-supports/#health-wellness

4. Learning Outcomes

By the end of this course, students will be able to:

1.   Identify and describe core concepts and principles required for high-quality instrument design and evaluation;

2.   Interpret and communicate research about instrument development and validation;

3.   Plan ways to develop and/or evaluate data collection instruments and measures used in education and psychology;

4.   Evaluate instruments developed by themselves or others by applying analysis techniques based on Critical Test Theory, Factor Analysis, and Item Response Theory.

5. Scope and Sequence of Course Topics

6. Course Assignments

While taking this course, the achievement of learning outcomes will be accomplished through the following assignments:

Class participation (20%)

Students are required to attend classes and participate in class discussions and small group activities. Complete attendance and active participation are required.

Assignment 1: Instrument Development (20%)

Deadline : Tues. May 21, 2024 (5pm)

Length: No more than 3 pages (tables and figures included; references and your instrument excluded) The paper should include the following:

1)   Description of the target construct, its definition, operationalization, intended use, and structure of the instrument

2)   A diagram of the instrument structure

3)   The instrument (including intro/instructions page and items) in the appendix (NOT as a link to the online instrument)

Assignment 2: Application of Classical Test Theory (20%)

Deadline : Tues. June 4, 2024 (5pm)

Length: No more than 3 pages (tables and figures included; references excluded)

Students will analyze one of the datasets provided using classical test theory and report their results. The paper should include the following:

1)   A brief introduction to the data source

2)   Formulate 3 questions requiring answers based on CTT and provide empirical evidence for each question.

3)   Use tables and graphs to present empirical evidence and provide your interpretations.

Assignment 3: Application of Factor Analysis (20%)

Deadline : Thurs. June 13, 2024 (5pm)

Length: No more than 4 pages (tables and figures included; references excluded)

Students will perform factor analysis using one of the datasets provided to investigate the internal structure (dimensionality) of the instrument. The paper should include the following:

1)   A brief introduction to the data source

2)   Formulate 3 questions requiring answers based on EFA/CFA and provide empirical evidence for each question. These questions should be the building blocks that are linearly linked in order to achieve the assignment’s goal (i.e., instrument validation and, if applicable, suggestions for

improvement).

3)   Use tables and graphs to present empirical evidence and provide your interpretations.

Assignment 4: A Critical Review of An Instrument Validation Study (20%)

Deadline : Thurs. June 20, 2024 (5pm)

Length: No more than 4 pages (tables and figures included; references excluded)

Students will identify a widely used instrument of interest and a peer-reviewed published study of its  development and/or validation that used either factor analysis or item response theory (or both). Then, prepare a critical review of the selected study with a focus on its method and result interpretation. Use  the course reading materials and/or relevant literature to support your opinions. The paper should

include the following:

1)   A brief introduction to the instrument (e.g., purpose, context, operationalization) 2)   Analytical approaches used to validate the instrument

3)   Critical evaluation of the validation study in terms of the concepts and principles covered in the course

4)   The paper reviewed (attach as a separate PDF file)

*Note all assignments: please follow the APA 7th style (e.g., double-spaced, pagination, reference list). Do not include your name in the header. No title page required.