Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

ESQREM 6661

Autumn 2022

Assignment 1

This assignment is designed to be an opportunity for you to show your ‘applied’ measurement skills. This is a pilot for the assignment so I expect there might be some clarity issues—if that is the case for you or your group, be sure to contact me. You can format the assignment however you like but submit all three scenarios in the same document.

Scenario 1 – Scale/Test Development

For this scenario, say you have determined you have a need for a test or scale for a research project (an action research project, a program outcome evaluation, a teaching/learning assessment, a needs assessment – tailor the scenario to your particular discipline). You decide to use that structured scale development process you learned in your measurement class. In general terms, describe how the process would unfold:

• Step 1: Identify the intended use/purpose for your test or scale

– Clinical/screening

– Academic assessments/test

– General population survey

– Outcome measurement and tracking

– Other

• Step 2: Determine what you want to measure

– Identify the domain/construct of interest

– Narrow, focus, and clarify the construct

• Step 3: Generate an item pool

– Use literature, existing tests/scales, focus groups, experts to assist in item identification/development

– Provide some example items

• Step 4: Determine the format for measurement

– Response scale (depending on scale type - knowledge, attitude, intent, etc.)

• Multiple choice for knowledge

• True-false for knowledge

• Essay for knowledge

• Likert scales for attitudes

• Semantic differential scales for attitudes

• Step 5: Have initial item pool reviewed by users and experts

– Describe how you would use a structured process for this step, e.g.,

• Cognitive interviewing

• Focus groups

• Step 6: Describe how you plan to pilot your scale with a development sample

– Carefully identify an accessible reasonably large pool of respondents who represent the population you are considering for scale use

– You should have at least 50 respondents in a pilot, but more is better

– Conditions of administration should be uniform; all respondents should receive the same directions and should be given the same amount of time to complete the instrument

Note: I am not looking for a dissertation here. There should be some detail and appropriate links to the literature, but I do not expect a lengthy treatment of these topics.

Scenario 2 – CTT Analysis of Dichotomous Data

Say you are working with that faculty member who is developing the exam designed to test the statistical knowledge of students entering the doctoral program. Recall that results from the exam will guide content in the basic statistics course. The exam will have true-false questions, multiple choice questions, and a set of applied problems. Finally, the instructor is using a carefully designed scale development process and is using methods from classical test theory and item response theory to assess both individual item performance and scale performance.

For this part of the assignment, you will be analyzing eight true-false items that potentially will be a part of the larger battery of questions. The items are:

Data were collected from 100 first year doctoral students. You will be using the R CTT program to do an initial analysis of the items and then follow with a more detailed look at item and scale performance using IRT methods (Assignment 2).

Details:

1. The data set is “exam t_f.csv” (located in the Assignment 1 Module)

2. Use the “stats t_f ctt.R” script (located in the Assignment 1 Module)

Answer the following questions (Use the Schultz et al., DeVellis and Shenge readings to guide your interpretations):

1. What is coefficient alpha for the item set? Interpret.

2. What is the SEM for these items. Compute and interpret a 95% confidence interval around one value of an observed score.

3. Fill in results from your item analysis in the following table:

4. What do each of these coefficients tell you about items. Are there any flags for concern? What is your overall assessment of these items based on classical test theory methods? What are your recommendations to the test development team?

Scenario 3 – CTT Analysis of Ordinal Data

Say you are consulting with a school district to assess risk and protective factors in a district wide needs assessment. You are particularly interested in a scale designed to assess academic motivation. The scale developers have reported that the scale has good psychometric properties and you’re interested to see if the scale works as advertised with your students. The scale is composed of six items and uses Likert responses to each question.

The Academic Motivation Scale (AMS) is composed of six questions:

Item 1: “I have a positive attitude towards school”,

Item 2: “I feel I have made the most of my school experiences so far”,

Item 3:” I like the challenges of learning new things in school”,

Item 4: “I am confident in my ability to manage my schoolwork”,

Item 5:” I feel my school experience is preparing me well for adulthood”, and

Item 6: “I have enjoyed my school experiences so far”.

The response scale for each item is Strongly disagree (1), Disagree (2), Neither agree or disagree (3), Agree (4), and Strongly agree (5). The summary scale score is a total of the six items resulting in a 6 to 30 scale score range with higher scores corresponding to higher levels of perceived academic motivation.

You will be using the R CTT program to do an initial analysis of the items and then follow with a more detailed look at item and scale performance using IRT methods (Assignment 2).

Details:

1. Use the R CTT program loaded in RStudio for the analysis

2. The data set is “motivation.csv” (located in the Assignment 1 Module)

3. Use the “motivation ctt.R” script (located in the Assignment 1 Module).

Answer the following questions (Use the Schultz et al., DeVellis and Shenge readings to guide your interpretations):

1. What is coefficient alpha for the item set? Interpret.

2. What is the SEM for these items. Compute and interpret a 95% confidence interval around one value of an observed score.