Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: daixieit

Information Technology Evaluation

ISYS3401

Semester 1, 2023

Take Home Examination

Long Release

Release: 9am, 12th June 2023 (Monday)

Submission: 9am, 17th June 2023 (Saturday)

GENERAL ASSESSMENT INFORMATION

Based on the following list of examination preparation reading list provided in the final week 13 lecture:

1. An international survey on AI in radiology in 1041 radiologists and radiology residents, Huisman et. al (2021),

 part 1: fear of replacement, knowledge, and attitude, https://pubmed.ncbi.nlm.nih.gov/33744991/

 part 2: expectations, hurdles to implementation, and education, https://link.springer.com/article/10.1007/s00330-021-07782-4

2. A systematic review of the diagnostic accuracy of artificial intelligence-based computer programs to analyze chest x-rays for pulmonary tuberculosis:

 Harris et al. (2019) https://pubmed.ncbi.nlm.nih.gov/31479448/

3. Guideline for good evaluation practice in health informatics (GEP-HI),

 Nykanen et al. (2011) https://pubmed.ncbi.nlm.nih.gov/21920809/

4. Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI)

 Talmon et al. (2009) https://pubmed.ncbi.nlm.nih.gov/18930696/

 Brender et al. (2013) Explanation and Elaboration, https://pubmed.ncbi.nlm.nih.gov/24155788/

This take-home exam requires the students to read carefully and understand the review study provided re: the use of AI for breast cancer screening. The goal is for you to assist a medical imaging services network to determine whether the adoption of the new AI technology for breast cancer screening by preparing a 3- part evaluation study. Evaluation Study to be use (attached):

 Freeman et al. (2021) Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy, https://www.bmj.com/content/374/bmj.n1872

PREPARATION YOUR EXAM SUBMISSION

Submission Pack:

1. One single Word or PDF file with all written parts (for Part 1, 2 and 3) combined in sequential order with clear page break inserted in-between each part.

a. Part 1: Review of Technology Evaluation (Executive summary)

b. Part 2: Pre-implementation evaluation (Evaluation protocol)

c. Part 3: Post-implementation evaluation (Evaluation Plan)

2. Three separate presentations, one zoom mp4 recording for each part. Consider each presentation is your briefs for different stages of the adoption timeline to the senior executives of the medical imaging services network. You will need to first produce a professional powerpoint presentation (with no more than 10 slides) for each part.

3. Prepare a .zip file that includes the 1 word or pdf document and the 3 presentation recordings. Name your file with your student number, unit of study number (i.e. ISYS3401), and “exam2023” wording. For example, the zip filename for your submission should look something like: s1010101_ISYS3401_exam2023.pptx”.

How to present your powerpoint using zoom:

 Start a New Meeting (having yourself being the only participant)

 Have video and audio turned on (because you need your face and voice recorded)

 Once everything is ready, present your power point slide show using “Share Screen”, and start recording in zoom (Please try to keep the presentation within 5 minutes)

 Once you have finished your presentation, stop recording and end the meeting.

 Zoom will take the next few minutes to convert the meeting to recording. You should be able to locate your recording in the zoom sub-folder. The file name of your recording should be “zoom_0.mp4”. Your presentation should look something like the image below.

ASSESSMENTS PACKAGE:

Part 1: Evaluating the Technology for implementation. (15%)

Based on what you learnt from reading Harris et al. (2019) paper, appraise and synthesise the findings shown in a recent review paper Freeman (et al. (2021) to determine the AI technology is ready for clinical use.

1. If not, what are the issues hindering? Discuss further if needed.

2. If yes, how certain are you? Discuss further if needed.

You are tasked to provide your overall recommendations with justifications based on evidence shown in the review paper.

Deliverables:

1. a written “executive report” length up to 3 pages (10%)

2. a recorded professional powerpoint presentation duration up to 5 minutes (5%)

Part 2: Pre-implementation Evaluation Plan. (10%)

Assume the technical performance of the AI technology has reached the required threshold, the organisation is considering a 6-month trial to see how the technology performs in practice. Anticipating the 6-month trial will start in 3 months, you are asked to evaluate before the implementation:

1. The “attitude” of radiologist towards the use of AI for Breast Screening

2. The “usability” of the technology using appropriate UX metrics from what you learnt in this unit.

By providing you access up to 10 radiologists across the organisation network, you are tasked to provide a suitable evaluation plan to assess the readiness of the radiologists for using AI in their breast cancer screening tasks; justify your plan based on usability issues you would like to address.

Deliverables:

1. a written “evaluation protocol” length up to 2 pages (6%)

2. a recorded professional powerpoint presentation duration up to 5 minutes (4%)

Part 3: Post-Implementation Evaluation. (15%)

Anticipating the AI technology will be implemented and integrated into the breast cancer screening service for a trial period of 6 months, you are tasked to develop a suitable survey instrument to evaluate user acceptance of the AI technology used in the breast screening task. The goal of the survey is to help the organisation to continue with the technology beyond 6 months. You are required to develop your evaluation based on the following documents:

1. Guideline for good evaluation practice in health informatics (GEP-HI) as template

2. An international survey on AI in radiology in 1041 radiologists and radiology residents, part 2: expectations, hurdles to implementation, and education as basis to develop your instruments.

Deliverables:

1. a written extensive “evaluation plan” length up to 4 pages (10%)

2. a recorded professional powerpoint presentation duration up to 5 minutes (5%)