Rationale and Description

Foundational to effective information interaction is an understanding of (a) the nature of human cognition, (b) the capabilities of the interactive technology, and (c) strategies for effective

communication of information to provide a bridge between humans and technology.

In this assessment you will explore these aspects by creating a conversational agent

based on the technology, Rasa. Your agent will be a knowledge-bot, designed to engage a human user in a conversation on a specific topic of your choice, in order to help them

learn more about that topic.

In undertaking the assignment, you will further develop the knowledge and techniques that

you have gained from interaction with lectures and participation in the workshops, and will

draw these together with relevant theory to design and implement your conversational agent.

The development process will be undertaken in an iterative process that ensures a working agent that develops in capability over the period of the assessment.

Learning  Outcomes

A  successful  completion  of  this  task  will  demonstrate:

1. An understanding of theory that is important for the design of conversational agents, and how key design principles can be implemented to improve the quality of

information interaction.

2  .   An  ability to  use  conversational agent technologies to develop a conversational agent that  implements  relevant  design  principles  in  an  effective  manner.

3 .   An ability to  plan and implement user testing that meaningfully informs the design of your  conversational  agent.

4. An ability to critically analyse an interaction in a reflective and theory-informed way,

identifying and describing key insights and demonstrating sound critical analysis skills.

Essential  Elements

This assessment is tiered.TTer         foundationtasks comprise basic operational knowledge, fundamental techniques, and the planning of user testing.Ter2T2)extensiontasks

comprise deeper thinking about how theory applies to design, extended techniques to

implement            designs, execution and analysis of user testing, and critical analysis of another chatbot.

For T1  (due week 4 –  see Canvas for exact date) you must demonstrate:

1 .   A basic knowledge of at least 3 key design principles ( 1 from each category) , and the ability to operationalise them in a conversational agent.

2.  An ability to use basic techniques for the creation of a conversational agent

(knowledge-bot) using Rasa which can respond to simple questions on a specific topic of interest*.

3.  An ability to demonstrate the application of knowledge of human interaction principles and user testing techniques in the planning of user testing of your chatbot.

This will be marked (formatively) as pass/fail, according to the T1 criteria sheet below. This  grade will not count towards your final grade, but will provide you with guidance whether you are meeting the requirements for the first stage of the assignment.

For T2  (due week 5 –  see Canvas for exact date) you must demonstrate:

1.  A detailed knowledge of at least 6 design principles (2 from each category which    may include those used in T1) by extending your conversational agent appropriately and   linking this work to relevant theory.

2. An ability to use more advanced techniques to extend the agent ’s interaction capability from simple questioning to more complex extended dialogue including ambiguous utterances.

3 .   A demonstrated ability to conduct a user interaction audit and user experience testing

and integrate the insights gained from the audit/testing into the final chatbot design.

4.  An ability to critically analyse AI generated responses, as well as another student ’s chatbot, with respect to design principles and theory.

Your work should clearly show how important design principles (as given by theory) can influence the design and implementation of a conversational agent.

Further details on the steps required for this assessment are outlined in the  ‘ requirements’section below.

*Note: your chatbot needs to be about a topic. It is not sufficient to create a chatbot that does something other than converses

about its “knowledge ”about a specific topic. For instance, bots that won’t meet the requirements of this assignment are things like appointment booking bots, or hotel finding bots, or weather forecasting bots. It can ’t be a bot that does a simple action; it needs to  be something that can converse meaningfully about a certain topic.

Detailed requirements

For Tier  1 ,  you will  be asked to submit three things:

1.  Achatbot (download your chatbot project as a zip file containing one trained model) . At a minimum, your submitted T1 bot must be able to respond to the following:

a  .        Who  created  you?   What  is  their  student  number?   What  is  their  email  address?

b.    Who were you designed to converse with? Who do you normally talk to?

c.   What topic do you know about? What can you tell me about _? What do you know about_?

d.   Why do you know about this topic? Do you like talking about _? e.   Can you tell me more about _? Is that all you know about?

2.  A justification of how your chatbot meets 3 design principles (1 from each category)

(approximately 500 words in the form of a PDF).

3.  A user testing plan (in the form of a PDF).

Tier 2 (T2) – Extension

1.  A chatbot (in the form of a zip file with one trained model). To create the chatbot, you should follow the following steps:

a.  Using ChatGPT or other generative AI, produce 20-25 response sets* in your “domain” file, and edit these to better align them with the design principles.

b.  Write an additional 15 or more response sets written by you. These will be more nuanced and specific than what can be generated by an AI model, and should   be written with the design principles in mind.

c.  Complete other necessary files and train your model, ensuring it runs without

bugs before submitting. Please note, your chatbot will be marked based on your trained model and the domain.yml file. Please only submit one trained model. If   there is more than one model in the folder, only the latest one will be used for

marking. Other .yml files are for reference purposes only.

2.  A critique and justification based on 6 design principles (2 from each category)

(approximately 1000 - 1250 words in the form of a PDF). This must include:

a.  A critique of the AI generated list of responses, using the design principles as a basis for your critique. To aid clarity in your critique, use examples.

b.  A record of AI generated list of responses with your corrections. You can include this as a table with columns ‘AI generated ’ and ‘corrected ’, or include them as a list with track-changes to show how you edited them.

c.  A justification of how you have amended the generated responses to better

meet the 6 design principles within the context of the interactions your chatbot is designed to have with users. Note: describe the process you undertook to    improve the generated intents and responses in the context of intended

interactions, and use examples when referring to changes made.

d.  A justification of how you have designed the rest of your chatbot (i.e., your own written responses included in the domain file, as well as the intents and stories  included in the other required files), providing evidence and arguments relating  to 6 design principles (2 from each category).

3.  A user testing audit and testing report (in the form of a PDF). This must include a list of issues identified (usually around 10), where each issue is documented by:

a.  Issue name (a few words to summarise the issue)

b.  Concise description of the issue, noting it’s impact on interaction (1-2 paragraphs)

c.  Example of the interaction which shows the issue (e.g., saved script or screenshot)

d.  Recommendation for improvement, noting how the recommendation will improve the interaction (1 paragraph)

*Note: response sets describe a response to a user query that has variations. For example, a response set to “hello” might include responses “hi” “hello, how can I help you” and “hey, what can I do for you today?” . In this case, all three would be included in the   response set. It is up to you to determine how many response variations (if any) you would like to include in each of your response sets, and this decision should be guided by your understanding of the design principles. Due to the scope of this assignment, not  all response sets are expected to have multiple response variations.

4.  A critical analysis of another student’s chatbot (approximately 500-750 words in the form of a PDF). This must include the following sections:

a.  Introduction: Write a short paragraph of a few sentences describing the chatbot, its name, knowledge topic, what design principles were used.

b.  Strengths/weakness: Concisely describe a comprehensive set of accurate, relevant and specific strengths and weaknesses together with appropriate  justifications.

c.  Personal reflection: Concisely describe your experiences and feelings from the interactions.

d.  Analysis of Design principles: For any 4 of the design principles, concisely state  your assessment of the level that the chatbot exhibits that design principle, e.g., “fully”, “somewhat”, “not at all”, together with an appropriate justification.

e.  Conclusion: Briefly summarize the insights of what you have learnt from this

critical analysis and how these can help improve your own chatbot. (note: this    doesn’t mean ideas you can copy from the other chatbot, rather, how have you learnt about best practices and common mistakes in chatbot design and how    can you apply these insights in your own design)

In writing the written elements of your assignment, ensure that you always justify your

assumptions, opinions and conclusions with evidence. In addition, please ensure your writing is: • Clear — Your writing must be easily understandable to a non-expert reader by avoiding uncommon terminology and abbreviations.

• Concise — You must express your ideas efficiently, so that key points are not

obscured by irrelevant material. In other words, always stick to the point you are trying to make without padding your writing with unnecessary words and sentences.

• Coherent — Your conclusions must follow logically from your assumptions.

• Convincing — The content that your present should be compelling and believable.

During the self-directed learning weeks, you will be provided with supporting material, including templates, to assist you in formatting your submission.

Referencing, use of AI and Academic Integrity

We expect that you will draw from some literature (at least some of the papers we have

referenced for you for the design principles) in the written components of this assignment. If you draw from any ideas from any source, you need to provide adequate referencing.

Plagiarism is taken very seriously at QUT and multiple methods are employed to detect it. Not

only is it a violation of the academic integrity policy to plagiarise; at a more basic level, it is

unethical to take someone else’s words or ideas and present them as your own. You will not be awarded any marks for sections that include copied text (even if the wording has been changed or rearranged), as the marks for someone else’s ideas and writing do not belong to you. In

addition, if the amount of work that is plagiarised reaches a certain level, we are required to report it.

Ideas that have been developed by others can be included in your work, as long as you

reference where they came from. If you do draw from external sources, such as research

papers, to support your claims and develop your arguments in written parts of this assignment, you will need to reference them appropriately. All references should be in APA format both in

the body of the report and in the reference list.

For information on APA referencing, please see the following link:

https://www.citewrite.qut.edu.au/cite/qutcite.html#apa

Use of AI Tools

A component of this assignment involves the use of AI tools. Note that these should only be used in the way described in the assignment guide. Using AI tools such as ChatGPT to

generate other parts of your assignment without substantially transforming/editing the text

yourself is not permissible, and unlikely to meet the requirements as graded according to the marking criteria.

Remember that these tools cannot attend the lecture or tutorials for you, nor do they have

human-level ability for genuine reflection and critical analysis and as such, are unable to meet the specific requirements of the assignments to the same quality as you can. Overly generic   content that does not relate to specifics of what is taught in the unit and required in this

assignment will not meet essential criteria for a passing grade.

Resources

The following resources will assist with the completion of this task:

.    Weekly videos and shared materials provided on Canvas for tutorials and assistance in using Rasa

.    The Design Principles document and associated readings

.    Templates provided via the Assessment 2 page on Canvas (uploaded in the self-directed learning period)

.    Slack to exchange code and discuss details of the task

.    Request tech help from the technical tutor via Slack or during drop-in sessions

Questions

Questions related to the assessment should be directed initially during the workshop or drop-in sessions or on the appropriate public slack channel. The teaching team may address these

questions publicly for the benefit of the whole class.

Please do not direct message (DM) assignment questions unless they are personally specific. Sensitive or private questions should be directed to the unit coordinator via email.

The teaching team will not be available to answer questions outside business hours, nor immediately before the assessment is due.

Marking Criteria

Although a pass/fail grade is provided for your formative submission (T1), receiving a pass for T1 does not guarantee a pass for your final submission. Regardless of the result of your T1

submission, your final (T2) submission will be assessed against the extended criteria (1-7)

provided below. The formative grade awarded for T1 serves only to help you determine whether you are on track at that stage in the assignment process.

You should note that the assessment does not require you to simply repeat contents of various teaching materials, but you provide evidence of your understanding of the materials and demonstrate your ability to apply that understanding in a way that is effectively communicated to the person marking your assignment.

You will not receive marks or percentages for this assessment. You will receive an overall

qualitative grade (e.g. pass - 4, high distinction - 7) based on the extent to which you meet the criteria.  If the final grade for the assignment calculated based on weighted criteria includes

decimal places, standard  rounding will be used for the final assessment grade (i.e., each assessment grade will be a whole number).

Criteria Sheets – Assessment 1 – IFN623 Human Information Interaction

Tier 1 (Formative feedback) criteria sheet