MARK 5828 Product Analytics

Individual Research Paper Instruction (39 Mark)

Description

        By doing group projects, you designed conversational flow, built chatbot and did chatbot A/B tests with your group members. Now, you have an independent ability to do chatbot research. For your individual research project, you will do the same type of A/B test with your own hypotheses.

        To test your hypothesis, you can use the chatbot which your group has developed. If you prefer, you can also build a small chatbot considering your hypotheses.

Deadline: W9 Friday 10 pm July 30

Marking Criteria https://docs.google.com/spreadsheets/d/19xv6z7Z6FitkYzuRoaqTw_N6k9rwpuAaJMucNykiMyU/edit?usp=sharing

Content:

<1> Title page (1 page)

a. Title of your paper

b. Author: Course Name, Tutorial session and group number (W15_1), name, and zID

c. Abstract: An executive summary of your research paper

<2> Body of your report (12~15 pages)

1. Hypothesis + Literature

        You will propose 2 new hypotheses. You need to explain your hypotheses logically and the importance of your hypotheses by citing either academic literature, industry reports, or news articles. The A/B test is straightforward. Hence, your hypotheses are the most important part of your research paper.

        One of the early applications of a chatbot is customer service like call centre chatbots. The call centre chatbots are getting smarter and now manage the complaints of customers more effectively by understanding the emotion of customers and answering with appropriate emotions. Now, companies are using chatbots more actively to attract new customers or recommend new products for them. You can choose any application area. Think about how a chatbot can do a better job to address the customer’s request more effectively. Recently, voice-bots such as voice assistants are quite popular. Some companies combine chatbots with VR (Virtual Reality) or AR (Augmented Reality).

        Read recent news articles or industry reports about chatbots. Then, you can easily find what issues are important for better chatbots. Depending on application and industry, the issues may be quite different. With your initial idea, read academic papers about chatbots or AI. You can find what research questions have been addressed and what research questions need to be investigated more. You can also learn how one can make and write hypotheses by citing relevant theories or finding in the previous academic papers.

1.1.How to generate hypotheses?

You need to figure out new important X or Y variables to make new important hypotheses. Before you decide on your hypotheses, please read many academic papers. Then, some hypotheses will come to your mind.

1.1.1. Examples of hypotheses

● Chatbot’s empathetic tone reduces users’ negative emotions.

● Chatbot’s passionate tone cheers users up and increases service satisfaction.

● Chatbot’s polite tone makes the user also talk more politely.

● Talkative chatbot increases consumer’s engagement.

● Chatbots that use intelligent words more easily persuade consumers to accept product recommendations.

● Localized voice assistants (e.g., chatbot with British (vs. American) accent) will increase purchase intention of British consumers.

1.1.2. Potential X variables (This is about your Hs)

● You can add new chatbot features.

● Or you can simply change conversational words, flow, style, tone, sequence, or the number of questions.

● Conversation styles (e.g., chatbots talk more politely or chatbot use emoticons frequently.)

● The different sequence of conversation (e.g., when to mention “sell product”? early or later?)

● Matching the types of chatbot and users (e.g. personality or emotion matching)

1.1.3. Potential Y variables

● During conversation

o The emotion of the user (e.g. Chatbot’s empathetic tone reduces users’ negative emotion).

o Shorter conversation duration (to save consumer’s ordering time)

● After finishing the conversation, you could ask users of

● Willingness to buy, book, or register

● Willingness to recommend this product to your friend, family, or neighbour

● How much you are satisfied with the conversation

● How much knowledgeable chatbot is

● Test at least 2 Y variables

1.1.4. Where to find hypotheses about chatbot research?

● Academic papers

o Every academic paper has research questions. They state why the research question is important to investigate and test their hypothesis using data. You can feel free to benchmark it.

o “Chatbot”, “AI” Dropbox folders which I shared

o https://medium.com/@ODSC/top-10-ai-chatbot-research-papers-from-axxiv-org-in-2019-1982dddabdb4

o https://www.topbots.com/most-important-conversational-ai-research/

o http://www.academia.edu/Documents/in/Chatbot

o https://paperswithcode.com/task/chatbot

o https://www.chatbots.org/papers/

o https://link.springer.com/article/10.1007/s12525-020-00414-7

o Google Scholar: https://scholar.google.com.au/

o Scopus: https://www.scopus.com/

o Management Science: https://pubsonline.informs.org/journal/mnsc

o Marketing Science: https://pubsonline.informs.org/journal/mksc

o Information System Research: https://pubsonline.informs.org/journal/isre

o Journal of Marketing Research: https://www.ama.org/publications/JournalOfMarketingResearch

o Service Science: https://pubsonline.informs.org/journal/serv

o You can also look at many other academic journals.

● Newspaper articles, blog, industry reports

o You can easily find many potential research questions about chatbot and AI by reading industry trends or applications about chatbot.

o https://sproutsocial.com/insights/chatbot-marketing-examples/

o https://www.hubspot.com/stories/chatbot-marketing-future

o https://www.wordstream.com/blog/ws/2017/10/04/chatbots

o https://www.artificial-solutions.com/chatbots

o https://mobilemonkey.com/blog/chatbot-marketing/

o https://buffer.com/resources/chatbot-marketing/

o https://sendpulse.com/support/glossary/chatbot-marketing

o https://www.intercom.com/blog/chatbot-marketing/

o https://towardsdatascience.com/how-conversational-chatbots-marketing-is-the-future-of-ecommerce-6743268caa11

o https://www.intercom.com/blog/chatbot-marketing/

1.2.The requirement of your Hypotheses (Hs)

(1) 2 Hypotheses ( in terms of X)

(2) Specific hypotheses: you need to make specific hypotheses, rather than exploration type of questions

a. Bad: which chatbot’s communication style is the most effective?

b. Good: Chatbot’s passionate tone increases service satisfaction.

(3) Testable hypotheses: To test your hypotheses, you need to be able to measure the outcome. Thus, make sure whether the necessary outcome can be measured.

(4) Important hypotheses: Justify the importance of your research question by citing newspaper articles, industry reports, blogs, or academic papers. You can benchmark how to write it by reading several academic research papers.

a. Either academic, industry (i.e. practical or managerial), or society (i.e. community) value is excellent. In the news article or industry report, they claim that some chatbot features are effective. These claims can be important research questions because industry people believe them from their field experience. However, the claims are often based on their experience, not a formal statistical test. Therefore, you can test whether the insight from industry is indeed effective or not.

b. One way to justify the importance of your research question is to cite relevant theory. For example, the colour could make emotional appeals. Warm and cool colours have different emotional appeals. There are a variety of psychological theories about colour effects. Such theories often can guide you to come up with new hypotheses. Also, by citing those theory papers, you can justify the importance of your research questions as well.

c. How to find such theory papers? The theory was often established a long time ago. It is hard to discover such an old theory paper directly. When you start to read recent articles, most papers cite relevant theory papers to support their hypothesis. If you think that those theory papers are relevant to your research questions, you can cite them.

d. Sometimes, two theories predict the opposite effect. This is a good opportunity for you. By citing those two theories, you can say that we want to test which theory works in the real marketplace.

e. Often, academic theory and industry people (e.g., marketing managers)’s beliefs are different. This is another excellent opportunity for you to test your research questions using your data.

f. Academic papers often suggest important future research topics. Then, you can cite these papers to justify the importance of the research questions.

g. Please do not make too obvious research questions that people can answer easily without any formal test.

(5) New (unique) hypotheses: your research idea (and your Hs) needs to be a unique one. In other words, please try to make new Hs from existing academic papers. One easy way is to read recent newspaper articles or industry reports about chatbots. Let’s support that voice assistants recently added new important features but no academic paper examined the effectiveness of these features. Then, this research is likely to be new.

2. Research Design

Depending on your hypotheses, your research design can be different.

2.1.Testing two new chatbot features

● The company often upgrades their chatbot with better features. They need to test whether their new feature is effective.

● In this case, you need to compare your baseline chatbot A and your new chatbot B with the new feature.

● If you use a simple A/B test, you can test only ONE new feature at each test.

● Thus, you make two additional chatbot B1 and B2 for your two hypotheses H1 and H2.

● In total, you need to have 3 versions of chatbots: A, B1, and B2.

● Note that your chatbot A, B1, and B2 are the same except the additional new features.

2.2.Testing two types of chatbots with two types of users

● Let’s suppose that company considers matching the right type of chatbot to different types of consumers. The examples of such types are gender, ethnicity, age, emotion, and personality.

● For example, you make a male chatbot A and a female chatbot B. Then, depending on the context, matching with the same genders might be more effective than the opposite matching with different genders. Or, the opposite matching might be better in other contexts.

● Example of hypotheses:

o For the recommendation of expensive products such as a car or a house, the opposite matching between a chatbot and a user is better for a user to accept the recommendation because one may consider that expert has a different type with himself.

o For the recommendation of entertaining products or services such as movies or music, matching the same types between a chatbot and a user is better for a user to accept the recommendation because one may think that similar person can know his taste.

● In this case, you need only two versions of chatbots A and B.

● Then, you invite two different types of users A and B (e.g., Females vs. Males, Old vs. Young, Extrovert vs. Introvert)

● Then, you have 4 conditions: 2 types of chatbots x 2 types of users.

3. Building Chatbot

● The baseline chatbot could be the chatbot of your group project. Or, you could make a very simple chatbot.

● How many chatbots? Depending on your research design, you might need 2 or 3 types’ chatbots.

● In your individual paper report, briefly describe each chatbot’s conversational flow by emphasizing the new features which you would propose. You can also put a screenshot of your chatbot. Please put only the key part of each chatbot communication in the body and put the others in the appendix. You will need to submit JSON files for your chatbot (Watson Assistant and Node-red). See the instruction at the end of this instruction about downloading JSON files from your chatbot.

● Note that new features do not have to be complex. Depending on your hypotheses, you could change just some words. For example, you can add more polite sentences if your hypothesis is about polite chatbot. (But, make sure that your Hs are new!)

● To save your time, you can export the JSON file of your chatbot A. Then, when you make a new chatbot B, you can import the JSON file of chatbot A and then revise it.

4. Experiment Design

4.1.Online survey form

● How to make Google Form: https://www.youtube.com/watch?v=BtoOHhA3aPQ

● Chatbot A/B test survey example: https://docs.google.com/forms/d/e/1FAIpQLSdhkJuBuF6bZf0k8SAcFFxh-7BdEnxW9wWGIDVL_ehd9aXUxg/viewform

● Create an interface of your chatbot

o IBM interface: https://cloud.ibm.com/docs/assistant?topic=assistant-deploy-web-link

o You could use another interface as long as you can get a web link. 

4.2.Outcome measures 

Make relevant survey questions to test your hypotheses. The examples are

● Example of satisfaction question

▪ (Quantitative) How much are you satisfied with the conversation with a chatbot?

● Not satisfied at all (1 to 7) highly satisfied

▪ (Qualitative) Tell me about your good or bad experience with the chatbot

● Example of knowledgeable chatbot question

▪ (Quantitative) How was the agent's knowledge?

● Not knowledgeable (1 to 7) Very knowledgeable

▪ (Qualitative) Tell me about your impression of the agent’s knowledge.

4.3.The sequence of chatbot in survey

● If everybody talks with chatbot A and then B, chatbot B might have a better result because users become familiar with chatbot communication after they talk with chatbot A.

● For a fair comparison, while some users talk with chatbot A and then B, others do with chatbot B and then A.

● If you have 3 versions of chatbots A, B1 and B2, ideally, you need to make 6 different sequences (= 3 x 2 x 1). However, since it takes time to invite many survey participants, let’s simplify it. Let’s show chatbot A first and then change the sequence of chatbot B1 and B2 as follows.

o A, B1, B2

o A, B2, B1

● In each survey, ask some necessary demographic questions such as gender, age, and major. You can also ask other questions about user personality and so on depending on your hypotheses.

● Survey 1

o Talk with chatbot A, and then answer Quantitative & Qualitative questions

o Talk with chatbot B1, and then answer Quantitative & Qualitative questions

o Talk with chatbot B2, and then answer Quantitative & Qualitative questions

● Survey 2

o Talk with chatbot A, and then answer Quantitative & Qualitative questions

o Talk with chatbot B2, and then answer Quantitative & Qualitative questions

o Talk with chatbot B1, and then answer Quantitative & Qualitative questions

4.4.Online survey

● Classmates will do the survey. Put your survey link in the “Individual Survey” in “Gr” G-sheet to any person except their group members. 5 links for Survey 1 and another 5 links for Survey 2. Then, email 6 students and ask them to answer the survey within 24 hrs. Therefore, each student will answer 10 surveys. If students already have 10 survey links, please do not put your survey link for those students. By answering the surveys, you can help and learn from your classmates. 

o https://docs.google.com/spreadsheets/d/1zgkLZZwVgIuBT4ZZM1F8uOec5yuNK-kLST0-erKIEaw/edit#gid=108759918

● You can also email your friends, family members or your company mentors to have more respondents.

● Survey deadline: W8 Tuesday 10 pm July 20, (If you receive a survey email but you don’t complete it within 48 hrs, the penalty is -1 point per hour).

● Please put the links for your online survey and chatbot in your paper.

4.5.Analysis

4.5.1. Demographics

● Summarize the demographics (gender, ….) of survey participants briefly.

4.5.2. Quantitative questions

● First, plot survey participant’s answers by comparing the answers from two chatbots. You could use two box plots. Or, you can use different types of plots.

● Survey participants will talk with several chatbots within the survey. Thus, we can test their answers using the paired t-test (p-value).

● https://pythonfordatascienceorg.wordpress.com/paired-samples-t-test-python/#:~:text=The%20paired%20sample%20t%2Dtest,difference%20between%202%20related%20variables.&text=Null%20hypothesis%20(H0)%3A,2%20is%20equal%20to%200.

● https://www.statisticssolutions.com/manova-analysis-paired-sample-t-test/#:~:text=The%20paired%20sample%20t%2Dtest,resulting%20in%20pairs%20of%20observations.

● What is paired t-test?

● Example: Satisfaction for chatbot A and B

o User1: 1 for A, 3 for B

o User2: 4 for A, 7 for B

● Step1: Make a difference

o User1: 3 - 1 = 2

o User2: 7 - 4 = 3

● Step2: Mean (Average) of difference: (2 + 3) / 2 = 2.5

● Step3: statistical test:

o Null hypothesis

▪ Whether the satisfaction score from chatbot A and B are equal

▪ In other words, it tests whether the difference in satisfaction scores from chatbot A and B is zero

o You want to reject this null hypothesis because you want to claim that your new chatbot B is better than chatbot A

o As the difference increases, the absolute value of t-value increases and its P-value decreases.

o If P-value < 0.05, we say that the difference is significantly different from zero. In other words, the satisfaction with chatbot B is significantly different from that of chatbot A.

● Python code for the paired t-test is posted. You can put raw data (i.e., participants’ answers). Then, the paired t-test function will do all the steps for you. You just need to check the P-value.

o https://colab.research.google.com/drive/1VRfhmeCeRgMia7ja11iETeNca3tzFFP4?usp=sharing

● In your paper, interpret your result properly based on the p-value.

4.5.3. Qualitative questions

● You can quote some comments.

o E.g. “I like the chatbot because it gives me relevant career option”

● You can manually count the number of “positive”, “neutral”, “negative” comments by survey participants for each chatbot. Then, you can compare the performance of the two chatbots.

● You don’t need a statistical test here.

5. Conclusion

● What are the key results? Does the result support your hypotheses? If not, explain why your hypotheses are not supported.

● Who will be the target organisation which would benefit from your analysis and suggestion? It does NOT have to be a commercial company. It can be a non-profit organisation or government.

6. Future Research

● By extending your current hypotheses, what other hypotheses do you want to test in the future?

● What other chatbot research do you want to do in the future?

<3> Reference (no page requirement)

- List academic papers, newspaper articles, blogs, or industry reports which you cite.

- How to reference? https://student.unsw.edu.au/referencing

<4> Appendixes (no page requirement)

- Put your survey questions here.

- Do not put irrelevant or unedited raw results.

Format

● Use word file (.doc), 12pt, 1.5 lines spacing, at least 2.5cm margins on all sides.

● Make sentences rather than bullet points.

● Revise your paper several times.

Submission

● Submit the word file of your report, chatbot, data, code, and survey into Moodle.

● File name: “PA_Research_tutorial session_zID_first_last name.doc” (e.g., “PA_Research_W15_zID_first_last name”.doc)

● .doc contains your research paper.

● .ipynb contains all relevant code to get the results in your report.

● .csv contains the dataset from your survey.

● .xlsx contains the link for survey and chatbot preview

● .JSON contains your chatbot. Submit all your chatbots.

● Late penalty: -1 mark per hour and -20% marks per day

Downloading Chatbot for Submission

1. Launch the watson assistant

2. The landing pages shows the list of assistants, click on the relevant assistant.

3. Now, you can see the dialog skill that is currently attached to the assistant. This is the brain (simply put)

4. Click on the three dots, on the dialog skill card and proceed to download. This is your assistant in a json format with all the trained intents, entities, and nodes. Your team will be submitting this json file (only the finalized one) as your chatbot (one per group).