Background

Data visualization has become an important tool for explorative data analysis as well as for presentation and communication of data in many application domains. A domain that has become increasingly data driven over the last decades are biosciences, and in particular when it comes to studies of the microbiome and other genome sequenced data. In this summative assessment, you are asked to  design and  implement  an  interactive  multiple  coordinated views visualization that support analysis of data from a study of the ocean microbiome, using different visualization methods.

The focus of the tasks in the assessment is on visualization of heterogeneous and multivariate (high dimensional) data, interactive visualization and multiple views, heuristic evaluation, and visualization of uncertainty.

Data context

The oceans are the largest cohesive eco-system on earth, and a greater understanding of this eco-system is important for the preservation of the planet as well as for understanding of how organisms have evolved since life began. The data that you will work with originates from a two-and-a-half-year expedition with the schooner Tara, during which oceanic samples were collected from 210 stations across the world oceans. If you are interested, you can read more about the expedition and ocean microbiome here: https://www.embl.org/topics/tara/

User context

The end user of the visualization that you will develop would typically be a microbiologist or another domain expert in a bioscience field. The aim of their analysis would be to increase their knowledge of the ocean microbiome, and analysis questions of particular interest may for example include:

•   Which microbes are detected at the highest levels overall in the oceans?

•   Which microbes are detected at the highest levels in certain regions of the oceans?

•   Are there differences in microbe detection levels that can be linked to other features of the oceanic samples, for example the geographic region, sample depth etc?

•   Are there differences between taxonomic levels, which can be linked to other features of the oceanic samples?

The data

You will  be provided with a set of different spreadsheets to work with, which have gone through some initial formatting and cleaning. The full dataset include data related to 135 samples that were taken from different oceanic regions.

The detection levels of 35,650 Operational Taxonomic Units (OTUs) were recorded for the individual samples. Detection levels are sometimes referred to as the abundance of the OTU. OTUs are close approximations of microbial species, which are extracted through clustering of DNA sequences, so you can think of an OTU as being the same as a microbial species (such as  a  bacterium) .  The  OTUs  also  have  an  associated  hierarchical  taxonomy  through  the biological classification system (https://en.wikipedia.org/wiki/Taxonomy_(biology)), and are often converted into higher levels in the taxonomy for analysis, since an OTU name generally has no biological meaning. Analysis is quite often carried out and reported at Genus level.

In addition to the OTU detection levels, there area range of contextual data associated with the samples (i.e. metadata) . From a data science and visualization perspective, the OTUs are generally treated as data variables (dimensions) and the samples are data items.

You will be provided with the following datasets, in comma separated file format (csv):

•   Tara_OTUtableTax_full.csv:  Each  row  in  this  file   corresponds  to  a   unique   OTU (microbial species). The first six columns include the taxonomic classification for each OTU at the following hierarchical levels: Domain, Phylum, Class, Order, Family, Genus. The original taxonomic classification of the OTUs included a lot of missing values, as a result of OTUs that were not identifiable at all levels in the taxonomy. The highest level where nearly all OTUs were identified was the Class level. Due to this, the missing values have been replaced with the Class name of the OTU, followed by (undef) (i.e. a Cyanobacteria OTU is referred to as Cyanobacteria(undef) at all levels where it has not been classified). The seventh column include a unique OTU-id, which has no biological meaning. The remaining columns each correspond to a sample, with a unique sample id as heading. The cells represent the relative detection level (relative abundance) of OTUs in samples as a percentage value, thus the sum of each column is 100%.

•   Tara_OTUtableTax_80CAb.csv:  This  file   includes   a  subset   of  the   same  data  as Tara_OTUtableTax_full.csv.  It  is  reduced  to  include  only  the  1400  most  abundant OTUs, which make up 80% of the total cumulative abundance of the full dataset.

•   Tara_OTUtableTax_80Cab_transp.csv:  This  file   includes   a  transposed  version   of Tara_OTUtableTax_80Cab.csv,  without  the  taxonomy.  In  this  dataset  each  row represent  a  sample  and  each  column   represent  an  OTU,  with  the  first  column representing the sample id.

•   Tara_SampleMeta.csv: Each row in the file correspond to a sample, with sample id’s that  are  identical  to  those   in  the   OTU  tables.  The   columns   include  contextual information      about      the      samples,      including:      SampleID,      Year,       Month, Latitude[degreesNorth],  Longitude[degreesEast], SamplingDepth[m],  LayerOfOrigin, MarinePelagicBiome, OceanAndSeaRegion, MarinePelagicProvince.

You can choose yourself which version of the OTU table to use, and are welcome to perform any data wrangling or modification using a tool of your choice prior to visualization.

The assignment

The coursework consists of three parts, which are detailed below. Submission and implementation details are provided at the end of the document.

Part 1: Interactive visualization using multiple coordinated views (60%)

The first and main task of the coursework is to design and implement an interactive multiple coordinated views visualization that support exploration of the Tara Ocean data, using one of the OTU tables and the sample metadata. The final multiple coordinated views visualization should be saved and submitted as an html page.

The aims of the visualization are to:

1.   Help the user understand overall abundance patterns and diversity in the oceans: the user would typically be interested in knowing which the most abundant microbes are, and if there are large variations in abundance between different microbes.

2.   Help the user understand some of the abundance patterns and diversity in the oceans in context of the sample meta data: e.g. Are there differences in abundance profiles between different sample classes? What does such differences tellus about the Ocean microbiome in context of the sample categories?

3.   Help the user get an overview of the microbiome while also being able to investigate details and patterns of potential interest in more detail: a user may, for example, be interested to know if there are differences between different taxonomic levels, to identify and explore patterns that are visible only in subsets of data, or to compare specific subsets of samples in more detail.

For full marks you are expected to include at least three views in your visualization, which are interactively coordinated and display different aspects of the data. You are also expected to take accessibility and user diversity into consideration.

Fill in the relevant parts of the submission table to demonstrate your approach to meeting the aims. You need to demonstrate in the table your use of visualization theory and principles in  the  design,  and  to  justify  design  choices  made.  You  are  expected  to  also  reflect  on alternative visualization approaches and methods, and how these could have been used.

Part 2: Uncertainty in data (10%)

Based on your visualization in part 1: Reflect on potential sources of uncertainty in the data, and how you could approach visualizing them. You do not have to implement anything for this but fill in the relevant part of the submission table.

Part 3: Heuristics evaluation (20%)

Based on your visualization in part 1: Reflect on how the visualization meet the visualization heuristics of Wall et al. (2019), and how you could modify the visualization to better meet these heuristics. You do not have to implement anything for this and should not carryout an evaluation with other participants but fill in the relevant part of the submission table.

Note: marking in this section is not based on if you meet the heuristic criteria, but on your understanding of how the heuristics could be met. Hence, not meeting a heuristic criterion but  having a good suggestion of how you could meet it may be marked equally high as meeting the heuristic.

Use of language/tools

You  must  use  Python  and  are  recommended  to  use  the  Altair  and  Pandas  packages  for creating the interactive multiple coordinated views visualization in part 1. You are allowed to use other Python visualization packages, although there will be limited technical support for them and you  must  make sure you are able to generate an  html version of the  multiple coordinated views visualization.

You are free to use any language or software of choice for any data wrangling that you need to  do.  Make  sure  to  detail  in  the  appendix  and  reference  list  in  your  submission  which packages and software you have used (Python and non-Python).

What to submit

Coursework

•   Submit in Canvas a single zip file including:

o Report: A document including the submission table with details and justification of your visualization and design choices, and a list of references to sources used to  carry  out  the   project,  in  pdf  format.   References  in  the   report  must  be consistently cited in a standard way.

o Visualization: The html page with your multiple coordinated views visualization from part 1 (note : this should not bean html version of a Jupyter Notebook, but an html-file saved using Altair’s ‘save’ functionality or similar).

o Code: Your Python code and the datasets that are loaded by the code.

The coursework submission deadline is 16:30 on Thursday 22rd February.

Oral presentation

Submit in Canvas a short (5-7 min) video demonstration of your visualization and its interactive features. The videos will be shared with others in the module when all have submitted. Video recordings can be made using, for example, Zoom or Microsoft Teams, by recording a meeting where you share your screen.

The video submission deadline is 16:30 on Friday 23rd February.

Marking Scheme

The following marking scheme will be used