List Processing TILT Information

Purpose

Data scientists and statisticians must be able to take data that is in its natural, messy form and transform it into tidy data, making and documenting the transformations and considering their impact on inference and interpretation of the data. Many data formats, such as JSON, XML, and YAML, do not assume a rectangular data format where observations are in rows and variables are in columns. There are also many situations where data may be nested or networked in its natural form. It is important to be able to convert between formats, identifying the critical pieces of information in any data structure and developing a strategy to convert the data into something that can be analyzed or visualized effectively. This assignment will help you understand and leverage list processing techniques to tidy nested and record-based data.

Skills

This assignment will help you practice the following skills which are important for being able to access and work with data:

• Identify the structure of JSON and XML files
• Use functional programming to apply functions to lists in order to extract and/or process data efficiently
• Transform record-based formats into one or more rectangular table(s) that may be linked by a key variable.
• Identify areas where quality control checks may be necessary when converting data between record-based and tabular formats

Knowledge

This assignment will help you to become familiar with important knowledge in this discipline:

• Record-based data structures
• Reading and constructing database schema (descriptions of variables in data tables)
• Functional programming techniques

Success Criteria

General Criteria

• Student’s name is included at top of assignment
• Document compiles on a different machine
• Compiled document is formatted well
• All code in the document is contained in appropriately formatted code chunks
• Compiled document does not include long sections of printed data, making use of head and tail commands to show only a few rows of data during data cleaning.
• Student answers are below the --- separator and above the next prompt, or in cases where questions are enumerated, student answers are indicated using the quote indicator, > and the placeholder text Your answer here has been removed and replaced with the answer.

Task specific Criteria

1. Warming Up
   1. Parse the file
      • Code chunk is added correctly
      • Three drwhoYYYY objects are created: drwho1963, drwho2005, and drwho2023
      • [ ]
   2. Examining List Data Structures
      • List structure is properly indented to show different levels of nesting
      • List structure is in valid markdown and renders as an unordered, nested list
      • Redundant data question is answered thoughtfully
   3. Develop a Strategy
      • Rectangular data structure contains all relevant information
      • Sketch is included successfully, either using one or more markdown tables or an image.
      • A reasonable sequence of operations are provided in an ordered list
   4. Implement Your Strategy
      • Code chunk is added correctly
      • JSON files are converted into tabular data
      • Variables in tabular data are correctly formatted using appropriate types
      • First 5 rows of each table are printed out
   5. Examining Episode Air Dates
      • Plot has appropriate axes, axis labels, and title
      • The geom used in the plot is appropriate for displaying the data
      • Plot shows length between adjacent episodes by season across all seasons
      • Different series of Dr. Who are handled successfully
      • Plot is described appropriately and any visually interesting patterns are mentioned.
2. Timey-Wimey Series and Episodes
   1. Setting Up
      • Code to read in files looks for JSON files in the working directory and then applies a function to read in data from each file (e.g. functional programming)
      • Processing code is very similar to that used in previous sections – that is, the code is flexible enough to be reused.
      • data table is appropriately named
      • data table generated is reasonably tidy and has appropriate types in the columns
   2. Air Time
      • Investigative code is provided and follows a logical sequence
      • Conclusions are drawn relating airtimes to target audiences.
      • Shows without airtimes (Torchwood: Web of Lies) are explained in a reasonable manner.
   3. Another Layer of JSON
      • JSON files are read in using a functional approach
      • Minimal server calls are used
      • Keys provided for joins are reasonable, and meet the technical definition of a key.
   4. Explore!
      • Question requires at least a couple of processing steps before it is possible to answer.
      • Graphics or tables are well formatted and selected to convey appropriate information.
      • Graphics or tables are explained with at least 2-3 sentences
      • Code makes use of functional programming.