Record-based data and List Processing

Complex Data structures

How would you encode this structure in python? how in R?

Lists

Lists are a superset of all data structures

• allow multiple data formats
• allow items that are lists

Lists are python’s basic data form:

[4, [2, [10, 13, [5, 6]], 7]]

(needs convention of [ starts children nodes and ] finishes list of children)

In R there are list objects list()

Tree in R

tree <- list(value = 4, children = 
               c(list(value = 2, children = 
                        c(list(value = 10, children = NULL), 
                          list(value = 13, children = 
                                 c(list(value = 5, children = NULL),
                                   list(value = 6, children = NULL))))), 
                 list(value = 7, children = NULL)))

str(tree)

List of 2
 $ value   : num 4
 $ children:List of 4
  ..$ value   : num 2
  ..$ children:List of 4
  .. ..$ value   : num 10
  .. ..$ children: NULL
  .. ..$ value   : num 13
  .. ..$ children:List of 4
  .. .. ..$ value   : num 5
  .. .. ..$ children: NULL
  .. .. ..$ value   : num 6
  .. .. ..$ children: NULL
  ..$ value   : num 7
  ..$ children: NULL

Trees

there are no native ‘tree’ objects in R or python (there are packages for sure), i.e. no standard implementation for trees

implementation as data frame:

# A tibble: 7 × 3
  node_id value children 
    <int> <dbl> <list>   
1       1     4 <dbl [2]>
2       2     2 <dbl [2]>
3       3     7 <NULL>   
4       4    10 <NULL>   
5       5    13 <dbl [2]>
6       6     5 <NULL>   
7       7     6 <NULL>   

Your Turn: Web Page Anatomy

Use the JSON specification at https://www.json.org/json-en.html to write out the tree

XML can be expressed as list

Coerce list to xml: as_xml_document

xml to list: xml2::as_list

We need to be able to navigate along lists

Working with lists

• [ accesses sub lists

• [[ accesses elements

• for well-structured lists more powerful tools: map (purrr)

Package purrr: https://purrr.tidyverse.org/reference/index.html

Example: Employee Data

Relational Overview

Identify the Structures - follow the hierarchy

xml_children recovers the children of an XML node

{xml_nodeset (6)}
[1] <EMPLOYEE_INFO>\n  <EMPLOYEE>\n    <ID>2824</ID>\n    <first>Michael</fir ...
[2] <BACKGROUND_CHECK>\n  <BACKGROUND_CHEC>\n    <ID>39089</ID>\n    <EMPLOYE ...
[3] <TAX_INFO>\n  <TAX_INF>\n    <ID>93941</ID>\n    <EMPLOYEE_ID>2824</EMPLO ...
[4] <TRAINING>\n  <TRAININ>\n    <ID>67510</ID>\n    <TRAINING_TYPE>3</TRAINI ...
[5] <EMP_AGREE>\n  <EMP_AGRE>\n    <ID>14107</ID>\n    <EMPLOYEE_ID>2824</EMP ...
[6] <ORG_REL>\n  <ORG_RE>\n    <ID>80344</ID>\n    <EMPLOYEE_ID>2824</EMPLOYE ...

Employee Data

{xml_node}
<EMPLOYEE>
[1] <ID>2824</ID>
[2] <first>Michael</first>
[3] <last>Phillips</last>
[4] <middle>E</middle>
[5] <address>341 Bonilla Extensions\nLake Jacktown, VA 72058</address>
[6] <phone>+1-701-028-0259x700</phone>
[7] <ssn>821097324</ssn>

Get all elements of one record

Your Turn

How do we need to modify the function make_row so that we can extract data for the background check?

Is your solution applicable to all the tables of the Employee database?

Solution

In practice we need to use some error handling