Web Scraping TILT Information

Purpose

In this assignment, you will decode file formats that are XML-based, pulling out relevant information programmatically and implementing quality-control measures for that data. The techniques you practice here may be applied to pull data out of documents with common formats, map data, and other information provided online, as XML and HTML documents are involved in everything online.

Skills

This assignment will help you practice the following skills which are important for being able to access and work with data:

• Reading and using XML data schemas
• Developing strategies to extract data from an XML document using XPATH and CSS selectors
• Extracting data from XML-coded documents
• Reformatting and tidying extracted data into a form that is workable for statistical analysis
• Implementing quality control measures for data assembled from sources which are not always completely consistent

Knowledge

This assignment will help you to become familiar with important knowledge in this discipline:

• XML data formats
• Tidy data formats
• Functional programming techniques
• Computational time considerations for different strategies for data extraction
• File access time considerations for different data extraction strategies

Success Criteria

Note: These criteria are provided to facilitate self reflection and evaluation. Assignment grades may be decided based on an instructor-selected subset of these criteria.

General Criteria

• Student’s name is included at top of assignment
• Document compiles on a different machine
• Compiled document is formatted well
• All code in the document is contained in appropriately formatted code chunks
• Compiled document does not include long sections of printed data, making use of head and tail commands to show only a few rows of data during data cleaning.
• Student answers are below the --- separator and above the next prompt, or in cases where questions are enumerated, student answers are indicated using the quote indicator, > and the placeholder text Your answer here has been removed and replaced with the answer.

Task-specific criteria

1. Warming Up
   1. Parsing
      • Code chunk reads in the RSS feed from the web address
      • RSS feed is saved as an xml-parseable object
   2. Extracting Information
      • Resulting data frame has columns title, link, summary, updated, category, and id
      • Code demonstrates accessing node attribute(s)
      • Code demonstrates accessing node content
   3. Is it XML?
      • Response lists requirements for valid XML files
      • Response evaluates each listed requirement of an XML file accurately
2. Cleaning the Data
   1. Tidy Data
      • Response identifies aspects of the data which are not tidy
      • Each aspect of the data which is identified as messy is related back to a violation of tidy criteria (one value in a cell, one variable in each column, one observation in each row).
   2. Tidy Planning
      • Steps to transition from current form to tidy form are listed in order
      • Listed steps are adequate to complete the transition
      • Each step contains the necessary information to identify a function and provide the required parameters for that function. For instance, if the step is “Separate values in column 3 into two separate columns”, the description should also include the delimiter or regular expression used to separate the values, and the names of the columns the data will be extracted into.
   3. Tidying, for real!
      • Code included in this step matches the planning in step 2
      • Code works correctly
      • Resulting data shown contains less than 10 entries
      • Data shown is formatted using a
3. Filing Forms
   1. Preparing
      • CSS selector for the table is correctly identified
      • CSS selector for the first link in the table is correctly identified
   2. Reading HTML Tables
      • Function extracts the .htm link from the table when provided with the filing detail URL
      • HTML addresses for all filed forms are obtained successfully
   3. Reading HTML Pages
      • Requisite data is extracted and stored in a data frame
      • (Suggested) Functions are used to modularize data extraction
      • Data extraction code effectively handles missing data without erroring out
   4. Assessing Market Value of Stock Sales
      • Aggregate market value of stock is converted to a numerical variable
      • Agg. market value of stock is used to make a histogram
      • Histogram y-axis indicates that values are counts
      • Histogram x-axis indicates that the plotted value is the aggregate market value of the stock
      • Histogram has an accurate title
      • If histogram values cover several orders of magnitude, an appropriate scale transformation is used
   5. Efficiency
      • Ordered list of steps is complete
      • Steps minimize the number of calls to the SEC server
      • Steps de-duplicate any repeated entries
      • Total number of calls to the SEC server is calculated correctly
      • Calculation of total number of calls is explained