APIs

URL Anatomy

\[\underbrace{\text{https}}_{scheme}\text{://}\underbrace{\text{archive.org}}_{endpoint}/\underbrace{\text{wayback/available}}_{path}?\underbrace{\text{url=nyt.com}}_{query}\]

\[\underbrace{\text{https}}_{scheme}\text{://}\underbrace{\text{srvanderplas.github.io}}_{endpoint}\text{/}\underbrace{\text{stat-computing-r-python/part-advanced-topics/05-APIs.html}}_{path}\underbrace{\text{#http-response-codes}}_{fragment}\]

URL Anatomy

\[\begin{align} scheme &\ \ \ \text{https}\text{://}\\ endpoint &\ \ \ \text{www.google.com}\\ path &\ \ \ \text{/search} \\ my\ actual\ query &\ \ \ \text{?q=parts+of+a+url} \\ possibly\ experiment\ version\ info?? &\ \ \ \text{&sca_esv=d64738122a810e98} \\ initiated\ via\ homepage &\ \ \ \text{&source=hp}\\ engine\ identifier\ (meaning\ unknown) &\ \ \ \text{&ei=aKPQaLvEMrmw0...}\\ internal\ link\ tracking\ info &\ \ \ \text{&ved=0ahUKEwi7mLm...}\\ my\ query\ again? &\ \ \ \text{&oq=parts+of+a+url}\\ Google\ Web\ Search\ Wizard &\ \ \ \text{&sclient=gws-wiz} \end{align}\]

Guide to Google search parameters

APIs without Authentication

library(httr2)
library(dplyr)
library(jsonlite)
url <- "https://openlibrary.org/search.json?q=the+lord+of+the+rings"

res <- request(url) |>
  req_perform()
res

tmp <- res |> resp_body_string() |> fromJSON()
# tmp[1:7]
tmp$docs |> select(author_key, author_name, title) |> head()
<httr2_response>
GET https://openlibrary.org/search.json?q=the+lord+of+the+rings
Status: 200 OK
Content-Type: application/json
Body: In memory (50738 bytes)
  author_key    author_name                               title
1   OL26320A J.R.R. Tolkien               The Lord of the Rings
2   OL26320A J.R.R. Tolkien                      The Two Towers
3   OL26320A J.R.R. Tolkien          The Fellowship of the Ring
4   OL26320A J.R.R. Tolkien              The Return of the King
5   OL26320A J.R.R. Tolkien Novels (Hobbit / Lord of the Rings)
6   OL26320A J.R.R. Tolkien                          The Hobbit
import requests
import pandas as pd

url = "https://openlibrary.org/search.json?q=the+lord+of+the+rings"
req = requests.get(url)
res = req.json()
pd.DataFrame.from_dict(res['docs']).head()
   author_key       author_name  ...                                title  subtitle
0  [OL26320A]  [J.R.R. Tolkien]  ...                The Lord of the Rings       NaN
1  [OL26320A]  [J.R.R. Tolkien]  ...                       The Two Towers       NaN
2  [OL26320A]  [J.R.R. Tolkien]  ...           The Fellowship of the Ring       NaN
3  [OL26320A]  [J.R.R. Tolkien]  ...               The Return of the King       NaN
4  [OL26320A]  [J.R.R. Tolkien]  ...  Novels (Hobbit / Lord of the Rings)       NaN

[5 rows x 17 columns]

API key authentication

library(httr2)
library(dplyr)
library(jsonlite)
api_key <- 1 # Test key for educational purposes
base <- "http://www.themealdb.com/api/json/v1/"
query <- "/search.php?s=Pizza"

url <- paste0(base, api_key, query)


res <- request(url) |>
  req_perform()
res

tmp <- res |> resp_body_string() |> parse_json()
<httr2_response>
GET https://www.themealdb.com/api/json/v1/1/search.php?s=Pizza
Status: 200 OK
Content-Type: application/json
Body: In memory (2563 bytes)
import requests
import pandas as pd
api_key = 1 # Test key for educational purposes
base = "http://www.themealdb.com/api/json/v1/"
query = "/search.php?s=Pizza"
url = base + str(api_key) + query
req = requests.get(url)
res = req.json()
res
{'meals': [{'idMeal': '53014', 'strMeal': 'Pizza Express Margherita', 'strMealAlternate': None, 'strCategory': 'Miscellaneous', 'strArea': 'Italian', 'strInstructions': '1 Preheat the oven to 230°C.\r\n\r\n2 Add the sugar and crumble the fresh yeast into warm water.\r\n\r\n3 Allow the mixture to stand for 10 – 15 minutes in a warm place (we find a windowsill on a sunny day works best) until froth develops on the surface.\r\n\r\n4 Sift the flour and salt into a large mixing bowl, make a well in the middle and pour in the yeast mixture and olive oil.\r\n\r\n5 Lightly flour your hands, and slowly mix the ingredients together until they bind.\r\n\r\n6 Generously dust your surface with flour.\r\n\r\n7 Throw down the dough and begin kneading for 10 minutes until smooth, silky and soft.\r\n\r\n8 Place in a lightly oiled, non-stick baking tray (we use a round one, but any shape will do!)\r\n\r\n9 Spread the passata on top making sure you go to the edge.\r\n\r\n10 Evenly place the mozzarella (or other cheese) on top, season with the oregano and black pepper, then drizzle with a little olive oil.\r\n\r\n11 Cook in the oven for 10 – 12 minutes until the cheese slightly colours.\r\n\r\n12 When ready, place the basil leaf on top and tuck in!', 'strMealThumb': 'https://www.themealdb.com/images/media/meals/x0lk931587671540.jpg', 'strTags': None, 'strYoutube': 'https://www.youtube.com/watch?v=Mt5lgUZRoUg', 'strIngredient1': 'Water', 'strIngredient2': 'Sugar', 'strIngredient3': 'Yeast', 'strIngredient4': 'Plain Flour', 'strIngredient5': 'Salt', 'strIngredient6': 'Olive Oil', 'strIngredient7': 'Passata', 'strIngredient8': 'Mozzarella', 'strIngredient9': 'Oregano', 'strIngredient10': 'Basil', 'strIngredient11': 'Black Pepper', 'strIngredient12': '', 'strIngredient13': '', 'strIngredient14': '', 'strIngredient15': '', 'strIngredient16': '', 'strIngredient17': '', 'strIngredient18': '', 'strIngredient19': '', 'strIngredient20': '', 'strMeasure1': '150ml', 'strMeasure2': '1 tsp ', 'strMeasure3': '15g', 'strMeasure4': '225g', 'strMeasure5': '1 1/2 tsp ', 'strMeasure6': 'Drizzle', 'strMeasure7': '80g', 'strMeasure8': '70g', 'strMeasure9': 'Peeled and Sliced', 'strMeasure10': 'Leaves', 'strMeasure11': 'Pinch', 'strMeasure12': ' ', 'strMeasure13': ' ', 'strMeasure14': ' ', 'strMeasure15': ' ', 'strMeasure16': ' ', 'strMeasure17': ' ', 'strMeasure18': ' ', 'strMeasure19': ' ', 'strMeasure20': ' ', 'strSource': 'https://www.dailymail.co.uk/femail/food/article-8240361/Pizza-Express-release-secret-recipe-Margherita-Pizza-make-home.html', 'strImageSource': None, 'strCreativeCommonsConfirmed': None, 'dateModified': None}]}

API key authentication

  • Generally, you’ll need to sign up to obtain an API key
  • Read the documentation to figure out how to pass in the key
    • as part of the query string
    • in an HTTP header value such as apiKey
    • as part of the path (prev example)

Your Turn

  • Pick one of the APIs at the end of the chapter in the textbook

  • Read the documentation

  • See what data you can pull out of it