Warm up

Before we introduce the data, let’s warm up with some simple exercises. Update the YAML of your R Markdown file with your information, knit, commit, and push your changes. Make sure to commit with a meaningful commit message. Then, go to your repo on GitHub and confirm that your changes are visible in your Rmd and md files. If anything is missing, commit and push again.

Packages

We’ll use the tidyverse package for much of the data wrangling and visualisation, the robotstxt package to check if we’re allowed to scrape the data, the rvest package for data scraping, and the scales package for better formatting of labels on visualisations. These packages are already installed for you. You can load them by running the following in your Console:

library(tidyverse)
library(robotstxt)
library(rvest)
library(scales)

Data

This assignment does not come with any prepared datasets. Instead you’ll be scraping the data!

Data collection via web scraping

The data come from OpenSecrets.org, a “website tracking the influence of money on U.S. politics, and how that money affects policy and citizens’ lives”. This website is hosted by The Center for Responsive Politics, which is a nonpartisan, independent nonprofit that “tracks money in U.S. politics and its effect on elections and public policy.”²

Before getting started, let’s check that a bot has permissions to access pages on this domain.

library(robotstxt)
paths_allowed("https://www.opensecrets.org")

## [1] TRUE

Our goal is to scrape data for contributions in all election years Open Secrets has data for. Since that means repeating a task many times, let’s first write a function that works on the first page. Confirm it works on a few others. We will be scraping from the page https://www.opensecrets.org/political-action-committees-pacs/foreign-connected-pacs/2022. Note that you can select the year from a drop down.

Then iterate it over pages for all years.

• Write a function called scrape_pac() that scrapes information from the Open Secrets webpage for foreign-connected PAC contributions in a given year. This function should

  • have one input: the URL of the webpage and should return a data frame.
  • rename variables scraped, using snake_case naming.
  • clean up the Country of Origin/Parent Company variable with str_squish().
  • add a new column to the data frame for year. We will want this information when we ultimately have data from all years, so this is a good time to keep track of it. Our function doesn’t take a year argument, but the year is embedded in the URL, so we can extract it out of there, and add it as a new column. Use the str_sub() function to extract the last 4 characters from the URL. You will probably want to look at the help for this function to figure out how to specify “last 4 characters”.

• Define the URLs for 2022. 2020, 2018, through 2000 contributions and more! This dataset is constantly updated and the dates may shift. Use the drop-down menu to make sure you cover all dates and keep these updated. Then, test your function using these URLs as inputs. Does the function seem to do what you expected it to do?

• Construct a vector called urls that contains the URLs for each webpage that contains information on foreign-connected PAC contributions for a given year.

• Map the scrape_pac() function over urls in a way that will result in a data frame called pac_all.

• Write the data frame to a csv file called pac-all.csv in the data folder.

✅⬆️ If you haven’t yet done so, now is definitely a good time to commit and push your changes to GitHub with an appropriate commit message (e.g. “Data scraping complete”). Make sure to commit and push all changed files so that your Git pane is cleared up afterwards.

1. In your R Markdown file, load pac-all.csv and report its number of observations and variables using inline code.

Data cleaning

In this section we clean the pac_all data frame to prepare it for analysis and visualization. We have two goals in data cleaning:

• Separate the country_parent into two such that country and parent company appear in different columns for country-level analysis.

• Convert contribution amounts in total, dems, and repubs from character strings to numeric values.

The following exercises walk you through how to make these fixes to the data.

2. Use the separate() function to separate country_parent into country and parent columns. Note that country and parent company names are separated by \ (which will need to be specified in your function) and also note that there are some entries where the \ sign appears twice and in these cases we want to only split the value at the first occurrence of \. This can be accomplished by setting the extra argument in to "merge" so that the cell is split into only 2 segments, e.g. we want "Denmark/Novo Nordisk A/S" to be split into "Denmark" and "Novo Nordisk A/S". (See help for separate() for more on this.) End your code chunk by printing out the top 10 rows of your data frame (if you just type the data frame name it should automatically do this for you).

3. Remove the character strings including $ and , signs in the total, dems,and repubs columns and convert these columns to numeric. End your code chunk by printing out the top 10 rows of your data frame (if you just type the data frame name it should automatically do this for you). A couple hints to help you out:

   • The $ character is a special character so it will need to be escaped.

   • Some contribution amounts are in the millions (e.g. Anheuser-Busch contributed a total of $1,510,897 in 2008). In this case we need to remove all occurrences of ,, which we can do by using str_remove_all() instead of str_remove().

🧶 ✅ ⬆️ Now is a good time to knit your document, commit, and push your changes to GitHub with an appropriate commit message. Make sure to commit and push all changed files so that your Git pane is cleared up afterwards.

Data visualization and interpretation

4. Create a line plot of total contributions from all foreign-connected PACs in the Canada and Mexico over the years. Once you have made the plot, write a brief interpretation of what the graph reveals. Few hints to help you out:

   • Filter for only Canada and Mexico.
   • Calculate sum of total contributions from PACs for each year for each country by using a sequence of group_by() then summarise().
   • Make a plot of total contributions (y-axis) by year (x-axis) where two lines identified by different colours represent each of Canada and Mexico.

⊕Note: The figure you create might look slightly different than this one if the data on the website has been updated recently.

4. Recreate the following visualisation. Once you have made the plot, write a brief interpretation of what the graph reveals. Note that these are only UK contributions. You will need to make use of functions from the scales package for axis labels as well as from ggplot2.

⊕Hint: You will need to use pivot_longer to create this graphic.

🧶 ✅ ⬆️ Knit, commit, and push your changes to GitHub with an appropriate commit message. Make sure to commit and push all changed files so that your Git pane is cleared up afterwards and review the md document on GitHub to make sure you’re happy with the final state of your work.