Scraping top 250 movies on IMDB

# Scraping top 250 movies on IMDB
## College of the Atlantic

---

# Top 250 movies on IMDB

---

## Top 250 movies on IMDB

Take a look at the source code, look for the tag `table` tag:
 
http://www.imdb.com/chart/top

.pull-left[
<img src="img/imdb-top-250.png" width="100%" style="display: block; margin: auto;" />
]
.pull-right[
<img src="img/imdb-top-250-source.png" width="94%" style="display: block; margin: auto;" />
]

---

## First check if you're allowed!

```r
library(robotstxt)
paths_allowed("http://www.imdb.com")
```

```
## [1] TRUE
```

vs. e.g.

```r
paths_allowed("http://www.facebook.com")
```

```
## [1] FALSE
```

---

## Plan

---

## Plan

1. Read the whole page

2. Scrape movie titles and save as `titles`

3. Scrape years movies were made in and save as `years`

4. Scrape IMDB ratings and save as `ratings`

5. Create a data frame called `imdb_top_250` with variables `title`, `year`, and `rating`

---

# Step 1. Read the whole page

---

## Read the whole page

```r
page <- read_html("https://www.imdb.com/chart/top/")
page
```

```
## {html_document}
## <html xmlns:og="http://ogp.me/ns#" xmlns:fb="http://www.facebook.com/2008/fbml">
## [1] <head>\n<meta http-equiv="Content-Type" content="text/html ...
## [2] <body id="styleguide-v2" class="fixed">\n <img ...
```

---

## A webpage in R

- Result is a list with 2 elements

```r
typeof(page)
```

```
## [1] "list"
```

- that we need to convert to something more familiar, like a data frame....

```r
class(page)
```

```
## [1] "xml_document" "xml_node"
```

---

# Step 2. Scrape movie titles and save as `titles`

---

## Scrape movie titles

---

## Scrape the nodes

```r
page %>%
  html_nodes(".titleColumn a")
```

```
## {xml_nodeset (250)}
## [1] <a href="/title/tt0111161/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [2] <a href="/title/tt0068646/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [3] <a href="/title/tt0468569/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [4] <a href="/title/tt0071562/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [5] <a href="/title/tt0050083/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [6] <a href="/title/tt0108052/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [7] <a href="/title/tt0167260/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [8] <a href="/title/tt0110912/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [9] <a href="/title/tt0120737/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [10] <a href="/title/tt0060196/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [11] <a href="/title/tt0109830/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [12] <a href="/title/tt0137523/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [13] <a href="/title/tt0167261/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [14] <a href="/title/tt1375666/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [15] <a href="/title/tt0080684/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
## [16] <a href="/title/tt0133093/?pf_rd_m=A2FGELUUNOQJNL&amp;pf_ ...
...
```
]
.pull-right[
<img src="img/titles.png" width="100%" style="display: block; margin: auto;" />
]

---

## Extract the text from the nodes

```r
page %>%
  html_nodes(".titleColumn a") %>%
  html_text()
```

```
## [1] "The Shawshank Redemption" 
## [2] "The Godfather" 
## [3] "The Dark Knight" 
## [4] "The Godfather Part II" 
## [5] "12 Angry Men" 
## [6] "Schindler's List" 
## [7] "The Lord of the Rings: The Return of the King" 
## [8] "Pulp Fiction" 
## [9] "The Lord of the Rings: The Fellowship of the Ring" 
## [10] "The Good, the Bad and the Ugly" 
## [11] "Forrest Gump" 
## [12] "Fight Club" 
## [13] "The Lord of the Rings: The Two Towers" 
## [14] "Inception" 
## [15] "Star Wars: Episode V - The Empire Strikes Back" 
## [16] "The Matrix" 
...
```
]
.pull-right[
<img src="img/titles.png" width="100%" style="display: block; margin: auto;" />
]

---

## Save as `titles`

```r
titles <- page %>%
 html_nodes(".titleColumn a") %>%
 html_text()

titles
```

```
## [1] "The Shawshank Redemption" 
## [2] "The Godfather" 
## [3] "The Dark Knight" 
## [4] "The Godfather Part II" 
## [5] "12 Angry Men" 
## [6] "Schindler's List" 
## [7] "The Lord of the Rings: The Return of the King" 
## [8] "Pulp Fiction" 
## [9] "The Lord of the Rings: The Fellowship of the Ring" 
## [10] "The Good, the Bad and the Ugly" 
## [11] "Forrest Gump" 
## [12] "Fight Club" 
## [13] "The Lord of the Rings: The Two Towers" 
## [14] "Inception" 
...
```
]
.pull-right[
<img src="img/titles.png" width="100%" style="display: block; margin: auto;" />
]

---

# Step 3. Scrape year movies were made and save as `years`

---

## Scrape years movies were made in

---

## Scrape the nodes

```r
page %>%
  html_nodes(".secondaryInfo")
```

```
## {xml_nodeset (250)}
## [1] (1994)
## [2] (1972)
## [3] (2008)
## [4] (1974)
## [5] (1957)
## [6] (1993)
## [7] (2003)
## [8] (1994)
## [9] (2001)
## [10] (1966)
## [11] (1994)
## [12] (1999)
## [13] (2002)
## [14] (2010)
## [15] (1980)
## [16] (1999)
...
```
]
.pull-right[
<img src="img/years.png" width="100%" style="display: block; margin: auto;" />
]

---

## Extract the text from the nodes

```r
page %>%
  html_nodes(".secondaryInfo") %>%
  html_text()
```

```
## [1] "(1994)" "(1972)" "(2008)" "(1974)" "(1957)" "(1993)"
## [7] "(2003)" "(1994)" "(2001)" "(1966)" "(1994)" "(1999)"
## [13] "(2002)" "(2010)" "(1980)" "(1999)" "(1990)" "(1975)"
## [19] "(1995)" "(1946)" "(1954)" "(1991)" "(1998)" "(2002)"
## [25] "(2014)" "(1997)" "(1999)" "(1977)" "(1991)" "(1985)"
## [31] "(2001)" "(2002)" "(1960)" "(2019)" "(1994)" "(1994)"
## [37] "(2000)" "(1998)" "(2006)" "(2014)" "(2006)" "(1995)"
## [43] "(1942)" "(1988)" "(1962)" "(2011)" "(1936)" "(1968)"
## [49] "(1988)" "(1954)" "(1979)" "(1931)" "(1979)" "(2000)"
## [55] "(2012)" "(1981)" "(2008)" "(2006)" "(1950)" "(1957)"
## [61] "(1980)" "(1940)" "(2018)" "(1957)" "(1986)" "(2018)"
## [67] "(1999)" "(1964)" "(2012)" "(2003)" "(2009)" "(1984)"
## [73] "(2017)" "(1995)" "(1995)" "(2019)" "(1981)" "(2019)"
## [79] "(1997)" "(1984)" "(1997)" "(2016)" "(1952)" "(2009)"
## [85] "(2000)" "(1963)" "(2010)" "(2018)" "(1983)" "(1968)"
## [91] "(2004)" "(1985)" "(1992)" "(2012)" "(1941)" "(1962)"
...
```
]
.pull-right[
<img src="img/years.png" width="100%" style="display: block; margin: auto;" />
]

---

## Clean up the text

We need to go from `"(1994)"` to `1994`:

- Remove `(` and `)`: string manipulation

- Convert to numeric: `as.numeric()`

---

## stringr

.pull-left-wide[
- **stringr** provides a cohesive set of functions designed to make working with strings as easy as possible
- Functions in stringr start with `str_*()`, e.g.
 - `str_remove()` to remove a pattern from a string
 
 ```r
 str_remove(string = "jello", pattern = "el")
 ```
 
 ```
 ## [1] "jlo"
 ```
 - `str_replace()` to replace a pattern with another
 
 ```r
 str_replace(string = "jello", pattern = "j", replacement = "h")
 ```
 
 ```
 ## [1] "hello"
 ```
]
.pull-right-narrow[
<img src="img/stringr.png" width="100%" style="display: block; margin: auto auto auto 0;" />
]

---

## Clean up the text

```r
page %>%
  html_nodes(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\\(") # remove (
```

```
##   [1] "1994)" "1972)" "2008)" "1974)" "1957)" "1993)" "2003)"
##   [8] "1994)" "2001)" "1966)" "1994)" "1999)" "2002)" "2010)"
##  [15] "1980)" "1999)" "1990)" "1975)" "1995)" "1946)" "1954)"
##  [22] "1991)" "1998)" "2002)" "2014)" "1997)" "1999)" "1977)"
##  [29] "1991)" "1985)" "2001)" "2002)" "1960)" "2019)" "1994)"
##  [36] "1994)" "2000)" "1998)" "2006)" "2014)" "2006)" "1995)"
##  [43] "1942)" "1988)" "1962)" "2011)" "1936)" "1968)" "1988)"
##  [50] "1954)" "1979)" "1931)" "1979)" "2000)" "2012)" "1981)"
##  [57] "2008)" "2006)" "1950)" "1957)" "1980)" "1940)" "2018)"
##  [64] "1957)" "1986)" "2018)" "1999)" "1964)" "2012)" "2003)"
##  [71] "2009)" "1984)" "2017)" "1995)" "1995)" "2019)" "1981)"
##  [78] "2019)" "1997)" "1984)" "1997)" "2016)" "1952)" "2009)"
##  [85] "2000)" "1963)" "2010)" "2018)" "1983)" "1968)" "2004)"
##  [92] "1985)" "1992)" "2012)" "1941)" "1962)" "1931)" "1952)"
##  [99] "1959)" "1958)" "1960)" "2001)" "1944)" "1971)" "1987)"
...
```

---

## Clean up the text

```r
page %>%
  html_nodes(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\$") %>% # remove (
  str_remove("\$") # remove )
```

```
##   [1] "1994" "1972" "2008" "1974" "1957" "1993" "2003" "1994"
##   [9] "2001" "1966" "1994" "1999" "2002" "2010" "1980" "1999"
##  [17] "1990" "1975" "1995" "1946" "1954" "1991" "1998" "2002"
##  [25] "2014" "1997" "1999" "1977" "1991" "1985" "2001" "2002"
##  [33] "1960" "2019" "1994" "1994" "2000" "1998" "2006" "2014"
##  [41] "2006" "1995" "1942" "1988" "1962" "2011" "1936" "1968"
##  [49] "1988" "1954" "1979" "1931" "1979" "2000" "2012" "1981"
##  [57] "2008" "2006" "1950" "1957" "1980" "1940" "2018" "1957"
##  [65] "1986" "2018" "1999" "1964" "2012" "2003" "2009" "1984"
##  [73] "2017" "1995" "1995" "2019" "1981" "2019" "1997" "1984"
##  [81] "1997" "2016" "1952" "2009" "2000" "1963" "2010" "2018"
##  [89] "1983" "1968" "2004" "1985" "1992" "2012" "1941" "1962"
##  [97] "1931" "1952" "1959" "1958" "1960" "2001" "1944" "1971"
## [105] "1987" "1983" "2020" "2010" "1995" "1962" "2009" "1973"
...
```

---

## Convert to numeric

```r
page %>%
  html_nodes(".secondaryInfo") %>%
  html_text() %>%
  str_remove("\$") %>% # remove (
  str_remove("\$") %>% # remove )
  as.numeric()
```

```
##   [1] 1994 1972 2008 1974 1957 1993 2003 1994 2001 1966 1994 1999
##  [13] 2002 2010 1980 1999 1990 1975 1995 1946 1954 1991 1998 2002
##  [25] 2014 1997 1999 1977 1991 1985 2001 2002 1960 2019 1994 1994
##  [37] 2000 1998 2006 2014 2006 1995 1942 1988 1962 2011 1936 1968
##  [49] 1988 1954 1979 1931 1979 2000 2012 1981 2008 2006 1950 1957
##  [61] 1980 1940 2018 1957 1986 2018 1999 1964 2012 2003 2009 1984
##  [73] 2017 1995 1995 2019 1981 2019 1997 1984 1997 2016 1952 2009
##  [85] 2000 1963 2010 2018 1983 1968 2004 1985 1992 2012 1941 1962
##  [97] 1931 1952 1959 1958 1960 2001 1944 1971 1987 1983 2020 2010
## [109] 1995 1962 2009 1973 2011 1927 1976 1997 1988 1989 2000 1948
## [121] 2007 2019 2022 2004 2016 1965 2005 1921 1959 2020 1950 2013
## [133] 2018 1961 1985 1995 1998 2007 2006 1992 1999 2010 2001 1993
## [145] 1961 1948 1975 2007 1963 2003 1950 1982 2021 2003 1980 1974
...
```

---

## Save as `years`

```r
years <- page %>%
 html_nodes(".secondaryInfo") %>%
 html_text() %>%
 str_remove("\$") %>% # remove (
 str_remove("\$") %>% # remove )
 as.numeric()

years
```

```
## [1] 1994 1972 2008 1974 1957 1993 2003 1994 2001 1966 1994 1999
## [13] 2002 2010 1980 1999 1990 1975 1995 1946 1954 1991 1998 2002
## [25] 2014 1997 1999 1977 1991 1985 2001 2002 1960 2019 1994 1994
## [37] 2000 1998 2006 2014 2006 1995 1942 1988 1962 2011 1936 1968
## [49] 1988 1954 1979 1931 1979 2000 2012 1981 2008 2006 1950 1957
## [61] 1980 1940 2018 1957 1986 2018 1999 1964 2012 2003 2009 1984
## [73] 2017 1995 1995 2019 1981 2019 1997 1984 1997 2016 1952 2009
## [85] 2000 1963 2010 2018 1983 1968 2004 1985 1992 2012 1941 1962
## [97] 1931 1952 1959 1958 1960 2001 1944 1971 1987 1983 2020 2010
## [109] 1995 1962 2009 1973 2011 1927 1976 1997 1988 1989 2000 1948
## [121] 2007 2019 2022 2004 2016 1965 2005 1921 1959 2020 1950 2013
...
```
]
.pull-right[
<img src="img/years.png" width="100%" style="display: block; margin: auto;" />
]

---

# Step 4. Scrape IMDB ratings and save as `ratings`

---

## Scrape IMDB ratings

---

## Scrape the nodes

```r
page %>%
  html_nodes("strong")
```

```
## {xml_nodeset (250)}
## [1] 9.2</ ...
## [2] 9.2</ ...
## [3] 9.0</ ...
## [4] 9.0</ ...
## [5] 9.0</st ...
## [6] 8.9</ ...
## [7] 8.9</ ...
## [8] 8.8</ ...
## [9] 8.8</ ...
## [10] 8.8</st ...
## [11] 8.8</ ...
## [12] 8.7</ ...
## [13] 8.7</ ...
## [14] 8.7</ ...
## [15] 8.7</ ...
## [16] 8.7</ ...
...
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Extract the text from the nodes

```r
page %>%
  html_nodes("strong") %>%
  html_text()
```

```
## [1] "9.2" "9.2" "9.0" "9.0" "9.0" "8.9" "8.9" "8.8" "8.8" "8.8"
## [11] "8.8" "8.7" "8.7" "8.7" "8.7" "8.7" "8.7" "8.6" "8.6" "8.6"
## [21] "8.6" "8.6" "8.6" "8.6" "8.6" "8.6" "8.6" "8.5" "8.5" "8.5"
## [31] "8.5" "8.5" "8.5" "8.5" "8.5" "8.5" "8.5" "8.5" "8.5" "8.5"
## [41] "8.5" "8.5" "8.5" "8.5" "8.5" "8.5" "8.4" "8.4" "8.4" "8.4"
## [51] "8.4" "8.4" "8.4" "8.4" "8.4" "8.4" "8.4" "8.4" "8.4" "8.4"
## [61] "8.4" "8.4" "8.4" "8.4" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3"
## [71] "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3"
## [81] "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3" "8.3"
## [91] "8.3" "8.3" "8.3" "8.3" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2"
## [101] "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2"
## [111] "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2"
## [121] "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2"
## [131] "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2" "8.2"
## [141] "8.2" "8.2" "8.2" "8.1" "8.1" "8.1" "8.1" "8.1" "8.1" "8.1"
## [151] "8.1" "8.1" "8.1" "8.1" "8.1" "8.1" "8.1" "8.1" "8.1" "8.1"
...
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Convert to numeric

```r
page %>%
  html_nodes("strong") %>%
  html_text() %>%
  as.numeric()
```

```
## [1] 9.2 9.2 9.0 9.0 9.0 8.9 8.9 8.8 8.8 8.8 8.8 8.7 8.7 8.7 8.7
## [16] 8.7 8.7 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.5 8.5 8.5
## [31] 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5
## [46] 8.5 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4
## [61] 8.4 8.4 8.4 8.4 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3
## [76] 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3
## [91] 8.3 8.3 8.3 8.3 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2
## [106] 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2
## [121] 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2
## [136] 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [151] 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [166] 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [181] 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [196] 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [211] 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0 8.0
...
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

## Save as `ratings`

```r
ratings <- page %>%
 html_nodes("strong") %>%
 html_text() %>%
 as.numeric()

ratings
```

```
## [1] 9.2 9.2 9.0 9.0 9.0 8.9 8.9 8.8 8.8 8.8 8.8 8.7 8.7 8.7 8.7
## [16] 8.7 8.7 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.6 8.5 8.5 8.5
## [31] 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5 8.5
## [46] 8.5 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4 8.4
## [61] 8.4 8.4 8.4 8.4 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3
## [76] 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3 8.3
## [91] 8.3 8.3 8.3 8.3 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2
## [106] 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2
## [121] 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2
## [136] 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.2 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [151] 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1
## [166] 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1 8.1
...
```
]
.pull-right[
<img src="img/ratings.png" width="100%" style="display: block; margin: auto;" />
]

---

# Step 5. Create a data frame called `imdb_top_250`

---

## Create a data frame: `imdb_top_250`

```r
imdb_top_250 <- tibble(
 title = titles, 
 year = years, 
 rating = ratings
 )

imdb_top_250
```

```
## # A tibble: 250 x 3
## title year rating
## <chr> <dbl> <dbl>
## 1 The Shawshank Redemption 1994 9.2
## 2 The Godfather 1972 9.2
## 3 The Dark Knight 2008 9 
## 4 The Godfather Part II 1974 9 
## 5 12 Angry Men 1957 9 
## 6 Schindler's List 1993 8.9
## # ... with 244 more rows
```

---

<div id="htmlwidget-1b4ff99564eb6e8884a5" style="width:100%;height:400px;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-1b4ff99564eb6e8884a5">{"x":{"filter":"none","vertical":false,"data":[["1","2","3","4","5","6","7","8","9","10","11","12","13","14","15","16","17","18","19","20","21","22","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","50","51","52","53","54","55","56","57","58","59","60","61","62","63","64","65","66","67","68","69","70","71","72","73","74","75","76","77","78","79","80","81","82","83","84","85","86","87","88","89","90","91","92","93","94","95","96","97","98","99","100","101","102","103","104","105","106","107","108","109","110","111","112","113","114","115","116","117","118","119","120","121","122","123","124","125","126","127","128","129","130","131","132","133","134","135","136","137","138","139","140","141","142","143","144","145","146","147","148","149","150","151","152","153","154","155","156","157","158","159","160","161","162","163","164","165","166","167","168","169","170","171","172","173","174","175","176","177","178","179","180","181","182","183","184","185","186","187","188","189","190","191","192","193","194","195","196","197","198","199","200","201","202","203","204","205","206","207","208","209","210","211","212","213","214","215","216","217","218","219","220","221","222","223","224","225","226","227","228","229","230","231","232","233","234","235","236","237","238","239","240","241","242","243","244","245","246","247","248","249","250"],["The Shawshank Redemption","The Godfather","The Dark Knight","The Godfather Part II","12 Angry Men","Schindler's List","The Lord of the Rings: The Return of the King","Pulp Fiction","The Lord of the Rings: The Fellowship of the Ring","The Good, the Bad and the Ugly","Forrest Gump","Fight Club","The Lord of the Rings: The Two Towers","Inception","Star Wars: Episode V - The Empire Strikes Back","The Matrix","Goodfellas","One Flew Over the Cuckoo's Nest","Se7en","It's a Wonderful Life","Seven Samurai","The Silence of the Lambs","Saving Private Ryan","City of God","Interstellar","Life Is Beautiful","The Green Mile","Star Wars","Terminator 2: Judgment Day","Back to the Future","Spirited Away","The Pianist","Psycho","Parasite","Léon: The Professional","The Lion King","Gladiator","American History X","The Departed","Whiplash","The Prestige","The Usual Suspects","Casablanca","Grave of the Fireflies","Harakiri","The Intouchables","Modern Times","Once Upon a Time in the West","Cinema Paradiso","Rear Window","Alien","City Lights","Apocalypse Now","Memento","Django Unchained","Raiders of the Lost Ark","WALL·E","The Lives of Others","Sunset Blvd.","Paths of Glory","The Shining","The Great Dictator","Avengers: Infinity War","Witness for the Prosecution","Aliens","Spider-Man: Into the Spider-Verse","American Beauty","Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb","The Dark Knight Rises","Oldboy","Inglourious Basterds","Amadeus","Coco","Toy Story","Braveheart","Joker","Das Boot","Avengers: Endgame","Princess Mononoke","Once Upon a Time in America","Good Will Hunting","Your Name.","Singin' in the Rain","3 Idiots","Requiem for a Dream","High and Low","Toy Story 3","Capernaum","Star Wars: Episode VI - Return of the Jedi","2001: A Space Odyssey","Eternal Sunshine of the Spotless Mind","Come and See","Reservoir Dogs","The Hunt","Citizen Kane","Lawrence of Arabia","M","Ikiru","North by Northwest","Vertigo","The Apartment","Amélie","Double Indemnity","A Clockwork Orange","Full Metal Jacket","Scarface","Hamilton","Incendies","Heat","To Kill a Mockingbird","Up","The Sting","A Separation","Metropolis","Taxi Driver","L.A. Confidential","Die Hard","Indiana Jones and the Last Crusade","Snatch","Bicycle Thieves","Like Stars on Earth","1917","Top Gun: Maverick","Downfall","Dangal","For a Few Dollars More","Batman Begins","The Kid","Some Like It Hot","The Father","All About Eve","The Wolf of Wall Street","Green Book","Judgment at Nuremberg","Ran","Casino","The Truman Show","There Will Be Blood","Pan's Labyrinth","Unforgiven","The Sixth Sense","Shutter Island","A Beautiful Mind","Jurassic Park","Yojimbo","The Treasure of the Sierra Madre","Monty Python and the Holy Grail","No Country for Old Men","The Great Escape","Kill Bill: Vol. 1","Rashomon","The Thing","Spider-Man: No Way Home","Finding Nemo","The Elephant Man","Chinatown","Raging Bull","V for Vendetta","Gone with the Wind","Lock, Stock and Two Smoking Barrels","Dial M for Murder","Inside Out","The Secret in Their Eyes","Howl's Moving Castle","John Wick: Chapter 4","Three Billboards Outside Ebbing, Missouri","Trainspotting","The Bridge on the River Kwai","Prisoners","Warrior","Fargo","Gran Torino","Catch Me If You Can","My Neighbor Totoro","Million Dollar Baby","Children of Heaven","Blade Runner","The Gold Rush","Klaus","Harry Potter and the Deathly Hallows: Part 2","Before Sunrise","12 Years a Slave","Ben-Hur","On the Waterfront","The Grand Budapest Hotel","Gone Girl","Wild Strawberries","The General","Barry Lyndon","In the Name of the Father","The Deer Hunter","The Third Man","Hacksaw Ridge","The Wages of Fear","Memories of Murder","Sherlock Jr.","Wild Tales","Mr. Smith Goes to Washington","Mad Max: Fury Road","Mary and Max","The Seventh Seal","How to Train Your Dragon","Monsters, Inc.","Dead Poets Society","Jaws","Room","The Big Lebowski","Tokyo Story","The Passion of Joan of Arc","Ford v Ferrari","Hotel Rwanda","Rocky","Platoon","Ratatouille","Spotlight","Logan","The Terminator","Stand by Me","Rush","Network","Before Sunset","Into the Wild","The Wizard of Oz","The Best Years of Our Lives","Groundhog Day","The Exorcist","The Incredibles","La haine","To Be or Not to Be","Pirates of the Caribbean: The Curse of the Black Pearl","Pather Panchali","Hachi: A Dog's Tale","The Grapes of Wrath","The Battle of Algiers","Jai Bhim","My Father and My Son","Amores Perros","Rebecca","Cool Hand Luke","The Handmaiden","The Sound of Music","The 400 Blows","It Happened One Night","Persona","Life of Brian","The Iron Giant","The Help","Aladdin","Dersu Uzala","Dances with Wolves"],[1994,1972,2008,1974,1957,1993,2003,1994,2001,1966,1994,1999,2002,2010,1980,1999,1990,1975,1995,1946,1954,1991,1998,2002,2014,1997,1999,1977,1991,1985,2001,2002,1960,2019,1994,1994,2000,1998,2006,2014,2006,1995,1942,1988,1962,2011,1936,1968,1988,1954,1979,1931,1979,2000,2012,1981,2008,2006,1950,1957,1980,1940,2018,1957,1986,2018,1999,1964,2012,2003,2009,1984,2017,1995,1995,2019,1981,2019,1997,1984,1997,2016,1952,2009,2000,1963,2010,2018,1983,1968,2004,1985,1992,2012,1941,1962,1931,1952,1959,1958,1960,2001,1944,1971,1987,1983,2020,2010,1995,1962,2009,1973,2011,1927,1976,1997,1988,1989,2000,1948,2007,2019,2022,2004,2016,1965,2005,1921,1959,2020,1950,2013,2018,1961,1985,1995,1998,2007,2006,1992,1999,2010,2001,1993,1961,1948,1975,2007,1963,2003,1950,1982,2021,2003,1980,1974,1980,2005,1939,1998,1954,2015,2009,2004,2023,2017,1996,1957,2013,2011,1996,2008,2002,1988,2004,1997,1982,1925,2019,2011,1995,2013,1959,1954,2014,2014,1957,1926,1975,1993,1978,1949,2016,1953,2003,1924,2014,1939,2015,2009,1957,2010,2001,1989,1975,2015,1998,1953,1928,2019,2004,1976,1986,2007,2015,2017,1984,1986,2013,1976,2004,2007,1939,1946,1993,1973,2004,1995,1942,2003,1955,2009,1940,1966,2021,2005,2000,1940,1967,2016,1965,1959,1934,1966,1979,1999,2011,1992,1975,1990],[9.2,9.2,9,9,9,8.9,8.9,8.8,8.8,8.8,8.8,8.7,8.7,8.7,8.7,8.7,8.7,8.6,8.6,8.6,8.6,8.6,8.6,8.6,8.6,8.6,8.6,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.5,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.4,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.3,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.2,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8.1,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th> <\/th>\n <th>title<\/th>\n <th>year<\/th>\n <th>rating<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"columnDefs":[{"className":"dt-right","targets":[2,3]},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false}},"evals":[],"jsHooks":[]}</script>

---

## Clean up / enhance

May or may not be a lot of work depending on how messy the data are

- See if you like what you got:

```r
glimpse(imdb_top_250)
```

```
## Rows: 250
## Columns: 3
## $ title <chr> "The Shawshank Redemption", "The Godfather", "Th~
## $ year <dbl> 1994, 1972, 2008, 1974, 1957, 1993, 2003, 1994, ~
## $ rating <dbl> 9.2, 9.2, 9.0, 9.0, 9.0, 8.9, 8.9, 8.8, 8.8, 8.8~
```

- Add a variable for rank

```r
imdb_top_250 <- imdb_top_250 %>%
 mutate(rank = 1:nrow(imdb_top_250)) %>%
 relocate(rank)
```

---

```
## # A tibble: 250 x 4
## rank title year rating
## <int> <chr> <dbl> <dbl>
## 1 1 The Shawshank Redemption 1994 9.2
## 2 2 The Godfather 1972 9.2
## 3 3 The Dark Knight 2008 9 
## 4 4 The Godfather Part II 1974 9 
## 5 5 12 Angry Men 1957 9 
## 6 6 Schindler's List 1993 8.9
## 7 7 The Lord of the Rings: The Return of the K~ 2003 8.9
## 8 8 Pulp Fiction 1994 8.8
## 9 9 The Lord of the Rings: The Fellowship of t~ 2001 8.8
## 10 10 The Good, the Bad and the Ugly 1966 8.8
## 11 11 Forrest Gump 1994 8.8
## 12 12 Fight Club 1999 8.7
## 13 13 The Lord of the Rings: The Two Towers 2002 8.7
## 14 14 Inception 2010 8.7
## 15 15 Star Wars: Episode V - The Empire Strikes ~ 1980 8.7
## 16 16 The Matrix 1999 8.7
## 17 17 Goodfellas 1990 8.7
## 18 18 One Flew Over the Cuckoo's Nest 1975 8.6
## 19 19 Se7en 1995 8.6
## 20 20 It's a Wonderful Life 1946 8.6
## # ... with 230 more rows
```

---

# What next?

---

```r
imdb_top_250 %>% 
  count(year, sort = TRUE)
```

```
## # A tibble: 87 x 2
## year n
## <dbl> <int>
## 1 1995 8
## 2 2004 7
## 3 1957 6
## 4 1999 6
## 5 2003 6
## 6 2009 6
## # ... with 81 more rows
```

---

```r
imdb_top_250 %>% 
  filter(year == 1995) %>%
  print(n = 8)
```

```
## # A tibble: 8 x 4
## rank title year rating
## <int> <chr> <dbl> <dbl>
## 1 19 Se7en 1995 8.6
## 2 42 The Usual Suspects 1995 8.5
## 3 74 Toy Story 1995 8.3
## 4 75 Braveheart 1995 8.3
## 5 109 Heat 1995 8.2
## 6 136 Casino 1995 8.2
## 7 181 Before Sunrise 1995 8.1
## 8 228 La haine 1995 8
```

---

.question[
Visualize the average yearly rating for movies that made it on the top 250 list over time.
]

.panelset[
.panel[.panel-name[Plot]
<img src="u2-d19-top-250-imdb_files/figure-html/unnamed-chunk-46-1.png" width="58%" style="display: block; margin: auto;" />
]
.panel[.panel-name[Code]

```r
imdb_top_250 %>% 
  group_by(year) %>%
  summarise(avg_score = mean(rating)) %>%
  ggplot(aes(y = avg_score, x = year)) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE) +
  labs(x = "Year", y = "Average score")
```
]
]

---
## Acknowledgements

* This course builds on the materials from [Data Science in a Box](https://datasciencebox.org/) developed by Mine Çetinkaya-Rundel and are adapted under the [Creative Commons Attribution Share Alike 4.0 International](https://github.com/rstudio-education/datascience-box/blob/master/LICENSE.md)