Collecting Data from the Web: Foundation of Webscraping
In Person
This course introduces the fundamentals of scraping user-generated content from the web using R or Python, taught over two 2-hour sessions. The first session focuses on the legal and ethical dimensions of web scraping, including questions around terms of service, copyright, privacy, consent, and the boundary between public and private data. We will also look at controversial research cases, consider why they were problematic, and reflect on what responsible web scraping might look like in practice. Alongside these discussions, participants will receive a short introduction to HTML structures and the tools available for scraping in R and Python. The second session will be hands-on, working through the process of scraping a selected static website step by step, before concluding with a discussion of the limitations of scraping and the challenges posed by more complex dynamic sites.
This course will be taught by Aybukee Atalay and Ponrawee Prasertsom
After taking part in this event, you may decide that you need some further help in applying what you have learnt to your research. If so, you can book a Data Surgery meeting with one of our training fellows.
More details about Data Surgeries.
Those who have registered to take part will receive an email with full details on how to get ready for this course.
If you’re new to this training event format, or to CDCS training events in general, read more on what to expect from CDCS training. Here you will also find details of our cancellation and no-show policy, which applies to this event
Level
Participants should have a basic familiarity with R or Python.
You should be comfortable installing and loading packages, working with data frames, and running simple code chunks.
For those working with R, some prior exposure to the Tidyverse syntax (e.g. using the pipe %>% and functions like mutate() or filter()) will make the hands-on parts easier to follow, though it is not strictly required.
If you want to learn the basics of Python or R, you can take our introductory courses Getting Started with Python for Research and Getting Started with R for Research.
No prior experience with web scraping or HTML is assumed.
Learning Outcomes
Make informed decisions about when web scraping is appropriate, and when alternative approaches (e.g., APIs, open data sources) should be used.
Explain the main legal and ethical considerations in web scraping, including terms of service, copyright, privacy, and consent.
Identify and interpret the HTML structure of a static web page.
Use R or Python tools to scrape and organise content from a simple static website.
Skills
By attending this course, you will familiarise yourself with the following skills
- Understanding and working with basic HTML structures
Using the R package rvest or the Python library Beautiful Soup to scrape static websites
Explore More Training
- From Images to Text: Working with OCR
- Foundations of Sentiment Analysis. Silent Disco
- A Pipeline for Data Wrangling and Manipulation with R
- BYOD (Bring Your Own Data) Fest
Return to the Training Homepage to see other available events
Room 4.35, Edinburgh Futures Institute
This room is on Level 4, in the North East side of the building.
When you enter via the level 2 East entrance on Middle Meadow Walk, the room will be on the 4th floor straight ahead.
When you enter via the level 2 North entrance on Lauriston Place underneath the clock tower, the room will be on the 4th floor to your left.
When you enter via the level 0 South entrance on Porters Walk (opposite Tribe Yoga), the room will be on the 4th floor to your right.
