Hands-on Lab : Web Scraping

Estimated time needed: 30 to 45 minutes

Objectives

In this lab you will perform the following:

• Extract information from a given web site
• Write the scraped data into a csv file.

Extract information from the given web site

You will extract the data from the below web site:

#this url contains the data you need to scrape
url = "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/Programming_Languages.html"

The data you need to scrape is the name of the programming language and average annual salary.
It is a good idea to open the url in your web broswer and study the contents of the web page before you start to scrape.

Import the required libraries

# Your code here
from bs4 import BeautifulSoup # this module helps in web scrapping.
import requests  # this module helps us to download a webpage
import numpy as np
import pandas as pd

Download the webpage at the url

#your code goes here
# get the contents of the webpage in text format and store in a variable called data
data  = requests.get(url).text

Create a soup object

#your code goes here
soup = BeautifulSoup(data, "html.parser")  # create a soup object using the variable 'data'

Scrape the Language name and annual average salary.

#your code goes here
#find a html table in the web page
table = soup.find('table') # in html table is represented by the tag <table>
lnas = []
for row in table.find_all('tr'): # in html table row is represented by the tag <tr>
    # Get all columns in each row.
    cols = row.find_all('td') # in html a column is represented by the tag <td>
    language_name = cols[1].getText()
    language_salary = cols[3].getText().replace(',', '').replace('$', '')
    lnas.append(f"{language_name},{language_salary}")
print(lnas)

['Language,Average Annual Salary', 'Python,114383', 'Java,101013', 'R,92037', 'Javascript,110981', 'Swift,130801', 'C++,113865', 'C#,88726', 'PHP,84727', 'SQL,84793', 'Go,94082']

Save the scrapped data into a file named popular-languages.csv

# your code goes here
with open('popular-languages.csv', 'w') as output:
    for line in lnas:
        print(line, file=output)

Authors

Ramesh Sannareddy

Other Contributors

Rav Ahuja

Change Log

Copyright © 2020 IBM Corporation. This notebook and its source code are released under the terms of the MIT License.