Web Scraping Lab

Estimated time needed: 30 minutes

Objectives

After completing this lab, you will be able to:

Download a webpage using requests module
Scrape all links from a webpage
Scrape all image URLs from a web page
Scrape data from html tables

Scrape www.ibm.com

Import the required modules and functions

from bs4 import BeautifulSoup # this module helps in web scrapping.
import requests  # this module helps us to download a webpage

Download the contents of the webpage

url = "http://www.ibm.com"

# get the contents of the webpage in text format and store in a variable called data
data  = requests.get(url).text

Create a soup object using the class BeautifulSoup

soup = BeautifulSoup(data, "html.parser")  # create a soup object using the variable 'data'

Scrape all links

for link in soup.find_all('a'):  # in html anchor/link is represented by the tag <a>
    print(link.get('href'))

https://www.ibm.com/case-studies/ibm-client-zero?lnk=hpls1us
https://www.ibm.com/products/watsonx-orchestrate?lnk=hpls2us
https://www.ibm.com/account/reg/us-en/signup?formid=urx-54227&lnk=hprc1us
https://www.ibm.com/think/insights/2026-resolutions-for-ai-and-technology-leaders?lnk=hprc2us
https://www.ibm.com/account/reg/us-en/signup?formid=urx-54067&lnk=hprc3us
https://www.ibm.com/thought-leadership/institute-business-value/us-en/report/enterprise-2030?lnk=hprc4us
https://www.ibm.com/case-studies/avid-solutions-international?lnk=hpcs1us
https://www.ibm.com/case-studies/scuderia-ferrari?lnk=hpcs2us
https://www.ibm.com/case-studies/pfizer?lnk=hpcs3us
https://www.ibm.com/account/reg/us-en/signup?formid=urx-54203&lnk=hppr1us
https://www.ibm.com/software?lnk=hpfp1us
https://www.ibm.com/solutions/ai-agents?lnk=hpfp2us
https://www.ibm.com/solutions/data-and-ai?lnk=hpfp3us
https://www.ibm.com/solutions/automation?lnk=hpfp4us
https://www.ibm.com/solutions/hybrid-cloud?lnk=hpfp5us
https://www.ibm.com/solutions/ai-models?lnk=hpfp6us
https://www.ibm.com/solutions/analytics?lnk=hpfp7us
https://www.ibm.com/solutions/security?lnk=hpfp8us
https://www.ibm.com/consulting?lnk=hpfp9us
None
https://www.ibm.com/about?lnk=hpii1us
https://www.ibm.com/history?lnk=hpii1us
https://research.ibm.com?lnk=hpii1us
https://www.ibm.com/quantum?lnk=hpii1us
https://www.ibm.com/careers?lnk=hpii1us
https://skillsbuild.org?lnk=hpii1us
https://newsroom.ibm.com/2026-01-20-ibm-and-the-recording-academy-strike-a-chord,-release-new-watsonx-digital-experiences-for-fans-and-members-around-2026-grammy-awards
https://newsroom.ibm.com/2026-01-19-e-and-ibm-unveil-enterprise-grade-agentic-AI-to-transform-governance-and-compliance
https://newsroom.ibm.com/2026-01-19-ibm-launches-enterprise-advantage-service-to-help-businesses-scale-agentic-ai
https://newsroom.ibm.com/2026-01-19-ibm-study-ai-poised-to-drive-smarter-business-growth-through-2030

Scrape all images

for link in soup.find_all('img'): # in html image is represented by the tag <img>
    print(link.get('src'))

https://assets.ibm.com/is/image/ibm/ibm2569_client_zero_procurement?ts=1768922014756&dpr=off
https://assets.ibm.com/is/image/ibm/ibm-homepage-agents-yellow-2?ts=1768922023434&dpr=off
https://assets.ibm.com/is/image/ibm/ibm-homepage-agents-yellow-4?ts=1768922035280&dpr=off
https://assets.ibm.com/is/image/ibm/1604-enteprise-2030-square_568x568?ts=1768922038960&dpr=off
https://assets.ibm.com/is/image/ibm/avid-solutions-leadspace?ts=1768922039686&dpr=off
https://assets.ibm.com/is/image/ibm/ibm_ferrari_video_thumb_01?ts=1768922040311&dpr=off
https://assets.ibm.com/is/image/ibm/pfizer_thumb?ts=1768922040865&dpr=off
https://assets.ibm.com/is/image/ibm/hybrid-cloud-promo-banner?ts=1768922041551&dpr=off
https://assets.ibm.com/is/image/ibm/watson-discovery_eye-connection_1x1_padding?ts=1768922047606&dpr=off

Scrape data from html tables

#The below URL contains a html table with data about colors and color codes.
URL = "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/HTMLColorCodes.html"

Before proceeding to scrape a website, you need to examine the contents, and the way data is organized on the website. Open the above URL in your browser and check how many rows and columns are there in the color table.

# get the contents of the webpage in text format and store in a variable called data
data  = requests.get(URL).text

soup = BeautifulSoup(data, "html.parser")

#find a html table in the web page
table = soup.find('table') # in html table is represented by the tag <table>

Get all rows from the table

for row in table.find_all('tr'): # in html table row is represented by the tag <tr>
    # Get all columns in each row.
    cols = row.find_all('td') # in html a column is represented by the tag <td>
    color_name = cols[2].getText() # store the value in column 3 as color_name
    color_code = cols[3].getText() # store the value in column 4 as color_code
    print("{}--->{}".format(color_name, color_code))

Color Name--->Hex Code#RRGGBB
lightsalmon--->#FFA07A
salmon--->#FA8072
darksalmon--->#E9967A
lightcoral--->#F08080
coral--->#FF7F50
tomato--->#FF6347
orangered--->#FF4500
gold--->#FFD700
orange--->#FFA500
darkorange--->#FF8C00
lightyellow--->#FFFFE0
lemonchiffon--->#FFFACD
papayawhip--->#FFEFD5
moccasin--->#FFE4B5
peachpuff--->#FFDAB9
palegoldenrod--->#EEE8AA
khaki--->#F0E68C
darkkhaki--->#BDB76B
yellow--->#FFFF00
lawngreen--->#7CFC00
chartreuse--->#7FFF00
limegreen--->#32CD32
lime--->#00FF00
forestgreen--->#228B22
green--->#008000
powderblue--->#B0E0E6
lightblue--->#ADD8E6
lightskyblue--->#87CEFA
skyblue--->#87CEEB
deepskyblue--->#00BFFF
lightsteelblue--->#B0C4DE
dodgerblue--->#1E90FF

Authors

Ramesh Sannareddy

Other Contributors

Rav Ahuja