{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " \n", " \"Skills\n", " \n", "

\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# **Web Scraping Lab**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Estimated time needed: **30** minutes\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Objectives\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After completing this lab, you will be able to:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Download a webpage using requests module\n", "* Scrape all links from a webpage\n", "* Scrape all image URLs from a web page\n", "* Scrape data from html tables\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Scrape www.ibm.com\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Import the required modules and functions\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from bs4 import BeautifulSoup # this module helps in web scrapping.\n", "import requests # this module helps us to download a webpage" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Download the contents of the webpage\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "url = \"http://www.ibm.com\"" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# get the contents of the webpage in text format and store in a variable called data\n", "data = requests.get(url).text " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a soup object using the class BeautifulSoup\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "soup = BeautifulSoup(data, \"html.parser\") # create a soup object using the variable 'data'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Scrape all links\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://www.ibm.com/case-studies/ibm-client-zero?lnk=hpls1us\n", "https://www.ibm.com/products/watsonx-orchestrate?lnk=hpls2us\n", "https://www.ibm.com/account/reg/us-en/signup?formid=urx-54227&lnk=hprc1us\n", "https://www.ibm.com/think/insights/2026-resolutions-for-ai-and-technology-leaders?lnk=hprc2us\n", "https://www.ibm.com/account/reg/us-en/signup?formid=urx-54067&lnk=hprc3us\n", "https://www.ibm.com/thought-leadership/institute-business-value/us-en/report/enterprise-2030?lnk=hprc4us\n", "https://www.ibm.com/case-studies/avid-solutions-international?lnk=hpcs1us\n", "https://www.ibm.com/case-studies/scuderia-ferrari?lnk=hpcs2us\n", "https://www.ibm.com/case-studies/pfizer?lnk=hpcs3us\n", "https://www.ibm.com/account/reg/us-en/signup?formid=urx-54203&lnk=hppr1us\n", "https://www.ibm.com/software?lnk=hpfp1us\n", "https://www.ibm.com/solutions/ai-agents?lnk=hpfp2us\n", "https://www.ibm.com/solutions/data-and-ai?lnk=hpfp3us\n", "https://www.ibm.com/solutions/automation?lnk=hpfp4us\n", "https://www.ibm.com/solutions/hybrid-cloud?lnk=hpfp5us\n", "https://www.ibm.com/solutions/ai-models?lnk=hpfp6us\n", "https://www.ibm.com/solutions/analytics?lnk=hpfp7us\n", "https://www.ibm.com/solutions/security?lnk=hpfp8us\n", "https://www.ibm.com/consulting?lnk=hpfp9us\n", "None\n", "https://www.ibm.com/about?lnk=hpii1us\n", "https://www.ibm.com/history?lnk=hpii1us\n", "https://research.ibm.com?lnk=hpii1us\n", "https://www.ibm.com/quantum?lnk=hpii1us\n", "https://www.ibm.com/careers?lnk=hpii1us\n", "https://skillsbuild.org?lnk=hpii1us\n", "https://newsroom.ibm.com/2026-01-20-ibm-and-the-recording-academy-strike-a-chord,-release-new-watsonx-digital-experiences-for-fans-and-members-around-2026-grammy-awards\n", "https://newsroom.ibm.com/2026-01-19-e-and-ibm-unveil-enterprise-grade-agentic-AI-to-transform-governance-and-compliance\n", "https://newsroom.ibm.com/2026-01-19-ibm-launches-enterprise-advantage-service-to-help-businesses-scale-agentic-ai\n", "https://newsroom.ibm.com/2026-01-19-ibm-study-ai-poised-to-drive-smarter-business-growth-through-2030\n" ] } ], "source": [ "for link in soup.find_all('a'): # in html anchor/link is represented by the tag \n", " print(link.get('href'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Scrape all images\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "https://assets.ibm.com/is/image/ibm/ibm2569_client_zero_procurement?ts=1768922014756&dpr=off\n", "https://assets.ibm.com/is/image/ibm/ibm-homepage-agents-yellow-2?ts=1768922023434&dpr=off\n", "https://assets.ibm.com/is/image/ibm/ibm-homepage-agents-yellow-4?ts=1768922035280&dpr=off\n", "https://assets.ibm.com/is/image/ibm/1604-enteprise-2030-square_568x568?ts=1768922038960&dpr=off\n", "https://assets.ibm.com/is/image/ibm/avid-solutions-leadspace?ts=1768922039686&dpr=off\n", "https://assets.ibm.com/is/image/ibm/ibm_ferrari_video_thumb_01?ts=1768922040311&dpr=off\n", "https://assets.ibm.com/is/image/ibm/pfizer_thumb?ts=1768922040865&dpr=off\n", "https://assets.ibm.com/is/image/ibm/hybrid-cloud-promo-banner?ts=1768922041551&dpr=off\n", "https://assets.ibm.com/is/image/ibm/watson-discovery_eye-connection_1x1_padding?ts=1768922047606&dpr=off\n" ] } ], "source": [ "for link in soup.find_all('img'): # in html image is represented by the tag \n", " print(link.get('src'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Scrape data from html tables\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "#The below URL contains a html table with data about colors and color codes.\n", "URL = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/HTMLColorCodes.html\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before proceeding to scrape a website, you need to examine the contents, and the way data is organized on the website. Open the above URL in your browser and check how many rows and columns are there in the color table.\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# get the contents of the webpage in text format and store in a variable called data\n", "data = requests.get(URL).text" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "soup = BeautifulSoup(data, \"html.parser\")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "#find a html table in the web page\n", "table = soup.find('table') # in html table is represented by the tag " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Get all rows from the table\n" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Color Name--->Hex Code#RRGGBB\n", "lightsalmon--->#FFA07A\n", "salmon--->#FA8072\n", "darksalmon--->#E9967A\n", "lightcoral--->#F08080\n", "coral--->#FF7F50\n", "tomato--->#FF6347\n", "orangered--->#FF4500\n", "gold--->#FFD700\n", "orange--->#FFA500\n", "darkorange--->#FF8C00\n", "lightyellow--->#FFFFE0\n", "lemonchiffon--->#FFFACD\n", "papayawhip--->#FFEFD5\n", "moccasin--->#FFE4B5\n", "peachpuff--->#FFDAB9\n", "palegoldenrod--->#EEE8AA\n", "khaki--->#F0E68C\n", "darkkhaki--->#BDB76B\n", "yellow--->#FFFF00\n", "lawngreen--->#7CFC00\n", "chartreuse--->#7FFF00\n", "limegreen--->#32CD32\n", "lime--->#00FF00\n", "forestgreen--->#228B22\n", "green--->#008000\n", "powderblue--->#B0E0E6\n", "lightblue--->#ADD8E6\n", "lightskyblue--->#87CEFA\n", "skyblue--->#87CEEB\n", "deepskyblue--->#00BFFF\n", "lightsteelblue--->#B0C4DE\n", "dodgerblue--->#1E90FF\n" ] } ], "source": [ "for row in table.find_all('tr'): # in html table row is represented by the tag \n", " # Get all columns in each row.\n", " cols = row.find_all('td') # in html a column is represented by the tag
\n", " color_name = cols[2].getText() # store the value in column 3 as color_name\n", " color_code = cols[3].getText() # store the value in column 4 as color_code\n", " print(\"{}--->{}\".format(color_name, color_code))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Authors\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ramesh Sannareddy\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Other Contributors\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rav Ahuja\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright © IBM Corporation. All rights reserved.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" }, "prev_pub_hash": "47f9ffc1b763f98e682ac3a01d2e9180987012a5e050cb2d84fe1df3c67f42fb" }, "nbformat": 4, "nbformat_minor": 4 }