{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " \n", " \"Skills\n", " \n", "

\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# **Collecting Job Data Using APIs**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Estimated time needed: **30** minutes\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Objectives\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After completing this lab, you will be able to:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Collect job data using Jobs API\n", "* Store the collected data into an excel spreadsheet.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ">Note: Before starting with the assignment make sure to read all the instructions and then move ahead with the coding part.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Instructions\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To run the actual lab, firstly you need to click on the [Jobs_API](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Jobs_API.ipynb) notebook link. The file contains flask code which is required to run the Jobs API data.\n", "\n", "Now, to run the code in the file that opens up follow the below steps.\n", "\n", "Step1: Download the file. \n", "\n", "Step2: Upload the file into your current Jupyter environment using the upload button in your Jupyter interface. Ensure that the file is in the same folder as your working .ipynb file.\n", "\n", "Step 2: If working in a local Jupyter environment, use the \"Upload\" button in your Jupyter interface to upload the Jobs_API notebook into the same folder as your current .ipynb file.\n", "\n", "\n", "\n", "Step3: Open the Jobs_API notebook, and run all the cells to start the Flask application. Once the server is running, you can access the API from the URL provided in the notebook.\n", "\n", "If you want to learn more about flask, which is optional, you can click on this link [here](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/FLASK_API.md.html).\n", "\n", "Once you run the flask code, you can start with your assignment.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dataset Used in this Assignment\n", "\n", "The dataset used in this lab comes from the following source: https://www.kaggle.com/promptcloud/jobs-on-naukricom under the under a **Public Domain license**.\n", "\n", "> Note: We are using a modified subset of that dataset for the lab, so to follow the lab instructions successfully please use the dataset provided with the lab, rather than the dataset from the original source.\n", "\n", "The original dataset is a csv. We have converted the csv to json as per the requirement of the lab.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Warm-Up Exercise\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Before you attempt the actual lab, here is a fully solved warmup exercise that will help you to learn how to access an API.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using an API, let us find out who currently are on the International Space Station (ISS).
The API at [http://api.open-notify.org/astros.json](http://api.open-notify.org/astros.json?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ) gives us the information of astronauts currently on ISS in json format.
\n", "You can read more about this API at [http://open-notify.org/Open-Notify-API/People-In-Space/](http://open-notify.org/Open-Notify-API/People-In-Space?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ)\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [] }, "outputs": [], "source": [ "import requests # you need this module to make an API call\n", "import pandas as pd" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [], "source": [ "response = requests.get(api_url) # Call the API using the get method and store the\n", " # output of the API call in a variable called response." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [], "source": [ "if response.ok: # if all is well() no errors, no network timeouts)\n", " data = response.json() # store the result in json format in a variable called data\n", " # the variable data is of type dictionary." ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'people': [{'craft': 'ISS', 'name': 'Oleg Kononenko'}, {'craft': 'ISS', 'name': 'Nikolai Chub'}, {'craft': 'ISS', 'name': 'Tracy Caldwell Dyson'}, {'craft': 'ISS', 'name': 'Matthew Dominick'}, {'craft': 'ISS', 'name': 'Michael Barratt'}, {'craft': 'ISS', 'name': 'Jeanette Epps'}, {'craft': 'ISS', 'name': 'Alexander Grebenkin'}, {'craft': 'ISS', 'name': 'Butch Wilmore'}, {'craft': 'ISS', 'name': 'Sunita Williams'}, {'craft': 'Tiangong', 'name': 'Li Guangsu'}, {'craft': 'Tiangong', 'name': 'Li Cong'}, {'craft': 'Tiangong', 'name': 'Ye Guangfu'}], 'number': 12, 'message': 'success'}\n" ] } ], "source": [ "print(data) # print the data just to check the output or for debugging" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Print the number of astronauts currently on ISS.\n" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "9\n" ] } ], "source": [ "people = data.get('people')\n", "ISS_astronauts = []\n", "for p in people:\n", " if p['craft'] == 'ISS':\n", " ISS_astronauts.append(p)\n", "print(len(ISS_astronauts))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Print the names of the astronauts currently on ISS.\n" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "There are 9 astronauts on ISS\n", "And their names are :\n", "Oleg Kononenko\n", "Nikolai Chub\n", "Tracy Caldwell Dyson\n", "Matthew Dominick\n", "Michael Barratt\n", "Jeanette Epps\n", "Alexander Grebenkin\n", "Butch Wilmore\n", "Sunita Williams\n" ] } ], "source": [ "print(\"There are {} astronauts on ISS\".format(len(ISS_astronauts)))\n", "print(\"And their names are :\")\n", "for astronaut in ISS_astronauts:\n", " print(astronaut.get('name'))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Hope the warmup was helpful. Good luck with your next lab!\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Lab: Collect Jobs Data using Jobs API\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Objective: Determine the number of jobs currently open for various technologies and for various locations\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Collect the number of job postings for the following locations using the API:\n", "\n", "* Los Angeles\n", "* New York\n", "* San Francisco\n", "* Washington DC\n", "* Seattle\n", "* Austin\n", "* Detroit\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "#Import required libraries\n", "import pandas as pd\n", "import json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json#### Write a function to get the number of jobs for the Python technology.
\n", "> Note: While using the lab you need to pass the **payload** information for the **params** attribute in the form of **key** **value** pairs.\n", " Refer the ungraded **rest api lab** in the course **Python for Data Science, AI & Development** link\n", " \n", " ##### The keys in the json are \n", " * Job Title\n", " \n", " * Job Experience Required\n", " \n", " * Key Skills\n", " \n", " * Role Category\n", " \n", " * Location\n", " \n", " * Functional Area\n", " \n", " * Industry\n", " \n", " * Role \n", " \n", "You can also view the json file contents from the following json URL.\n", "\n" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [], "source": [ "api_url=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json\"\n", "\n", "def get_number_of_jobs_T(technology):\n", " #your code goes here\n", " \n", " response = requests.get(api_url, params={\"Key Skills\": technology})\n", " if response.ok: # if all is well() no errors, no network timeouts)\n", " data = response.json()\n", " number_of_jobs = 0\n", " for j in data:\n", " if technology.lower() in j['Key Skills'].lower().replace('| ', '|').replace(' |', '|').strip().split('|'):\n", " number_of_jobs += 1\n", " return technology, number_of_jobs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Calling the function for Python and checking if it works.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "('Python', 1165)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_number_of_jobs_T(\"Python\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Write a function to find number of jobs in US for a location of your choice\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [], "source": [ " def get_number_of_jobs_L(location):\n", " \n", " #your coe goes here\n", " response = requests.get(api_url, params={\"Location\": location})\n", " if response.ok: # if all is well() no errors, no network timeouts)\n", " data = response.json()\n", " number_of_jobs = 0\n", " for j in data:\n", " if location.lower() in j['Location'].lower().strip():\n", " number_of_jobs += 1\n", " return location, number_of_jobs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Call the function for Los Angeles and check if it is working.\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "('Los Angeles', 640)" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#your code goes here\n", "get_number_of_jobs_L('Los Angeles')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Store the results in an excel file\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Call the API for all the given technologies above and write the results in an excel spreadsheet.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you do not know how create excel file using python, double click here for **hints**.\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a python list of all technologies for which you need to find the number of jobs postings.\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "tags": [] }, "outputs": [], "source": [ "#your code goes here\n", "technologies = ['C', 'C#', 'C++', 'Java', 'JavaScript', 'Python', 'Scala', 'Oracle', 'SQL Server', 'MySQL Server', 'PostgreSQL', 'MongoDB']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Import libraries required to create excel spreadsheet\n" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: xlrd in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (1.2.0)\n", "Collecting openpyxl\n", " Downloading openpyxl-3.1.3-py2.py3-none-any.whl (251 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m251.3/251.3 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting et-xmlfile (from openpyxl)\n", " Downloading et_xmlfile-1.1.0-py3-none-any.whl (4.7 kB)\n", "Installing collected packages: et-xmlfile, openpyxl\n", "Successfully installed et-xmlfile-1.1.0 openpyxl-3.1.3\n" ] } ], "source": [ "# your code goes here\n", "!pip install xlrd\n", "!pip install openpyxl\n", "import xlrd, openpyxl" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Create a workbook and select the active worksheet\n" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [] }, "outputs": [], "source": [ "# your code goes here\n", "wb = openpyxl.Workbook() # create a workbook object\n", "ws = wb.active # use the active worksheet" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Find the number of jobs postings for each of the technology in the above list.\n", "Write the technology name and the number of jobs postings into the excel spreadsheet.\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [], "source": [ "#your code goes here\n", "for t in technologies:\n", " ws.append(get_number_of_jobs_T(t)) # add a row with two columns" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Save into an excel spreadsheet named **job-postings.xlsx**.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "tags": [] }, "outputs": [], "source": [ "#your code goes here\n", "wb.save(\"job-postings.xlsx\") # save the workbook into a file called countries.xlsx" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### In the similar way, you can try for below given technologies and results can be stored in an excel sheet.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Collect the number of job postings for the following languages using the API:\n", "\n", "* C\n", "* C#\n", "* C++\n", "* Java\n", "* JavaScript\n", "* Python\n", "* Scala\n", "* Oracle\n", "* SQL Server\n", "* MySQL Server\n", "* PostgreSQL\n", "* MongoDB\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Authors\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ayushi Jain\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Other Contributors\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rav Ahuja\n", "\n", "Lakshmi Holla\n", "\n", "Malika\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright © IBM Corporation.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" }, "prev_pub_hash": "61a35a07ad98492b710274ae0e52a0fdce2221e88e366133dd4a20370680fa8f" }, "nbformat": 4, "nbformat_minor": 4 }