{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "

\n", " \n", " \"Skills\n", " \n", "

\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# **Impute Missing Values**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Estimated time needed: **30** minutes\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this lab, you will practice essential data wrangling techniques using the Stack Overflow survey dataset. The primary focus is on handling missing data and ensuring data quality. You will:\n", "\n", "- **Load the Data:** Import the dataset into a DataFrame using the pandas library.\n", "\n", "- **Clean the Data:** Identify and remove duplicate entries to maintain data integrity.\n", "\n", "- **Handle Missing Values:** Detect missing values, impute them with appropriate strategies, and verify the imputation to create a complete and reliable dataset for analysis.\n", "\n", "This lab equips you with the skills to effectively preprocess and clean real-world datasets, a crucial step in any data analysis project.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Objectives\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this lab, you will perform the following:\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Identify missing values in the dataset.\n", "\n", "- Apply techniques to impute missing values in the dataset.\n", " \n", "- Use suitable techniques to normalize data in the dataset.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-----\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Install needed library\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true, "jupyter": { "outputs_hidden": true } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Collecting pandas\n", " Downloading pandas-3.0.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (79 kB)\n", "Collecting numpy>=1.26.0 (from pandas)\n", " Downloading numpy-2.4.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.12/site-packages (from pandas) (2.9.0.post0)\n", "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.12/site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n", "Downloading pandas-3.0.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (10.9 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.9/10.9 MB\u001b[0m \u001b[31m165.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hDownloading numpy-2.4.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.4 MB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.4/16.4 MB\u001b[0m \u001b[31m170.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "Installing collected packages: numpy, pandas\n", "Successfully installed numpy-2.4.1 pandas-3.0.0\n" ] } ], "source": [ "!pip install pandas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Import Required Libraries\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: Load the Dataset Into a Dataframe\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### **Read Data**\n", "

\n", "The functions below will download the dataset into your browser:\n", "

\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " ResponseId MainBranch Age \\\n", "0 1 I am a developer by profession Under 18 years old \n", "1 2 I am a developer by profession 35-44 years old \n", "2 3 I am a developer by profession 45-54 years old \n", "3 4 I am learning to code 18-24 years old \n", "4 5 I am a developer by profession 18-24 years old \n", "\n", " Employment RemoteWork Check \\\n", "0 Employed, full-time Remote Apples \n", "1 Employed, full-time Remote Apples \n", "2 Employed, full-time Remote Apples \n", "3 Student, full-time NaN Apples \n", "4 Student, full-time NaN Apples \n", "\n", " CodingActivities \\\n", "0 Hobby \n", "1 Hobby;Contribute to open-source projects;Other... \n", "2 Hobby;Contribute to open-source projects;Other... \n", "3 NaN \n", "4 NaN \n", "\n", " EdLevel \\\n", "0 Primary/elementary school \n", "1 Bachelor’s degree (B.A., B.S., B.Eng., etc.) \n", "2 Master’s degree (M.A., M.S., M.Eng., MBA, etc.) \n", "3 Some college/university study without earning ... \n", "4 Secondary school (e.g. American high school, G... \n", "\n", " LearnCode \\\n", "0 Books / Physical media \n", "1 Books / Physical media;Colleague;On the job tr... \n", "2 Books / Physical media;Colleague;On the job tr... \n", "3 Other online resources (e.g., videos, blogs, f... \n", "4 Other online resources (e.g., videos, blogs, f... \n", "\n", " LearnCodeOnline ... JobSatPoints_6 \\\n", "0 NaN ... NaN \n", "1 Technical documentation;Blogs;Books;Written Tu... ... 0.0 \n", "2 Technical documentation;Blogs;Books;Written Tu... ... NaN \n", "3 Stack Overflow;How-to videos;Interactive tutorial ... NaN \n", "4 Technical documentation;Blogs;Written Tutorial... ... NaN \n", "\n", " JobSatPoints_7 JobSatPoints_8 JobSatPoints_9 JobSatPoints_10 \\\n", "0 NaN NaN NaN NaN \n", "1 0.0 0.0 0.0 0.0 \n", "2 NaN NaN NaN NaN \n", "3 NaN NaN NaN NaN \n", "4 NaN NaN NaN NaN \n", "\n", " JobSatPoints_11 SurveyLength SurveyEase ConvertedCompYearly JobSat \n", "0 NaN NaN NaN NaN NaN \n", "1 0.0 NaN NaN NaN NaN \n", "2 NaN Appropriate in length Easy NaN NaN \n", "3 NaN Too long Easy NaN NaN \n", "4 NaN Too short Easy NaN NaN \n", "\n", "[5 rows x 114 columns]\n", "Initial dataframe: (65437, 114)\n" ] } ], "source": [ "file_path = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/n01PQ9pSmiRX6520flujwQ/survey-data.csv\"\n", "df = pd.read_csv(file_path)\n", "\n", "# Display the first few rows to ensure it loaded correctly\n", "print(df.head())\n", "print('Initial dataframe:', df.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 3. Finding and Removing Duplicates\n", "##### Task 1: Identify duplicate rows in the dataset.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fraction of duplicates in each column\n", "ResponseId: 0.0\n", "LanguageHaveWorkedWith: 0.6353\n", "LanguageWantToWorkWith: 0.652\n", "ToolsTechHaveWorkedWith: 0.6961\n", "ToolsTechWantToWorkWith: 0.7471\n" ] } ], "source": [ "## Write your code here\n", "frdupl = []\n", "for col in df.columns:\n", " nunique = df[col].nunique(dropna=False) # number of unique values for this column\n", " frdupl.append((col, round(((df[col].size - nunique) / df[col].size), 4)))\n", "frdupl = sorted(frdupl, key=lambda n:n[1])\n", "print('Fraction of duplicates in each column')\n", "for i in range(5):\n", " print(*frdupl[i], sep=': ')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Task 2: Remove the duplicate rows from the dataframe.\n", "\n" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of duplicate rows: 487\n", "Final dataframe: (64950, 114)\n" ] } ], "source": [ "## Write your code here\n", "subset_cols = list(df.drop(['ResponseId'], axis=1).columns)\n", "dup_rows = df[df.duplicated(subset=subset_cols)]\n", "print('Number of duplicate rows:', dup_rows.shape[0])\n", "df_d = df.drop_duplicates(subset=subset_cols)\n", "print('Final dataframe:', df_d.shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 4: Finding Missing Values\n", "##### Task 3: Find the missing values for all columns.\n" ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fraction of missing values in each column\n", "AINextMuch less integrated 0.9823\n", "AINextLess integrated 0.9637\n", "AINextNo change 0.8076\n", "AINextMuch more integrated 0.7931\n", "EmbeddedAdmired 0.7424\n", "dtype: float64\n", "RemoteWork: 0.1624\n" ] } ], "source": [ "## Write your code here\n", "frnan = df_d.isna().mean().sort_values(ascending=False)\n", "print('Fraction of missing values in each column')\n", "print(frnan.round(decimals=4).head())\n", "print('RemoteWork:', frnan.RemoteWork.round(decimals=4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Task 4: Find out how many rows are missing in the column RemoteWork.\n" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "10546\n" ] } ], "source": [ "## Write your code here\n", "print(df_d.RemoteWork.isna().sum())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 5. Imputing Missing Values\n", "##### Task 5: Find the value counts for the column RemoteWork.\n" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RemoteWork\n", "Hybrid (some remote, some in-person) 22850\n", "Remote 20732\n", "In-person 10822\n", "NaN 10546\n", "Name: count, dtype: int64\n" ] } ], "source": [ "## Write your code here\n", "print(df_d.RemoteWork.value_counts(dropna=False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Task 6: Identify the most frequent (majority) value in the RemoteWork column.\n", "\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Hybrid (some remote, some in-person)\n" ] } ], "source": [ "## Write your code here\n", "print(df_d.RemoteWork.describe().top)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Task 7: Impute (replace) all the empty rows in the column RemoteWork with the majority value.\n", "\n" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "count 64950\n", "unique 3\n", "top Hybrid (some remote, some in-person)\n", "freq 33396\n", "Name: RemoteWork, dtype: object\n" ] } ], "source": [ "## Write your code here\n", "df_dm = df_d.copy()\n", "common_RemoteWork = df_dm['RemoteWork'].value_counts().idxmax()\n", "df_dm['RemoteWork'] = df_dm['RemoteWork'].fillna(common_RemoteWork)\n", "print(df_dm['RemoteWork'].describe())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "##### Task 8: Check for any compensation-related columns and describe their distribution.\n", "\n" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Index: 64950 entries, 0 to 65436\n", "Data columns (total 2 columns):\n", " # Column Non-Null Count Dtype \n", "--- ------ -------------- ----- \n", " 0 CompTotal 33740 non-null float64\n", " 1 ConvertedCompYearly 23435 non-null float64\n", "dtypes: float64(2)\n", "memory usage: 1.5 MB\n", "None\n", " CompTotal ConvertedCompYearly\n", "count 3.374000e+04 2.343500e+04\n", "mean 2.963841e+145 8.615529e+04\n", "std 5.444117e+147 1.867570e+05\n", "min 0.000000e+00 1.000000e+00\n", "25% 6.000000e+04 3.271200e+04\n", "50% 1.100000e+05 6.500000e+04\n", "75% 2.500000e+05 1.079715e+05\n", "max 1.000000e+150 1.625660e+07\n" ] } ], "source": [ "## Write your code here\n", "print(df_dm[['CompTotal', 'ConvertedCompYearly']].info())\n", "print(df_dm[['CompTotal', 'ConvertedCompYearly']].describe())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Summary \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**In this lab, you focused on imputing missing values in the dataset.**\n", "\n", "- Use the pandas.read_csv() function to load a dataset from a CSV file into a DataFrame.\n", "\n", "- Download the dataset if it's not available online and specify the correct file path.\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Copyright © IBM Corporation. All rights reserved.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.5" }, "prev_pub_hash": "70ab641719bca2be0bdcb38f6a8b5de7851b6e9c28d41b9407096c62e74916a6" }, "nbformat": 4, "nbformat_minor": 4 }