{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DtNBZFHO3M7n"
   },
   "source": [
    "# **TikTok Project**\n",
    "**Course 3 - The Power of Statistics**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j-u8hxWUk64w"
   },
   "source": [
    "You are a data professional at TikTok. The current project is reaching its midpoint; a project proposal, Python coding work, and exploratory data analysis have all been completed.\n",
    "\n",
    "The team has reviewed the results of the exploratory data analysis and the previous executive summary the team prepared. You received an email from Orion Rainier, Data Scientist at TikTok, with your next assignment: determine and conduct the necessary hypothesis tests and statistical analysis for the TikTok classification project.\n",
    "\n",
    "A notebook was structured and prepared to help you in this project. Please complete the following questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rgSbVJvomcVa"
   },
   "source": [
    "\n",
    "# **Course 3 End-of-course project: Data exploration and hypothesis testing**\n",
    "\n",
    "In this activity, you will explore the data provided and conduct hypothesis testing.\n",
    "<br/>\n",
    "\n",
    "**The purpose** of this project is to demonstrate knowledge of how to prepare, create, and analyze hypothesis tests.\n",
    "\n",
    "**The goal** is to apply descriptive and inferential statistics, probability distributions, and hypothesis testing in Python.\n",
    "<br/>\n",
    "\n",
    "*This activity has three parts:*\n",
    "\n",
    "**Part 1:** Imports and data loading\n",
    "* What data packages will be necessary for hypothesis testing?\n",
    "\n",
    "**Part 2:** Conduct hypothesis testing\n",
    "* How will descriptive statistics help you analyze your data?\n",
    "\n",
    "* How will you formulate your null hypothesis and alternative hypothesis?\n",
    "\n",
    "**Part 3:** Communicate insights with stakeholders\n",
    "\n",
    "* What key business insight(s) emerge from your hypothesis test?\n",
    "\n",
    "* What business recommendations do you propose based on your results?\n",
    "\n",
    "<br/>\n",
    "\n",
    "Follow the instructions and answer the questions below to complete the activity. Then, complete an executive summary using the questions listed on the PACE Strategy Document.\n",
    "\n",
    "Be sure to complete this activity before moving on. The next course item will provide you with a completed exemplar to compare to your own work.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qwXf61KLibBj"
   },
   "source": [
    "# **Data exploration and hypothesis testing**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N8qrn_UAzX6j"
   },
   "source": [
    "<img src=\"images/Pace.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "# **PACE stages**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pGaoqOgZiiR_"
   },
   "source": [
    "Throughout these project notebooks, you'll see references to the problem-solving framework PACE. The following notebook components are labeled with the respective PACE stage: Plan, Analyze, Construct, and Execute."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Bb1HNu3FzX6k"
   },
   "source": [
    "<img src=\"images/Plan.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "\n",
    "## **PACE: Plan**\n",
    "\n",
    "Consider the questions in your PACE Strategy Document and those below to craft your response.\n",
    "\n",
    "1. What is your research question for this data project? Later on, you will need to formulate the null and alternative hypotheses as the first step of your hypothesis test. Consider your research question now, at the start of this task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iHhO44UenHcC"
   },
   "source": [
    "To determine whether there is a statistically significant difference in the average number of views for TikTok videos posted by verified accounts versus unverified accounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Kp3uDhhSg_qu"
   },
   "source": [
    "*Complete the following steps to perform statistical analysis of your data:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xgQEdIN7f1Jq"
   },
   "source": [
    "### **Task 1. Imports and Data Loading**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ehehvUzFj0-e"
   },
   "source": [
    "Import packages and libraries needed to compute descriptive statistics and conduct a hypothesis test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e6cryW4WgiIL"
   },
   "source": [
    "<details>\n",
    "  <summary><h4><strong>Hint:</strong></h4></summary>\n",
    "\n",
    "Be sure to import `pandas`, `numpy`, `matplotlib.pyplot`, `seaborn`, and `scipy`.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "id": "1Q-GL2UrgWWG"
   },
   "outputs": [],
   "source": [
    "# Import packages for data manipulation\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Import packages for data visualization\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Import packages for statistical analysis/hypothesis testing\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IHSa7TiTkg78"
   },
   "source": [
    "Load the dataset.\n",
    "\n",
    "**Note:** As shown in this cell, the dataset has been automatically loaded in for you. You do not need to download the .csv file, or provide more code, in order to access the dataset and proceed with this lab. Please continue with this activity by completing the following instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "HLhV9YACjlC4"
   },
   "outputs": [],
   "source": [
    "# Load dataset into dataframe\n",
    "data = pd.read_csv(\"tiktok_dataset.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c8paWsdKzX6o"
   },
   "source": [
    "<img src=\"images/Analyze.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "<img src=\"images/Construct.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "## **PACE: Analyze and Construct**\n",
    "\n",
    "Consider the questions in your PACE Strategy Document and those below to craft your response:\n",
    "1. Data professionals use descriptive statistics for Exploratory Data Analysis. How can computing descriptive statistics help you learn more about your data in this stage of your analysis?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AKY3ms4LsEsU"
   },
   "source": [
    "Descriptive statistics are used to explore and summarize data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UKhYTdVhf5B2"
   },
   "source": [
    "### **Task 2. Data exploration**\n",
    "\n",
    "Use descriptive statistics to conduct Exploratory Data Analysis (EDA).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aBLu1iBMr6Bd"
   },
   "source": [
    "<details>\n",
    "  <summary><h4><strong>Hint:</strong></h4></summary>\n",
    "\n",
    "Refer back to *Self Review Descriptive Statistics* for this step-by-step proccess.\n",
    "\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JBHl90JIRuXk"
   },
   "source": [
    "Inspect the first five rows of the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "2rfk95MLp4a_"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>#</th>\n",
       "      <th>claim_status</th>\n",
       "      <th>video_id</th>\n",
       "      <th>video_duration_sec</th>\n",
       "      <th>video_transcription_text</th>\n",
       "      <th>verified_status</th>\n",
       "      <th>author_ban_status</th>\n",
       "      <th>video_view_count</th>\n",
       "      <th>video_like_count</th>\n",
       "      <th>video_share_count</th>\n",
       "      <th>video_download_count</th>\n",
       "      <th>video_comment_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>claim</td>\n",
       "      <td>7017666017</td>\n",
       "      <td>59</td>\n",
       "      <td>someone shared with me that drone deliveries a...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>under review</td>\n",
       "      <td>343296.0</td>\n",
       "      <td>19425.0</td>\n",
       "      <td>241.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>claim</td>\n",
       "      <td>4014381136</td>\n",
       "      <td>32</td>\n",
       "      <td>someone shared with me that there are more mic...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>140877.0</td>\n",
       "      <td>77355.0</td>\n",
       "      <td>19034.0</td>\n",
       "      <td>1161.0</td>\n",
       "      <td>684.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>claim</td>\n",
       "      <td>9859838091</td>\n",
       "      <td>31</td>\n",
       "      <td>someone shared with me that american industria...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>902185.0</td>\n",
       "      <td>97690.0</td>\n",
       "      <td>2858.0</td>\n",
       "      <td>833.0</td>\n",
       "      <td>329.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>claim</td>\n",
       "      <td>1866847991</td>\n",
       "      <td>25</td>\n",
       "      <td>someone shared with me that the metro of st. p...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>437506.0</td>\n",
       "      <td>239954.0</td>\n",
       "      <td>34812.0</td>\n",
       "      <td>1234.0</td>\n",
       "      <td>584.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>claim</td>\n",
       "      <td>7105231098</td>\n",
       "      <td>19</td>\n",
       "      <td>someone shared with me that the number of busi...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>56167.0</td>\n",
       "      <td>34987.0</td>\n",
       "      <td>4110.0</td>\n",
       "      <td>547.0</td>\n",
       "      <td>152.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   # claim_status    video_id  video_duration_sec  \\\n",
       "0  1        claim  7017666017                  59   \n",
       "1  2        claim  4014381136                  32   \n",
       "2  3        claim  9859838091                  31   \n",
       "3  4        claim  1866847991                  25   \n",
       "4  5        claim  7105231098                  19   \n",
       "\n",
       "                            video_transcription_text verified_status  \\\n",
       "0  someone shared with me that drone deliveries a...    not verified   \n",
       "1  someone shared with me that there are more mic...    not verified   \n",
       "2  someone shared with me that american industria...    not verified   \n",
       "3  someone shared with me that the metro of st. p...    not verified   \n",
       "4  someone shared with me that the number of busi...    not verified   \n",
       "\n",
       "  author_ban_status  video_view_count  video_like_count  video_share_count  \\\n",
       "0      under review          343296.0           19425.0              241.0   \n",
       "1            active          140877.0           77355.0            19034.0   \n",
       "2            active          902185.0           97690.0             2858.0   \n",
       "3            active          437506.0          239954.0            34812.0   \n",
       "4            active           56167.0           34987.0             4110.0   \n",
       "\n",
       "   video_download_count  video_comment_count  \n",
       "0                   1.0                  0.0  \n",
       "1                1161.0                684.0  \n",
       "2                 833.0                329.0  \n",
       "3                1234.0                584.0  \n",
       "4                 547.0                152.0  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display first few rows\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "70vpvcv1gArp"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>#</th>\n",
       "      <th>claim_status</th>\n",
       "      <th>video_id</th>\n",
       "      <th>video_duration_sec</th>\n",
       "      <th>video_transcription_text</th>\n",
       "      <th>verified_status</th>\n",
       "      <th>author_ban_status</th>\n",
       "      <th>video_view_count</th>\n",
       "      <th>video_like_count</th>\n",
       "      <th>video_share_count</th>\n",
       "      <th>video_download_count</th>\n",
       "      <th>video_comment_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>19382.000000</td>\n",
       "      <td>19084</td>\n",
       "      <td>1.938200e+04</td>\n",
       "      <td>19382.000000</td>\n",
       "      <td>19084</td>\n",
       "      <td>19382</td>\n",
       "      <td>19382</td>\n",
       "      <td>19084.000000</td>\n",
       "      <td>19084.000000</td>\n",
       "      <td>19084.000000</td>\n",
       "      <td>19084.000000</td>\n",
       "      <td>19084.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19012</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>NaN</td>\n",
       "      <td>claim</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>a friend read  in the media a claim that badmi...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>NaN</td>\n",
       "      <td>9608</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>18142</td>\n",
       "      <td>15663</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>9691.500000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.627454e+09</td>\n",
       "      <td>32.421732</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>254708.558688</td>\n",
       "      <td>84304.636030</td>\n",
       "      <td>16735.248323</td>\n",
       "      <td>1049.429627</td>\n",
       "      <td>349.312146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>5595.245794</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.536440e+09</td>\n",
       "      <td>16.229967</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>322893.280814</td>\n",
       "      <td>133420.546814</td>\n",
       "      <td>32036.174350</td>\n",
       "      <td>2004.299894</td>\n",
       "      <td>799.638865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.234959e+09</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>4846.250000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.430417e+09</td>\n",
       "      <td>18.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4942.500000</td>\n",
       "      <td>810.750000</td>\n",
       "      <td>115.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>9691.500000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.618664e+09</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9954.500000</td>\n",
       "      <td>3403.500000</td>\n",
       "      <td>717.000000</td>\n",
       "      <td>46.000000</td>\n",
       "      <td>9.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>14536.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.843960e+09</td>\n",
       "      <td>47.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>504327.000000</td>\n",
       "      <td>125020.000000</td>\n",
       "      <td>18222.000000</td>\n",
       "      <td>1156.250000</td>\n",
       "      <td>292.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>19382.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.999873e+09</td>\n",
       "      <td>60.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>999817.000000</td>\n",
       "      <td>657830.000000</td>\n",
       "      <td>256130.000000</td>\n",
       "      <td>14994.000000</td>\n",
       "      <td>9599.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   # claim_status      video_id  video_duration_sec  \\\n",
       "count   19382.000000        19084  1.938200e+04        19382.000000   \n",
       "unique           NaN            2           NaN                 NaN   \n",
       "top              NaN        claim           NaN                 NaN   \n",
       "freq             NaN         9608           NaN                 NaN   \n",
       "mean     9691.500000          NaN  5.627454e+09           32.421732   \n",
       "std      5595.245794          NaN  2.536440e+09           16.229967   \n",
       "min         1.000000          NaN  1.234959e+09            5.000000   \n",
       "25%      4846.250000          NaN  3.430417e+09           18.000000   \n",
       "50%      9691.500000          NaN  5.618664e+09           32.000000   \n",
       "75%     14536.750000          NaN  7.843960e+09           47.000000   \n",
       "max     19382.000000          NaN  9.999873e+09           60.000000   \n",
       "\n",
       "                                 video_transcription_text verified_status  \\\n",
       "count                                               19084           19382   \n",
       "unique                                              19012               2   \n",
       "top     a friend read  in the media a claim that badmi...    not verified   \n",
       "freq                                                    2           18142   \n",
       "mean                                                  NaN             NaN   \n",
       "std                                                   NaN             NaN   \n",
       "min                                                   NaN             NaN   \n",
       "25%                                                   NaN             NaN   \n",
       "50%                                                   NaN             NaN   \n",
       "75%                                                   NaN             NaN   \n",
       "max                                                   NaN             NaN   \n",
       "\n",
       "       author_ban_status  video_view_count  video_like_count  \\\n",
       "count              19382      19084.000000      19084.000000   \n",
       "unique                 3               NaN               NaN   \n",
       "top               active               NaN               NaN   \n",
       "freq               15663               NaN               NaN   \n",
       "mean                 NaN     254708.558688      84304.636030   \n",
       "std                  NaN     322893.280814     133420.546814   \n",
       "min                  NaN         20.000000          0.000000   \n",
       "25%                  NaN       4942.500000        810.750000   \n",
       "50%                  NaN       9954.500000       3403.500000   \n",
       "75%                  NaN     504327.000000     125020.000000   \n",
       "max                  NaN     999817.000000     657830.000000   \n",
       "\n",
       "        video_share_count  video_download_count  video_comment_count  \n",
       "count        19084.000000          19084.000000         19084.000000  \n",
       "unique                NaN                   NaN                  NaN  \n",
       "top                   NaN                   NaN                  NaN  \n",
       "freq                  NaN                   NaN                  NaN  \n",
       "mean         16735.248323           1049.429627           349.312146  \n",
       "std          32036.174350           2004.299894           799.638865  \n",
       "min              0.000000              0.000000             0.000000  \n",
       "25%            115.000000              7.000000             1.000000  \n",
       "50%            717.000000             46.000000             9.000000  \n",
       "75%          18222.000000           1156.250000           292.000000  \n",
       "max         256130.000000          14994.000000          9599.000000  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a table of descriptive statistics about the data\n",
    "data.describe(include=\"all\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9OpeNQDdyIT6"
   },
   "source": [
    "Check for and handle missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "BrunHcfa7xnT"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "#                             0\n",
       "claim_status                298\n",
       "video_id                      0\n",
       "video_duration_sec            0\n",
       "video_transcription_text    298\n",
       "verified_status               0\n",
       "author_ban_status             0\n",
       "video_view_count            298\n",
       "video_like_count            298\n",
       "video_share_count           298\n",
       "video_download_count        298\n",
       "video_comment_count         298\n",
       "dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for missing values\n",
    "data.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "SHSj1Hma914I"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 19084 entries, 0 to 19083\n",
      "Data columns (total 12 columns):\n",
      " #   Column                    Non-Null Count  Dtype  \n",
      "---  ------                    --------------  -----  \n",
      " 0   #                         19084 non-null  int64  \n",
      " 1   claim_status              19084 non-null  object \n",
      " 2   video_id                  19084 non-null  int64  \n",
      " 3   video_duration_sec        19084 non-null  int64  \n",
      " 4   video_transcription_text  19084 non-null  object \n",
      " 5   verified_status           19084 non-null  object \n",
      " 6   author_ban_status         19084 non-null  object \n",
      " 7   video_view_count          19084 non-null  float64\n",
      " 8   video_like_count          19084 non-null  float64\n",
      " 9   video_share_count         19084 non-null  float64\n",
      " 10  video_download_count      19084 non-null  float64\n",
      " 11  video_comment_count       19084 non-null  float64\n",
      "dtypes: float64(5), int64(3), object(4)\n",
      "memory usage: 1.9+ MB\n"
     ]
    }
   ],
   "source": [
    "# Drop rows with missing values\n",
    "data_m = data.dropna()\n",
    "data_m.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "ZYUF3xap9_Fa"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>#</th>\n",
       "      <th>claim_status</th>\n",
       "      <th>video_id</th>\n",
       "      <th>video_duration_sec</th>\n",
       "      <th>video_transcription_text</th>\n",
       "      <th>verified_status</th>\n",
       "      <th>author_ban_status</th>\n",
       "      <th>video_view_count</th>\n",
       "      <th>video_like_count</th>\n",
       "      <th>video_share_count</th>\n",
       "      <th>video_download_count</th>\n",
       "      <th>video_comment_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>claim</td>\n",
       "      <td>7017666017</td>\n",
       "      <td>59</td>\n",
       "      <td>someone shared with me that drone deliveries a...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>under review</td>\n",
       "      <td>343296.0</td>\n",
       "      <td>19425.0</td>\n",
       "      <td>241.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>claim</td>\n",
       "      <td>4014381136</td>\n",
       "      <td>32</td>\n",
       "      <td>someone shared with me that there are more mic...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>140877.0</td>\n",
       "      <td>77355.0</td>\n",
       "      <td>19034.0</td>\n",
       "      <td>1161.0</td>\n",
       "      <td>684.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>claim</td>\n",
       "      <td>9859838091</td>\n",
       "      <td>31</td>\n",
       "      <td>someone shared with me that american industria...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>902185.0</td>\n",
       "      <td>97690.0</td>\n",
       "      <td>2858.0</td>\n",
       "      <td>833.0</td>\n",
       "      <td>329.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>claim</td>\n",
       "      <td>1866847991</td>\n",
       "      <td>25</td>\n",
       "      <td>someone shared with me that the metro of st. p...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>437506.0</td>\n",
       "      <td>239954.0</td>\n",
       "      <td>34812.0</td>\n",
       "      <td>1234.0</td>\n",
       "      <td>584.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>claim</td>\n",
       "      <td>7105231098</td>\n",
       "      <td>19</td>\n",
       "      <td>someone shared with me that the number of busi...</td>\n",
       "      <td>not verified</td>\n",
       "      <td>active</td>\n",
       "      <td>56167.0</td>\n",
       "      <td>34987.0</td>\n",
       "      <td>4110.0</td>\n",
       "      <td>547.0</td>\n",
       "      <td>152.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   # claim_status    video_id  video_duration_sec  \\\n",
       "0  1        claim  7017666017                  59   \n",
       "1  2        claim  4014381136                  32   \n",
       "2  3        claim  9859838091                  31   \n",
       "3  4        claim  1866847991                  25   \n",
       "4  5        claim  7105231098                  19   \n",
       "\n",
       "                            video_transcription_text verified_status  \\\n",
       "0  someone shared with me that drone deliveries a...    not verified   \n",
       "1  someone shared with me that there are more mic...    not verified   \n",
       "2  someone shared with me that american industria...    not verified   \n",
       "3  someone shared with me that the metro of st. p...    not verified   \n",
       "4  someone shared with me that the number of busi...    not verified   \n",
       "\n",
       "  author_ban_status  video_view_count  video_like_count  video_share_count  \\\n",
       "0      under review          343296.0           19425.0              241.0   \n",
       "1            active          140877.0           77355.0            19034.0   \n",
       "2            active          902185.0           97690.0             2858.0   \n",
       "3            active          437506.0          239954.0            34812.0   \n",
       "4            active           56167.0           34987.0             4110.0   \n",
       "\n",
       "   video_download_count  video_comment_count  \n",
       "0                   1.0                  0.0  \n",
       "1                1161.0                684.0  \n",
       "2                 833.0                329.0  \n",
       "3                1234.0                584.0  \n",
       "4                 547.0                152.0  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display first few rows after handling missing values\n",
    "data_m.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-fFiNg4hdGBK"
   },
   "source": [
    "You are interested in the relationship between `verified_status` and `video_view_count`. One approach is to examine the mean value of `video_view_count` for each group of `verified_status` in the sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "2LGEWrdVXgI2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "verified_status\n",
       "not verified    265663.785339\n",
       "verified         91439.164167\n",
       "Name: video_view_count, dtype: float64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the mean `video_view_count` for each group in `verified_status`\n",
    "data_m.groupby(\"verified_status\")[\"video_view_count\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QpFRQDaNzX6p"
   },
   "source": [
    "### **Task 3. Hypothesis testing**\n",
    "\n",
    "Before you conduct your hypothesis test, consider the following questions where applicable to complete your code response:\n",
    "\n",
    "1. Recall the difference between the null hypothesis and the alternative hypotheses. What are your hypotheses for this data project?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ViBd7pL-tYHY"
   },
   "source": [
    "**Null hypothesis**: There is no difference in number of views between TikTok videos posted by verified accounts and TikTok videos posted by unverified accounts (any observed difference in the sample data is due to chance or sampling variability).<br>\n",
    "**Alternative hypothesis**: There is a difference in number of views between TikTok videos posted by verified accounts and TikTok videos posted by unverified accounts (any observed difference in the sample data is due to an actual difference in the corresponding population means)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7Mn5GJ460HUS"
   },
   "source": [
    "\n",
    "\n",
    "Your goal in this step is to conduct a two-sample t-test. Recall the steps for conducting a hypothesis test:\n",
    "\n",
    "\n",
    "1.   State the null hypothesis and the alternative hypothesis\n",
    "2.   Choose a signficance level\n",
    "3.   Find the p-value\n",
    "4.   Reject or fail to reject the null hypothesis\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "phXxr74vVPrs"
   },
   "source": [
    "$H_0$: There is no difference in the average video view count for TikTok videos posted by verified accounts versus unverified accounts.\n",
    "\n",
    "$H_A$: There is a difference in the average video view count for TikTok videos posted by verified accounts versus unverified accounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IVHpnfvNeTUA"
   },
   "source": [
    "You choose 5% as the significance level and proceed with a two-sample t-test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "rdlsXg7Dhbrd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Ttest_indResult(statistic=25.499441780633777, pvalue=2.6088823687177823e-120)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Conduct a two-sample t-test to compare means\n",
    "significance_level = 0.05\n",
    "a = data_m[data_m[\"verified_status\"] == \"not verified\"][\"video_view_count\"]\n",
    "b = data_m[data_m[\"verified_status\"] == \"verified\"][\"video_view_count\"]\n",
    "stats.ttest_ind(a=a, b=b, equal_var=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BmCJLuE2vwpI"
   },
   "source": [
    "**Question:** Based on the p-value you got above, do you reject or fail to reject the null hypothesis?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64Wq_VSaVsNm"
   },
   "source": [
    "With a p-value (2.6e-120) being less than 0.05 (as the significance level is 5%), we reject the null hypothesis in favor of the alternative hypothesis.\n",
    "\n",
    "Therefore, we can conclude at the 5% significance level that there is a statistically significant difference in the average video view count for TikTok videos posted by verified accounts versus unverified accounts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ab4e-W3mzX6r"
   },
   "source": [
    "<img src=\"images/Execute.png\" width=\"100\" height=\"100\" align=left>\n",
    "\n",
    "## **PACE: Execute**\n",
    "\n",
    "Consider the questions in your PACE Strategy Documentto reflect on the Execute stage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h1PRfiD3hcwE"
   },
   "source": [
    "## **Step 4: Communicate insights with stakeholders**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ItreFaZKhvVw"
   },
   "source": [
    "*Ask yourself the following questions:*\n",
    "\n",
    "1. What business insight(s) can you draw from the result of your hypothesis test?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lL__ZEvhtQIH"
   },
   "source": [
    "We can conclude at the 5% significance level that non-verified users published more TikTok videos in which a claim/opinion has been made. <br>\n",
    "It would be interesting to investigate the main cause of this difference.\n",
    "Also, it seems that encouraging customers to verify their accounts can decrease the amount of content that needs to be reviewed by moderators.<br>\n",
    "The next step will be to build a regression model on verified_status. A regression model is the natural next step because the end goal is to make predictions on claim status. A regression model for verified_status can help analyze user behavior in this group of verified users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations!** You've completed this lab. However, you may not notice a green check mark next to this item on Coursera's platform. Please continue your progress regardless of the check mark. Just click on the \"save\" icon at the top of this notebook to ensure your work has been logged."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
