TikTok Project

Course 3 - The Power of Statistics

You are a data professional at TikTok. The current project is reaching its midpoint; a project proposal, Python coding work, and exploratory data analysis have all been completed.

The team has reviewed the results of the exploratory data analysis and the previous executive summary the team prepared. You received an email from Orion Rainier, Data Scientist at TikTok, with your next assignment: determine and conduct the necessary hypothesis tests and statistical analysis for the TikTok classification project.

A notebook was structured and prepared to help you in this project. Please complete the following questions.

Course 3 End-of-course project: Data exploration and hypothesis testing

In this activity, you will explore the data provided and conduct hypothesis testing.

The purpose of this project is to demonstrate knowledge of how to prepare, create, and analyze hypothesis tests.

The goal is to apply descriptive and inferential statistics, probability distributions, and hypothesis testing in Python.

This activity has three parts:

Part 1: Imports and data loading

• What data packages will be necessary for hypothesis testing?

Part 2: Conduct hypothesis testing

• How will descriptive statistics help you analyze your data?

• How will you formulate your null hypothesis and alternative hypothesis?

Part 3: Communicate insights with stakeholders

• What key business insight(s) emerge from your hypothesis test?

• What business recommendations do you propose based on your results?

Follow the instructions and answer the questions below to complete the activity. Then, complete an executive summary using the questions listed on the PACE Strategy Document.

Be sure to complete this activity before moving on. The next course item will provide you with a completed exemplar to compare to your own work.

Data exploration and hypothesis testing

PACE stages

Throughout these project notebooks, you'll see references to the problem-solving framework PACE. The following notebook components are labeled with the respective PACE stage: Plan, Analyze, Construct, and Execute.

PACE: Plan

Consider the questions in your PACE Strategy Document and those below to craft your response.

1. What is your research question for this data project? Later on, you will need to formulate the null and alternative hypotheses as the first step of your hypothesis test. Consider your research question now, at the start of this task.

To determine whether there is a statistically significant difference in the average number of views for TikTok videos posted by verified accounts versus unverified accounts.

Complete the following steps to perform statistical analysis of your data:

Task 1. Imports and Data Loading

Import packages and libraries needed to compute descriptive statistics and conduct a hypothesis test.

Hint:

Be sure to import pandas, numpy, matplotlib.pyplot, seaborn, and scipy.

# Import packages for data manipulation
import pandas as pd
import numpy as np

# Import packages for data visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Import packages for statistical analysis/hypothesis testing
from scipy import stats

Load the dataset.

Note: As shown in this cell, the dataset has been automatically loaded in for you. You do not need to download the .csv file, or provide more code, in order to access the dataset and proceed with this lab. Please continue with this activity by completing the following instructions.

# Load dataset into dataframe
data = pd.read_csv("tiktok_dataset.csv")

PACE: Analyze and Construct

Consider the questions in your PACE Strategy Document and those below to craft your response:

1. Data professionals use descriptive statistics for Exploratory Data Analysis. How can computing descriptive statistics help you learn more about your data in this stage of your analysis?

Descriptive statistics are used to explore and summarize data.

Task 2. Data exploration

Use descriptive statistics to conduct Exploratory Data Analysis (EDA).

Hint:

Refer back to Self Review Descriptive Statistics for this step-by-step proccess.

Inspect the first five rows of the dataframe.

# Display first few rows
data.head()

# Generate a table of descriptive statistics about the data
data.describe(include="all")

Check for and handle missing values.

# Check for missing values
data.isna().sum()

#                             0
claim_status                298
video_id                      0
video_duration_sec            0
video_transcription_text    298
verified_status               0
author_ban_status             0
video_view_count            298
video_like_count            298
video_share_count           298
video_download_count        298
video_comment_count         298
dtype: int64

# Drop rows with missing values
data_m = data.dropna()
data_m.info()

<class 'pandas.core.frame.DataFrame'>
Int64Index: 19084 entries, 0 to 19083
Data columns (total 12 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   #                         19084 non-null  int64  
 1   claim_status              19084 non-null  object 
 2   video_id                  19084 non-null  int64  
 3   video_duration_sec        19084 non-null  int64  
 4   video_transcription_text  19084 non-null  object 
 5   verified_status           19084 non-null  object 
 6   author_ban_status         19084 non-null  object 
 7   video_view_count          19084 non-null  float64
 8   video_like_count          19084 non-null  float64
 9   video_share_count         19084 non-null  float64
 10  video_download_count      19084 non-null  float64
 11  video_comment_count       19084 non-null  float64
dtypes: float64(5), int64(3), object(4)
memory usage: 1.9+ MB

# Display first few rows after handling missing values
data_m.head()

You are interested in the relationship between verified_status and video_view_count. One approach is to examine the mean value of video_view_count for each group of verified_status in the sample data.

# Compute the mean `video_view_count` for each group in `verified_status`
data_m.groupby("verified_status")["video_view_count"].mean()

verified_status
not verified    265663.785339
verified         91439.164167
Name: video_view_count, dtype: float64

Task 3. Hypothesis testing

Before you conduct your hypothesis test, consider the following questions where applicable to complete your code response:

1. Recall the difference between the null hypothesis and the alternative hypotheses. What are your hypotheses for this data project?

Null hypothesis: There is no difference in number of views between TikTok videos posted by verified accounts and TikTok videos posted by unverified accounts (any observed difference in the sample data is due to chance or sampling variability).
Alternative hypothesis: There is a difference in number of views between TikTok videos posted by verified accounts and TikTok videos posted by unverified accounts (any observed difference in the sample data is due to an actual difference in the corresponding population means).

Your goal in this step is to conduct a two-sample t-test. Recall the steps for conducting a hypothesis test:

1. State the null hypothesis and the alternative hypothesis
2. Choose a signficance level
3. Find the p-value
4. Reject or fail to reject the null hypothesis

$H_0$: There is no difference in the average video view count for TikTok videos posted by verified accounts versus unverified accounts.

$H_A$: There is a difference in the average video view count for TikTok videos posted by verified accounts versus unverified accounts.

You choose 5% as the significance level and proceed with a two-sample t-test.

# Conduct a two-sample t-test to compare means
significance_level = 0.05
a = data_m[data_m["verified_status"] == "not verified"]["video_view_count"]
b = data_m[data_m["verified_status"] == "verified"]["video_view_count"]
stats.ttest_ind(a=a, b=b, equal_var=False)

Ttest_indResult(statistic=25.499441780633777, pvalue=2.6088823687177823e-120)

Question: Based on the p-value you got above, do you reject or fail to reject the null hypothesis?

With a p-value (2.6e-120) being less than 0.05 (as the significance level is 5%), we reject the null hypothesis in favor of the alternative hypothesis.

Therefore, we can conclude at the 5% significance level that there is a statistically significant difference in the average video view count for TikTok videos posted by verified accounts versus unverified accounts.

PACE: Execute

Consider the questions in your PACE Strategy Documentto reflect on the Execute stage.

Step 4: Communicate insights with stakeholders

Ask yourself the following questions:

1. What business insight(s) can you draw from the result of your hypothesis test?

We can conclude at the 5% significance level that non-verified users published more TikTok videos in which a claim/opinion has been made.
It would be interesting to investigate the main cause of this difference. Also, it seems that encouraging customers to verify their accounts can decrease the amount of content that needs to be reviewed by moderators.
The next step will be to build a regression model on verified_status. A regression model is the natural next step because the end goal is to make predictions on claim status. A regression model for verified_status can help analyze user behavior in this group of verified users.

Congratulations! You've completed this lab. However, you may not notice a green check mark next to this item on Coursera's platform. Please continue your progress regardless of the check mark. Just click on the "save" icon at the top of this notebook to ensure your work has been logged.