{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "85835aec", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/home/shezin/.local/lib/python3.8/site-packages/pandas/core/computation/expressions.py:20: UserWarning: Pandas requires version '2.7.3' or newer of 'numexpr' (version '2.7.1' currently installed).\n", " from pandas.core.computation.check import NUMEXPR_INSTALLED\n", "/usr/local/lib/python3.8/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.24.4\n", " warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n" ] } ], "source": [ "import numpy as np\n", "import pandas as pd\n", "import os\n", "\n", "from sklearn.model_selection import train_test_split, RandomizedSearchCV\n", "\n", "import lightgbm as lgb\n", "from scipy.stats import randint as sp_randint, uniform as sp_uniform\n", "\n", "from sklearn.model_selection import StratifiedKFold\n", "from sklearn.base import clone\n", "\n", "\n", "from sklearn.metrics import (\n", " roc_auc_score, recall_score, precision_score, accuracy_score, \n", " confusion_matrix, fbeta_score, precision_recall_curve, roc_curve\n", ")" ] }, { "cell_type": "code", "execution_count": 42, "id": "3772bff3", "metadata": {}, "outputs": [], "source": [ "# Load cleaned dataframe\n", "df = pd.read_pickle(\"../data/prepared/NSDUH_2023_prepared.pkl\") # <- change filename\n", "\n", "# Binary target and weights\n", "TARGET = \"IRAMDEYR\"\n", "WEIGHT = \"ANALWT2_C\"\n", "\n", "df[TARGET] = (df[TARGET] == 1).astype(int)\n", "y = df[TARGET]\n", "w = df[WEIGHT]\n", "\n", "# Load tiers\n", "from tiering_preparation import TIERS" ] }, { "cell_type": "code", "execution_count": 43, "id": "e0cb6876", "metadata": {}, "outputs": [], "source": [ "# ----- choose which tier to run -----\n", "tier_name = \"Tier_3_Personalized\" # \"Tier_1_Basic\" or \"Tier_2_Clinical\" or \"Tier_3_Personalized\"\n", "feature_list = TIERS[tier_name]\n", "\n", "X = df[feature_list]\n", "\n", "X_train, X_test, y_train, y_test, w_train, w_test = train_test_split(\n", " X, y, w,\n", " test_size=0.2,\n", " stratify=y,\n", " random_state=42,\n", ")" ] }, { "cell_type": "markdown", "id": "4fcd76e7", "metadata": {}, "source": [ "### 1. LightGBM base model and hyperparameter search space\n", "\n", "In this step, we define:\n", "\n", "1. A **base LightGBM classifier** that knows our problem is **binary** and **class-imbalanced**, and \n", "2. A set of **hyperparameter distributions** that `RandomizedSearchCV` will sample from when searching for good models." ] }, { "cell_type": "code", "execution_count": 44, "id": "56c6744f", "metadata": {}, "outputs": [], "source": [ "# Base LightGBM model\n", "base_lgb = lgb.LGBMClassifier(\n", " objective=\"binary\",\n", " boosting_type=\"gbdt\",\n", " class_weight=\"balanced\",\n", " random_state=42,\n", ")\n", "\n", "# Parameter distributions for random search\n", "param_dist = {\n", " \"num_leaves\": sp_randint(31, 128),\n", " \"max_depth\": sp_randint(3, 12),\n", " \"learning_rate\": sp_uniform(0.005, 0.05), # 0.01–0.10\n", " \"n_estimators\": sp_randint(600, 1500),\n", " \"min_child_samples\": sp_randint(5, 45),\n", " \"subsample\": sp_uniform(0.6, 0.4), # 0.6–1.0\n", " \"colsample_bytree\": sp_uniform(0.6, 0.4), # 0.6–1.0\n", "}\n" ] }, { "cell_type": "markdown", "id": "916fb947", "metadata": {}, "source": [ "### 2. Hyperparameter tuning using RandomizedSearchCV (Recall-optimized model)\n", "\n", "In this step, we run a **RandomizedSearchCV** to find the LightGBM hyperparameters that maximize **recall** during 5-fold cross-validation.\n", "\n", "The goal here is to create a model that **catches as many MDE cases as possible**, which matches our clinical objective of minimizing false negatives (important for screening and triage).\n", "\n", "#### Why Randomized Search?\n", "Instead of exhaustively trying every possible combination (GridSearch), RandomizedSearchCV:\n", "- Samples random combinations from the parameter distributions we defined earlier \n", "- Allows exploring a *broad* hyperparameter space \n", "- Is **much faster** than GridSearch \n", "- Often finds equally good (or better) results with fewer evaluations \n", "\n", "#### Why optimize for recall?\n", "- MDE cases are relatively rare in the population \n", "- Missing a positive case is clinically costly \n", "- High recall ensures that we correctly identify the majority of true MDE cases \n", "- Later, we will tune thresholds to improve NPV even further \n", "\n", "The code below runs the search and returns the **best model** based on mean recall across folds." ] }, { "cell_type": "code", "execution_count": 45, "id": "90793160", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 5 folds for each of 40 candidates, totalling 200 fits\n", "Best params (Recall optimized):\n", "{'colsample_bytree': 0.6028265220878869, 'learning_rate': 0.006153121252070788, 'max_depth': 5, 'min_child_samples': 7, 'n_estimators': 1084, 'num_leaves': 81, 'subsample': 0.8721230154351118}\n" ] } ], "source": [ "random_search_recall = RandomizedSearchCV(\n", " estimator=base_lgb,\n", " param_distributions=param_dist,\n", " n_iter=40,\n", " scoring=\"recall\",\n", " cv=5,\n", " n_jobs=-1,\n", " random_state=42,\n", " verbose=1,\n", ")\n", "\n", "random_search_recall.fit(X_train, y_train, sample_weight=w_train)\n", "best_lgb_recall = random_search_recall.best_estimator_\n", "\n", "from joblib import dump\n", "model_path = f\"../models/lightgbm_{tier_name.lower()}_recall.joblib\"\n", "dump(best_lgb_recall, model_path)\n", "\n", "print(\"Best params (Recall optimized):\")\n", "print(random_search_recall.best_params_)" ] }, { "cell_type": "markdown", "id": "512bff21", "metadata": {}, "source": [ "### 3. (NOT USING) Hyperparameter tuning using RandomizedSearchCV (AUC-optimized model) \n", "\n", "In this step, we run a second hyperparameter search - this time optimizing for **ROC AUC** instead of recall.\n", "\n", "#### Why optimize AUC separately?\n", "While the recall-optimized model is best for **catching as many MDE cases as possible**, the AUC-optimized model helps us:\n", "\n", "- Identify the model that **best separates positive vs negative cases overall** \n", "- Improve the **ranking quality** of predicted probabilities \n", "- Ensure that the model produces **meaningful risk scores**, which is important for threshold tuning later \n", "- Compare two complementary optimization strategies (Recall vs AUC), which strengthens our study methodology " ] }, { "cell_type": "code", "execution_count": 6, "id": "eca0341b", "metadata": {}, "outputs": [], "source": [ "# random_search_auc = RandomizedSearchCV(\n", "# estimator=base_lgb,\n", "# param_distributions=param_dist,\n", "# n_iter=40,\n", "# scoring=\"roc_auc\",\n", "# cv=5,\n", "# n_jobs=-1,\n", "# random_state=42,\n", "# verbose=1,\n", "# )\n", "\n", "# random_search_auc.fit(X_train, y_train, sample_weight=w_train)\n", "# best_lgb_auc = random_search_auc.best_estimator_\n", "\n", "# print(\"Best params (AUC optimized):\")\n", "# print(random_search_auc.best_params_)\n" ] }, { "cell_type": "markdown", "id": "9142873e", "metadata": {}, "source": [ "### 4. Evaluation helper function " ] }, { "cell_type": "code", "execution_count": 46, "id": "a5b6b098", "metadata": {}, "outputs": [], "source": [ "def evaluate_model(y_true, preds, probs):\n", " tn, fp, fn, tp = confusion_matrix(y_true, preds).ravel()\n", "\n", " specificity = tn / (tn + fp)\n", " sensitivity = recall_score(y_true, preds) # same as recall\n", " ppv = precision_score(y_true, preds)\n", " npv = tn / (tn + fn)\n", "\n", " return {\n", " \"Accuracy\": accuracy_score(y_true, preds),\n", " \"Sensitivity\": sensitivity,\n", " \"Specificity\": specificity,\n", " \"PPV\": ppv,\n", " \"NPV\": npv,\n", " \"AUC\": roc_auc_score(y_true, probs),\n", " \"F1\": fbeta_score(y_true, preds, beta=1),\n", " \"F2\": fbeta_score(y_true, preds, beta=2),\n", " }" ] }, { "cell_type": "markdown", "id": "ffe7108c", "metadata": {}, "source": [ "### 5. Saving CV fold wise Results, Mean and SD (only on Train data) - threshold=0.5" ] }, { "cell_type": "code", "execution_count": 47, "id": "00130057", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved fold table to: ../results/lightgbm_5fold_threshold0.50_by_tier.csv\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
FoldMetricTierFold 1Fold 2Fold 3Fold 4Fold 5MeanStd Dev
0AccuracyTier_3_Personalized0.8579400.8573680.8546490.8537910.8623550.8572210.003009
1SensitivityTier_3_Personalized0.8861080.8500000.8887500.8750000.8785980.8756910.013770
2SpecificityTier_3_Personalized0.8543050.8583200.8502420.8510500.8602580.8548350.003931
3PPVTier_3_Personalized0.4397520.4367370.4340660.4315660.4479900.4380220.005680
4NPVTier_3_Personalized0.9830860.9779130.9833710.9813710.9821100.9815700.001962
5AUCTier_3_Personalized0.9380990.9256060.9357700.9323090.9390410.9341650.004869
6F1Tier_3_Personalized0.5877960.5770050.5832650.5780350.5934070.5839010.006130
7F2Tier_3_Personalized0.7365790.7147360.7348080.7258400.7369310.7297790.008538
\n", "
" ], "text/plain": [ "Fold Metric Tier Fold 1 Fold 2 Fold 3 \\\n", "0 Accuracy Tier_3_Personalized 0.857940 0.857368 0.854649 \n", "1 Sensitivity Tier_3_Personalized 0.886108 0.850000 0.888750 \n", "2 Specificity Tier_3_Personalized 0.854305 0.858320 0.850242 \n", "3 PPV Tier_3_Personalized 0.439752 0.436737 0.434066 \n", "4 NPV Tier_3_Personalized 0.983086 0.977913 0.983371 \n", "5 AUC Tier_3_Personalized 0.938099 0.925606 0.935770 \n", "6 F1 Tier_3_Personalized 0.587796 0.577005 0.583265 \n", "7 F2 Tier_3_Personalized 0.736579 0.714736 0.734808 \n", "\n", "Fold Fold 4 Fold 5 Mean Std Dev \n", "0 0.853791 0.862355 0.857221 0.003009 \n", "1 0.875000 0.878598 0.875691 0.013770 \n", "2 0.851050 0.860258 0.854835 0.003931 \n", "3 0.431566 0.447990 0.438022 0.005680 \n", "4 0.981371 0.982110 0.981570 0.001962 \n", "5 0.932309 0.939041 0.934165 0.004869 \n", "6 0.578035 0.593407 0.583901 0.006130 \n", "7 0.725840 0.736931 0.729779 0.008538 " ] }, "execution_count": 47, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 5-fold CV table on TRAIN set (no test leakage)\n", "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)\n", "\n", "metrics_order = [\"Accuracy\", \"Sensitivity\", \"Specificity\", \"PPV\", \"NPV\", \"AUC\", \"F1\", \"F2\"]\n", "fold_rows = []\n", "\n", "for fold_i, (tr_idx, val_idx) in enumerate(skf.split(X_train, y_train), start=1):\n", " X_tr, X_val = X_train.iloc[tr_idx], X_train.iloc[val_idx]\n", " y_tr, y_val = y_train.iloc[tr_idx], y_train.iloc[val_idx]\n", " w_tr = w_train.iloc[tr_idx]\n", "\n", " model = clone(best_lgb_recall) \n", " model.fit(X_tr, y_tr, sample_weight=w_tr)\n", "\n", " proba = model.predict_proba(X_val)[:, 1]\n", " pred = (proba >= 0.5).astype(int)\n", "\n", " m = evaluate_model(y_val, pred, proba)\n", " m[\"Fold\"] = f\"Fold {fold_i}\"\n", " fold_rows.append(m)\n", "\n", "# Table: rows=Metric, cols=Fold1..Fold5\n", "fold_df = pd.DataFrame(fold_rows).set_index(\"Fold\")[metrics_order].T\n", "fold_df[\"Mean\"] = fold_df.mean(axis=1)\n", "fold_df[\"Std Dev\"] = fold_df.std(axis=1)\n", "\n", "# Adds tier column \n", "fold_df.insert(0, \"Tier\", tier_name)\n", "\n", "# Metric as a column\n", "fold_df = fold_df.reset_index().rename(columns={\"index\": \"Metric\"})\n", "\n", "# Save\n", "output_csv = \"../results/lightgbm_5fold_threshold0.50_by_tier.csv\"\n", "\n", "if os.path.exists(output_csv):\n", " fold_df.to_csv(output_csv, mode=\"a\", header=False, index=False)\n", "else:\n", " fold_df.to_csv(output_csv, index=False)\n", "\n", "print(f\"Saved fold table to: {output_csv}\")\n", "fold_df" ] }, { "cell_type": "markdown", "id": "70b835b0", "metadata": {}, "source": [ "### 6. Evaluate Recall-optimized and AUC-optimized models on the test set (threshold = 0.5)\n", "\n", "After performing two separate hyperparameter searches one maximizing **Recall** and one maximizing **AUC** we now evaluate both tuned models on the **held-out test set**.\n", "\n", "The goal is to compare how each model behaves using the **default decision threshold of 0.5**, before we perform threshold tuning." ] }, { "cell_type": "code", "execution_count": 48, "id": "b7cd0e11", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Recall-optimized (threshold 0.5):\n", "{'Accuracy': 0.8542000457770657, 'Sensitivity': 0.8648648648648649, 'Specificity': 0.8528233621914976, 'PPV': 0.4313529705441837, 'NPV': 0.9799554565701559, 'AUC': 0.9327786243408417, 'F1': 0.575616255829447, 'F2': 0.720120020003334}\n" ] } ], "source": [ "# Recall-optimized model\n", "proba_recall = best_lgb_recall.predict_proba(X_test)[:, 1]\n", "pred_recall = (proba_recall >= 0.5).astype(int)\n", "metrics_recall = evaluate_model(y_test, pred_recall, proba_recall)\n", "\n", "print(\"Recall-optimized (threshold 0.5):\")\n", "print(metrics_recall)\n", "\n", "# # AUC-optimized model\n", "# proba_auc = best_lgb_auc.predict_proba(X_test)[:, 1]\n", "# pred_auc = (proba_auc >= 0.5).astype(int)\n", "# metrics_auc = evaluate_model(y_test, pred_auc, proba_auc)\n", "\n", "# print(\"\\nAUC-optimized (threshold 0.5):\")\n", "# print(metrics_auc)\n" ] }, { "cell_type": "markdown", "id": "23660615", "metadata": {}, "source": [ "### 6.1. Saving metrics results of TEST DATA to /results/lightgbm_testset_metrics_by_tier.csv" ] }, { "cell_type": "code", "execution_count": 49, "id": "071f7f70", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved TEST metrics for Tier_3_Personalized to ../results/lightgbm_testset_metrics_by_tier.csv\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MetricTierValue
0AccuracyTier_3_Personalized0.854200
1SensitivityTier_3_Personalized0.864865
2SpecificityTier_3_Personalized0.852823
3PPVTier_3_Personalized0.431353
4NPVTier_3_Personalized0.979955
5AUCTier_3_Personalized0.932779
6F1Tier_3_Personalized0.575616
7F2Tier_3_Personalized0.720120
\n", "
" ], "text/plain": [ " Metric Tier Value\n", "0 Accuracy Tier_3_Personalized 0.854200\n", "1 Sensitivity Tier_3_Personalized 0.864865\n", "2 Specificity Tier_3_Personalized 0.852823\n", "3 PPV Tier_3_Personalized 0.431353\n", "4 NPV Tier_3_Personalized 0.979955\n", "5 AUC Tier_3_Personalized 0.932779\n", "6 F1 Tier_3_Personalized 0.575616\n", "7 F2 Tier_3_Personalized 0.720120" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "tier_name = \"Tier_3_Personalized\" # change for desired tier Tier_1_Basic / Tier_2_Clinical / Tier_3_Personalized\n", "\n", "test_df = pd.DataFrame({\n", " \"Metric\": list(metrics_recall.keys()),\n", " \"Tier\": [tier_name] * len(metrics_recall),\n", " \"Value\": list(metrics_recall.values())\n", "})\n", "\n", "\n", "test_df[\"Value\"] = test_df[\"Value\"].astype(float).round(6)\n", "\n", "output_csv = \"../results/lightgbm_testset_metrics_by_tier.csv\"\n", "\n", "if os.path.exists(output_csv):\n", " test_df.to_csv(output_csv, mode=\"a\", header=False, index=False)\n", "else:\n", " test_df.to_csv(output_csv, index=False)\n", "\n", "print(f\"Saved TEST metrics for {tier_name} to {output_csv}\")\n", "test_df" ] }, { "cell_type": "markdown", "id": "d5b1538f", "metadata": {}, "source": [ "### 7. Threshold tuning to optimize Sensitivity and NPV for screening - on selected model (Recall or AUC optimized)" ] }, { "cell_type": "code", "execution_count": 50, "id": "2462121d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AccuracySensitivitySpecificityPPVNPVAUCF1F2threshold
00.6761270.9769770.6372920.2579960.9953580.9327790.4081970.6273300.10
10.7197300.9689690.6875570.2858830.9942080.9327790.4415050.6556490.15
20.7492560.9519520.7230910.3073690.9914950.9327790.4646960.6706630.20
30.7752350.9399400.7539730.3302850.9898220.9327790.4888080.6865040.25
40.7965210.9279280.7795580.3520700.9882060.9327790.5104630.6992000.30
50.8148320.9129130.8021710.3733120.9861800.9327790.5299240.7081850.35
60.8300530.8948950.8216820.3931400.9837560.9327790.5462880.7129190.40
70.8434420.8798800.8387390.4132580.9818480.9327790.5623800.7177850.45
80.8542000.8648650.8528230.4313530.9799550.9327790.5756160.7201200.50
90.8625540.8478480.8644530.4467300.9777840.9327790.5851470.7187710.55
\n", "
" ], "text/plain": [ " Accuracy Sensitivity Specificity PPV NPV AUC F1 \\\n", "0 0.676127 0.976977 0.637292 0.257996 0.995358 0.932779 0.408197 \n", "1 0.719730 0.968969 0.687557 0.285883 0.994208 0.932779 0.441505 \n", "2 0.749256 0.951952 0.723091 0.307369 0.991495 0.932779 0.464696 \n", "3 0.775235 0.939940 0.753973 0.330285 0.989822 0.932779 0.488808 \n", "4 0.796521 0.927928 0.779558 0.352070 0.988206 0.932779 0.510463 \n", "5 0.814832 0.912913 0.802171 0.373312 0.986180 0.932779 0.529924 \n", "6 0.830053 0.894895 0.821682 0.393140 0.983756 0.932779 0.546288 \n", "7 0.843442 0.879880 0.838739 0.413258 0.981848 0.932779 0.562380 \n", "8 0.854200 0.864865 0.852823 0.431353 0.979955 0.932779 0.575616 \n", "9 0.862554 0.847848 0.864453 0.446730 0.977784 0.932779 0.585147 \n", "\n", " F2 threshold \n", "0 0.627330 0.10 \n", "1 0.655649 0.15 \n", "2 0.670663 0.20 \n", "3 0.686504 0.25 \n", "4 0.699200 0.30 \n", "5 0.708185 0.35 \n", "6 0.712919 0.40 \n", "7 0.717785 0.45 \n", "8 0.720120 0.50 \n", "9 0.718771 0.55 " ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Choose which tuned model to threshold-tune\n", "final_model = best_lgb_recall # best_lgb_recall or best_lgb_auc - Only using best_lgb_recall\n", "\n", "# Get probabilities on test set\n", "y_proba_test = final_model.predict_proba(X_test)[:, 1]\n", "\n", "thresholds = np.arange(0.1, 0.91, 0.05)\n", "\n", "rows = []\n", "for t in thresholds:\n", " preds_t = (y_proba_test >= t).astype(int)\n", " m = evaluate_model(y_test, preds_t, y_proba_test)\n", " m[\"threshold\"] = t\n", " rows.append(m)\n", "\n", "metrics_df = pd.DataFrame(rows)\n", "metrics_df_sorted = metrics_df.sort_values(\"Sensitivity\", ascending=False)\n", "\n", "metrics_df_sorted.head(10)\n" ] }, { "cell_type": "markdown", "id": "5356ceee", "metadata": {}, "source": [ "### 8. Picking Best Threshold, prioritizing Recall, NPV - if both higher then prioritizing Specificity, F2 " ] }, { "cell_type": "code", "execution_count": 51, "id": "6edcb1e4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best threshold selected: 0.3500000000000001\n", "Metrics at best threshold:\n", "{'Accuracy': 0.8148317692835889, 'Sensitivity': 0.9129129129129129, 'Specificity': 0.8021708231037602, 'PPV': 0.3733115022513303, 'NPV': 0.9861795075456712, 'AUC': 0.9327786243408417, 'F1': 0.5299244625217896, 'F2': 0.7081845006988662, 'threshold': 0.3500000000000001}\n" ] } ], "source": [ "# BEST threshold row: prioritize Recall, NPV\n", "screening_candidates = metrics_df[\n", " (metrics_df[\"Sensitivity\"] >= 0.90) &\n", " (metrics_df[\"NPV\"] >= 0.95)\n", "]\n", "\n", "if screening_candidates.empty:\n", " best_row = metrics_df.sort_values(\n", " [\"Sensitivity\", \"NPV\", \"Specificity\"],\n", " ascending=[False, False, False]\n", " ).iloc[0]\n", "else:\n", " best_row = screening_candidates.sort_values(\n", " by=[\"Specificity\", \"F2\"],\n", " ascending=[False, False]\n", " ).iloc[0]\n", "\n", "\n", "best_threshold = float(best_row[\"threshold\"])\n", "print(\"Best threshold selected:\", best_threshold)\n", "print(\"Metrics at best threshold:\")\n", "print(best_row.to_dict())\n" ] }, { "cell_type": "code", "execution_count": 52, "id": "2c57af91", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
AccuracySensitivitySpecificityPPVNPVAUCF1F2thresholdtiertuned_forn_features
00.8542000.8648650.8528230.4313530.9799550.9327790.5756160.7201200.50Tier_3_Personalizedrecall19
10.8148320.9129130.8021710.3733120.9861800.9327790.5299240.7081850.35Tier_3_Personalizedrecall19
\n", "
" ], "text/plain": [ " Accuracy Sensitivity Specificity PPV NPV AUC F1 \\\n", "0 0.854200 0.864865 0.852823 0.431353 0.979955 0.932779 0.575616 \n", "1 0.814832 0.912913 0.802171 0.373312 0.986180 0.932779 0.529924 \n", "\n", " F2 threshold tier tuned_for n_features \n", "0 0.720120 0.50 Tier_3_Personalized recall 19 \n", "1 0.708185 0.35 Tier_3_Personalized recall 19 " ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Row 1: threshold = 0.50 \n", "row_05 = metrics_recall.copy()\n", "row_05[\"threshold\"] = 0.50\n", "row_05[\"tier\"] = tier_name\n", "row_05[\"tuned_for\"] = \"recall\"\n", "row_05[\"n_features\"] = len(feature_list)\n", "\n", "# Row 2: best threshold from sweep\n", "row_best = best_row.to_dict()\n", "row_best[\"tier\"] = tier_name\n", "row_best[\"tuned_for\"] = \"recall\"\n", "row_best[\"n_features\"] = len(feature_list)\n", "\n", "two_rows_df = pd.DataFrame([row_05, row_best])\n", "\n", "two_rows_df" ] }, { "cell_type": "markdown", "id": "f9e79ea2", "metadata": {}, "source": [ "### 9. Saving Results csv to results folder - two rows for each tier" ] }, { "cell_type": "code", "execution_count": 53, "id": "94b77a8a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Appended 2 rows to: ../results/lightgbm_two_rows_per_tier.csv\n" ] } ], "source": [ "output_csv = \"../results/lightgbm_two_rows_per_tier.csv\"\n", "\n", "# Nice column order (optional)\n", "front_cols = [\"tier\", \"tuned_for\", \"n_features\", \"threshold\"]\n", "other_cols = [c for c in two_rows_df.columns if c not in front_cols]\n", "two_rows_df = two_rows_df[front_cols + other_cols]\n", "\n", "if os.path.exists(output_csv):\n", " two_rows_df.to_csv(output_csv, mode=\"a\", header=False, index=False)\n", "else:\n", " two_rows_df.to_csv(output_csv, index=False)\n", "\n", "print(f\"Appended 2 rows to: {output_csv}\")\n" ] }, { "cell_type": "markdown", "id": "47aeb8f0", "metadata": {}, "source": [ "### 10. (NOT USING) Saving results to lightgbm_all_tiers_results.csv\n", "\n", "Note: For all three tiers, before running please change the tier_name in 3rd code block to desired tier.\n", "Also change final_model in the above code block to desired tuned mode, whether recall or auc \n", "Saving for both tuned model for recall and AUC" ] }, { "cell_type": "code", "execution_count": 15, "id": "29450cde", "metadata": {}, "outputs": [], "source": [ "# # Explicitly record which model was used\n", "# metrics_df_sorted[\"tier\"] = tier_name\n", "# metrics_df_sorted[\"model\"] = \"LightGBM\"\n", "# metrics_df_sorted[\"tuned_for\"] = \"recall\" # <-- IMPORTANT to change recall or auc tuned model results\n", "# metrics_df_sorted[\"n_features\"] = len(feature_list)\n", "\n", "# cols = [\"tier\", \"model\", \"tuned_for\", \"n_features\", \"threshold\"] + [\n", "# c for c in metrics_df_sorted.columns\n", "# if c not in [\"tier\", \"model\", \"tuned_for\", \"n_features\", \"threshold\"]\n", "# ]\n", "# metrics_df_sorted = metrics_df_sorted[cols]\n", "\n", "# # Output file\n", "# output_csv = \"../results/lightgbm_all_tiers_results.csv\"\n", "\n", "# # Append if exists, otherwise create\n", "# if os.path.exists(output_csv):\n", "# metrics_df_sorted.to_csv(output_csv, mode=\"a\", header=False, index=False)\n", "# else:\n", "# metrics_df_sorted.to_csv(output_csv, index=False)\n", "\n", "# # blank row after each tuned model, readability\n", "# blank_row = pd.DataFrame(\n", "# [{col: \"\" for col in metrics_df_sorted.columns}]\n", "# )\n", "\n", "# blank_row.to_csv(output_csv, mode=\"a\", header=False, index=False)\n", "\n", "# print(f\"Results appended to {output_csv}\")\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }