{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Model Citizen](https://www.model-citizen.org/brand/logo-color.png)\n",
    "\n",
    "# EDA Notebook Template\n",
    "\n",
    "A structured first pass at a new dataset: shape, types, missingness,\n",
    "distributions, duplicates/grain, and correlations \u2014 in that order, so you\n",
    "build up trust in the data before you build anything on top of it.\n",
    "\n",
    "Replace `DATA_PATH` below and run top to bottom. Delete sections you don't\n",
    "need; don't skip the grain check \u2014 it's the one most EDA templates forget,\n",
    "and the one that catches the most expensive bugs later.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "pd.set_option(\"display.max_columns\", 100)\n",
    "pd.set_option(\"display.width\", 120)\n",
    "plt.rcParams[\"figure.figsize\"] = (9, 4)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH = \"data/your_file.csv\"  # <- point this at your data\n",
    "\n",
    "df = pd.read_csv(DATA_PATH)\n",
    "df.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Shape & dtypes\n",
    "\n",
    "Confirm the file loaded the way you expect before profiling anything \u2014\n",
    "wrong dtypes here (a date read as a string, an ID read as a float) will\n",
    "quietly break every step after this one.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"{df.shape[0]:,} rows x {df.shape[1]} columns\")\n",
    "df.dtypes\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Missingness\n",
    "\n",
    "A column that's 40% null isn't necessarily broken \u2014 but you need to know\n",
    "*before* you build a metric on top of it, not after someone asks why the\n",
    "number looks off.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "missing = (df.isna().mean() * 100).sort_values(ascending=False)\n",
    "missing = missing[missing > 0]\n",
    "\n",
    "if len(missing):\n",
    "    missing.plot(kind=\"barh\", title=\"% missing by column\")\n",
    "    plt.xlabel(\"% missing\")\n",
    "    plt.tight_layout()\n",
    "else:\n",
    "    print(\"No missing values found.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Distributions\n",
    "\n",
    "Describe first, then look \u2014 summary statistics catch outliers that a chart alone can hide in a busy dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe(include='number').T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_cols = df.select_dtypes(include=\"number\").columns\n",
    "\n",
    "for col in numeric_cols:\n",
    "    df[col].hist(bins=40)\n",
    "    plt.title(col)\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Duplicates & grain check\n",
    "\n",
    "This is the step most EDA templates skip, and the one that matters most.\n",
    "Before you trust a single number derived from this table, answer: **what\n",
    "does one row mean?** If you can't state the grain in one sentence, don't\n",
    "build on this table yet \u2014 declare it first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace with the column(s) you believe should uniquely identify a row.\n",
    "candidate_key = [\"id\"]\n",
    "\n",
    "dupe_count = df.duplicated(subset=candidate_key).sum()\n",
    "print(f\"{dupe_count:,} duplicate rows on candidate key {candidate_key}\")\n",
    "\n",
    "if dupe_count:\n",
    "    df[df.duplicated(subset=candidate_key, keep=False)].sort_values(candidate_key).head(20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Correlations\n",
    "\n",
    "Useful for spotting redundant or leaking features \u2014 not a substitute for understanding *why* two columns move together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corr = df.select_dtypes(include=\"number\").corr()\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(8, 6))\n",
    "im = ax.imshow(corr, cmap=\"coolwarm\", vmin=-1, vmax=1)\n",
    "ax.set_xticks(range(len(corr.columns)))\n",
    "ax.set_yticks(range(len(corr.columns)))\n",
    "ax.set_xticklabels(corr.columns, rotation=90)\n",
    "ax.set_yticklabels(corr.columns)\n",
    "fig.colorbar(im)\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes / next steps\n",
    "\n",
    "- [ ] Grain declared and duplicates resolved (or explained)\n",
    "- [ ] Missingness understood \u2014 is it random, or does it mean something?\n",
    "- [ ] Outliers investigated, not just noted\n",
    "- [ ] Ready to model, or need another round with whoever owns this source?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Built by [Model Citizen](https://www.model-citizen.org) \u2014 no signup wall, no strings.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}