Machine Learning 600 Study Hub

Chapter 1 Extended Code Example: Iris Classification Starter Pipeline

This commented example follows the guide's Python-library exercise: load the Iris data with Pandas, split it with Scikit-learn, train Logistic Regression, and evaluate the result.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Step 1: Load the guide's sample Iris dataset.
# Each row is one flower observation; the species column is the known label.
df = pd.read_csv(
    "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv"
)

# Step 2: Explore before modelling.
# describe() gives count, mean, standard deviation, and range for numeric features.
print(df.head())
print(df.describe())

# Step 3: Separate input features X from output label y.
# Supervised learning needs both known inputs and known outputs during training.
X = df.drop(columns=["species"])
y = df["species"]

# Step 4: Split the data so the model is tested on unseen observations.
# random_state makes the split repeatable when revising or debugging.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Step 5: Train a classification model.
# LogisticRegression is used here as a simple, interpretable baseline.
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

# Step 6: Evaluate correctness.
# Accuracy is useful here because Iris classes are fairly balanced, but the confusion
# matrix still shows which classes were mixed up.
y_pred = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Confusion matrix:\n", confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

2.4 PCA: What Actually Happens to the Features?

PCA is dimensionality reduction, but the guide is clear that it is not simple column deletion. PCA standardizes the feature space, finds directions of maximum variance, rotates the data onto those directions, and then keeps the strongest components.

• Why reduce dimensions: high-dimensional data increases computation, raises overfitting risk, and makes relationships harder to interpret.
• What components are: principal components are new axes and are linear combinations of the original features.
• Why scaling matters: PCA is variance-based, so unscaled features with large ranges can dominate the components.
• How to choose components: use the scree plot elbow, a 90-95% cumulative explained-variance threshold, or cross-validation.

Chapter 2 Extended Code Example: Cleaning, Scaling, and PCA

This example joins the guide's missing-value, scaling, and PCA sections into one practical preprocessing workflow.

import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Step 1: Create a small structured dataset.
# Each row is an observation and each column is a feature.
data = {
    "Age": [25, 30, np.nan, 45, 50, 38],
    "Salary": [50000, np.nan, 60000, 80000, 90000, 72000],
    "SpendingScore": [62, 70, 65, 40, 35, 50],
    "OnlineVisits": [12, 15, 13, 4, 3, 8],
}
df = pd.DataFrame(data)

# Step 2: Split before fitting preprocessing objects.
# This avoids test-data leakage: the test set must not influence imputation or scaling.
X_train, X_test = train_test_split(df, test_size=0.33, random_state=42)

# Step 3: Build a preprocessing pipeline.
# SimpleImputer fills missing values with training-column means.
# StandardScaler gives each feature comparable mean and variance.
# PCA then rotates the scaled features into principal components.
preprocess = make_pipeline(
    SimpleImputer(strategy="mean"),
    StandardScaler(),
    PCA(n_components=2)
)

# Step 4: Fit only on training data, then transform both sets in the same way.
X_train_pca = preprocess.fit_transform(X_train)
X_test_pca = preprocess.transform(X_test)

# Step 5: Inspect explained variance to decide whether enough information remains.
pca = preprocess.named_steps["pca"]
print("Training data after PCA:\n", X_train_pca)
print("Test data after PCA:\n", X_test_pca)
print("Explained variance ratio:", pca.explained_variance_ratio_)
print("Total retained variance:", pca.explained_variance_ratio_.sum())

Chapter 3 Extended Code Example: Compare Regression Models

This example follows the guide's house-price exercise: preprocess a dataset, train linear, polynomial, and decision tree regressors, then compare R2 and MAE.

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.tree import DecisionTreeRegressor

# Step 1: Small house-price dataset based on the guide's square-footage example.
# Price is measured in thousands, so 320 means $320 000.
df = pd.DataFrame({
    "SquareFootage": [1200, 1500, 1800, 2100, 2500, 2800, 3200, np.nan],
    "Bedrooms": [2, 3, 3, 4, 4, 4, 5, 3],
    "Age": [18, 12, 10, 8, 5, 4, 2, 15],
    "Price": [220, 270, 320, 370, 420, 455, 510, 300],
})

# Step 2: Separate features and continuous target.
X = df.drop(columns=["Price"])
y = df["Price"]

# Step 3: Split before fitting imputers, scalers, or models.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# Step 4: Define three models from the chapter.
# Linear regression: interpretable straight-line baseline.
linear_model = make_pipeline(
    SimpleImputer(strategy="mean"),
    StandardScaler(),
    LinearRegression()
)

# Polynomial regression: adds curved feature terms before fitting LinearRegression.
poly_model = make_pipeline(
    SimpleImputer(strategy="mean"),
    StandardScaler(),
    PolynomialFeatures(degree=2, include_bias=False),
    LinearRegression()
)

# Decision tree regression: rule-based non-linear model.
# max_depth is pre-pruning; it limits complexity to reduce overfitting.
tree_model = make_pipeline(
    SimpleImputer(strategy="mean"),
    DecisionTreeRegressor(max_depth=3, random_state=42)
)

models = {
    "Linear": linear_model,
    "Polynomial degree 2": poly_model,
    "Decision tree": tree_model,
}

# Step 5: Train, predict, and compare the same metrics for each model.
for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print(name)
    print("  R2:", round(r2_score(y_test, y_pred), 3))
    print("  MAE:", round(mean_absolute_error(y_test, y_pred), 3))

# Step 6: Predict a new house.
# The same fitted preprocessing steps are applied automatically by each pipeline.
new_house = pd.DataFrame({
    "SquareFootage": [2000],
    "Bedrooms": [3],
    "Age": [9],
})
print("Linear prediction:", linear_model.predict(new_house)[0])

Classification Algorithm Selector

The chapter groups several common classifiers. For exam answers, name what the algorithm uses to classify and what problem can weaken it.

Chapter 4 Extended Code Example: Logistic Regression, k-NN, and SVM

This example uses a labelled dataset and compares the chapter's main classifiers with shared metrics.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

# Step 1: Load the Iris dataset.
# It has four input measurements and three known species labels.
iris = load_iris()
X = iris.data
y = iris.target

# Step 2: Split into training and testing sets.
# The test set estimates how well each model generalizes.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Step 3: Build classifiers from the chapter.
# Logistic Regression: interpretable linear baseline.
# k-NN: distance-based majority vote, so scaling is important.
# SVM: margin-based classifier; RBF kernel can handle non-linear boundaries.
models = {
    "Logistic Regression": make_pipeline(
        StandardScaler(),
        LogisticRegression(max_iter=300, C=1.0)
    ),
    "k-NN": make_pipeline(
        StandardScaler(),
        KNeighborsClassifier(n_neighbors=5)
    ),
    "SVM RBF": make_pipeline(
        StandardScaler(),
        SVC(kernel="rbf", C=1.0, gamma="scale")
    ),
}

# Step 4: Train and evaluate using the same test set.
for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print("\n", name)
    print("Accuracy:", round(accuracy_score(y_test, y_pred), 3))
    print("Confusion matrix:\n", confusion_matrix(y_test, y_pred))
    print(classification_report(y_test, y_pred, target_names=iris.target_names))

# Step 5: Predict one new flower.
# The nested list is one row with four feature values.
new_iris = [[5.0, 2.9, 1.0, 0.2]]
predicted_class = models["k-NN"].predict(new_iris)[0]
print("Predicted iris species:", iris.target_names[predicted_class])

Hierarchical Clustering and DBSCAN: Two Different Ways to Control Granularity

Agglomerative clustering starts with every point as its own cluster and repeatedly merges the two most similar clusters. DBSCAN starts from density: enough nearby points create a core region, sparse points become boundaries or noise.

• Ward linkage: minimizes the increase in within-cluster variance and often gives similar-sized clusters.
• Average linkage: merges clusters with the smallest average distance between their points.
• Complete linkage: merges based on the smallest maximum distance between any two points in the clusters.
• Dendrogram: shows the merge history and helps inspect possible cut points.
• DBSCAN eps: too small can mark everything as noise; too large can merge everything into one cluster.

Chapter 5 Extended Code Example: Customer Segmentation With Three Clustering Methods

This example follows the guide's customer-segmentation scenario: scale customer features, apply k-means, agglomerative clustering, and DBSCAN, then compare internal metrics.

import numpy as np
import pandas as pd
from sklearn.cluster import AgglomerativeClustering, DBSCAN, KMeans
from sklearn.metrics import davies_bouldin_score, silhouette_score
from sklearn.preprocessing import StandardScaler

# Step 1: Generate guide-style customer data.
# Features: age, annual income, and spending score.
np.random.seed(42)
data = np.random.rand(200, 3) * [50, 100000, 100]
df = pd.DataFrame(data, columns=["Age", "Annual_Income", "Spending_Score"])

# Step 2: Scale features before distance-based clustering.
# Without scaling, Annual_Income would dominate Age and Spending_Score.
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df)

# Step 3: Fit the three clustering algorithms from the chapter.
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
hierarchical = AgglomerativeClustering(n_clusters=3, linkage="ward")
dbscan = DBSCAN(eps=0.5, min_samples=5)

df["KMeans_Cluster"] = kmeans.fit_predict(X_scaled)
df["Hierarchical_Cluster"] = hierarchical.fit_predict(X_scaled)
df["DBSCAN_Cluster"] = dbscan.fit_predict(X_scaled)

# Step 4: Evaluate when possible.
# Silhouette and Davies-Bouldin require at least two non-noise clusters.
def evaluate_clusters(name, labels):
    unique_labels = set(labels)
    non_noise_labels = unique_labels - {-1}

    print("\n", name)
    print("Cluster labels:", sorted(unique_labels))
    print("Cluster sizes:", pd.Series(labels).value_counts().sort_index().to_dict())

    if len(non_noise_labels) >= 2:
        print("Silhouette:", round(silhouette_score(X_scaled, labels), 3))
        print("Davies-Bouldin:", round(davies_bouldin_score(X_scaled, labels), 3))
    else:
        print("Not enough non-noise clusters for internal scores.")

evaluate_clusters("k-Means", df["KMeans_Cluster"])
evaluate_clusters("Hierarchical", df["Hierarchical_Cluster"])
evaluate_clusters("DBSCAN", df["DBSCAN_Cluster"])

# Step 5: Inspect cluster profiles instead of trusting numeric labels.
# This is where the arbitrary labels become meaningful business descriptions.
profile = df.groupby("KMeans_Cluster")[["Age", "Annual_Income", "Spending_Score"]].mean()
print("\nk-Means customer profiles:\n", profile)

# Step 6: Remember that DBSCAN label -1 means noise/outlier.
noise_customers = df[df["DBSCAN_Cluster"] == -1]
print("DBSCAN noise count:", len(noise_customers))