Machine Learning 600 Study Hub

Chapter 1 Extended Code Example: Iris Classification Starter Pipeline

This commented example follows the guide's Python-library exercise: load the bundled Iris data into a Pandas table (no internet needed), split it with Scikit-learn, train Logistic Regression, and evaluate the result.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Step 1: Load the guide's sample Iris dataset.
# Each row is one flower observation; target is the known species code.
iris = load_iris(as_frame=True)
df = iris.frame
print(dict(enumerate(iris.target_names)))  # code-to-species key

# Step 2: Explore before modelling.
# describe() gives count, mean, standard deviation, and range for numeric features.
print(df.head())
print(df.describe())

# Step 3: Separate input features X from output label y.
# Supervised learning needs both known inputs and known outputs during training.
X = df.drop(columns=["target"])
y = df["target"]

# Step 4: Split the data so the model is tested on unseen observations.
# random_state makes the split repeatable when revising or debugging.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Step 5: Train a classification model.
# LogisticRegression is used here as a simple, interpretable baseline.
model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

# Step 6: Evaluate correctness.
# Accuracy is useful here because Iris classes are fairly balanced, but the confusion
# matrix still shows which classes were mixed up.
y_pred = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))
print("Confusion matrix:\n", confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))

2.4 PCA: What Actually Happens to the Features?

PCA is not simple column deletion. Scikit-learn PCA centres the input but does not automatically scale features to unit variance. Use a separately fitted StandardScaler when comparable feature scales are appropriate; PCA then finds directions of maximum variance and projects data onto the retained directions. High variance does not necessarily mean high predictive value.

• Why reduce dimensions: high-dimensional data increases computation, raises overfitting risk, and makes relationships harder to interpret.
• What components are: principal components are new axes and are linear combinations of the original features.
• Why scaling matters: PCA is variance-based, so unscaled features with large ranges can dominate the components.
• How to choose components: use the scree plot elbow, a 90-95% cumulative explained-variance threshold, or cross-validation.

Chapter 2 Extended Code Example: Cleaning, Scaling, and PCA

This example joins the guide's missing-value, scaling, and PCA sections into one practical preprocessing workflow.

import numpy as np
import pandas as pd
from sklearn.decomposition import PCA
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Step 1: Create a small structured dataset.
# Each row is an observation and each column is a feature.
data = {
    "Age": [25, 30, np.nan, 45, 50, 38],
    "Salary": [50000, np.nan, 60000, 80000, 90000, 72000],
    "SpendingScore": [62, 70, 65, 40, 35, 50],
    "OnlineVisits": [12, 15, 13, 4, 3, 8],
}
df = pd.DataFrame(data)

# Step 2: Split before fitting preprocessing objects.
# This avoids test-data leakage: the test set must not influence imputation or scaling.
X_train, X_test = train_test_split(df, test_size=0.33, random_state=42)

# Step 3: Build a preprocessing pipeline.
# SimpleImputer fills missing values with training-column means.
# StandardScaler gives each feature comparable mean and variance.
# PCA then rotates the scaled features into principal components.
preprocess = make_pipeline(
    SimpleImputer(strategy="mean"),
    StandardScaler(),
    PCA(n_components=2)
)

# Step 4: Fit only on training data, then transform both sets in the same way.
X_train_pca = preprocess.fit_transform(X_train)
X_test_pca = preprocess.transform(X_test)

# Step 5: Inspect explained variance to decide whether enough information remains.
pca = preprocess.named_steps["pca"]
print("Training data after PCA:\n", X_train_pca)
print("Test data after PCA:\n", X_test_pca)
print("Explained variance ratio:", pca.explained_variance_ratio_)
print("Total retained variance:", pca.explained_variance_ratio_.sum())

Chapter 3 Extended Code Example: Compare Regression Models

This small example demonstrates the mechanics of preprocessing and comparing R2 and MAE. It is not a reliable benchmark: eight houses leave only two test observations, and polynomial expansion is especially prone to overfitting. Use a larger dataset and validation for meaningful comparison.

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.tree import DecisionTreeRegressor

# Step 1: Small house-price dataset based on the guide's square-footage example.
# Price is measured in thousands, so 320 means $320 000.
df = pd.DataFrame({
    "SquareFootage": [1200, 1500, 1800, 2100, 2500, 2800, 3200, np.nan],
    "Bedrooms": [2, 3, 3, 4, 4, 4, 5, 3],
    "Age": [18, 12, 10, 8, 5, 4, 2, 15],
    "Price": [220, 270, 320, 370, 420, 455, 510, 300],
})

# Step 2: Separate features and continuous target.
X = df.drop(columns=["Price"])
y = df["Price"]

# Step 3: Split before fitting imputers, scalers, or models.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# Step 4: Define three models from the chapter.
# Linear regression: interpretable straight-line baseline.
linear_model = make_pipeline(
    SimpleImputer(strategy="mean"),
    StandardScaler(),
    LinearRegression()
)

# Polynomial regression: adds curved feature terms before fitting LinearRegression.
poly_model = make_pipeline(
    SimpleImputer(strategy="mean"),
    StandardScaler(),
    PolynomialFeatures(degree=2, include_bias=False),
    LinearRegression()
)

# Decision tree regression: rule-based non-linear model.
# max_depth is pre-pruning; it limits complexity to reduce overfitting.
tree_model = make_pipeline(
    SimpleImputer(strategy="mean"),
    DecisionTreeRegressor(max_depth=3, random_state=42)
)

models = {
    "Linear": linear_model,
    "Polynomial degree 2": poly_model,
    "Decision tree": tree_model,
}

# Step 5: Train, predict, and compare the same metrics for each model.
for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print(name)
    print("  R2:", round(r2_score(y_test, y_pred), 3))
    print("  MAE:", round(mean_absolute_error(y_test, y_pred), 3))

# Step 6: Predict a new house.
# The same fitted preprocessing steps are applied automatically by each pipeline.
new_house = pd.DataFrame({
    "SquareFootage": [2000],
    "Bedrooms": [3],
    "Age": [9],
})
print("Linear prediction:", linear_model.predict(new_house)[0])

Classification Algorithm Selector

The chapter groups several common classifiers. For exam answers, name what the algorithm uses to classify and what problem can weaken it.

Chapter 4 Extended Code Example: Logistic Regression, k-NN, and SVM

This example uses a labelled dataset and compares the chapter's main classifiers with shared metrics.

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

# Step 1: Load the Iris dataset.
# It has four input measurements and three known species labels.
iris = load_iris()
X = iris.data
y = iris.target

# Step 2: Split into training and testing sets.
# The test set estimates how well each model generalizes.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Step 3: Build classifiers from the chapter.
# Logistic Regression: interpretable linear baseline.
# k-NN: distance-based majority vote, so scaling is important.
# SVM: margin-based classifier; RBF kernel can handle non-linear boundaries.
models = {
    "Logistic Regression": make_pipeline(
        StandardScaler(),
        LogisticRegression(max_iter=300, C=1.0)
    ),
    "k-NN": make_pipeline(
        StandardScaler(),
        KNeighborsClassifier(n_neighbors=5)
    ),
    "SVM RBF": make_pipeline(
        StandardScaler(),
        SVC(kernel="rbf", C=1.0, gamma="scale")
    ),
}

# Step 4: Train and evaluate using the same test set.
for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    print("\n", name)
    print("Accuracy:", round(accuracy_score(y_test, y_pred), 3))
    print("Confusion matrix:\n", confusion_matrix(y_test, y_pred))
    print(classification_report(y_test, y_pred, target_names=iris.target_names))

# Step 5: Predict one new flower.
# The nested list is one row with four feature values.
new_iris = [[5.0, 2.9, 1.0, 0.2]]
predicted_class = models["k-NN"].predict(new_iris)[0]
print("Predicted iris species:", iris.target_names[predicted_class])

Hierarchical Clustering and DBSCAN: Two Different Ways to Control Granularity

Agglomerative clustering starts with every point as its own cluster and repeatedly merges the two most similar clusters. DBSCAN starts from density: enough nearby points create a core region, sparse points become boundaries or noise.

• Ward linkage: minimizes the increase in within-cluster variance and often gives similar-sized clusters.
• Average linkage: merges clusters with the smallest average distance between their points.
• Complete linkage: merges based on the smallest maximum distance between any two points in the clusters.
• Dendrogram: shows the merge history and helps inspect possible cut points.
• DBSCAN eps: too small can mark everything as noise; too large can merge everything into one cluster.

Chapter 5 Extended Code Example: Customer Segmentation With Three Clustering Methods

This example follows the guide's customer-segmentation scenario: scale customer features, apply k-means, agglomerative clustering, and DBSCAN, then compare internal metrics.

import numpy as np
import pandas as pd
from sklearn.cluster import AgglomerativeClustering, DBSCAN, KMeans
from sklearn.metrics import davies_bouldin_score, silhouette_score
from sklearn.preprocessing import StandardScaler

# Step 1: Generate guide-style customer data.
# Features: age, annual income, and spending score.
np.random.seed(42)
data = np.random.rand(200, 3) * [50, 100000, 100]
df = pd.DataFrame(data, columns=["Age", "Annual_Income", "Spending_Score"])

# Step 2: Scale features before distance-based clustering.
# Without scaling, Annual_Income would dominate Age and Spending_Score.
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df)

# Step 3: Fit the three clustering algorithms from the chapter.
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
hierarchical = AgglomerativeClustering(n_clusters=3, linkage="ward")
dbscan = DBSCAN(eps=0.5, min_samples=5)

df["KMeans_Cluster"] = kmeans.fit_predict(X_scaled)
df["Hierarchical_Cluster"] = hierarchical.fit_predict(X_scaled)
df["DBSCAN_Cluster"] = dbscan.fit_predict(X_scaled)

# Step 4: Evaluate when possible.
# Silhouette and Davies-Bouldin require at least two non-noise clusters.
def evaluate_clusters(name, labels):
    unique_labels = set(labels)
    non_noise_labels = unique_labels - {-1}

    print("\n", name)
    print("Cluster labels:", sorted(unique_labels))
    print("Cluster sizes:", pd.Series(labels).value_counts().sort_index().to_dict())

    keep = np.asarray(labels) != -1
    clean_labels = np.asarray(labels)[keep]
    if 2 <= len(non_noise_labels) < keep.sum():
        print("Silhouette (non-noise):", round(silhouette_score(X_scaled[keep], clean_labels), 3))
        print("Davies-Bouldin (non-noise):", round(davies_bouldin_score(X_scaled[keep], clean_labels), 3))
    else:
        print("Need 2 clusters and at least 1 extra non-noise sample for internal scores.")

evaluate_clusters("k-Means", df["KMeans_Cluster"])
evaluate_clusters("Hierarchical", df["Hierarchical_Cluster"])
evaluate_clusters("DBSCAN", df["DBSCAN_Cluster"])

# Step 5: Inspect cluster profiles instead of trusting numeric labels.
# This is where the arbitrary labels become meaningful business descriptions.
profile = df.groupby("KMeans_Cluster")[["Age", "Annual_Income", "Spending_Score"]].mean()
print("\nk-Means customer profiles:\n", profile)

# Step 6: Remember that DBSCAN label -1 means noise/outlier.
noise_customers = df[df["DBSCAN_Cluster"] == -1]
print("DBSCAN noise count:", len(noise_customers))

Chapter 5 guide practice: all ten tasks

The guide's closing exercises ask you to compare clustering on Iris or Digits, inspect a dendrogram, test DBSCAN on noisy shapes, compare PCA and scaling choices, visualise with t-SNE, engineer features, detect anomalies, choose k with an elbow plot, and compare a Gaussian mixture with k-means. These are wider-module practice tasks, not additions to the earlier CA scope. Source: Machine Learning 600 Study Guide, §5.6, PDF pp. 161–166 (printed pp. 159–164).

1. k-means: vary n_clusters; compare silhouette and, when true labels exist, ARI. Cluster numbers need not match class numbers.
2. Hierarchical: inspect merge distances in a dendrogram; choose a cut and compare assignments with k-means.
3. DBSCAN: vary eps and min_samples; count label -1 as noise and inspect irregular shapes.
4. PCA: plot two components, report explained_variance_ratio_, then compare clustering on original and reduced features.
5. Preprocessing: compare original, scaled and row-normalised features. Normalizer scales each sample; RobustScaler scales columns robustly, so they answer different questions.
6. t-SNE: use a 2D embedding for visual exploration; local neighbours matter more than global axis distances, and the embedding is not a fitted predictor.
7. Feature engineering: add justified interactions, then compare clusters using the same evaluation method; extra features can also make distances less useful.
8. Isolation Forest: inspect predictions (-1 = anomaly) and compare the flagged points with DBSCAN noise or known synthetic outliers.
9. Elbow: plot k-means inertia_ over a range of k; an elbow is a judgement aid, not a guaranteed optimum.
10. Gaussian mixture: compare hard labels and predict_proba membership probabilities with k-means; a mixture assumes Gaussian components.

from sklearn.datasets import load_iris, make_moons
from sklearn.cluster import KMeans, DBSCAN, AgglomerativeClustering
from sklearn.decomposition import PCA
from sklearn.ensemble import IsolationForest
from sklearn.manifold import TSNE
from sklearn.metrics import adjusted_rand_score, silhouette_score
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import Normalizer, PolynomialFeatures, StandardScaler
from scipy.cluster.hierarchy import dendrogram, linkage
import matplotlib.pyplot as plt
import numpy as np

X, true_species = load_iris(return_X_y=True)
X = StandardScaler().fit_transform(X)
km = KMeans(n_clusters=3, random_state=42, n_init=10).fit(X)
print("ARI:", adjusted_rand_score(true_species, km.labels_))
print("Silhouette:", silhouette_score(X, km.labels_))
print("K-means centers:", km.cluster_centers_.shape)

# A dendrogram is drawn from a linkage matrix, not from model labels.
tree = linkage(X, method="ward")
dendrogram(tree, truncate_mode="lastp", p=12)
plt.title("Last 12 merges")
plt.show()
hierarchical = AgglomerativeClustering(n_clusters=3).fit_predict(X)

pca = PCA(n_components=2).fit(X)
Z = pca.transform(X)
print("PCA retained variance:", pca.explained_variance_ratio_.sum())
print("PCA-space silhouette:", silhouette_score(Z, KMeans(n_clusters=3, random_state=42, n_init=10).fit_predict(Z)))
unit_rows = Normalizer().fit_transform(X)
print("Row-normalised silhouette:", silhouette_score(unit_rows, KMeans(n_clusters=3, random_state=42, n_init=10).fit_predict(unit_rows)))
X_interactions = PolynomialFeatures(degree=2, include_bias=False).fit_transform(X)
print("Engineered-feature silhouette:", silhouette_score(X_interactions, KMeans(n_clusters=3, random_state=42, n_init=10).fit_predict(X_interactions)))
embedding = TSNE(n_components=2, random_state=42, perplexity=30).fit_transform(X)
plt.scatter(embedding[:, 0], embedding[:, 1], c=true_species)
plt.title("t-SNE view (colours shown only for interpretation)")
plt.show()

moons, _ = make_moons(n_samples=200, noise=0.07, random_state=42)
db_labels = DBSCAN(eps=0.2, min_samples=5).fit_predict(moons)
print("DBSCAN noise:", np.sum(db_labels == -1))
keep = db_labels != -1
if 2 <= np.unique(db_labels[keep]).size < keep.sum():
    print("DBSCAN silhouette without noise:", silhouette_score(moons[keep], db_labels[keep]))
anomaly_labels = IsolationForest(random_state=42, contamination=0.05).fit_predict(moons)
print("Isolation Forest flags:", np.sum(anomaly_labels == -1))

ks = range(1, 9)
inertias = [KMeans(n_clusters=k, random_state=42, n_init=10).fit(X).inertia_ for k in ks]
plt.plot(list(ks), inertias, marker="o")
plt.xlabel("k")
plt.ylabel("Within-cluster sum of squares")
plt.show()
gmm = GaussianMixture(n_components=3, random_state=42).fit(X)
print("GMM agreement with species:", adjusted_rand_score(true_species, gmm.predict(X)))
print("GMM memberships for first row:", gmm.predict_proba(X[:1]))

The snippet demonstrates the APIs on bundled or generated data. For the guide's full comparison task, also inspect cluster profiles and repeat evaluation after PCA or justified feature engineering. Compare silhouette only when the evaluated labels contain at least two clusters and fewer clusters than samples.