Machine Learning 600 Study Hub

1.1 Overview of Machine Learning

Machine learning is a subfield of artificial intelligence focused on models that learn from data to make predictions without being explicitly programmed for every case. It is used in healthcare, finance, marketing, autonomous systems, cybersecurity, education, agriculture, and entertainment.

1.2 Modeling

A model represents relationships between variables. In ML, the model learns those relationships from data and can improve as more data becomes available.

• Business model: profit from revenue and expenses.
• Recipe model: servings linked to ingredient quantities.
• Poker model: probability of winning based on revealed cards.

1.3 Types of Machine Learning

The guide separates machine learning by the nature of the available data and output.

1.4 Overfitting and Underfitting

Underfitting happens when a model is too simple to capture the real pattern. Overfitting happens when a model is so complex that it learns noise instead of the underlying pattern.

1.5 Correctness

Correctness is not just accuracy. The guide uses the confusion matrix to separate prediction outcomes into true positives, false positives, false negatives, and true negatives.

• Precision: of the predicted positives, how many were correct?
• Recall: of the actual positives, how many were found?
• F1 score: harmonic mean that balances precision and recall.

1.6 The Bias-Variance Tradeoff

Bias is error from a model being too simple. Variance is error from a model being too sensitive to training-data fluctuations. A good model balances both so it generalizes to new data.

1.7 Feature Engineering

Feature extraction transforms raw data into usable numerical representations. Feature selection identifies the most relevant features so the model avoids unnecessary complexity.

• Extraction: text to vectors, image pixels or edges to numeric features.
• Selection: statistical filter methods, wrapper methods, recursive feature elimination, forward selection.

1.8 Real-World Applications

The guide lists ML uses in healthcare, finance, retail, transportation, manufacturing, cybersecurity, NLP, education, agriculture, and entertainment.

1.9 Introduction to Python ML Libraries

The guide introduces NumPy for arrays and numerical computing, Pandas for DataFrame-based dataset work, Scikit-learn for ML models, and Matplotlib for visualizations.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

df = pd.read_csv("iris.csv")
X = df.drop(columns=["species"])
y = df["species"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = LogisticRegression()
model.fit(X_train, y_train)
print(model.score(X_test, y_test))

2.1 Understanding Datasets

Before applying ML algorithms, understand the data type, structure, and quality. Structured data is organised in rows and columns. Unstructured data includes text, images, audio, and video.

2.2 Data Cleaning

Data cleaning handles missing values, inconsistencies, errors, outliers, and duplicates so the dataset becomes reliable enough for model training.

• Missing Completely at Random: missingness is independent of observed and missing data.
• Missing at Random: missingness depends on observed data.
• Missing Not at Random: missingness depends on the missing value itself.

from sklearn.impute import SimpleImputer

imputer = SimpleImputer(strategy="mean")
df.iloc[:, :] = imputer.fit_transform(df)

2.2.2-2.2.3 Outliers and Duplicates

Outliers are unusually distant values. They can be errors or genuine extreme cases. Duplicates repeat observations and can skew the model.

import numpy as np

data = np.array([10, 12, 13, 15, 1000])
filtered_data = data[data < np.percentile(data, 95)]

df.drop_duplicates(inplace=True)

2.3 Feature Scaling and Normalization

Scaling puts features on comparable ranges. It is important when a model uses distances, gradients, or margins, as with k-NN and SVM.

from sklearn.preprocessing import MinMaxScaler, StandardScaler

data = [[100, 200], [300, 400], [500, 600]]

scaled = MinMaxScaler().fit_transform(data)
standardized = StandardScaler().fit_transform(data)

2.4 Principal Component Analysis

PCA is a dimensionality reduction technique that creates principal components, which are new axes capturing maximum variance. It reduces high-dimensional data while preserving as much useful information as possible.

• Reduces computational cost.
• Can reduce overfitting risk by removing noisy dimensions.
• Improves interpretability and visualization for high-dimensional data.

2.4.3 Choosing Principal Components

The explained variance ratio tells how much variance each principal component captures. The cumulative explained variance helps decide how many components to keep.

2.5 Applications of PCA

The guide applies PCA to visualizing high-dimensional data such as the breast cancer dataset and to image feature extraction through eigenfaces. PCA can produce a better representation than raw pixel distances for some analysis tasks.

3.1 What Is Supervised Learning?

Supervised learning uses labeled input-output pairs. The model learns a function from features to a target and adjusts its parameters to minimize prediction error.

• Labeled data: each example has known features and a known target.
• Learning function: the model approximates the relationship.
• Error minimization: training reduces the gap between predictions and actual values.

3.2 Regression

Regression is supervised learning for continuous numerical outcomes. The guide uses examples such as house prices, stock prices, and sales revenue forecasting.

3.3 Linear Regression

Linear regression assumes a straight-line relationship between independent variables and the target. The slope shows how much the output changes for a unit change in the input; the intercept is the predicted output when the input is zero.

import numpy as np
from sklearn.linear_model import LinearRegression

X = np.array([1200, 1500, 1800, 2100, 2500]).reshape(-1, 1)
y = np.array([220, 270, 320, 370, 420])

model = LinearRegression()
model.fit(X, y)
print(model.predict([[2000]]))

3.4 Polynomial Regression

Polynomial regression handles non-linear relationships by adding higher-degree terms. It is more flexible than linear regression but can overfit when the degree is too high.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

poly_model = make_pipeline(
    PolynomialFeatures(degree=2),
    LinearRegression()
)
poly_model.fit(X, y)

3.5 Decision Tree Regression

Decision tree regression splits the data into branches based on feature values. Each leaf predicts a value, usually the average outcome in that region.

• Easy to visualize for small trees.
• Does not require feature scaling.
• Can overfit if the tree grows too deep.
• Cannot extrapolate beyond the range of the training data.

3.6 Hands-On Exercise: House Price Prediction

The practical task is to load a housing dataset, preprocess missing values and scales, train linear, polynomial, and decision tree models, evaluate with R2 score and MAE, and visualize predictions against actual values.

4.1 Classification

Classification predicts categorical outcomes. A decision boundary separates categories in feature space. Common applications include fraud detection, automated decision-making, and medical diagnostics.

4.2 Logistic Regression

Logistic regression is a linear classification algorithm that predicts the probability of class membership. It is commonly used as an interpretable baseline for binary classification.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(accuracy_score(y_test, y_pred))

4.3 k-Nearest Neighbors

k-NN stores the training dataset and classifies a new point by looking at the majority class among its k closest neighbors, often using Euclidean distance.

• Lazy learning: no heavy training phase.
• Distance based: scaling matters.
• Curse of dimensionality: performance can weaken with many features.

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
print(knn.score(X_test, y_test))

4.4 Support Vector Machines

SVM finds a hyperplane that maximizes the margin between classes. Support vectors are the closest points that define this margin. Kernel methods such as RBF help with non-linear separation.

4.4.1 Strengths, Weaknesses, and Parameters

Linear models train and predict quickly, scale well to large and sparse datasets, and are easy to express mathematically. Their coefficient interpretation can become unclear when features are highly correlated.

• Use L1 regularization when only a few features are expected to matter.
• Use L2 regularization as the usual default when many features may contribute.
• Search regularization values on a logarithmic scale.

4.4.2 Evaluation Metrics for Classification

Classification metrics explain different kinds of correctness.

5.1 Introduction

Unsupervised learning finds patterns in unlabeled data. It is useful when labels are unavailable but hidden structure may still support segmentation, anomaly detection, or exploration.

5.2 Clustering

Clustering groups similar data points so intra-cluster similarity is high and inter-cluster similarity is low. Cluster labels are arbitrary: cluster 0 does not inherently mean anything until you inspect the group.

5.3 k-Means Clustering

k-Means is a centroid-based algorithm. It chooses k centroids, assigns points to the nearest centroid, updates centroids to the mean of assigned points, and repeats until cluster assignments stop changing.

1. Choose the number of clusters k.
2. Randomly initialize k centroids.
3. Assign each point to its nearest centroid.
4. Update each centroid using the mean of its assigned points.
5. Repeat until convergence.

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

df_scaled = StandardScaler().fit_transform(df)
kmeans = KMeans(n_clusters=3, random_state=42)
df["Cluster"] = kmeans.fit_predict(df_scaled)

5.4 Hierarchical Clustering

Hierarchical clustering creates nested clusters shown as a tree-like dendrogram. Agglomerative clustering starts with each point as its own cluster and merges the most similar clusters until the stopping condition is reached.

• Ward: minimizes variance increase and often creates similar-sized clusters.
• Average: uses average distance between all points in two clusters.
• Complete: uses the smallest maximum distance between points in two clusters.

5.5 Density-Based Clustering (DBSCAN)

DBSCAN identifies dense regions and separates noise points. It does not require a predefined k and can find arbitrary-shaped clusters.

• eps: neighborhood radius.
• min_samples: minimum points needed to form a dense region.
• Noise: points not assigned to a dense cluster.

from sklearn.cluster import DBSCAN

dbscan = DBSCAN(eps=0.5, min_samples=5)
df["DBSCAN_Cluster"] = dbscan.fit_predict(df_scaled)

5.6 Clustering Evaluation Metrics

Because clustering is unsupervised, ordinary accuracy is not the main evaluation method. The guide lists internal evaluation methods.

5.6.1 Failure Cases of k-Means

k-Means can fail even when the correct number of clusters is known because it assumes clusters are represented well by centers, have simple convex shapes, and have similar diameter.