Introduction

Machine learning is more than simply computing averages or performing some data manipulation. It actually involves making predictions about new observations based on previous information.

Types

• Supervised (Predictive Analytics) There does exists a predefined relationship to exploit between some inputs (predictors) and one or more outputs (responses or outcomes), so that an algorithm is able to build a function that tries to approximate in some sense, and that will be used to map new inputs with no known outputs.
  • Classification for Qualitative Output(s)
    • k-Nearest Neighbors (kNN)
    • Naive Bayes
    • Logistic Regression
    • Recursive Partitioning (or Decision Tree)
    • Random Forest
    • Support Vector Machine
  • Regression for Quantitative Output(s)
    • Linear Regression
    • Lasso Regression
    • Ridge Regression
    • Poisson Regression (counting, discrete data)
• Unsupervised (Pattern Discovery) There is no concepts of predictor and response,
  • Dimensionality Reduction for Feature Selection
    • Principal Components (PCA)
    • Factor Analysis (FA)
    • Manifold Learning (IsoMap)
  • Clustering for grouping objects together such that the objects are similar within each group, and dissimilar between different groups. Notice that there is no prior knowledge of what the resulting groups could or should look like.
    • k-means, k-medians, k-modes, k-prototypes
    • Hierarchical
    • Mean-shift
    • DBSCAN
  • Anomaly Detection for Outlier Analysis
    • Isolation Forests
  • Data Imputation for Missing Values
  • Natural Language Processing ==> Topic Modeling

Use cases

• Spam Filtering. Predictors: word frequency, character frequency, the amount of sequential capital letters
• Fraud Detection
• Credit Scoring
• Customer Segmentation
• Shopping Basket Analysis
• Content Tagging
• Image/Text Recognition
• Recommender System
• Medical Diagnosis

R most known packages

• caret: Classification And REgression Training
• mlr:
• glmnet
• rpart / party Recursive Partitioning and Regression Trees
• ROCR Visualizing the performance of scoring classifiers
• e1071
• randomForest
• nnet
• igraph Collection of Network Analysis tools
• kernlab
• neuralnet
• h2o
• tensorflow Deep Learning
• keras Deep Learning

Steps for Supervided Models

• EDA: preprocess and explore the data

• dataset splitting
  When the model should be built only on a of the available observations. The remaining units should be used to assess the predictive power of the model. In order to have a fair distribution of the output variable in each set, it's important that the units are sampled randomly from the dataset, or equivalently shuffle the dataset beforehand and then extract sequentially.

• Data splitting into validation / train / test partitions

• model selection and validation, using default Hyper-Parameters

• fit the model on the training set

• fine-tune ==> act on the Hyper-Parameters of the model structure to find the best configuration

• generate prediction values apllying the model on the test set

• evaluate the model

  • Classification: Confusion Matrix, accuracy, precision (1-T1), recall (1-T2)
  • Regression: mean absolute error, median absolute error, R^2 score

• interpret the results

In problems that have a random aspect, the set.seed(n) function should be used to enforce reproducibility. After fixing the seed, the random numbers that are generated are always the same, and so should be all the subsequent results.

R built-in functions

• lm is a generic function to build linear models
• predict takes a model (1st arg) and some unseen observations (2nd arg), then returns the predicted outcomes from the model for those observations.
• kmeans performs k-means clustering
• set.seed enforce reproducibility in problems that have a random aspect function. If you fix the seed, the random numbers that are generated are always the same.
• rpart

Examples

• load the packages

  library(data.table)
  library(broom)
  library(rpart)

• when following a supervised algorithm, it's important to first split randomly the available data into a training and a test subdataset, while assuring reproducibility:

  set.seed(n)
  dts <- fread('/path/to/filename.csv')
  pct_train <- n
  units_train <- sample.int(nrow(dts), pct_train * nrow(dts)) 
  dts_train <- dts[units_train]
  dts_test  <- dts[!units_train]
  all.equal(dts, rbind(dts_train, dts_test))

  An alternative way is to first shuffle the dataset, and then extract .

  shuffled <- sample(dts)
  n_train <- round(nrow(shuffled) * pct_train)
  dts_train <- shuffled[1:n_train]
  dts_test  <- shuffled[(n_train + 1):nrow(shuffled)]
  all.equal(shuffled, rbind(dts_train, dts_test))

• classification: kNN

  fit <- 
  pred <- predict(fit, dts_test)
  table(dts$y, pred) # Confusion Matrix

• classification: Naive Bayes

  fit <- 
  pred <- predict(fit, dts_test)
  conf_mat <- table(dts$y, pred) # Confusion Matrix

• classification: recursive partitioning (decision tree)

  fit <- rpart(y ~ x1 + ... + xn, data = dts_train, method = "class")
  pred <- predict(fit, dts_test, type = 'class')
  conf_mat <- table(dts_test$y, pred)
  TP <- conf[1, 1] 
  FN <- conf[1, 2] 
  FP <- conf[2, 1] 
  TN <- conf[2, 2] 
  accuracy <- TP / (TP + FP + TN + FN)
  precision <- TP / (TP + FP)
  recall <- TP / (TP + FN)

• classification: logistic regression

  fit <- lm(y ~ x1 + ... + xn, data = dts_train, family = 'binomial')
  pred <- predict(fit, dts_test)
  table(dts$y, pred) # Confusion Matrix

• linear regression

  fit <- lm(y ~ x1 + ... + xn + x1*x2 + ..., data = dts_train)
  broom::tidy(fit)
  pred <- predict(fit, dts_test)
  rmse <- sqrt( 1 / nrow(dts_test) * sum( (dts$y - pred)^2 ) )

• clustering

  set.seed(1)
  groups <- kmeans(dts, n)
  table(z, groups$cluster)
  plot(dts, col = groups$cluster)
  points(groups$centers, pch = 22, bg = c(1, 2), cex = 2)
  groups$tot.withinss / groups$betweenss

Performance Measures

Bias vs Variability Trade Off

Overfitting vs Underfitting

Confusion Matrix

In the case of supervised learning, it's possible to compare the outcomes of the model with the truth, and the four possible situations leads to build what is called as Confusion Matrix:

Out of the n cases:

• the classifier predicted in total TPP = TP + FP positives and TPN = TN + FN negatives
• in reality, there has been TAP = TP + FN positives and TAN = TN + FP negatives.

From the above , the following metrics can be quickly calculated:

• Accuracy (TP + TN) / n Overall, how often is the classifier correct?
• Error Rate (FP + FN) / n = 1 - Accuracy Overall, how often is the classifier wrong? Also called Misclassification Rate
• Precision TP / (TP + FP) = TP / TPP How often does it predict a correct positive?
• Recall or Sensitivity TP / (TP + FN) = TP / TAP How often does it predict an actual positive?
• False Positive Rate FP / (FP + TN) = FP / TAN How often does it predict a wrong negative?
• Specificity TN / (FP + TN) = TN / TAN How often does it predict an actual negative?
• Prevalence TP + FN / n = TAP / n How many positives actually occur in the sample?

A particular attention should be on the concepts of accuracy and precision. Many people use these terms interchangeably. However, these words have different meanings in statistics:

• accuracy refers to how closely a measurement or observation comes to measuring a true value, since measurements and observations are always subject to error. It's a similar notion to the unbiasedness of a statistical estimator.
• precision refers to how closely repeated measurements or observations come to duplicating measured or observed values. It's a similar notion to the variability of a statistical estimator.

Cross Validation

A better way to assess the predicticve power of a supervised algorithm is to use all units both in the training and test phases alternatively. This process is called k-fold Cross-Validation. The original dataset is split k times, with the accuracy calculated for each fold. The mean of these accuracies forms a more robust estimation of the model's true accuracy of predicting unseen data, because it is less dependent on the choice of training and test sets.

# Set the number of folds
n_fold <- n
# Set the number of units in each fold
k_units <- round( (1/n_fold) * nrow(dts) )
# Set random seed for reproducibility
set.seed(n)
# Shuffle the original dataset for fair representation
shuffled <- sample(dts)
# Initialize the vector storing the folds accuracies
accs <- rep(0, n_fold)
# run the model over a loop
for (i in 1:n_fold){
  # Calculate the indices that address the current test set
  k_idx <- ((i-1) * k_units + 1):(i * k_units)
  # Form the train set
  dts_train <- shuffled[-k_idx,]
  # Form the test set
  dts_test <- shuffled[k_idx,]
  # Build the model 
  tree <- rpart(y ~ ., dts_train, method = 'class')
  # Make a prediction based on the test set
  pred <- predict(tree, dts_test, type = 'class')
  # Calculate the confusion matrix
  conf <- table(dts_test$y, pred)
  # Assign the accuracy to the ith index in accs
  accs[i] <- sum( diag(conf) ) / sum(conf)
}
# Print out the mean of accs as final accuracy of the model
mean(accs)

Receiver Operating Characteristic (ROC) Curve

RMSE

When using a regression algorithm, a standard measure for assessing the quality of the resulting model is the Root Mean Squared Error (RMSE) which the Mean distance between the estimates and the regression line and therefore represent a measure of the error incurred in substituting the true value with its prediction: the lower the value for the RMSE, the better the model.

However, as a standalone number, the RMSE doesn't tell anything meaningful, as it is expressed in the units of the outcome variable. In order to derive its meaning, it should be compared to the RMSE of a different model for the same problem. It should be noted that in general, adding complexity to a model, in this case adding predictors, decreases the value of the RMSE

Dunn's Index

Clustering algorithms are based on:

• maximising the similarity within groups. We can measure this by using:
  • Within Sum of Squares (WSS)
  • Average Diameter of clusters
    The smaller the better
• minimizing the similarity between groups. We can measure this by using:
  • Between Sum of Squares (BSS)
  • Average Intercluster Distances
    The higher the better
    Notice that the Total Sum of Squares (TSS) is simply the sum of Within and Between sums: TSS = BSS + WSS.

A popular measure of performance for different clustering algorithm is the Dunn's Index, which is defined as the ratio between the
Minimal Intercluster Distance and the Maximal Diameter. Using a similar approach, another measure of performance is defined as the ratio between the Within cluster Sum of Squares and the Between cluster Sum of Squares, so WSS/BSS. With both the above measures, the smaller they are the clusters are well seperated and overall compact.

Algorithms

k Nearest Neighbors (kNN)

Naive Bayes

Decision Tree

Automatic Interaction Detection, a wide range of methods has been suggested that is usually termed Recursive Partitioning or Decision trees or tree(-structured) models. Particularly influential has been and still are the following algorithms

• CART, Classification And Regression Trees
• CHAID, Chi-square Automatic Interaction Detector
• CTree, Conditional Inference
• MoB, Model-Based
• EvTree, Evolutionary Learning of Globally Optimal Classification and Regression Trees
• Ensemble Methods
  • Bagging
  • Random Forest
  • Gradient boosting
    • Xgboost
    • GBM
      Decision trees are of two main types:
• Classification tree, when the response variable is categorical, or a discretized numeric.
• Regression tree, when the response variable is numeric.

Within R the list of prominent packages includes rpart (CART), mvpart (multivariate CART), party (CTree, MOB), partykit (CTree, MOB)

Basic Terminology used with Decision trees:

• Root Node: It represents entire population or sample and this further gets divided into two or more homogeneous sets.
• Splitting: It is a process of dividing a node into two or more sub-nodes.
• Decision Node: When a sub-node splits into further sub-nodes, then it is called decision node.
• Leaf or Terminal Node: Nodes do not split is called Leaf or Terminal node.
• Branch / Sub-Tree: A sub section of entire tree is called branch or sub-tree.
• Parent vs Child Nodes: A node, which is divided into sub-nodes is called parent node of sub-nodes where as sub-nodes are the child of parent node.
• Pruning: When we remove sub-nodes of a decision node, this process is called pruning. You can say opposite process of splitting.

Regression Tree

Clustering

An important part in machine learning is understanding and interpreting the results. In the case of clustering, visualization is key, and the simplest way to achieve it is by plotting the features in the dataset and coloring the points based on their corresponding cluster. The cluster centroids are typically added as a good representations of all the observations in a cluster. They are often used to summarize your clusters.

Interpretabilty and Explainability

In the context of Modeling and Machine Learning systems, interpretability is the ability to explain or to present a machine learning model in understandable terms to a human. As long as a model is having no significant impact in the real world, its interpretability doesn’t matter as much. But when there are implications involved based on a model’s prediction, be it financial or social, the concept of interpretability becomes paramount. Machine learning models are being increasingly used to make decisions that affect people’s lives. With this power comes a responsibility to ensure, for example, that the model predictions are fair and not discriminating. It’s not enough to know if a model works, we need to know how it works. A model can often work in a right way using multiple, even infinite, different combination of inputs. We want to choose the one(s) that has the least negative impact, And to be able to do that we first need to be sure that the model can be interepreted, and to build some techniques that (try to) explain how different features have a different effect on the prediction of a model.

Permutation Importance

Partial Dependence Plots

SHAP Values

Advanced Uses of SHAP Values

LIME: Locally Interpretable Model-Agnostic Explanations