How to tune hyperparameters with Python and scikit-learn

In last week’s post, I introduced the k-NN machine learning algorithm which we then applied to the task of image classification.

Using the k-NN algorithm, we obtained 57.58% classification accuracy on the Kaggle Dogs vs. Cats dataset challenge:

Figure 1:Classifying an image as whether itcontains a dog or a cat.

The question is: “Can we do better?”

Of course we can!Obtaining higher accuracy for nearly any machine learning algorithm boils down to tweaking various knobs and levels.

In the case of k-NN, we can tune k , the number of nearest neighbors. We can also tune our distance metric/similarity function as well.

Of course, hyperparameter tuning has implications outside of the k-NN algorithm as well. In the context of Deep Learning and Convolutional Neural Networks, we can easily have hundreds of various hyperparameters to tune and play with (although in practice we try to limit the number of variables to tune to a small handful), each affecting our overall classification to some (potentially unknown) degree.

Because of this, it’s important to understand the concept of hyperparameter tuning and how your choice in hyperparameters can dramatically impact your classification accuracy.

Looking for the source code to this post?

Jump right to the downloads section. How to tune hyperparameters with python and scikit-learn

In the remainder of today’s tutorial, I’ll be demonstrating how to tune k-NN hyperparameters for the Dogs vs. Cats dataset . We’ll start with a discussion on what hyperparameters are , followed by viewing a concrete example on tuning k-NN hyperparameters.

We’ll then explore how to tune k-NN hyperparameters using two search methods: Grid Search and Randomized Search.

As our results will demonstrate, we can improveour classification accuracy from 57.58% to over 64%!

What are hyperparameters?

Hyperparameters are simply the knobs and levels you pull and turn when building a machine learning classifier. The process of tuning hyperparameters is more formally called hyperparameter optimization .

So what’s the difference between a normal “model parameter” and a “hyperparameter”?

Well, a standard “model parameter” is normally an internal variable that is optimized in some fashion. In the context of Linear Regression, Logistic Regression, and Support Vector Machines, we wouldthink of parameters as the weight vector coefficients found by the learning algorithm.

On the other hand, “hyperparameters” are normally set by a human designer or tuned via algorithmic approaches. Examples of hyperparameters include the number of neighbors k in the k-Nearest Neighbor algorithm, the learning rate alpha of a Neural Network, or the number of filters learned in a given convolutional layerin a CNN.

In general, model parameters are optimized according to some loss function , while hyperparameters are instead searched for by exploring various settings to see which values provided the highest level of accuracy.

Because of this, it tends to be easier to tune model parameters (since we’re optimizing some objective function based on our training data) whereas hyperparameterscanrequire a nearly blind search to find optimal ones.

k-NN hyperparameters

As a concrete example of tuning hyperparameters, let’s consider the k-Nearest Neighbor classification algorithm . For your standard k-NN implementation, there are two primary hyperparameters that you’ll want to tune:

The number of neighbors k . The distance metric/similarity function.

Both of these values can dramatically affect the accuracy of your k-NN classifier. To demonstrate this in the context of image classification, let’s apply hyperparameter tuning to our Kaggle Dogs vs. Cats dataset from last week.

Open up a new file, name it knn_tune . py , and insert the following code:

# import the necessary packages from sklearn.neighborsimport KNeighborsClassifier from sklearn.grid_searchimport RandomizedSearchCV from sklearn.grid_searchimport GridSearchCV from sklearn.cross_validationimport train_test_split from imutilsimport paths import numpyas np import argparse import imutils import time import cv2 import os

Lines 2-12start by importing our required Python packages. We’ll be making heavy use of the scikit-learn library , so if you do not have it installed, make sure you follow these instructions .

We’ll also be using my personal imutils library , so make sure you have it installed as well:

$ pipinstallimutils

Next, we’ll define our extract_color_histogram function:

def extract_color_histogram(image, bins=(8, 8, 8)): # extract a 3D color histogram from the HSV color space using # the supplied number of `bins` per channel hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV) hist = cv2.calcHist([hsv], [0, 1, 2], None, bins, [0, 180, 0, 256, 0, 256]) # handle normalizing the histogram if we are using OpenCV 2.4.X if imutils.is_cv2(): hist = cv2.normalize(hist) # otherwise, perform "in place" normalization in OpenCV 3 (I # personally hate the way this is done else: cv2.normalize(hist, hist) # return the flattened histogram as the feature vector return hist.flatten()

This function accepts an input image along with a number of bins for each channel of the image.

We convert the image to the HSV color space and compute a 3D color histogram to characterize the color distribution of the image ( Lines 17-19 ).

This histogram is then flattened into a single 8 x 8 x 8 = 512-d feature vector that is returned to the calling function.

For a more detailed review of this method, please refer to last week’s blog post .

# construct the argument parse and parse the arguments ap = argparse.ArgumentParser() ap.add_argument("-d", "--dataset", required=True, help="path to input dataset") ap.add_argument("-j", "--jobs", type=int, default=-1, help="# of jobs for k-NN distance (-1 uses all available cores)") args = vars(ap.parse_args()) # grab the list of images that we'll be describing print("[INFO] describing images...") imagePaths = list(paths.list_images(args["dataset"])) # initialize the data matrix and labels list data = [] labels = [] Lines 34-39handle parsing our command line arguments. We only need twoswitches h