Tuning hyperparameters

The algorithm hyperparameters outline the model’s structure. The model performance can depend on the choice of hyperparameters.

There is no best choice for a hyperparameter value that fits all models and all problems. Hyperparameters must be optimized within the context of each machine learning project.

The process of searching for the ideal architecture for the model is determined by experimenting to figure out what works best for your project. Define the range of possible values for all hyperparameters and evaluative criteria.

Hyperparameters address these model design questions:

What is the maximum depth allowed for my decision tree? A decision tree can yield good results for moderate tree depth and have very bad performance for very deep trees.
What is the minimum number of samples required at a leaf node in my decision tree?
How many trees should I include in my random forest?
How many layers should I have in my neural network?
What should I set my learning rate to for gradient descent?

Start with a wide hyperparameter space and iteratively refine their values in the highest-performing region of the hyperparameter space.

The time required to train and test a model can depend on the choice of hyperparameters. Some hyperparameters are conditional upon the value of others, for example, the size of each hidden layer in a neural network can be conditional upon the number of layers.