while working with aisaratuners was the fact that I had to introduce minimal

7 min readDec 13, 2020

It is unanimously agreed that model training is more about iterating and tweaking little knobs called hyperparameters to squeeze out the best from your model.

Photo by yinka adeoti on Unsplash. This picture has way fewer knobs to tweak than a typical deep learning model!

During the first three months of my machine learning journey, I quickly realized that I could not train a model that makes the most out of my data; I had to try thousands of different hyperparameter combinations and select the best model for myself.

At that moment, I knew about techniques like GridSearchCV from scikit-learn but I knew that it was not efficient and needed a lot of computing power and time. I needed a more efficient algorithm for keras; eventually, I stumbled upon two hyperparameter tuning libraries: aisaratuners and keras-tuner.

In this article, I will compare both these libraries and report my findings so you can have a better idea of which library is for you!

Aisaratuners library is a dynamic hyperparameter tuning library that tunes your model’s hyperparameters to give you the most optimized model with the best hyperparameters. Initially, it uses Latin Hypercube Sampling, a sophisticated statistical technique to pick hyperparameters, then uses its API for SOTA pattern recognition. In the end, you get the hyperparameters for the optimized model.

Enough talking, let’s test the library to see if it actually gets the job done.

We will use the Fashion MNIST dataset since it is easy to work with and is still slightly difficult to get 90%+ validation accuracy. To keep this article concise, I will only include code relevant to the library and not include the code for loading and preprocessing data. You can refer to the notebooks at the end of this article.

We will train a rather simple but dynamic model, with the help of aisaratuners library. Here we go.

To get started, we need to pip install the package:

We only need two imports from aisaratuners library to get started, yes, it is that simple!

https://www.reddit.com/r/NFLwk14LivHd/new/
https://www.reddit.com/r/NFLwk14LivHd/top/
https://www.reddit.com/r/NFLwk14LivHd/hot/
https://www.reddit.com/r/NFLwk14LivHd/rising/
https://www.reddit.com/r/NFLwk14LivHd/gilded/
https://www.reddit.com/r/NFLwk14LivHd/wiki/index/

We instantiate our Hp object which is a container for all the hyperparameters we wish to tune. You can set min/max ranges, as well.

Next, in a function containing our model, we will include the Hp object and all the ranges we have defined:

One thing I like while working with aisaratuners was the fact that I had to introduce minimal and understandable changes to the model. We just need to use the Hp object we created and access any random number we set in the numrange method in the above cell.

Let’s now run the optimizer and see how the algorithm performs:

We have set 3 rounds, each with 5 trials. This means in any round, the algorithm will sample 5 sets of hyperparameters from the solution space and train models. This is depicted in Fig. 1 above. In each round, the model will try to reduce this solution space. Also, we have set two criteria, one to maximize the validation accuracy and one to minimize validation loss. The mode argument is set to ‘p’ to use the free available API for aisaratuners to function. You could use a commercial API to use as well. Find more information here.

Please note that this time will vary for everyone and I was running the optimization using Google Colab, which has a variety of GPUs.

It took me less than 15 minutes (14.81) minutes to acquire the best model. That is amazing. It usually would take me at least two hours of manual tuning. Instead, we automatically tuned for the best model using aisaratuners in much less time.

The five trials of each round are labeled. My findings were:

Orange: Some variance (e.g. 48% validation accuracy) in the first round, but overall, the accuracies were definitely in the higher region. This is impressive! Aisaratuners algorithm did well in initial sampling.
Green: In the second round, the search space was reduced well enough because all the models were now achieving 90%+ validation accuracies.
Yellow: Even now, when typically I would stop trying to achieve better results, aisaratuners still managed to squeeze a one per cent increase on the validation set.

This was exactly what I needed: an automatic hyperparameter tuner. What exceeded my expectations were the visualizations this library provided me with. The above information can be visualized in a picture-worth-a-thousand-words manner. Interactive visualizations are labeled in the captions.

This image gives you a sense of what the optimizer did under the hood. The regions where the lines are dense indicates the best value in the specified range for that hyperparameter. For example, aisaratuners knows that the best number of convolutional filters is 150 and the optimum learning rate for the model is around 0.0001454. How cool is that!

That concludes our tutorial for aisaratuners. As with all things, there are some improvements I would suggest:

Including step size to the range functions will allow for more control of ranges
Including Boolean customizability to have control over things like optimizer choice (SGD, Adam, RMSProp, etc.)
Add the ability to conduct one trial multiple times. Usually, a model has a slight (2% or so) variance in the accuracy, multiple executions will make sure you can take an average of the accuracies. Sure, this will slow the optimization process but will make the results much more accurate. I think this tradeoff is sometimes worth it.
Creating well-formatted, easy-to-use documentation.

Aisaratuners definitely worked as claimed, the model I got was accurate and the visualizations were great. But let’s compare its algorithm with keras-tuner.

Keras-tuner is also a hyperparameter tuning library specifically made for keras. One of the most popular algorithms is RandomSearch.

RandomSearch is the basic way of testing out different hyperparameter combinations. It randomly samples hyperparameters and tries to converge on the best model. This is still better than grid search, though.

You require Python 3.6 and Tensorflow 2.0 for keras-tuner to work.

All code relevant to the library is shared. The rest of the code can be found linked at the end of the article.

Syntactically, both the libraries are very similar. I will try my best to tune to the same hyperparameters as I did using aisaratuners, and also make sure to keep the number of trials and epochs the same. I want the comparison to be fair.

Without further ado, let’s dive into the code. Let’s install the dependencies and set the log directory first:

Unlike aisaratuners, you do not need to contain all our hyperparameters in a predefined container object. Instead, you simply create a model function and make it dynamic as shown below:

The syntax is very similar, although keras-tuner does give you a step option. I find that helpful.

Since keras-tuner does not include the ability to set rounds and trials as aisaratuners has, I have set the number of trials to 15 (equal to total trials aisaratuners had in three rounds of five trials each). This also makes sense since the keras-tuner tuning algorithm is random and cannot intelligently reduce the search space in each round. Furthermore, I like the ability to conduct multiple executions per trail to reduce variance in each trial’s result; however, we are going to keep that at 1, so as to produce the fairest conditions.

Please note that this time will vary for everyone and I was running the optimization using Google Colab, which has a variety of GPUs. For the comparisons, I made sure I was using Google Colab’s Tesla T4 GPU instance. You can always check your GPU instance in Colab using !nvidia-smi . In case you need a different GPU instance, you can factory reset your GPU runtime.

So, 19.38 minutes. 15 trials. Not bad, not great, either. Compared to aisaratuners which took 14.81 minutes, keras-tuner completed the optimization much slower. But maybe, it has converged to a better validation accuracy? How much better should the validation accuracy be to compensate for increased optimization time?

It turns out, keras-tuner did get a better validation accuracy — only by 0.27%. One thing I did not like about keras-tuner’s algorithm is that it was unpredictable. For instance, in the seventh trial, one can achieve 92% validation accuracy and in the tenth trial, it could be around 73%. There was not a converging trend as in aisaratuners results (shown in Fig. 2). One would not be exactly sure whether one has got the best hyperparameters mid-way through the optimization process. In other words, had I started this comparison with keras-tuners instead of aisaratuners, no doubt it would have taken me longer. I would then not stop at 15 trials since I did not know if the tuner had reached the highest accuracy due to lack of convergence or ease of visualization.

Here is how you get the summary of results, get the best model’s hyperparameters or get the best model itself:

I cannot show the output of since it was poorly formatted. You can always view the notebook shared at the end of the article.

You can also produce some visualizations using keras-tuner of the tuning process; however, they are complicated to produce. They require using TensorBoard and make use of the logs directory.

I would like to pass on some remarks about keras-tuner before making a final verdict. These are:

That concludes our comprehensive comparison of the two libraries. We looked at their somewhat different syntax. Using each library, we trained a very similar convolutional neural network on the Fashion MNIST dataset to record the optimization results and the time it took for the algorithms to complete the optimization.

In the end, I chose to stick with the aisaratuners library. I recommend this library because it is much easier to configure, and runs on Python ≥ 3.6. It optimizes faster and the interactive visualizations make the tuning process easy to grasp and more fun. If my suggestions are implemented into their library, it would only make things better.

while working with aisaratuners was the fact that I had to introduce minimal

Written by Xetel