Heuristic hyperparameter optimization for neural networks

One of the crucial steps of training a neural network model is the process of fine-tuning its hyperparameters. This process can be time-consuming and hard to be done properly by hand. In this paper we explore the usage of selected heuristic algorithms: Covariance Matrix Adaptation Evolution Strategy (CMA-ES), Differential Evolution Strategy (DES) and jSO for the hyperparameter fine-tuning task. Results for various datasets are presented. An improvement in models' performance is observed through the usage of fine-tuned hyperparameters.