Hyper-Parameter Tuning based on Random Search for DenseNet Optimization

Deep learning is a machine learning technology that is currently experiencing rapid development. One of the deep learning architecture models is Densely Connected Convolutional Networks (DenseNet), which connects each layer along with feature maps to all next layers. In other words, the next layer receives input feature maps from all previous layers. Because of this density architecture, the training time is longer and requires much memory. One of the ways to overcome these problems is to optimize using the batching strategy method. This batching method uses adaptive batch size with the ratio of learning rate during training to get faster training time without reducing accuracy. To obtain the optimal performance of this batching, a proper selection of hyper-parameter learning rate and batch size is needed. In this study, we perform tuning this hyper-parameter to get the optimal value. The selection of hyper-parameters uses a random search to select candidates for learning rate and batch size, and then an experiment is conducted on the candidates. Random search randomly selects the hyper-parameters, then training and scoring to the hyper-parameters. Our experiment results show that the lower limit of the batch size used is 64, while the optimal learning rate range is in the range of 0.1 - 0.3. The average accuracy obtained from using this hyper-parameter is 95%. Our experiments run on GPU RTX 2080 Ti with CIFAR-10 and CIFAR-100 datasets.