Keras: Introduction to Mini-Batch Gradient Descent
Stochastic gradient descent adalah metoda yang banyak digunakan untuk train deep learning model.
Ada tiga varian utama gradient descent dan bisa membingungkan mana yang akan digunakan. Dalam tulisan ini, anda akan menemukan satu jenis gradient descent yang harus anda gunakan secara umum dan cara mengkonfigurasinya.
Setelah selesai membaca tulisan ini, anda akan tahu:
- Apa itu gradient descent dan bagaimana kerjanya pada tingkat tinggi.
- Apakah batch, stochastic, dan mini-batch gradient descent serta manfaat dan keterbatasan masing-masing metode.
- Mini-batch gradient descent adalah metode yang relatif paling aman dan cara mengkonfigurasinya pada aplikasi anda.
Tutorial Overview
Tutorial dibagi dalam tiga (3) bagian, yaitu:
- Apakah Gradient Descent?
- Perbandingan antara 3 Type Gradient Descent
- Care mengkonfigurasi Mini-Batch Gradient Descent
Apakah Gradient Descent?
Gradient descent adalah algoritma optimasi yang sering digunakan untuk menemukan weight atau koefisien algoritma machine learning, seperti artificial neural networks dan logistic regression.
Ini bekerja dengan meminta model membuat prediksi pada data training dan menggunakan kesalahan pada prediksi untuk memperbarui model sedemikian rupa agar bisa mengurangi kesalahan.
Tujuan algoritma adalah untuk menemukan parameter model (mis. Koefisien atau weight) yang meminimalkan kesalahan model pada dataset training. Algoritma melakukan ini dengan membuat perubahan pada model yang memindahkannya sepanjang gradien atau kemiringan kesalahan ke bawah menuju nilai kesalahan minimum. Hal ini menyebabkan nama algoritma "gradient descent."
Pseudocode di bawah ini merangkum algoritma gradient descent:
model = initialization(...) n_epochs = ... train_data = ... for i in n_epochs: train_data = shuffle(train_data) X, y = split(train_data) predictions = predict(X, model) error = calculate_error(y, predictions) model = update_model(model, error)
Untuk informasi lebih lanjut bisa membaca:
- Keras: Gradient Descent For Machine Learning
- Keras: How to Implement Linear Regression with Stochastic Gradient Descent from Scratch with Python
Membandingkan 3 Type Gradient Descent
Gradient descent dapat bervariasi dalam hal jumlah pola training yang digunakan untuk menghitung kesalahan; yang nantinya digunakan untuk memperbarui model.
Jumlah pola yang digunakan untuk menghitung kesalahan termasuk seberapa stabil gradien yang digunakan untuk memperbarui model. Kita akan melihat bahwa ada tarik ulur dalam konfigurasi gradient descent akan efisiensi komputasi dengan ketepatan dari gradien error.
Tiga variasi utama dari gradient descent adalah batch, stochastic, dan mini-batch.
Mari kita melihat lebih dekat pada masing-masing.
Apakah Stochastic Gradient Descent?
Stochastic gradient descent, kadang di singkat SGD, adalah varian dari algoritma gradient descent yang menghitung error dan memperbarui model untuk setiap contoh dalam dataset training.
Update model untuk setiap contoh training berarti bahwa stochastic gradient descent disebut online machine learning algorithm.
Upsides
- The frequent updates immediately give an insight into the performance of the model and the rate of improvement.
- This variant of gradient descent may be the simplest to understand and implement, especially for beginners.
- The increased model update frequency can result in faster learning on some problems.
- The noisy update process can allow the model to avoid local minima (e.g. premature convergence).
Downsides
- Updating the model so frequently is more computationally expensive than other configurations of gradient descent, taking significantly longer to train models on large datasets.
- The frequent updates can result in a noisy gradient signal, which may cause the model parameters and in turn the model error to jump around (have a higher variance over training epochs).
- The noisy learning process down the error gradient can also make it hard for the algorithm to settle on an error minimum for the model.
Apakah Batch Gradient Descent?
Batch gradient descent is a variation of the gradient descent algorithm that calculates the error for each example in the training dataset, but only updates the model after all training examples have been evaluated.
One cycle through the entire training dataset is called a training epoch. Therefore, it is often said that batch gradient descent performs model updates at the end of each training epoch.
Upsides
- Fewer updates to the model means this variant of gradient descent is more computationally efficient than stochastic gradient descent.
- The decreased update frequency results in a more stable error gradient and may result in a more stable convergence on some problems.
- The separation of the calculation of prediction errors and the model update lends the algorithm to parallel processing based implementations.
Downsides
- The more stable error gradient may result in premature convergence of the model to a less optimal set of parameters.
- The updates at the end of the training epoch require the additional complexity of accumulating prediction errors across all training examples.
- Commonly, batch gradient descent is implemented in such a way that it requires the entire training dataset in memory and available to the algorithm.
- Model updates, and in turn training speed, may become very slow for large datasets.
Apakah Mini-Batch Gradient Descent?
Mini-batch gradient descent is a variation of the gradient descent algorithm that splits the training dataset into small batches that are used to calculate model error and update model coefficients.
Implementations may choose to sum the gradient over the mini-batch which further reduces the variance of the gradient.
Mini-batch gradient descent seeks to find a balance between the robustness of stochastic gradient descent and the efficiency of batch gradient descent. It is the most common implementation of gradient descent used in the field of deep learning.
Upsides
- The model update frequency is higher than batch gradient descent which allows for a more robust convergence, avoiding local minima.
- The batched updates provide a computationally more efficient process than stochastic gradient descent.
- The batching allows both the efficiency of not having all training data in memory and algorithm implementations.
Downsides
- Mini-batch requires the configuration of an additional “mini-batch size” hyperparameter for the learning algorithm.
- Error information must be accumulated across mini-batches of training examples like batch gradient descent.
How to Configure Mini-Batch Gradient Descent
Mini-batch gradient descent is the recommended variant of gradient descent for most applications, especially in deep learning.
Mini-batch sizes, commonly called “batch sizes” for brevity, are often tuned to an aspect of the computational architecture on which the implementation is being executed. Such as a power of two that fits the memory requirements of the GPU or CPU hardware like 32, 64, 128, 256, and so on.
Batch size is a slider on the learning process.
- Small values give a learning process that converges quickly at the cost of noise in the training process.
- Large values give a learning process that converges slowly with accurate estimates of the error gradient.
Tip 1: A good default for batch size might be 32.
… [batch size] is typically chosen between 1 and a few hundreds, e.g. [batch size] = 32 is a good default value, with values above 10 taking advantage of the speedup of matrix-matrix products over matrix-vector products. — Practical recommendations for gradient-based training of deep architectures, 2012
Update 2018: here is another paper supporting a batch size of 32, here’s the quote (m is batch size):
The presented results confirm that using small batch sizes achieves the best training stability and generalization performance, for a given computational cost, across a wide range of experiments. In all cases the best results have been obtained with batch sizes m = 32 or smaller, often as small as m = 2 or m = 4. — Revisiting Small Batch Training for Deep Neural Networks, 2018.
Tip 2: It is a good idea to review learning curves of model validation error against training time with different batch sizes when tuning the batch size.
… it can be optimized separately of the other hyperparameters, by comparing training curves (training and validation error vs amount of training time), after the other hyper-parameters (except learning rate) have been selected.
Tip 3: Tune batch size and learning rate after tuning all other hyperparameters.
… [batch size] and [learning rate] may slightly interact with other hyper-parameters so both should be re-optimized at the end. Once [batch size] is selected, it can generally be fixed while the other hyper-parameters can be further optimized (except for a momentum hyper-parameter, if one is used).
Further Reading
This section provides more resources on the topic if you are looking go deeper. Related Posts
Gradient Descent for Machine Learning How to Implement Linear Regression with Stochastic Gradient Descent from Scratch with Python
Additional Reading
Stochastic gradient descent on Wikipedia Online machine learning on Wikipedia An overview of gradient descent optimization algorithms Practical recommendations for gradient-based training of deep architectures, 2012 Efficient Mini-batch Training for Stochastic Optimization, 2014 In deep learning, why don’t we use the whole training set to compute the gradient? on Quora Optimization Methods for Large-Scale Machine Learning, 2016
Summary
In this post, you discovered the gradient descent algorithm and the version that you should use in practice.
Specifically, you learned:
- What gradient descent is and how it works from a high level.
- What batch, stochastic, and mini-batch gradient descent are and the benefits and limitations of each method.
- That mini-batch gradient descent is the go-to method and how to configure it on your applications.
Referensi