Till Rohrmann created FLINK-2162:
------------------------------------
Summary: Implement adaptive learning rate strategies for SGD
Key: FLINK-2162
URL: https://issues.apache.org/jira/browse/FLINK-2162
Project: Flink
Issue Type: Improvement
Components: Machine Learning Library
Reporter: Till Rohrmann
Priority: Minor
At the moment, the SGD implementation uses a simple adaptive learning rate
strategy, {{adaptedLearningRate = initialLearningRate / sqrt(iterationNumber)
}}, which makes the optimization algorithm sensitive to the setting of the
{{initialLearningRate}}. If this value is chosen wrongly, then the SGD might
become instable.
There are better ways to calculate the learning rate [1] such as Adagrad [3],
Adadelta [4], SGD with momentum [5] others [2]. They promise to result in more
stable optimization algorithms which don't require so much hyperparameter
tweaking. It might be worthwhile to investigate these approaches.
Resources:
[1] [http://imgur.com/a/Hqolp]
[2] [http://cs.stanford.edu/people/karpathy/convnetjs/demo/trainers.html]
[3] [http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf]
[4] [http://www.matthewzeiler.com/pubs/googleTR2012/googleTR2012.pdf]
[5] [http://www.willamette.edu/~gorr/classes/cs449/momrate.html]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
