I'm using Keras (with TensorFlow). It turns out this is an artifact of the way the Keras library does model validation. From the FAQ <https://keras.io/getting-started/faq/#why-is-the-training-loss-much-higher-than-the-testing-loss>
A Keras model has two modes: training and testing. Regularization mechanisms, such as Dropout and L1/L2 weight regularization, are turned off at testing time. Besides, the training loss is the average of the losses over each batch of training data. Because your model is changing over time, the loss over the first batches of an epoch is generally higher than over the last batches. On the other hand, the testing loss for an epoch is computed using the model as it is at the end of the epoch, resulting in a lower loss. On Sun, Oct 27, 2019 at 1:15 PM James Bowery <[email protected]> wrote: > I'm seeing a rather strange phenomenon in training an LSTM on a time > series. I'm training it on early data and testing on later data. After > say 100 epochs the test data produces lower error than the train data. > This could just be a coincidence, but since the test data is about 25% of a > total dataset of about 1200 cases, it still seems kind of strange. > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + delivery > options <https://agi.topicbox.com/groups/agi/subscription> Permalink > <https://agi.topicbox.com/groups/agi/Tbd9e6404771d6b20-M2c6dbde57c746032a10e7777> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tbd9e6404771d6b20-M826b1c33c69628d026fcd731 Delivery options: https://agi.topicbox.com/groups/agi/subscription
