Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/967
The problem is somewhere in Storm's windowing functionality. The time that
it initially recognizes is too far in the future and causes it to mark the
messages sent in as expired. This only occurs intermittently.
Here you can see test messages generated with the last timestamp being
1521756035817.
```
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for
message with timestamp=1521756035759
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for
message with timestamp=1521756035802
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 0 route(s) for
message with timestamp=1521756035806
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for
message; profile=example2, entity=10.0.0.2, timestamp=1521756035807
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
message with timestamp=1521756035807
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for
message; profile=example2, entity=10.0.0.2, timestamp=1521756035808
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
message with timestamp=1521756035808
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for
message; profile=example2, entity=10.0.0.2, timestamp=1521756035813
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
message with timestamp=1521756035813
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for
message; profile=example2, entity=10.0.0.3, timestamp=1521756035814
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
message with timestamp=1521756035814
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for
message; profile=example2, entity=10.0.0.3, timestamp=1521756035816
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
message with timestamp=1521756035816
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:195 - Found route for
message; profile=example2, entity=10.0.0.3, timestamp=1521756035817
2018-03-22 22:00:35 DEBUG ProfileSplitterBolt:201 - Found 1 route(s) for
message with timestamp=1521756035817
```
The first timestamp that Storm recognizes is 1521756041122, which is 5.3
seconds ahead of the latest timestamp in the data.
```
2018-03-22 22:00:41 DEBUG WindowManager:189 - Scan events, eviction policy
TimeEvictionPolicy{windowLength=5000, referenceTime=1521756041122}
```
Storm then marks these messages as expired and the Profiler never sees them.
```
2018-03-22 22:00:41 DEBUG WindowManager:212 - [6] events expired from
window.
2018-03-22 22:00:41 DEBUG WindowManager:214 - invoking
windowLifecycleListener.onExpiry
2018-03-22 22:00:41 DEBUG WindowManager:147 - No events in the window,
skipping onActivation
```
Epic test failure.
---
