Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5886#issuecomment-98985622
I think this may be the problem scenario:
- Log file last modified at time 80
- `lastModifiedTime` begins at 90 due to other log files
- Scan starts at 100
- Log file is excluded from the update (80 < 100)
- Log file is modified at time 101
- Scan finds another log file modified at time 102
- Scan ends and `lastModifiedTime` becomes 102
- The next run won't pick up the log file since 101 < 102 but it had
modifications
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
