Review Request 48811: SAMZA-968 - SequenceFileHdfsFileWriter does not close file properly

2016-06-16 Thread Benjamin Smith
--- Unit tests pass successfully. Performing manual test in our code base to ensure that files close properly (we currently see them remain open and several hour delays in propogating writes to HDFS) Open to suggestions for how to write a unit test for this use case. Thanks, Benjamin Smith

Re: Bug in SequenceFileHdfsFileWriter

2016-06-16 Thread Benjamin Smith
t: Re: Bug in SequenceFileHdfsFileWriter Hi, Benjamin, Thanks a lot for reporting this! It makes sense from reading the posts. Could you open a JIRA? Are you interested in assigning to yourself and contribute the fix? Thanks a lot again! -Yi On Thu, Jun 16, 2016 at 9:52 AM, Benjamin Smit

Bug in SequenceFileHdfsFileWriter

2016-06-16 Thread Benjamin Smith
Hello, I am working on a project where we are integrating Samza and Hive. As part of this project, we ran into an issue where sequence files written from Samza were taking a long time (hours) to completely sync with HDFS. After some Googling and digging into the code, it appears that the issue