Re: Data not visible to new readers after SequenceFile.writer.hflush()

This is Hadoop's unit test for sequence file exercising hsync/hflush: https://github.com/apache/hadoop/blob/61df1b27a797efd094328c7d9141b9e157e01bf4/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestHSync.java#L151 On Mon, Jan 27, 2025 at 9:53 AM Wei-Chiu Ch

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Do you have a unit test to reproduce? Note that for performance reasons, the actual HDFS hflush/hsync implementation does not update visible length at NameNode. The data is flushed to DataNode though. So the writer would need to record the client side visible length, and pass it to the reader, if

Re: Data not visible to new readers after SequenceFile.writer.hflush()

Hi Sébastien, I replied to your same question on the hadoop-user@ thread. Let's keep the discussion there unless we discover there is some kind of HDFS bug to discuss. Thank you! Chris Nauroth On Mon, Jan 27, 2025 at 6:31 AM Sébastien Rebecchi wrote: > Hello, > > I got this issue using hadoop

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/807/ [Jan 25, 2025, 3:47:27 PM] (github) HADOOP-19397. Update LICENSE-binary with jersey 2 details (#7315). Contributed by PJ Fanning [Jan 26, 2025, 1:27:19 AM] (github) HADOOP-19236. Incorporate VolcanoE

Data not visible to new readers after SequenceFile.writer.hflush()

Hello, I got this issue using hadoop client with both hadoop 2.9.1 and 3.4.1 (client 2.9.1 to write to HDFS 2.9.1 etc). For SequenceFile.Writer class, after doing hflush(), data is not visible to new readers, it is visible only after doing close(). The doc asserts that data must be visible to new

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1835/ [Jan 26, 2025, 1:51:04 AM] (github) Revert "HADOOP-19236. Incorporate VolcanoEngine Cloud TOS File System Impleme…" (#7327) [Jan 26, 2025, 5:07:28 AM] (Chris Nauroth) HADOOP-19401: Improve error mess

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1650/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc