----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43732/ -----------------------------------------------------------
(Updated Feb. 25, 2016, 4:50 p.m.) Review request for samza. Changes ------- The new patch addressing all issues Yi brought up. Repository: samza Description ------- https://issues.apache.org/jira/browse/SAMZA-876 Implemented AvroDataFileHdfsWriter fashioned loosely after BinarySequenceFileHDFSWriter. Exposed several RocksDb configuration options (recommended in RocksDb tuning guide): rocksdb.log.level rocksdb.log.keepfilenum rocksdb.log.timetoroll rocksdb.log.maxfilesize rocksdb.bloomfilter.bits rocksdb.max.background.compactions rocksdb.max.background.flushes rocksdb.num.write.buffers rocksdb.target.file.size.base rocksdb.max.bytes.level.base Diffs (updated) ----- docs/learn/documentation/versioned/hdfs/producer.md cfd22c6 docs/learn/documentation/versioned/jobs/configuration-table.html 6705530 gradle/dependency-versions.gradle 52e25aa samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/HdfsConfig.scala 7993119 samza-hdfs/src/main/scala/org/apache/samza/system/hdfs/writer/AvroDataFileHdfsWriter.scala PRE-CREATION samza-hdfs/src/test/resources/samza-hdfs-test-batch-job-avro.properties PRE-CREATION samza-hdfs/src/test/resources/samza-hdfs-test-job-avro.properties PRE-CREATION samza-hdfs/src/test/scala/org/apache/samza/system/hdfs/TestHdfsSystemProducerTestSuite.scala c4b04a1 Diff: https://reviews.apache.org/r/43732/diff/ Testing ------- I am using AvroDataFileHdfsWriter at the end of my pipeline. I feed the generated avro files to Apache Samoa. Have processed millions of records successfully. The RocksDb config changes are older and were used and verified to be working when originally implemented. Thanks, Edi Bice
