Re: Review Request 51142: SAMZA-967: HDFS System Consumer

> On Sept. 13, 2016, 1:37 a.m., Yi Pan (Data Infrastructure) wrote: > > samza-hdfs/src/main/java/org/apache/samza/system/hdfs/partitioner/DirectoryPartitioner.java, > > line 83 > > > > > > Question: this seems to be

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

> On Sept. 13, 2016, 1:37 a.m., Yi Pan (Data Infrastructure) wrote: > > samza-hdfs/src/main/java/org/apache/samza/system/hdfs/reader/AvroFileHdfsReader.java, > > line 24 > > > > > > One concern I had w/ this HdfsAvro

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

> On Sept. 13, 2016, 10:03 p.m., Jagadish Venkatraman wrote: > > samza-operator/src/main/java/org/apache/samza/operators/api/MessageStreams.java, > > line 37 > > > > > > What do you think about adding a getSystemStr

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

> On Sept. 13, 2016, 10:03 p.m., Jagadish Venkatraman wrote: > > samza-operator/src/test/java/org/apache/samza/task/JoinOperatorTask.java, > > line 47 > > > > > > The current implementation seems to rely on state ma

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

> On Sept. 13, 2016, 10:03 p.m., Jagadish Venkatraman wrote: > > samza-operator/src/test/java/org/apache/samza/task/JoinOperatorTask.java, > > line 48 > > > > > > Curious as to why should users rely on `Join.join` t

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47835/ --- (Updated Sept. 14, 2016, 8:53 a.m.) Review request for samza, Boris Shkolnik, C

Issue with consuming non-existent topics in 0.10.1

We are testing an upgrade to 0.10.1 from 0.9.1 and noticed a regression. When starting a stream job that consumes a topic that does not yet exist, the job dies with the following exception: Exception in thread "main" java.lang.IllegalArgumentException: No tasks found. Likely due to no input pa

SIGSEGV in RocksDB when killing jobs

While testing with Samza 0.10.1 I noticed the following crash whenever I would kill a job that uses a RocksDB store: # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x7eff66b6c27e, pid=20315, tid=139636974364416 # # JRE version: Java(TM) SE Runtim

Re: Review Request 51142: SAMZA-967: HDFS System Consumer

> On Sept. 13, 2016, 12:33 a.m., Yi Pan (Data Infrastructure) wrote: > > samza-hdfs/src/main/java/org/apache/samza/system/hdfs/HdfsSystemConsumer.java, > > line 142 > > > > > > Isn't it clearer to have one loop like

Re: SIGSEGV in RocksDB when killing jobs

Hi, Tommy, Thanks for reporting this. Definitely we can be more defensive in coding here. I just wonder what's the specific reason for you to call RocksDB store close() explicitly? As you see that SamzaContainer#shutdownStores already calling flush() and close() automatically. Does it work for you

Re: Issue with consuming non-existent topics in 0.10.1

Hi, Tommy, Could you open a JIRA for this one? Also, could you include the Kafka broker version in this test? Thanks! -Yi On Wed, Sep 14, 2016 at 6:06 AM, Tommy Becker wrote: > We are testing an upgrade to 0.10.1 from 0.9.1 and noticed a regression. > When starting a stream job that consumes

Re: SIGSEGV in RocksDB when killing jobs

Thanks for the reply, Yi. It does indeed work if we remove calls to KeyValueStore.close(). As for why we're doing that, again mostly just for housekeeping. I haven't seen any guidance either way on whether StreamTasks should close the stores they use, and was unaware this was automatic. We can

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47835/#review148908 --- This review turned out not to be so massive as I expected (lots of

Re: Review Request 47835: SAMZA-914: Initial draft for Java programming APIs on operators supporting DAGs

> On Sept. 14, 2016, 7:03 p.m., Chris Pettitt wrote: > > This review turned out not to be so massive as I expected (lots of > > deletes). I don't see any serious issues. There are some minor cosmetic > > issues and some subjective stuff that you can take or leave. I tried to not > > mark subje

Re: Issue with consuming non-existent topics in 0.10.1

Thanks for the response, and done. https://issues.apache.org/jira/browse/SAMZA-1018 On 09/14/2016 01:14 PM, Yi Pan wrote: Hi, Tommy, Could you open a JIRA for this one? Also, could you include the Kafka broker version in this test? Thanks! -Yi On Wed, Sep 14, 2016 at 6:06 AM, Tommy Becker

Re: Review Request 51703: Enable passing of Configs and MetricsRegistry into Monitor objects.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51703/ --- (Updated Sept. 14, 2016, 8:37 p.m.) Review request for samza, Boris Shkolnik, J

Re: Review Request 51703: Enable passing of Configs and MetricsRegistry into Monitor objects.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51703/#review148764 --- samza-rest/src/main/java/org/apache/samza/monitor/DefaultMonitorF

Re: Review Request 51703: Enable passing of Configs and MetricsRegistry into Monitor objects.

> On Sept. 13, 2016, 3:53 p.m., Jake Maes wrote: > > samza-rest/src/main/java/org/apache/samza/monitor/config/PropertiesMonitorConfigFactory.java, > > line 37 > > > > > > I think there are a couple high-level issues