Re: Allow user-specified class-loader

Have you figured out a workflow where you can run jobs from rhe repl during development?

Re: Allow user-specified class-loader

We will release some open source when it's ready but the basic idea is to define a single task that implements all the interfaces, and have that delegate to clojure handlers. On Feb 10, 2016 4:39 PM, "Jacob Maes" wrote: > Hey Andy, > > That's good news! Do you mind sharing the revelation for any

Re: Zombie writers protection

Hi, Rick and John, Thanks for the great discussion! As Jacob said, we realized the possible drawbacks relying solely on YARN for process liveness detection as well and that's why SAMZA-871 was opened. Please help to comment on the JIRA so that we can track the discussion and move the design proces

Re: Allow user-specified class-loader

Why aren't you just writing samza tasks directly in clojure using gen-class and implementing the samza interfaces? Works great for us. Much simpler. > On Feb 10, 2016, at 7:39 PM, Jacob Maes wrote: > > Hey Andy, > > That's good news! Do you mind sharing the revelation for any other Samza > use

Re: Allow user-specified class-loader

Hey Andy, That's good news! Do you mind sharing the revelation for any other Samza users in Clojure-land? -Jake On Wed, Feb 10, 2016 at 4:07 PM, Andy Chambers < andy.chamb...@fundingcircle.com> wrote: > My co-worker has figured out how to do this without any changes required in > samza. > Andy

Re: Allow user-specified class-loader

My co-worker has figured out how to do this without any changes required in samza. Andy Chambers | Full-Stack Software Developer andy.chamb...@fundingcircle.com | 707-205-6502 (m) 747 Front St, 4th Fl | San Francisco, CA 94111 Our Mission: T o build a better financial world Unless speci

Re: HTTP-based Elasticsearch system producer and reusable task

Hi Yi, Please see my comment inline. On Tue, Feb 9, 2016 at 10:08 PM, Yi Pan wrote: > Hi, Roger, > > Got it! I would like to understand more on the SystemProducer API changes > required by #1 and #2. Could you elaborate a bit more? > > For #1, IndexRequestFactory interface ( https://github.com/

Re: Zombie writers protection

Jake, Not my question, I was just adding my 2 cents :) John, it’s not that yarn is responsible for maintaining 1 instance of each container, samza has an abstract management layer that defers this to yarn, but some people bypass yarn all together and manage their containers themselves or run on

Re: Zombie writers protection

Hey Rick, If I understand your question, the goal is really to make sure there are no orphaned containers that continue to run "off the books". The newly added SAMZA-871 describes a heart beat mechanism to make sure orphaned containers actually get killed. Also, the YARN Node Manager Restart cap

Re: Zombie writers protection

To second Rick's point. Its less about malicious actors, but rather containers thought to be lost due to a network partition popping up later and starting to write to the change log. I assume from Rick's response that yarn is responsible for ensure only one version of each container is running and

Re: Zombie writers protection

Security wouldn’t stop zombie processes from writing to kafka. I had this problem with yarn before where the container thought it was killing jobs but they never actually died, and in fact continued to write to kafka. > On Feb 10, 2016, at 4:23 PM, Jagadish Venkatraman > wrote: > > Hi John >

Re: Zombie writers protection

Hi John Currently there is no authorization on who writes to Kafka. There is a Kafka security proposal that the kafka community is working on. https://cwiki.apache.org/confluence/display/KAFKA/Security Building this into Samza may entail expensive coordination (to prevent other jobs). Since, jobs

Zombie writers protection

Greetings, I have general design question i did not see addressed in the docs. Basically how does samza guarantee a single writer for each changelog partition. Because of strong ordering assumption of these changelog, how do you protect against zombie processes writing to the changelog with out of

Re: Compression type Snappy is not linked with the binary

Navina, It's not my code changes. I started off fresh on a different but similar RHEL 6.6 box. Cloned and built latest from master and changed only gradle/dependencies jackson 1.9.3 and yarn 2.7.1 (as I used to do since day 1 to get jobs running on my cluster). All my KV store jobs fail with sam

Re: Review Request 43350: SAMZA-867 Fix job restart/shutdown in the event of a node outage.

> On Feb. 10, 2016, 7:48 a.m., Yi Pan (Data Infrastructure) wrote: > > samza-yarn/src/main/java/org/apache/samza/job/yarn/SamzaTaskManager.java, > > line 166 > > > > > > nit: I would prefer to keep UNUSED_CONTAINER_

Re: Review Request 43350: SAMZA-867 Fix job restart/shutdown in the event of a node outage.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43350/ --- (Updated Feb. 10, 2016, 3:57 p.m.) Review request for samza, Navina Ramesh, Jag

Re: Review Request 43350: SAMZA-867 Fix job restart/shutdown in the event of a node outage.

> On Feb. 10, 2016, 7:19 a.m., Yi Pan (Data Infrastructure) wrote: > > First question: is this an independent patch or depend on RB 43074? It depends on 43074, but I see you've committed that one now, so this one should be good to go now. - Jake -

Re: Compression type Snappy is not linked with the binary

Navina, I created a pull request some time back and it's still probably the easiest way to see what I changed. Let me know if you still prefer a patch file attachment. https://github.com/apache/samza/pull/6 Thanks, Edi On Tue, Feb 9, 2016 at 6:09 PM, Navina Ramesh wrote: > Thanks, Edi. > Not

Re: Review Request 43404: SAMZA-851: update Hello-Samza w/ CDH tutorial documentation

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43404/#review118657 --- Ship it! Ship It! - Stanislav Los On Feb. 10, 2016, 5:21 a.m