There was an interesting thread a while back from I believe the netflix guys
about running ThreadJobFactory in production.
> On Mar 2, 2016, at 4:20 PM, Robert Crim wrote:
>
> Hi,
>
> We're currently working on a solution that allows us to run Samza jobs on
> Mesos. This seems to be going wel
This is why we went the route of using clojure from the start and not calling
out to clojure from a java task.
> On Feb 10, 2016, at 9:11 PM, Andy Chambers
> wrote:
>
> Have you figured out a workflow where you can run jobs from rhe repl during
> development?
signature.asc
Description: Mes
Why aren't you just writing samza tasks directly in clojure using gen-class and
implementing the samza interfaces? Works great for us. Much simpler.
> On Feb 10, 2016, at 7:39 PM, Jacob Maes wrote:
>
> Hey Andy,
>
> That's good news! Do you mind sharing the revelation for any other Samza
> use
PM, Jacob Maes wrote:
>
> Hey Rick,
>
> If I understand your question, the goal is really to make sure there are no
> orphaned containers that continue to run "off the books".
>
> The newly added SAMZA-871 describes a heart beat mechanism to make sure
> orphaned cont
Security wouldn’t stop zombie processes from writing to kafka. I had this
problem with yarn before where the container thought it was killing jobs but
they never actually died, and in fact continued to write to kafka.
> On Feb 10, 2016, at 4:23 PM, Jagadish Venkatraman
> wrote:
>
> Hi John
>
Can I ask why you’re using samza and kafka for this?
> On Feb 4, 2016, at 7:41 PM, ramesh shanmugam
> wrote:
>
> Hi Team,
>
> We have use case that destination system accept only 160 messages per
> minute.
>
> Planning to use Samza with Kafka. Looking for out of box throttling feature
> but
There isn't really anything completely off the shelf that I've found. We put
everything in a Kafka topic and then have a job that puts the data in opentsdb.
> On Jan 26, 2016, at 12:53 AM, Elias Levy wrote:
>
> I am wondering if there are any off-the-shelf systems for consuming Samza
> metrics,
t; via a
end of stream message. It also seems like any windowing aggregation that
assumes there is no more data while streaming would need to be disabled. I
wasn't able to find much discussion on the topic, so looking for some
pointers.
Thanks
Rick
HI Navina,
It stopped happening once I deleted an old checkpoint topic. I think in the
rapid development cycle my checkpoints became invalid. If it happens again I
will save the logs.
Thanks!
> On Dec 21, 2015, at 2:14 PM, Navina Ramesh
> wrote:
>
> Hi Rick,
> Can you share
Hi all,
I just started seeing these errors the other day. I am heavily refactoring my
code, but it works locally. I’m wondering if anyone has seen this error when
deploying to yarn.
This is in stderr log on my application master.
Exception in thread "AMRM Callback Handler Thread"
org.apache.h
task downstream with another job that can signal when each
sub-task has completed.
Perhaps someone else has a better solution.
HTH,
Rick
> On Dec 2, 2015, at 1:41 AM, Anton Polyakov wrote:
>
> Hi Rick
>
> But processing pipeline is more than 1 job working in parallel, not
>
The simplest way would be to have your job send a message to another kafka
topic when it’s done.
Rick
> On Dec 1, 2015, at 3:44 PM, Anton Polyakov wrote:
>
> Hi
>
> I am looking at Samza to process some incoming stream of trades. Processing
> pipeline is a complex DA
So I left it running in this state for 15 minutes and then it started up.
Usually the app takes about 10 seconds to initialize.
> On Nov 20, 2015, at 10:23 AM, Rick Mangi wrote:
>
> Just started getting this error this morning. On startup the consumer trying
> to reload
Just started getting this error this morning. On startup the consumer trying to
reload the kv-store changelog starts throwing this exception and keeps
restarting and failing.
2015-11-20 10:16:19 [main] TaskStorageManager [DEBUG] Stopping consumers for
stores.
2015-11-20 10:16:19 [main] Broker
I’m not sure you can do this for the purposes of consuming messages faster, but
I’m interested to hear if I’m mistaken.
You could spawn threads to do other work within a job though.
> On Nov 18, 2015, at 11:40 PM, Chen Song wrote:
>
> Thanks Navina
>
> So theoretically I can create a threa
That patch seems to have fixed the problem.
> On Nov 18, 2015, at 3:43 PM, Rick Mangi wrote:
>
> Sorry. Just read the bug. Yes, that makes sense. I deleted a bunch of topics
> and then hit this.
>
>
>> On Nov 18, 2015, at 3:42 PM, Rick Mangi wrote:
>>
&g
Sorry. Just read the bug. Yes, that makes sense. I deleted a bunch of topics
and then hit this.
> On Nov 18, 2015, at 3:42 PM, Rick Mangi wrote:
>
> I take that back, it happened again. Will try your patch.
>
>
>> On Nov 18, 2015, at 3:36 PM, Rick Mangi wrote:
>>
I take that back, it happened again. Will try your patch.
> On Nov 18, 2015, at 3:36 PM, Rick Mangi wrote:
>
> I seem to have solved it by only specifying a single zookeeper node in my job
> config. Maybe a race condition of some sort?
>
>
>> On Nov 18, 2015, a
I seem to have solved it by only specifying a single zookeeper node in my job
config. Maybe a race condition of some sort?
> On Nov 18, 2015, at 2:37 PM, Yi Pan wrote:
>
> Hi, Rick,
>
> I think that you are running into SAMZA-754. I have a RB available for it
> already.
Hi, getting things working on samza 0.10.0 finally :)
I’m seeing the following error about 1/4 of the time from run-job.sh when
starting jobs:
[yarnmaster01] out: 2015-11-17 14:56:00 KafkaSystemAdmin$ [INFO] Got metadata:
Map(__samza_coordinator_t-key-grouper_dev -> SystemStreamMetadata
[strea
when things go
wrong the entire topology doesn’t go down. We’re building the backbone for a
bunch of other teams to work off of and having everything go down at once, even
for a code deployment, just doesn’t work for us.
My 2 cents :)
Rick
> On Nov 13, 2015, at 10:26 AM, Aram Mkrtch
Thanks YI,
I opened SAMZA-815 <https://issues.apache.org/jira/browse/SAMZA-815>
Rick
> On Nov 13, 2015, at 2:08 AM, Yi Pan wrote:
>
> Hi, Rick,
>
> Yes, please open a JIRA w/ your configuration, deployment set up and
> sequence, and logs from JobRunner.
>
>
Hi Yi,
I pulled from master and built this morning.
Yes, that’s the output from JobRunner. I also tried setting a job.id to see if
this was an issue migrating from an old task checkpoint topic but I got the
same result.
Would you like me to open a jira ticket?
Thanks,
Rick
> On Nov
that would mean we can’t migrate completely until the
10.0 release unless we want to run snapshot code in production.
Thanks!
Rick
signature.asc
Description: Message signed with OpenPGP using GPGMail
Ah, much better :)
That worked. Thanks!
> On Nov 5, 2015, at 4:42 PM, Navina Ramesh wrote:
>
> Rick,
> I think you have to swap the order of your statements in order to generate
> the documentation. Please see my previous response in this email thread.
>
> Thanks!
> N
I openend https://issues.apache.org/jira/browse/SAMZA-809
<https://issues.apache.org/jira/browse/SAMZA-809>
Thanks Navina!
> On Nov 5, 2015, at 3:50 PM, Rick Mangi wrote:
>
> I can generate the docs but the links in the left hand nav to
> documentation/configurati
t;
> Thanks!
> Navina
>
> On Thu, Nov 5, 2015 at 12:41 PM, Rick Mangi wrote:
>
>> Beautiful.
>>
>> Yeah, I’m trying to build the docs right now to view the new config
>> options and it’s not working :)
>>
>> Looking forward to the release.
>
Beautiful.
Yeah, I’m trying to build the docs right now to view the new config options and
it’s not working :)
Looking forward to the release.
Rick
> On Nov 5, 2015, at 3:39 PM, Navina Ramesh wrote:
>
> Hi Rick,
> Yes. A lot has changed :) And precisely, why it is taking so l
Hi,
I’m wondering when the 0.10.0 release is planned for? So much has changed :)
Thanks,
Rick
signature.asc
Description: Message signed with OpenPGP using GPGMail
container would handle that key in the stream, then hit
a service on that node (potentially there could be more than 1 container on the
node of course…).
Thanks,
Rick
signature.asc
Description: Message signed with OpenPGP using GPGMail
Nope, we left it out.
Rick
> On Nov 3, 2015, at 11:04 AM, John Tipper wrote:
>
> Thanks Rick, much appreciated. Did you have to set
> yarn.resourcemanager.hostname or can you leave this out?
>
> Thanks,
>
> John
>
> ____
Hi John,
We just got this set up last week. I haven’t fully tested the failover but it
certainly works for testing out our samza jobs on a pretty large cluster.
All of the server names are fqdn in our live yarn-site.xml
Hope this helps,
Rick
yarn.resourcemanager.ha.enabled
true
Hi Jason,
It depends on how you’re deploying. I deploy on yarn via cloudera and my
logging statements wind up in the container log for each node.
You need to instantiate a logger for your class before writing to it. Look at
the log4j docs to see the right way to do that.
Rick
> On Ju
ote:
>
> I was almost there. Got it now. Thanks for your help Rick.
>
>
>
>
> Cheers,
>
>
>
>
> Jason
>
>
>
>
>
>
>
>
>
> On Friday, Jun 26, 2558 at 11:43, Rick Mangi , wrote:
> Hey Jason,
>
>
> If you
cation master gui.
hth,
Rick
> On Jun 26, 2015, at 1:39 PM, ja...@marketingscience.co wrote:
>
> Hello,
>
>
> I am working on a basic Samza task that pulls from one Kafka topic and writes
> to another. This task runs in Yarn but the Output topic does not contain any
&
35 matches
Mail list logo
