Out of memory error on samza application container

2016-12-04 Thread Shekar Tippur
Samza yarn application container is getting killed due to out of memory error. Wondering of you guys can give me some pointers on how to troubleshoot this error. Background: I had to change the Kafka log location and restart kafka. There is no change in application or yarn configuration. org.apach

Re: Samza kinesis implementation

2016-09-13 Thread Shekar Tippur
aid that's all the guidance I can offer for now. Please let us know > if you have any questions. We would to be happy to review your design and > code contribution! > > Thanks! > Navina > > On Tue, Sep 13, 2016 at 5:17 PM, Shekar Tippur wrote: > > > Hello, > &g

Samza kinesis implementation

2016-09-13 Thread Shekar Tippur
Hello, I am looking for direction on implementing samza over Kinesis. I see that jira ticket is in unresolved state. https://issues.apache.org/jira/plugins/servlet/mobile#issue/SAMZA-489 I also saw that with Samza release 0.10 this implementation is in place. Appreciate any pointers on this. S

Re: Samza yarn job - cannot bind to local host

2016-08-10 Thread Shekar Tippur
Just following on the same thread, is there a feature/bug to force the jmxrmi port? This would be helpful to get the container metrics. - Shekar On Sat, Aug 6, 2016 at 6:14 PM, Shekar Tippur wrote: > Thanks a lot guys. It was a system problem. /etc/hosts file had wrong > permissions. loc

Re: Samza yarn job - cannot bind to local host

2016-08-06 Thread Shekar Tippur
r details about your cluster you can provide would be useful. > > -Jake > > On Fri, Aug 5, 2016 at 7:33 AM, Shekar Tippur wrote: > > > Any pointers on this please. I am completely blocked. > > > > - Shekar > > > > On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur wrote: > > > > > This server is not connected to vpn. > > > > > > - Shekar > > > > > >

Re: Samza yarn job - cannot bind to local host

2016-08-05 Thread Shekar Tippur
Any pointers on this please. I am completely blocked. - Shekar On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur wrote: > This server is not connected to vpn. > > - Shekar >

Re: Samza yarn job - cannot bind to local host

2016-08-04 Thread Shekar Tippur
This server is not connected to vpn. - Shekar

Re: Samza yarn job - cannot bind to local host

2016-08-04 Thread Shekar Tippur
Is there anyway we can disable this? - Shekar On Thu, Aug 4, 2016 at 4:02 PM, Shekar Tippur wrote: > Yi, > > I can do a curl or telnet to localhost but it is hard to simulate as it > spawns random ports. > The job fails. > > Here is the application log: > > 2016-08

Re: Samza yarn job - cannot bind to local host

2016-08-04 Thread Shekar Tippur
04 20:41:00 JmxServer [INFO] According to Util.getLocalHost.getHostName we are sprdargas102 - Shekar On Thu, Aug 4, 2016 at 2:23 PM, Yi Pan wrote: > Hi, Shekar, > > Please see my replies below: > > On Thu, Aug 4, 2016 at 12:18 PM, Shekar Tippur wrote: > > > Sorry .. g

Re: Samza yarn job - cannot bind to local host

2016-08-04 Thread Shekar Tippur
Sorry .. got trigger happy. Couple of observations: 1. rmi port seem to be random ones Even with task.opts and setting the port to 1099 2. Do I need to start rmiregistry manually? 3. I dont have any firewall settings. On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur wrote: > Yi > Than

Re: Samza yarn job - cannot bind to local host

2016-08-04 Thread Shekar Tippur
e your > configuration, especially task.opts? > > -Yi > > On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur wrote: > > > I am trying to submit a Samza job to yarn and I get a error: > > > > Exception in thread "main" java.io.IOException: Cannot bind to

Samza yarn job - cannot bind to local host

2016-08-03 Thread Shekar Tippur
I am trying to submit a Samza job to yarn and I get a error: Exception in thread "main" java.io.IOException: Cannot bind to URL [rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested excep

Re: Samza stops processing

2016-06-14 Thread Shekar Tippur
Weird, I see a response http://mail-archives.apache.org/mod_mbox/samza-dev/201606.mbox/browser but I dint get a email. Anyways, Jagadish - Thanks for responding. Yes. We were able to produce a message via console producer and consume it. - Shekar On Thu, Jun 9, 2016 at 1:42 AM, Shekar Tippur

Re: Samza - Invalid ContainerId: container_e09_1465882867895_0003_01_000001

2016-06-14 Thread Shekar Tippur
Looks like there is a Jira ticket with no conclusion https://issues.apache.org/jira/browse/SAMZA-750 Is there a workaround without upgrading the code? On Mon, Jun 13, 2016 at 11:25 PM, Shekar Tippur wrote: > > > Hello, > > Running Samza 8.0 with latest hadoop, I get the below

Re: Samza - Invalid ContainerId: container_e09_1465882867895_0003_01_000001

2016-06-13 Thread Shekar Tippur
> > Hello, > > Running Samza 8.0 with latest hadoop, I get the below exception. What do I > need to change to get this to work? > > [root@bus01 yarn]# /usr/hdp/2.3.4.7-4/hadoop-yarn/bin/yarn version > > Hadoop 2.7.1.2.3.4.7-4 > > > Exception in thread "main" java.lang.IllegalArgumentException: Inva

Samza - Invalid ContainerId: container_e09_1465882867895_0003_01_000001

2016-06-13 Thread Shekar Tippur
Hello, Running Samza 8.0 with latest hadoop, I get the below exception. What do I need to change to get this to work? [root@bus01 yarn]# /usr/hdp/2.3.4.7-4/hadoop-yarn/bin/yarn version Hadoop 2.7.1.2.3.4.7-4 Exception in thread "main" java.lang.IllegalArgumentException: Invalid ContainerId: co

Samza stops processing

2016-06-09 Thread Shekar Tippur
I have a strange issue. We just replaced failed zookeeper server. We killed the job in the interim and resubmitted the job to yarn. The events were getting processed but at some point, the events stopped processing. I have checked RM and application logs but they seem to be fine. We then restarte

Samza stops processing

2016-06-09 Thread Shekar Tippur
I have a strange issue. We just replaced failed zookeeper server. We killed the job in the interim and resubmitted the job to yarn. The events were getting processed but at some point, the events stopped processing. I have checked RM and application logs but they seem to be fine. We then restarte

Re: New Samza blog published - http://engineering.linkedin.com/performance/benchmarking-apache-samza-12-million-messages-second-single-node

2015-08-25 Thread Shekar Tippur
Ed, Nice work. On Tue, Aug 25, 2015 at 10:48 PM, Roger Hoover wrote: > Thanks for sharing! > > Tao, did you use YARN to run 15 containers or is there a way to have them > statically divide up the tasks? > > > > On Mon, Aug 24, 2015 at 12:03 PM, Ed Yakabosky < > eyakabo...@linkedin.com.invalid>

Re: Kill All Jobs

2015-08-06 Thread Shekar Tippur
Thanks Jordi. This really helps. - Shekar On Thu, Aug 6, 2015 at 12:21 AM, Jordi Blasi Uribarri wrote: > Hi, > > As a little present (and I know this is not the way to get the code in the > project, but I am new to this sharing). I just made a simple script to kill > all the jobs running in Sam

Re: Samza and sliding window

2015-07-24 Thread Shekar Tippur
in my local environment and verified > that the iterator is functioning with store.all(). > > -Yi > > On Thu, Jul 23, 2015 at 4:26 PM, Shekar Tippur wrote: > > > Yi, > > > > In my case, I am able to append to the key but I am not able to get the > > s

Re: Samza and sliding window

2015-07-23 Thread Shekar Tippur
Yi, In my case, I am able to append to the key but I am not able to get the store and iterate through. If you look at http://pastebin.com/fKGpHwW6, line 146, I am able to get the store value. but in window routine - line 187, I am unable to get the values from store. - Shekar

Re: Samza and sliding window

2015-07-23 Thread Shekar Tippur
Yi, I am new to Scala. While it is readable, I am not sure where you are incrementing the count per application? - Shekar On Wed, Jul 22, 2015 at 5:20 PM, Shekar Tippur wrote: > Thanks Yi. I got the pastebin link. > I am looking at it. > > Shekar > On Jul 22, 2015 5:09 PM,

Re: Samza and sliding window

2015-07-22 Thread Shekar Tippur
Thanks Yi. I got the pastebin link. I am looking at it. Shekar On Jul 22, 2015 5:09 PM, "Yi Pan" wrote: > Hmm... did you get my pastebin post? If not, I can send the diff directly > to your gmail account. > > On Wed, Jul 22, 2015 at 11:23 AM, Shekar Tippur wrote: >

Re: Samza and sliding window

2015-07-22 Thread Shekar Tippur
Yi, I am not sure if attachments gets filtered when sent to the group. I have seen this earlier as well. - Shekar

Re: Samza and sliding window

2015-07-22 Thread Shekar Tippur
ched diff file. > > If you have any further questions, please let me know. > > -Yi > > On Mon, Jul 20, 2015 at 12:35 PM, Shekar Tippur wrote: > >> Yi, >> >> Here is the config: >> http://pastebin.com/mCALEACs >> >> - Shekar >> >> O

Re: Samza and sliding window

2015-07-20 Thread Shekar Tippur
gt; On Mon, Jul 20, 2015 at 11:11 AM, Shekar Tippur wrote: > > > Yi, > > > > Thanks a lot. > > > > - Shekar > > >

Re: Samza and sliding window

2015-07-20 Thread Shekar Tippur
Yi, Thanks a lot. - Shekar

Re: Samza and sliding window

2015-07-17 Thread Shekar Tippur
this weekend. > > Thanks! > > -Yi > > On Fri, Jul 17, 2015 at 1:31 PM, Shekar Tippur wrote: > > > Any takers on this please? > > > > - Shekar > > >

Re: Samza and sliding window

2015-07-17 Thread Shekar Tippur
Any takers on this please? - Shekar

Re: Samza and sliding window

2015-07-08 Thread Shekar Tippur
:46 WindowTask [INFO] Processing Key 2015-07-08 17:36:46 WindowTask [INFO] Processing Key 2015-07-08 17:36:46 WindowTask [INFO] Processing Key On Mon, Jul 6, 2015 at 7:46 PM, Shekar Tippur wrote: > Yi, > > I see incoming messages. I see that the counts are getting aggregated as > wel

Re: Samza and sliding window

2015-07-06 Thread Shekar Tippur
? > 2) Is there any messages in the changelog topic? > > Could you also try to change the log4j level to DEBUG to see whether we can > see something in the log? > > Thanks! > > -Yi > > On Mon, Jul 6, 2015 at 4:43 PM, Shekar Tippur wrote: > > > Martin, > >

Re: Samza and sliding window

2015-07-06 Thread Shekar Tippur
treamTask is single-threaded, so you > don't have to worry about concurrency. If you want to delete all keys in > the store, you can do so. > > Martin > > On 3 Jul 2015, at 17:46, Shekar Tippur wrote: > > > Any answer on how to get all the kv values and reinitialis

Re: Samza and sliding window

2015-07-04 Thread Shekar Tippur
Any takers on this please? - Shekar On Fri, Jul 3, 2015 at 9:46 AM, Shekar Tippur wrote: > Any answer on how to get all the kv values and reinitialise the kv store? > > Had one more question on implementing sliding window. > > If i use a kv store like rocksdb, and I use y

Re: Samza and sliding window

2015-07-03 Thread Shekar Tippur
numbers get skewed? Is that a right assessment? On Thu, Jul 2, 2015 at 5:47 PM, Shekar Tippur wrote: > Also, next.getValue() or next.getKey() does not yield anything. > >KeyValueIterator i = store.all(); > > while(i.hasNext()){ > >Entry next = i.next(); > >

Re: Samza and sliding window

2015-07-02 Thread Shekar Tippur
Also, next.getValue() or next.getKey() does not yield anything. KeyValueIterator i = store.all(); while(i.hasNext()){ Entry next = i.next(); log.info("Removed Key", next.getValue()); } On Thu, Jul 2, 2015 at 5:36 PM, Shekar Tippur wrote: > Yi, &g

Re: Samza and sliding window

2015-07-02 Thread Shekar Tippur
on on the iterator, could you send us the log w/ the > exception? > > Thanks! > > -Yi > > On Thu, Jul 2, 2015 at 4:36 PM, Shekar Tippur wrote: > > > Yi, > > > > Looks like it is working now. There was a redundant line in the config. > > > > I am

Re: Samza and sliding window

2015-07-02 Thread Shekar Tippur
tsSeen = 0; i.close(); } How do I retrieve the key and is there a way to remove it? i.remove throws an exception. - Shekar On Wed, Jul 1, 2015 at 7:25 PM, Shekar Tippur wrote: > Yi, > > Here is my config file: > http://pastebin.com/Kf3C9E0h > > - S >

Re: Samza and sliding window

2015-07-01 Thread Shekar Tippur
Yi, Here is my config file: http://pastebin.com/Kf3C9E0h - S

Re: Samza and sliding window

2015-07-01 Thread Shekar Tippur
stores.store.msg.serde=string stores.store.changelog=kafka.argos-parser Is this correct? I am still getting the same exception. - Shekar On Wed, Jul 1, 2015 at 5:37 PM, Shekar Tippur wrote: > I do have this in init as well... > > public void init(Config config, TaskContext context) { >

Re: Samza and sliding window

2015-07-01 Thread Shekar Tippur
I do have this in init as well... public void init(Config config, TaskContext context) { store = (KeyValueStore) context.getStore("store"); } You are right. These are primitive types but I was trying to address this exception: Exception in thread "main" org.apache.samza.SamzaException: Mus

Re: Samza and sliding window

2015-07-01 Thread Shekar Tippur
KeyValueStore store; On Wed, Jul 1, 2015 at 3:55 PM, Yan Fang wrote: > So do you use the "store-name" as the kv storage name in your StreamTask > code? > > Fang, Yan > yanfang...@gmail.com > > On Wed, Jul 1, 2015 at 3:41 PM, Shekar Tippur wrote: > > > Yan, >

Re: Samza and sliding window

2015-07-01 Thread Shekar Tippur
Yan, yes. I do have it. - Shekar On Wed, Jul 1, 2015 at 3:09 PM, Yan Fang wrote: > Do you have > > serializers.registry.json.class > =org.apache.samza.serializers.JsonSerdeFactory > > in your config file? > > > Fang, Yan > yanfang...@gmail.com > > On Wed, J

Re: Samza and sliding window

2015-07-01 Thread Shekar Tippur
la:63) at org.apache.samza.job.JobRunner.run(JobRunner.scala:62) at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37) at org.apache.samza.job.JobRunner.main(JobRunner.scala) On Mon, Jun 29, 2015 at 12:41 PM, Shekar Tippur wrote: > Yi, > > My use case is more of the latter. Your explanation makes sens

Re: Samza and sliding window

2015-06-29 Thread Shekar Tippur
re, keyed by application name. Then, > resetting counter to zero can be done when next time the application > counter is updated again) > > Hope that makes sense. > > -Yi > > On Mon, Jun 29, 2015 at 10:06 AM, Shekar Tippur wrote: > > > Benjamin, > > > >

Re: Samza and sliding window

2015-06-29 Thread Shekar Tippur
Benjamin, Thanks for the explanation. We dont have any specific partition scheme as yet. We just have 2 topics - raw and processed and we use default partitioning scheme. Can you share any code snippet so I can understand it better? - Shekar

Re: Samza and sliding window

2015-06-29 Thread Shekar Tippur
r Samza > task implementation you should maintain windowing counts for each > application. > > Thanks > Milinda > > On Sun, Jun 28, 2015 at 8:48 PM, Shekar Tippur wrote: > > > Milinda, > > > > I see that the document you mentioned addresses win

Re: Samza and sliding window

2015-06-28 Thread Shekar Tippur
, Shekar Tippur wrote: > Never mind. I see it here: > > http://samza.apache.org/learn/documentation/0.8/container/windowing.html > > Thanks again Milinda. > > - Shekar > > On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur wrote: > >> Thanks Milinda. >> Is this

Re: Samza and sliding window

2015-06-26 Thread Shekar Tippur
Never mind. I see it here: http://samza.apache.org/learn/documentation/0.8/container/windowing.html Thanks again Milinda. - Shekar On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur wrote: > Thanks Milinda. > Is this feature available on 0.8 version of Samza? > > - Shekar > &

Re: Samza and sliding window

2015-06-26 Thread Shekar Tippur
; any helpers or utilities to handle window state maintenance. You have to > implement it on top of local storage or if you don't won't fault tolerance > you can keep the state in-memory too (as long as the state fit in memory). > > Thanks > Milinda > > On Fri, Jun 26,

Re: Samza and sliding window

2015-06-26 Thread Shekar Tippur
t; RocksDB, or even just in-memory? > > When you say "another topic", is this the topic consumed by the same Samza > job as your 5-minutes-job, or in a separate job? What is the relation > between the topic and the application name? > > Thanks, > > Fang, Ya

Samza and sliding window

2015-06-26 Thread Shekar Tippur
Hello, My apologies if I have raised it earlier. Here is the use case: I have a stream that is partitioned based on application name. I want to be able to count hte number of events happening for that particular application in the past 5 minutes (sliding window) and update either another topic or a

Re: 3 processed message per incoming message

2015-06-24 Thread Shekar Tippur
hanks, > > Fang, Yan > yanfang...@gmail.com > > On Tue, Jun 23, 2015 at 12:40 PM, Shekar Tippur wrote: > > > Yan, > > > > I have restarted Yarn but I still see the same issue. I will post this > > question on Yarn mailing list. > > yarn-...@hadoop.apache.org > > > > - Shekar > > >

Re: 3 processed message per incoming message

2015-06-23 Thread Shekar Tippur
Yan, I have restarted Yarn but I still see the same issue. I will post this question on Yarn mailing list. yarn-...@hadoop.apache.org - Shekar

Re: 3 processed message per incoming message

2015-06-22 Thread Shekar Tippur
Looks like when I kill an application ID, the next application ID in the pending queue takes over. As there are 2 running jobs, I see that for every in coming event, there are 2 outputs. a. How do I force to have just 1 running job b. How to force the number of application pending to a smaller numb

Re: 3 processed message per incoming message

2015-06-22 Thread Shekar Tippur
Yan, As suggested, I killed one of the application id's but looks like it respawns another one. Guessing I need to kill the application master and restart it? Cluster MetricsApps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory UsedMemory TotalMemory ReservedVCores UsedVCor

Re: [DISCUSS] Samza 0.9.1 release

2015-06-16 Thread Shekar Tippur
+1 On Jun 16, 2015 6:39 PM, "Percy Wegmann" wrote: > Thank you! > > Sent using CloudMagic< > https://cloudmagic.com/k/d/mailapp?ct=pi&cv=6.0.64&pv=8.2> > > On Tue, Jun 16, 2015 at 8:11 PM, Chris Riccomini > wrote: > +1 Here. > > On Tue, Jun 16, 2015 at 12:01 PM, Guozhang Wang > wrote: > > > Coo

Re: [DISCUSS] Samza 0.9.1 release

2015-06-16 Thread Shekar Tippur
Wang, I have not caught up but can you please highlight if there are any feature additions as well? - Shekar On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang wrote: > Hi all, > > We have been running a couple of our jobs against `0.9.1` branch last week > at LinkedIn with some critical bug fixe

Re: 3 processed message per incoming message

2015-06-16 Thread Shekar Tippur
This is what I see on Yarn monitoring page: As we can see, there are 9998 apps pending. There is some 10k limit we are hitting. I see only 1 app running. Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores Rese

3 processed message per incoming message

2015-06-15 Thread Shekar Tippur
Hello, I have a scenario where for each incoming message, I see 3 corresponding processed message. I am suspecting that I may have submitted the same job multiple number of times to Yarn. If this is the case, a. How do I detect it b. How do I kill redundant jobs? Could there be other reasons for

Re: Yarn jobs in accepted state

2015-05-20 Thread Shekar Tippur
- Shekar On Wed, May 20, 2015 at 9:35 PM, Guozhang Wang wrote: > Hello Shekar, > > The Apache mailing list blocks most attachments, could you send a link of > the screenshot here? > > Guozhang > > On Wed, May 20, 2015 at 7:14 PM, Shekar Tippur wrote: > > > Hello

Yarn jobs in accepted state

2015-05-20 Thread Shekar Tippur
Hello, After submitting Samza job to Yarn, I see a lot of jobs in accepted state. Please see the attached screenshot. Wondering if this is due to any missed setting. I see the jobs progressing but would it cause any harm? - Shekar

Re: Yarn redundancy

2015-05-15 Thread Shekar Tippur
; check 1) when you take the RM down, does the backup RM runs successfully? > > 2) if the backup RM runs successfully, can you see the Samza Application > > run in the Yarn UI (such as, localhost:8088?) 3) if can not see it, what > > does Samza's log say? > > > > Than

Re: Log rotation on Samza/yarn logs

2015-05-14 Thread Shekar Tippur
lingFileAppender. You > can > > change the log4j.xml to config as you want. Usually, daily base is good > > enough. > > > > Thanks, > > > > Fang, Yan > > yanfang...@gmail.com > > > > On Thu, May 14, 2015 at 10:56 AM, Shekar Tippur > wrote:

Re: Yarn redundancy

2015-05-14 Thread Shekar Tippur
I am not sure if I missed anything. - Shekar On Thu, May 14, 2015 at 3:06 PM, Yan Fang wrote: > Is the HA set correctly? The log looks like it's in the YARN setting side. > > Fang, Yan > yanfang...@gmail.com > > On Thu, May 14, 2015 at 12:29 PM, Shekar Tippur wrote: >

Re: Yarn redundancy

2015-05-14 Thread Shekar Tippur
Other observation I forgot to mention is that if I kill the rm and nm process, samza job seem to run properly. Only when 01 server is rebooted, I seem to encounter this error and as a result, no jobs get processed. - Shekar On Thu, May 14, 2015 at 12:14 PM, Shekar Tippur wrote: > Hello, &g

Yarn redundancy

2015-05-14 Thread Shekar Tippur
Hello, I have setup redundancy on resource manager based on this doc https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html I then shut down server 1 and was expecting that 02 server would take over. Instead I see this error. I am not sure if I am missing someth

Re: tkafka.common.ReplicaNotAvailableException on application logs

2015-05-14 Thread Shekar Tippur
the error logs this topic only has one replica 1018019532, which was down > and hence not available. > > Guozhang > > On Thu, May 14, 2015 at 5:16 AM, Shekar Tippur wrote: > > > Here is what I see on Kafka log: > > > > [2015-05-14 04:11:27,752] E

Log rotation on Samza/yarn logs

2015-05-14 Thread Shekar Tippur
Hello, How are the Samza/yarn logs rotated as a best practice? We seem to be filling our disk. Should I resort to a Linux logrotate utility or is there any log4j config we can leverage? - Shekar

Re: tkafka.common.ReplicaNotAvailableException on application logs

2015-05-14 Thread Shekar Tippur
, Shekar Tippur wrote: > Here is the complete log: > > http://pastebin.com/nX7twETm > > Interesting, I see a leader not available exception instead of the earlier > one. > > ./container_1431601903660_0001_01_02/samza-container-0.log:2015-05-14 > 04:53:41 BrokerParti

Re: tkafka.common.ReplicaNotAvailableException on application logs

2015-05-14 Thread Shekar Tippur
te the entire stacktrace/log? Where there any other errors ? > On Wed, May 13, 2015 at 6:04 PM Shekar Tippur wrote: > > > Hello, > > > > I seem to come across a issue with replication. We have 2 nodes where > Kafka > > and yarn run. > > > > We have enabled

tkafka.common.ReplicaNotAvailableException on application logs

2015-05-13 Thread Shekar Tippur
Hello, I seem to come across a issue with replication. We have 2 nodes where Kafka and yarn run. We have enabled replication factor on Kafka (Replication factor = 2). For testing redundancy, we shutdown broker01 server. On the yarn application logs, we see the exception kafka.common.ReplicaNotAva

Re: How do you serve the data computed by Samza?

2015-04-01 Thread Shekar Tippur
Infrastructure Engineer > Distributed Data Systems > LinkedIn > > f...@linkedin.com > linkedin.com/in/felixgv > > > From: Shekar Tippur [ctip...@gmail.com] > Sent: Wednesday, April 01, 2015 6:54 AM > To: dev@samza.apache.org &g

Re: Samza on Yarn

2015-04-01 Thread Shekar Tippur
Fri, Mar 13, 2015 at 1:28 PM, Shekar Tippur wrote: > After adding classpath to yarn-site.xml, I found that the jars that were > created for argos (extension of Samza) was not part of the tar.gz file that > was exposed to Http request. I changed the post install script on rpm to >

Re: How do you serve the data computed by Samza?

2015-04-01 Thread Shekar Tippur
V store and serving >> against it, creating a materialized view in the same fashion. >> >> I imagine that a (forthcoming?) transaction model in Kafka >> <https://cwiki.apache.org/confluence/display/KAFKA/ >> Transactional+Messaging+in+Kafka> >> could >>

Re: Kafka Question

2015-03-31 Thread Shekar Tippur
exist in > the 0.8.0 release. > > Cheers, > Chris > > On Mon, Mar 30, 2015 at 5:26 PM, Shekar Tippur wrote: > > > One more thing .. > > > > We have this configured as well. > > > > > > > systems.kafka.producer.metadata.broker.list=sprfarga

Re: Kafka Question

2015-03-30 Thread Shekar Tippur
One more thing .. We have this configured as well. systems.kafka.producer.metadata.broker.list=sprfargas102:6667,sprfargas103:6667 On Mon, Mar 30, 2015 at 5:08 PM, Shekar Tippur wrote: > Hello, > > I realise this is a Kafka question. Since there are quite a few Kafka > experts

Kafka Question

2015-03-30 Thread Shekar Tippur
Hello, I realise this is a Kafka question. Since there are quite a few Kafka experts here and the answer to this question can help Samza community as well, bringing this question here. We are testing Kafka redundancy (just 2 brokers for now). We have increased replication factor to 2 servers (Bo

Re: How do you serve the data computed by Samza?

2015-03-27 Thread Shekar Tippur
Felix/Jordan, 1 - 2 is exactly what I was looking for as well. I want to expose webservices call to Kafka/samza. As there is no concept of a session, I was wondering how to send back enriched data to the web services request. Or am I way off on this? Meaning, is this a completely wrong use case to

Re: Kafka partition key

2015-03-26 Thread Shekar Tippur
your dataset > by > > ‘columns’? > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic > > ? > > > > Richard > > > > > On Mar 26, 2015, at 8:22 AM, Shekar Tippur wrote: > > > > > >

Kafka partition key

2015-03-26 Thread Shekar Tippur
Hello, Want to confirm a basic understanding of Kafka. If I have a dataset that needs to be partitioned by 4 columns, then the progression is {topic1:partition_key1} -> {Group by samza on partition_key1} -> {topic2:partition_key2} -> {Group by samza on partition_key2} -> {topic3:partition_key3} -

Re: Submitting yarn job with custom properties

2015-03-24 Thread Shekar Tippur
e (HDFS, HTTP, etc), and have your job pull it > from there. > > Cheers, > Chris > > On Mon, Mar 23, 2015 at 4:10 PM, Shekar Tippur wrote: > > > I would like to decouple Samza properties with the custom ones (if > > possible). > > > > - Shekar > > &

Re: Submitting yarn job with custom properties

2015-03-23 Thread Shekar Tippur
I would like to decouple Samza properties with the custom ones (if possible). - Shekar On Mon, Mar 23, 2015 at 3:12 PM, Shekar Tippur wrote: > Hello, > > I have a custom properties file with name value pairs. > > name1 value1 > name2 value2 > name3 value3 > > I wa

Submitting yarn job with custom properties

2015-03-23 Thread Shekar Tippur
Hello, I have a custom properties file with name value pairs. name1 value1 name2 value2 name3 value3 I want to pass this to Yarn job. What is the best way to achieve this? This works well when we declare locally or thread job. - Shekar

Re: Samza on Yarn

2015-03-13 Thread Shekar Tippur
redundancy now. - Shekar On Fri, Mar 13, 2015 at 1:20 PM, Chris Riccomini wrote: > Hey Shekar, > > Awesome, thanks! Would love to get any doc updates that would be useful. > > Curious: what was wrong? > > Cheers, > Chris > > On Fri, Mar 13, 2015 at 1:00 PM, Shekar T

Re: Samza on Yarn

2015-03-13 Thread Shekar Tippur
_0001_02 exited with exitCode: 1 > > The _02 suffix indicates a non-AM failure (i.e. the Samza container > failed, not the Samza AM). Can you check the AM logs, and find the http:// > ... > link to the container logs? It should give a hint about why the container > failed

Re: Samza on Yarn

2015-03-12 Thread Shekar Tippur
java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. Looks like a common issue with yarn but not sure how to resolve as yet. - Shekar On Thu, Mar 12, 2015 at 1:44 PM, Shekar Tippur wrote: > Chris - Here it is. >

Re: Samza on Yarn

2015-03-12 Thread Shekar Tippur
oesn't, could you paste the line where the YARN NM is starting up, and > > dumps the full classpath? > > > > Cheers, > > Chris > > > > On Thu, Mar 12, 2015 at 10:17 AM, Shekar Tippur > wrote: > > > >> I think all these jars are in place (

Re: Samza on Yarn

2015-03-12 Thread Shekar Tippur
a patch, that'd be great! :) > > Cheers, > Chris > > On Wed, Mar 11, 2015 at 9:43 PM, Shekar Tippur wrote: > > > Here is the corresponding log: > > > > 2015-03-11 20:43:09,665 INFO [AsyncDispatcher event handler] > > localizer.LocalizedResource (Localize

Re: Samza on Yarn

2015-03-11 Thread Shekar Tippur
reTask.java:400) at java.util.concurrent.FutureTask.get(FutureTask.java:187) at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1030) at org.apache.hadoop.ipc.Client.call(Client.java:1384) ... 8 more On Wed, Mar 11, 2015 at 4:56 PM, Shekar Tippur wrote: > Hello, > &

Samza on Yarn

2015-03-11 Thread Shekar Tippur
Hello, Sorry to reopen this topic. I had setup yarn couple of months ago and cant seem to replicate this now. I see that I have done everything listed here http://samza.apache.org/learn/tutorials/0.7.0/run-in-multi-node-yarn.html I see this error on the application side Application application_

Re: Deserializing issue

2015-02-19 Thread Shekar Tippur
your json structure is fine… it’s just in the Map. > > Richard > > > On Feb 19, 2015, at 4:00 PM, Shekar Tippur wrote: > > > > Hello, > > > > As I am trying to extend what we have to other sources, I changed some > > code. Looks like deserializ

Deserializing issue

2015-02-19 Thread Shekar Tippur
Hello, As I am trying to extend what we have to other sources, I changed some code. Looks like deserialization is broken. Here is a sample stream input: {"creation_time":"2015-02-19T04:03:51Z","alert_policy_name":"Alert Policy","application_name":"PROD AWS","business_unit_l1":"CEG"} envelope.g

Re: Versioning beyond 0.9.0

2015-01-31 Thread Shekar Tippur
+1 On Sat, Jan 31, 2015 at 4:48 AM, Garry Turkington < g.turking...@improvedigital.com> wrote: > +1 > > -Original Message- > From: sriram [mailto:sriram@gmail.com] > Sent: 30 January 2015 23:29 > To: dev@samza.apache.org > Subject: Re: Versioning beyond 0.9.0 > > +1 > > On Fri, Jan 30

Re: Samza has graduated!

2015-01-23 Thread Shekar Tippur
Fantastic news. A big thanks to Chris and team. - Shekar On Thu, Jan 22, 2015 at 1:58 PM, Yan Fang wrote: > This is great !! So happy to see it grow and graduate ! ( And of course, > keep growing!) > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 > > On Thu, Jan 22, 2015 at 1:18 PM, sri