worker client connection issue

2015-03-18 Thread Tousif
Hello, I see 1 worker on each supervisor node working for some period of time and than workers disconnects and all three workers started on one single supervisor. I have 3 node zk, 3 kafka brokers and 3 supervisors . i see these messages . Any config i need to tune? 2015-03-19 00:40:47 o.a.z.

Re: reasons for topology hanging

2015-03-18 Thread Harsha
local mode is more for development debugging mode. It has in-process zookeeper I am not sure how well that can handle few hundred meg per minute. --  Harsha On March 18, 2015 at 8:30:00 PM, clay teahouse (clayteaho...@gmail.com) wrote: Hi All, What could be the reasons for a topology hanging,

reasons for topology hanging

2015-03-18 Thread clay teahouse
Hi All, What could be the reasons for a topology hanging, under a somewhat heavy load (a few hundred mb per minute)? There is no error in the logs. I am using kafkaspout to pull data from kafka and a simple bolt to stream the data. My spout max pending is set to 1024. My topology is running in loc

Re: How to programatically activate a spout from inside the topology

2015-03-18 Thread xujiaxj
Many thanks, Jens. This is exactly what I need. -Jia On Wed, Mar 18, 2015 at 10:49 AM, Jens-U. Mozdzen wrote: > Hi Jia, > Zitat von xuji...@gmail.com: >> Hi, >> >> >> I am building a topology, in which it first needs to read some >> persisted data (accounts, recovery point, etc) before all th

Re: How to programatically activate a spout from inside the topology

2015-03-18 Thread Jens-U. Mozdzen
Hi Jia, Zitat von xuji...@gmail.com: Hi, I am building a topology, in which it first needs to read some persisted data (accounts, recovery point, etc) before all the bolts can start processing tuples. Ideally the spout starts to emit tuples only after all the required data is read int

How to programatically activate a spout from inside the topology

2015-03-18 Thread xujiaxj
Hi, I am building a topology, in which it first needs to read some persisted data (accounts, recovery point, etc) before all the bolts can start processing tuples. Ideally the spout starts to emit tuples only after all the required data is read into memory (maybe on spouts, maybe on bolts).

Re: too many open files error when use storm ui REST api

2015-03-18 Thread Harsha
Check your ulimit and increase if its too low and see this if happens again. --  Harsha On March 18, 2015 at 8:16:38 AM, hjh (apply...@163.com) wrote: Check your ulimit and increase if its too low and see this if happens again. --  Harsha

Re: How to use storm's HiveBolt

2015-03-18 Thread Grant Overby (groverby)
If storm and hive run as different users, check your permissions in HDFS. The storm user will need permissions to write to your table (this is likely under /apps/hive/warehouse ). Similarly, the hive user will need to be able to access and modify the files created by the storm user. This can cau

too many open files error when use storm ui REST api

2015-03-18 Thread hjh
Hello, I am using storm ui rest api to get some metrics. However when doing so, after some time the ui.log shows too man open files error. Using lsof -p and see a lot of information like "can't identify protocol". The python code used to fetch the data is list below. Could anyone help me pleas