Re: Unit Testing Topologies

Hi, for integration testing you can look at https://github.com/xumingming/storm-lib/blob/master/src/jvm/storm/TestingApiDemo.java On Mon, Jun 29, 2015 at 12:11 PM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > I would recommend to write a unit test for each spout/bolt and an > integr

Re: Storm Message Flow Question

1 and b3 > in w3)? > > Simply put, does field grouping groups messages in whole topology? or only > groups in a single worker? > > Thanks, > Baek > > > > > > *Seungtack Baek | Precocity, LLC* > > Tel/Direct: (972) 378-1030 | Mobile: (214) 477-5715 > > *s

Re: Storm Message Flow Question

Hi, Seungtack! Distribution of messages will be depends only from grouping (in case of "shuffe grouping", Tuples are randomly distributed across the all bolt's tasks in a way such that each bolt is guaranteed to get an equal number of tuples. Best regards, Dmytro Dragan On Jun 8, 2015 07:12, "Seu

Re: Storm 0.9.4 cluster running on Amazon EC2 crashes with an unexpected Netty error

Hello Nick, Please check Security Groups for any limitations. Also you can setting up "storm.local.hostname" in storm.yml on worker machine for using only public IPs. Best regards, Dmytro Dragan On May 22, 2015 18:38, "Nick R. Katsipoulakis" wrote: > Hello Jeff, > > No, It seems that EC2 works

RE: 1 spout emit 2 tuples to 2 different bolt

Hi, You need go through https://storm.apache.org/documentation/Concepts.html Best regards, Dmytro Dragan On May 22, 2015 07:24, "prasad ch" wrote: > No , what ever spout emits all tuples will go all bolts . > but as per your requirement you consider required tuple in corresponding > bolt!. > >

Re: sharing data between tasks in the same bolt or spout

Hi, for 1 worker you can create a class with your static thread-safe collection and thread-safe initialization method (Double Checked Locking & volatile, for example) which return collection reference . In bolt`s prepare() method run init(). In this way all bolts will have the same ref to collecti

Re: Decreasing Complete latency with growing number of executors

ined they can process 2245 tuples / s > (note the values aren't precise to make things more legible). This means > it takes 0.445 seconds to process your tuples. Average complete latency is > 0.222ms. > > On Wed, May 20, 2015 at 8:12 AM, Dima Dragan > wrote: > >> Na

Re: Decreasing Complete latency with growing number of executors

p from more bolts) then you will see the > complete latency drop. > On May 20, 2015 4:01 AM, "Dima Dragan" wrote: > >> Thank you, Jeffrey and Devang for your answers. >> >> Jeffrey, as far as I use shuffle grouping, I think, network serialization >

Re: Decreasing Complete latency with growing number of executors

> enters the worker until it leaves the worker? I don't recall. >> >> I bet a firm definition of the latency terms would shed some light. >> >> Thank you for your time! >> >> + >> Jeff Maass >> linkedin.com/in/jeffmaass

Re: Decreasing Complete latency with growing number of executors

and the characteristics of the io. You > increased executors by 32x and each executor's throughput dropped by 5x, so > it makes sense that latency will drop. > On May 19, 2015 9:54 AM, "Dima Dragan" wrote: > >> Hi everyone, >> >> I have found a strange behavior in

Decreasing Complete latency with growing number of executors

Hi everyone, I have found a strange behavior in topology metrics. Let`s say, we have 1 node, 2-core machine. simple Storm topology Spout A -> Bolt B -> Bolt C Bolt B splits message on 320 parts and emits (shuffle grouping) each to Bolt C. Also Bolts B and C make some read/write operations to db

Re: Storm topology getting stuck

Hi Eran, Have you checked storm UI metrics? Is there capacity overload? Also please check log files for errors. Best regards, Dmytro Dragan On May 14, 2015 08:45, "Eran Chinthaka Withana" wrote: > Hi Nathan > > No I still didn't try jstack > > But I'm just wondering whether this is something t

Re: JDBC connection pool shared by bolts

Hi, You can create static class with thread-safe singleton initialization of connection pool and provide public access for reference. Take pool reference in prepare method. Take connection,execute smth, give it back in execute method. So every worker will get it's own pool. Best regards, Dmytro

Re: write data to file then bulk copy to database

) { > try { > > ps.close(); > } catch > (SQLException ex) { > } > } > } > } > } > > > Any idea? > > thanks > > On Fri, Dec 5

Re: write data to file then bulk copy to database

Hi, I think better approach is creating static class with thread-safe singleton initialization of connection pool (for example, I use HikariCP for SQL db and JedisPool for Redis) and provide public access for getting connection from it to bolt. Init pool in prepare method, take connection,execute

Re: Topology Code in C ?

Hi, I think this is answer to your question: http://stackoverflow.com/questions/19108769/how-to-write-c-spout-bolt-on-storm-and-thrift-usage-in-storm Best regards, Dmytro Dragan Hi, I am planning to use Apache Storm for communicating between multiples processes of different Virtual machines with

Re: Shared static data between bolts

Hi Eyal, I used the same approach as Michael. As far as I understand, your shared collection should be changeable. So do not forget use concurrent collection. Solution for copies by worker are: 1) Represent one collection as group of collections (collection per worker), using storm grouping ma

How to set up STORM_TEST_TIMEOUT_MS programatically in 0.9.3?

Hi All, Can anyone show snippet (example) how to use it? Is it work only with SimulatedTime mode? -- Best regards, Dmytro Dragan

Re: How to increase supervisor slots in apache storm 0.9.2

Hi Sridhar, I think you should make rebalancing after changing storm.yaml. Check please: http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#add-another-worker-node Best regards, Dmytro Dragan On Oct 8, 2014 9:56 AM, "Sridhar G" wrote: > Hi All, > > > > We are unable to inc