Re: How to increase supervisor slots in apache storm 0.9.2

[Jan Sipke van der Veen]

Hello Sridhar,

I think adding that to your storm.yaml should suffice. Have you restarted
the supervisor afterwards? How much memory does your machine have? Each
slot is essentially a new Java virtual machine which takes up memory.

Best regards,
Jan Sipke van der Veen
On Oct 8, 2014 8:56 AM, "Sridhar G"  wrote:

> Hi All,
>
>
>
> We are unable to increase the number of supervisor slots . We tried to add
> following lines in storm.yaml file but did not work. Please suggest us .
>
>
>
> supervisor.slots.ports:
>
> - 6700
>
> - 6701
>
> - 6702
>
> - 6703
>
> - 6704
>
> - 6705
>
> - 6706
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
>
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message. Although Microland has taken reasonable
> precautions to ensure no viruses are present in this email, Microland
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments. Computer viruses can be transmitted via email.
> Recipient should check the email and any attachments for the presence of
> viruses before using them. Any views or opinions are solely those of the
> author and do not necessarily represent those of Microland.
>
>
> This email may be monitored.
>

Re: How to increase supervisor slots in apache storm 0.9.2

[Dima Dragan]

Hi Sridhar,

I think you should make rebalancing after changing storm.yaml.

Check please:
http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#add-another-worker-node

Best regards,
Dmytro Dragan
On Oct 8, 2014 9:56 AM, "Sridhar G"  wrote:

> Hi All,
>
>
>
> We are unable to increase the number of supervisor slots . We tried to add
> following lines in storm.yaml file but did not work. Please suggest us .
>
>
>
> supervisor.slots.ports:
>
> - 6700
>
> - 6701
>
> - 6702
>
> - 6703
>
> - 6704
>
> - 6705
>
> - 6706
>
>
>
> Regards,
>
> Sridhar
>
> *Believe in the impossible and remove the improbable*
>
>
>
> The information contained in this transmission may contain privileged and
> confidential information of Microland Limited, including information
> protected by privacy laws. It is intended only for the use of Microland
> Limited. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution, or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message. Although Microland has taken reasonable
> precautions to ensure no viruses are present in this email, Microland
> cannot accept responsibility for any loss or damage arising from the use of
> this email or attachments. Computer viruses can be transmitted via email.
> Recipient should check the email and any attachments for the presence of
> viruses before using them. Any views or opinions are solely those of the
> author and do not necessarily represent those of Microland.
>
>
> This email may be monitored.
>

RE: How to increase supervisor slots in apache storm 0.9.2

[Sridhar G]

Hi All,

 

Getting following error :

 

in 'reader', line 41, column 1:

supervisor.slots.ports:

^

 

at 
org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)

at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)

at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)

at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:108)

at 
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)

at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)

at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)

at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:148)

at backtype.storm.utils.Utils.readStormConfig(Utils.java:195)

at backtype.storm.config$read_storm_config.invoke(config.clj:121)

at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)

at clojure.lang.AFn.applyToHelper(AFn.java:161)

at clojure.lang.AFn.applyTo(AFn.java:151)

at backtype.storm.command.config_value.main(Unknown Source) 
   Exception in thread "main" 
expected '', but found BlockMappingStart

in 'reader', line 41, column 1:

supervisor.slots.ports:

^

 

at 
org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)

at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)

at org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)

at org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:108)

at 
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)

at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)

at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)

at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:148)

at backtype.storm.utils.Utils.readStormConfig(Utils.java:195)

at backtype.storm.config$read_storm_config.invoke(config.clj:121)

at backtype.storm.command.config_value$_main.invoke(config_value.clj:22)

at clojure.lang.AFn.applyToHelper(AFn.java:161)

at clojure.lang.AFn.applyTo(AFn.java:151)

at backtype.storm.command.config_value.main(Unknown Source)

 

Regards,

Sridhar

Believe in the impossible and remove the improbable

 

From: Dima Dragan [mailto:dima.dra...@belleron.net] 
Sent: Wednesday, October 8, 2014 12:36 PM
To: user@storm.apache.org
Subject: Re: How to increase supervisor slots in apache storm 0.9.2

 

Hi Sridhar,

I think you should make rebalancing after changing storm.yaml.

Check please:
http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#add-another-worker-node

Best regards,
Dmytro Dragan

On Oct 8, 2014 9:56 AM, "Sridhar G"  wrote:

Hi All,

 

We are unable to increase the number of supervisor slots . We tried to add 
following lines in storm.yaml file but did not work. Please suggest us .

 

supervisor.slots.ports:

- 6700

- 6701

- 6702

- 6703

- 6704

- 6705

- 6706

 

Regards,

Sridhar

Believe in the impossible and remove the improbable

 

The information contained in this transmission may contain privileged and 
confidential information of Microland Limited, including information protected 
by privacy laws. It is intended only for the use of Microland Limited. If you 
are not the intended recipient, you are hereby notified that any review, 
dissemination, distribution, or duplication of this communication is strictly 
prohibited. If you are not the intended recipient, please contact the sender by 
reply email and destroy all copies of the original message. Although Microland 
has taken reasonable precautions to ensure no viruses are present in this 
email, Microland cannot accept responsibility for any loss or damage arising 
from the use of this email or attachments. Computer viruses can be transmitted 
via email. Recipient should check the email and any attachments for the 
presence of viruses before using them. Any views or opinions are solely those 
of the author and do not necessarily represent those of Microland. 

 

This email may be monitored.

Re: Is storm a good choice for our architecture?

[Albert Vila]

Now we are using Gearman, don't know if anyone succeed using Gearman to
populate the Spouts.

And one of my questions is if it's possible to isolate the execution of
some bolts on a specific machine, so all keyword extraction for document X
is done by the same machine it crawled it. Or maybe I should not be concern
about network traffic on a Storm cluster.

Regards

Albert

On 7 October 2014 13:26, padma priya chitturi 
wrote:

> Storm would be feasible to your business problem. You could actually
> design the topology in  such a way that few bolts would be doing the job of
> keyword extraction, another set of bolts doing language detection etc etc.
> You can apply you clusterin g and classification algorithms of Mahout on
> streams of data processed by bolts.
>
> But only thing that i am concerned is if your data would be coming from
> some datasource like kafka, that would be great. I don't think spouts
> reading data from files would be  the best fit.
>
> Regards,
> Padma Ch
>
> On Tue, Oct 7, 2014 at 3:56 PM, Albert Vila 
> wrote:
>
>> Hi
>>
>> I just came across Storm when I was trying to find solutions to scale our
>> current architecture.
>>
>> We are currently downloading and processing 6M documents per day from
>> online and social media. We have a different workflow for each type of
>> document, but some of the steps are keyword extraction, language detection,
>> clustering, classification, indexation,  We are using Gearman to
>> dispatch the job to workers.
>>
>> I'm wondering if we could integrate Storm on the current workflow and if
>> it's feasible. One of our main discussions are if we have to go to a fully
>> distributed architecture or to a semi-distributed one. I mean, distribute
>> everything or process some steps on the same machine (crawling, keyword
>> extraction, language detection, indexation). We don't know which one scales
>> more, each one has pros and cont.
>>
>> Now we have a semi-distributed one as we had network problems taking into
>> account the amount of data we were moving around. So now, all documents
>> crawled on server X, later on are dispatched through Gearman to the same
>> server, having all data on a Memcached locally.
>>
>> What do you think?
>> It's feasible to migrate to a Storm cluster?
>> Should we take into account the traffic among the Storm cluster?
>> Is there a way to isolate some bolts to be processed on the same machine
>> grouped by some field?
>>
>> Any help or comment will be appreciate. And If someone has had a similar
>> problem and has knowledge about the architecture approach will be more than
>> welcomed.
>>
>> Thanks
>>
>> Albert
>>
>
>


-- 
*Albert Vila*
R&D Manager & Software Developer


Tél. : +34 972 982 968

*www.augure.com*  | *Blog.* Reputation in action
 | *Twitter. *@AugureSpain

*Skype *: albert.vila | *Access map.* Augure Girona

Re: How to increase supervisor slots in apache storm 0.9.2

[Yuri Kostin]

You might have an extra characters there, spaces, tabs, etc. Try 
http://yamllint.com

On Oct 8, 2014, at 2:06 AM, Sridhar G  wrote:

> Hi All,
>  
> Getting following error :
>  
> in 'reader', line 41, column 1:
> supervisor.slots.ports:
> ^
>  
> at 
> org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)
> at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> at 
> org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
> at 
> org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:108)
> at 
> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)
> at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
> at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)
> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:148)
> at backtype.storm.utils.Utils.readStormConfig(Utils.java:195)
> at backtype.storm.config$read_storm_config.invoke(config.clj:121)
> at 
> backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
> at clojure.lang.AFn.applyToHelper(AFn.java:161)
> at clojure.lang.AFn.applyTo(AFn.java:151)
> at backtype.storm.command.config_value.main(Unknown Source)   
>  Exception in thread 
> "main" expected '', but found BlockMappingStart
> in 'reader', line 41, column 1:
> supervisor.slots.ports:
> ^
>  
> at 
> org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)
> at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
> at 
> org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)
> at 
> org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:108)
> at 
> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:120)
> at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)
> at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)
> at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:148)
> at backtype.storm.utils.Utils.readStormConfig(Utils.java:195)
> at backtype.storm.config$read_storm_config.invoke(config.clj:121)
> at 
> backtype.storm.command.config_value$_main.invoke(config_value.clj:22)
> at clojure.lang.AFn.applyToHelper(AFn.java:161)
> at clojure.lang.AFn.applyTo(AFn.java:151)
> at backtype.storm.command.config_value.main(Unknown Source)
>  
> Regards,
> Sridhar
> Believe in the impossible and remove the improbable
>  
> From: Dima Dragan [mailto:dima.dra...@belleron.net] 
> Sent: Wednesday, October 8, 2014 12:36 PM
> To: user@storm.apache.org
> Subject: Re: How to increase supervisor slots in apache storm 0.9.2
>  
> Hi Sridhar,
> 
> I think you should make rebalancing after changing storm.yaml.
> 
> Check please:
> http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#add-another-worker-node
> 
> Best regards,
> Dmytro Dragan
> 
> On Oct 8, 2014 9:56 AM, "Sridhar G"  wrote:
> Hi All,
>  
> We are unable to increase the number of supervisor slots . We tried to add 
> following lines in storm.yaml file but did not work. Please suggest us .
>  
> supervisor.slots.ports:
> - 6700
> - 6701
> - 6702
> - 6703
> - 6704
> - 6705
> - 6706
>  
> Regards,
> Sridhar
> Believe in the impossible and remove the improbable
>  
> The information contained in this transmission may contain privileged and 
> confidential information of Microland Limited, including information 
> protected by privacy laws. It is intended only for the use of Microland 
> Limited. If you are not the intended recipient, you are hereby notified that 
> any review, dissemination, distribution, or duplication of this communication 
> is strictly prohibited. If you are not the intended recipient, please contact 
> the sender by reply email and destroy all copies of the original message. 
> Although Microland has taken reasonable precautions to ensure no viruses are 
> present in this email, Microland cannot accept responsibility for any loss or 
> damage arising from the use of this email or attachments. Computer viruses 
> can be transmitted via email. Recipient should check the email and any 
> attachments for the presence of viruses before using them. Any views or 
> opinions are solely those of the author and do not necessarily represent 
> those of Microland. 
> 
>  
> 
> This email may be monitored.
> 

Re: Is storm a good choice for our architecture?

[Onur Ünlü]

Hi Albert,

You can use "local or shuffle grouping".

https://storm.incubator.apache.org/documentation/Concepts.html (stream
groupings)

Onur


On Wed, Oct 8, 2014 at 10:11 AM, Albert Vila  wrote:

> Now we are using Gearman, don't know if anyone succeed using Gearman to
> populate the Spouts.
>
> And one of my questions is if it's possible to isolate the execution of
> some bolts on a specific machine, so all keyword extraction for document X
> is done by the same machine it crawled it. Or maybe I should not be concern
> about network traffic on a Storm cluster.
>
> Regards
>
> Albert
>
> On 7 October 2014 13:26, padma priya chitturi 
> wrote:
>
>> Storm would be feasible to your business problem. You could actually
>> design the topology in  such a way that few bolts would be doing the job of
>> keyword extraction, another set of bolts doing language detection etc etc.
>> You can apply you clusterin g and classification algorithms of Mahout on
>> streams of data processed by bolts.
>>
>> But only thing that i am concerned is if your data would be coming from
>> some datasource like kafka, that would be great. I don't think spouts
>> reading data from files would be  the best fit.
>>
>> Regards,
>> Padma Ch
>>
>> On Tue, Oct 7, 2014 at 3:56 PM, Albert Vila 
>> wrote:
>>
>>> Hi
>>>
>>> I just came across Storm when I was trying to find solutions to scale
>>> our current architecture.
>>>
>>> We are currently downloading and processing 6M documents per day from
>>> online and social media. We have a different workflow for each type of
>>> document, but some of the steps are keyword extraction, language detection,
>>> clustering, classification, indexation,  We are using Gearman to
>>> dispatch the job to workers.
>>>
>>> I'm wondering if we could integrate Storm on the current workflow and if
>>> it's feasible. One of our main discussions are if we have to go to a fully
>>> distributed architecture or to a semi-distributed one. I mean, distribute
>>> everything or process some steps on the same machine (crawling, keyword
>>> extraction, language detection, indexation). We don't know which one scales
>>> more, each one has pros and cont.
>>>
>>> Now we have a semi-distributed one as we had network problems taking
>>> into account the amount of data we were moving around. So now, all
>>> documents crawled on server X, later on are dispatched through Gearman to
>>> the same server, having all data on a Memcached locally.
>>>
>>> What do you think?
>>> It's feasible to migrate to a Storm cluster?
>>> Should we take into account the traffic among the Storm cluster?
>>> Is there a way to isolate some bolts to be processed on the same machine
>>> grouped by some field?
>>>
>>> Any help or comment will be appreciate. And If someone has had a similar
>>> problem and has knowledge about the architecture approach will be more than
>>> welcomed.
>>>
>>> Thanks
>>>
>>> Albert
>>>
>>
>>
>
>
> --
> *Albert Vila*
> R&D Manager & Software Developer
>
>
> Tél. : +34 972 982 968
>
> *www.augure.com*  | *Blog.* Reputation in action
>  | *Twitter. *@AugureSpain
> 
> *Skype *: albert.vila | *Access map.* Augure Girona
> 
>



-- 
Onur Ünlü

RE: How to increase supervisor slots in apache storm 0.9.2

[Sridhar G]

I get following error when do storm rebalance 

 

 

Exception in thread "main" java.lang.IllegalArgumentException: Field
supervisor.slots.ports must be an Iterable of java.lang.Number

at
backtype.storm.ConfigValidation$1.validateField(ConfigValidation.java:63
)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:57)

at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at
clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)

at
clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)

at
backtype.storm.config$validate_configs_with_schemas.invoke(config.clj:11
7)

at
backtype.storm.config$read_storm_config.invoke(config.clj:122)

at
backtype.storm.command.rebalance$_main.doInvoke(rebalance.clj:43)

at clojure.lang.RestFn.invoke(RestFn.java:397)

at clojure.lang.AFn.applyToHelper(AFn.java:159)

at clojure.lang.RestFn.applyTo(RestFn.java:132)

at backtype.storm.command.rebalance.main(Unknown Source)

 

Regards,

Sridhar

Believe in the impossible and remove the improbable

 

From: Yuri Kostin [mailto:kost...@gmail.com] 
Sent: Wednesday, October 8, 2014 12:42 PM
To: user@storm.apache.org
Subject: Re: How to increase supervisor slots in apache storm 0.9.2

 

You might have an extra characters there, spaces, tabs, etc. Try
http://yamllint.com

 

On Oct 8, 2014, at 2:06 AM, Sridhar G  wrote:





Hi All,

 

Getting following error :

 

in 'reader', line 41, column 1:

supervisor.slots.ports:

^

 

at
org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserIm
pl.java:225)

at
org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)

at
org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)

at
org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:108)

at
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstru
ctor.java:120)

at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)

at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)

at
backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:148)

at backtype.storm.utils.Utils.readStormConfig(Utils.java:195)

at
backtype.storm.config$read_storm_config.invoke(config.clj:121)

at
backtype.storm.command.config_value$_main.invoke(config_value.clj:22)

at clojure.lang.AFn.applyToHelper(AFn.java:161)

at clojure.lang.AFn.applyTo(AFn.java:151)

at backtype.storm.command.config_value.main(Unknown Source)
Exception in thread "main" expected '', but found
BlockMappingStart

in 'reader', line 41, column 1:

supervisor.slots.ports:

^

 

at
org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserIm
pl.java:225)

at
org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)

at
org.yaml.snakeyaml.parser.ParserImpl.checkEvent(ParserImpl.java:143)

at
org.yaml.snakeyaml.composer.Composer.getSingleNode(Composer.java:108)

at
org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstru
ctor.java:120)

at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:481)

at org.yaml.snakeyaml.Yaml.load(Yaml.java:424)

at
backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:148)

at backtype.storm.utils.Utils.readStormConfig(Utils.java:195)

at
backtype.storm.config$read_storm_config.invoke(config.clj:121)

at
backtype.storm.command.config_value$_main.invoke(config_value.clj:22)

at clojure.lang.AFn.applyToHelper(AFn.java:161)

at clojure.lang.AFn.applyTo(AFn.java:151)

at backtype.storm.command.config_value.main(Unknown Source)

 

Regards,

Sridhar

Believe in the impossible and remove the improbable

 

From: Dima Dragan [mailto:dima.dra...@belleron.net] 
Sent: Wednesday, October 8, 2014 12:36 PM
To: user@storm.apache.org
Subject: Re: How to increase supervisor slots in apache storm 0.9.2

 

Hi Sridhar,

I think you should make rebalancing after changing storm.yaml.

Check please:
http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/#
add-another-worker-node
 

Best regards,
Dmytro Dragan

On Oct 8, 2014 9:56 AM, "Sridhar G" mailto:sridh...@microland.com> > wrote:

Hi All,

 

We are unable to increase the number of supervisor slots . We tried to
add following lines in storm.yaml file but did not work. Please suggest
us .

 

supervisor.slots.ports:

- 6700

- 6701

- 6702

- 6703

- 6704

- 6705

- 6706

 

Regards,

Sridhar

Believe in the impossible and remove the improbable

 

The inform

Confuse on setting spout name and get topology name in Trident topology

[vlhf刘海峰]

Hi,

I tried to use name() method to name the nodes in a trident topology, and I saw 
the generated bolts then has a name like b{N}-{name} in storm UI, but the spout 
was still shown as spout0. How to name the spout node?

Another question, in trident topology, via context.getStormId I can get the 
topology id, but is there any API to get the topology name?

Thanks

Re: Is storm a good choice for our architecture?

[Richards Peter]

You can use custom scheduler to execute specific tasks on a particular
machine. Please check the following mail thread:

https://groups.google.com/forum/m/#!topic/storm-user/n_-HZWSdcmM

Regards,
Richards Peter.

Multiple nimbus configuration possible?

[An Tran]

Hi, 

I am wondering if it's possible to set up storm with multiple nimbus in its 
configuration or not.  If possible, is it as simple as adding it to the 
storm.yml file? If this feature is not supported yet is there plans to support 
it in the near future?

Thanks,

An

Re: Multiple nimbus configuration possible?

[Harsha]

An,
  Its not possible to have multiple nimbus running at this point.
  Here is the JIRA for nimbus HA
  https://issues.apache.org/jira/browse/STORM-166 which is tracking
  the work to enable it.
Thanks,
Harsha

On Wed, Oct 8, 2014, at 03:37 AM, An Tran wrote:
> Hi, 
> 
> I am wondering if it's possible to set up storm with multiple nimbus in
> its configuration or not.  If possible, is it as simple as adding it to
> the storm.yml file? If this feature is not supported yet is there plans
> to support it in the near future?
> 
> Thanks,
> 
> An

Re: Multiple nimbus configuration possible?

[Harsha]

An,
  Its not possible to have multiple nimbus running at this point.
  Here is the JIRA for nimbus HA
  https://issues.apache.org/jira/browse/STORM-166 which is tracking
  the work to enable it.
Thanks,
Harsha

On Wed, Oct 8, 2014, at 03:37 AM, An Tran wrote:
> Hi, 
> 
> I am wondering if it's possible to set up storm with multiple nimbus in
> its configuration or not.  If possible, is it as simple as adding it to
> the storm.yml file? If this feature is not supported yet is there plans
> to support it in the near future?
> 
> Thanks,
> 
> An

java.lang.NullPointerException, at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor

[Stefano Pugnetti]

Hi all!

I am trying to use an HDFS state in a Trident topology, but I always get 
the exception I report below. I tried the following config values


topology.max.spout.pending=1
topology.trident.batch.emit.interval.millis=(500, then 1500, then 5000)

but nothing changed. If I remove the partitionPersist from my code, the 
topology runs without any problems. I originally had a parallelHint of 
3, now removed (I also cleared the zookeeper data directory between 
different trials). I am using storm-hdfs 0.1.2 and storm 
0.9.2-incubating, and running hadoop 2.5.1.


I am actually writing my code in clojure, but that's been no problem up 
to now.


Can anyone suggest where to look for my mistake?

Thanks a lot!

Stefano

===

2014-10-08 15:32:33 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: java.lang.NullPointerException
	at 
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]

at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.lang.NullPointerException: null
	at 
storm.trident.tuple.TridentTupleView.getValueByPointer(TridentTupleView.java:326) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
storm.trident.tuple.TridentTupleView.getValueByField(TridentTupleView.java:277) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
org.apache.storm.hdfs.trident.format.DelimitedRecordFormat.format(DelimitedRecordFormat.java:75) 
~[stormjar.jar:na]
	at 
org.apache.storm.hdfs.trident.HdfsState$HdfsFileOptions.execute(HdfsState.java:157) 
~[stormjar.jar:na]
	at 
org.apache.storm.hdfs.trident.HdfsState.updateState(HdfsState.java:289) 
~[stormjar.jar:na]
	at 
org.apache.storm.hdfs.trident.HdfsUpdater.updateState(HdfsUpdater.java:12) 
~[stormjar.jar:na]
	at 
org.apache.storm.hdfs.trident.HdfsUpdater.updateState(HdfsUpdater.java:9) ~[stormjar.jar:na]
	at 
storm.trident.planner.processor.PartitionPersistProcessor.finishBatch(PartitionPersistProcessor.java:98) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
storm.trident.planner.SubtopologyBolt.finishBatch(SubtopologyBolt.java:152) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
storm.trident.topology.TridentBoltExecutor.finishBatch(TridentBoltExecutor.java:252) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
storm.trident.topology.TridentBoltExecutor.checkFinish(TridentBoltExecutor.java:285) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
storm.trident.topology.TridentBoltExecutor.execute(TridentBoltExecutor.java:359) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]

... 6 common frames omitted
2014-10-08 15:32:33 b.s.d.executor [ERROR]
java.lang.RuntimeException: java.lang.NullPointerException
	at 
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at 
backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
	at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]

at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.lang.NullPointerException: null
	at 
storm.trident.tuple.TridentTupleView.getValueByPointer(TridentTupleView.java:326) 
~[storm-core-0.9.2-incubating.jar:0.9.2-incubati

Kryo

[Klausen Schaefersinho]

Hi,

Every time I want to test a simple topology I get the following error which
seems to be caused by different versions of the Kryo dependency. I have a
dependency on my code on kryo (2.17), which I thought is the same version
as Storm. But obviously not...

Here is the exception:



b.s.d.worker - Error on initialization of server mk-worker
java.lang.NoSuchMethodError:
backtype.storm.serialization.DefaultKryoFactory$KryoSerializableDefault.setReferences(Z)Z
at
backtype.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:49)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.serialization.KryoValuesDeserializer.(KryoValuesDeserializer.java:33)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.serialization.KryoTupleDeserializer.(KryoTupleDeserializer.java:39)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_executor_data$fn__3189.invoke(executor.clj:212)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$assoc_apply_self.invoke(util.clj:816)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
b.s.d.worker - Error on initialization of server mk-worker
java.lang.NoSuchMethodError:
backtype.storm.serialization.DefaultKryoFactory$KryoSerializableDefault.setReferences(Z)Z
at
backtype.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:49)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.serialization.KryoValuesDeserializer.(KryoValuesDeserializer.java:33)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.serialization.KryoTupleDeserializer.(KryoTupleDeserializer.java:39)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$mk_executor_data$fn__3189.invoke(executor.clj:212)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.util$assoc_apply_self.invoke(util.clj:816)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]


Best regards,

Klaus

Re: Kryo

[Nathan Leung]

0.9.2 release upgrades to kryo 2.21.

On Wed, Oct 8, 2014 at 12:13 PM, Klausen Schaefersinho <
klaus.schaef...@gmail.com> wrote:

> Hi,
>
> Every time I want to test a simple topology I get the following error
> which seems to be caused by different versions of the Kryo dependency. I
> have a dependency on my code on kryo (2.17), which I thought is the same
> version as Storm. But obviously not...
>
> Here is the exception:
>
>
>
> b.s.d.worker - Error on initialization of server mk-worker
> java.lang.NoSuchMethodError:
> backtype.storm.serialization.DefaultKryoFactory$KryoSerializableDefault.setReferences(Z)Z
> at
> backtype.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:49)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.serialization.KryoValuesDeserializer.(KryoValuesDeserializer.java:33)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.serialization.KryoTupleDeserializer.(KryoTupleDeserializer.java:39)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_executor_data$fn__3189.invoke(executor.clj:212)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$assoc_apply_self.invoke(util.clj:816)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> b.s.d.worker - Error on initialization of server mk-worker
> java.lang.NoSuchMethodError:
> backtype.storm.serialization.DefaultKryoFactory$KryoSerializableDefault.setReferences(Z)Z
> at
> backtype.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:49)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.serialization.KryoValuesDeserializer.(KryoValuesDeserializer.java:33)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.serialization.KryoTupleDeserializer.(KryoTupleDeserializer.java:39)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.executor$mk_executor_data$fn__3189.invoke(executor.clj:212)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.util$assoc_apply_self.invoke(util.clj:816)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>
>
> Best regards,
>
> Klaus
>

Re: Kryo

[Curtis Allen]

You might find this interesting
http://stackoverflow.com/questions/24244689/kryo-compatibility-in-storm-0-9-1and-titan-graph-database-0-4-2-with-apache-cass/24246845#24246845

On Wed, Oct 8, 2014 at 10:16 AM, Nathan Leung  wrote:

> 0.9.2 release upgrades to kryo 2.21.
>
> On Wed, Oct 8, 2014 at 12:13 PM, Klausen Schaefersinho <
> klaus.schaef...@gmail.com> wrote:
>
>> Hi,
>>
>> Every time I want to test a simple topology I get the following error
>> which seems to be caused by different versions of the Kryo dependency. I
>> have a dependency on my code on kryo (2.17), which I thought is the same
>> version as Storm. But obviously not...
>>
>> Here is the exception:
>>
>>
>>
>> b.s.d.worker - Error on initialization of server mk-worker
>> java.lang.NoSuchMethodError:
>> backtype.storm.serialization.DefaultKryoFactory$KryoSerializableDefault.setReferences(Z)Z
>> at
>> backtype.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:49)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.serialization.KryoValuesDeserializer.(KryoValuesDeserializer.java:33)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.serialization.KryoTupleDeserializer.(KryoTupleDeserializer.java:39)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.daemon.executor$mk_executor_data$fn__3189.invoke(executor.clj:212)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.util$assoc_apply_self.invoke(util.clj:816)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> b.s.d.worker - Error on initialization of server mk-worker
>> java.lang.NoSuchMethodError:
>> backtype.storm.serialization.DefaultKryoFactory$KryoSerializableDefault.setReferences(Z)Z
>> at
>> backtype.storm.serialization.DefaultKryoFactory.getKryo(DefaultKryoFactory.java:49)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.serialization.SerializationFactory.getKryo(SerializationFactory.java:51)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.serialization.KryoValuesDeserializer.(KryoValuesDeserializer.java:33)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.serialization.KryoTupleDeserializer.(KryoTupleDeserializer.java:39)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at
>> backtype.storm.daemon.executor$mk_executor_data$fn__3189.invoke(executor.clj:212)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>> at backtype.storm.util$assoc_apply_self.invoke(util.clj:816)
>> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
>>
>>
>> Best regards,
>>
>> Klaus
>>
>
>