Hi,
We are using Apache-storm 0.9.2 and the storm-kafka
(version 0.9.0-wip16a-scala292) which has support for Kafka 0.7.
I am trying to understand the failure handling of Kafka spout in a
particular scenario.
I have 4 workers, 1 running 1 executor of the Kafka spout, 1 running 1
executor of Bolt
Hi,
Few minutes after I sent out this mail, I noticed that the system recovered
and all pending messages were processed. However, I can't explain why it
took a long time to recover (almost 30 minutes after the erroneous bolt
died.)
One thing I noticed is that the Kafka spout re-established connec
Thanks Bobby!
Noam
On 25/09/14 16:35, Bobby Evans wrote:
0.9.1 to 0.9.2 is not possible to do a rolling upgrade. The data structures
stored in Zk and on disk have changed in a non-backwards compatible way. Most
EBFs we do rolling upgrades. For them we install the new yinst package on all
of
You can indeed register metrics in trident. In functions, override prepare(),
which gives you a TridentOperationContext. In aggregators, use the
IMetricsContext you get in makeState().
--
Yuval Oren
N3TWORK
> On Sep 26, 2014, at 4:21 PM, Raphael Hsieh wrote:
>
> I've been following the tutori
Gezim,
I manage a storm topology within our AWS env. However I use Chef, not lein. Not
sure how much help I can be.
To deploy a topology, I execute (from within Chef scripts) a Bash script which
runs ../bin/storm kill and then ../bin/storm jar . That’s how I handle it, and so far it’s worked
On a side note, if you try to register a metric in IRichSpout used in
Trident topology, you will get a RunTimeException.
See https://issues.apache.org/jira/browse/STORM-197
On Mon, Sep 29, 2014 at 5:39 PM, Yuval Oren wrote:
> You can indeed register metrics in trident. In functions, override
>
Hello,
How can I get the task ID of a bolt in a topology? I need to get it in
order to perform an emitDirect() from a Spout.
Thanks,
Nick
You get a handle to TopologyContext in the prepare method of your
spout/bolt.
http://nathanmarz.github.io/storm/doc/backtype/storm/task/TopologyContext.html#getThisTaskId()
On Mon, Sep 29, 2014 at 11:37 AM, Nick Katsipoulakis
wrote:
> Hello,
>
> How can I get the task ID of a bolt in a topology
Thank you very much Nathan.
Nick
On Mon, Sep 29, 2014 at 11:43 AM, Nathan Leung wrote:
> You get a handle to TopologyContext in the prepare method of your
> spout/bolt.
>
>
> http://nathanmarz.github.io/storm/doc/backtype/storm/task/TopologyContext.html#getThisTaskId()
>
> On Mon, Sep 29, 2014
Thanks Nathan for the explanation. Is there a way to get the actual no of
messages processed in trident from the storm gui or using the storm api ?
On Sat, Sep 27, 2014 at 9:20 PM, Nathan Marz wrote:
> Trident executes a batch every 500ms (by default). A batch involves a
> bunch of coordination
Thank you Yuval and Yair, this has been extremely helpful.
I am now running in to a problem where Storm thinks that the metric has
already been registered. I think this might be because of the parrallelism
in my topology, but I'm not sure. Do you have any ideas as to why this
might be?
I'm receivin
Actually nevermind, I think I figured it out. I was calling the same
function twice haha.
Thanks
On Mon, Sep 29, 2014 at 1:19 PM, Raphael Hsieh wrote:
> Thank you Yuval and Yair, this has been extremely helpful.
> I am now running in to a problem where Storm thinks that the metric has
> already
What is the benefit of using a metrics consumer and generating metrics, as
opposed to just creating a new bolt to log metrics ?
How does it affect the speed of the topology?
from what I can tell, registering a metricsConsumer will effectively spawn
a new bolt. Only this bolt has access to metrics
What is the benefit of using a metrics consumer and generating metrics, as
opposed to just creating a new bolt to log metrics ?
How does it affect the speed of the topology?
from what I can tell, registering a metricsConsumer will effectively spawn
a new bolt. Only this bolt has access to metrics
Think about the convenience that storm builtin metric utils brought to you.
You don’t need to handle time bucket yourself, or even calculate average.
发件人: Raphael Hsieh [mailto:raffihs...@gmail.com]
发送时间: 2014年9月30日 6:19
收件人: user@storm.apache.org
主题: Metrics consumer vs new bolt
What is the ben
15 matches
Mail list logo
