BTW, on friday I cleared all the messages manually in kafka for that particular topic by deleting the kafka-log and zk log. Is storm zk not pointing to the correct index for the messages that I am posting on kafka? how do i clear zk logs in storm as I have bunch of log files in zk logs.
-- Kushan Maskey 817.403.7500 M. Miller & Associates <http://mmillerassociates.com/> [email protected] On Sun, Nov 9, 2014 at 9:51 AM, Kushan Maskey < [email protected]> wrote: > I am using apache-storm-0.9.2-incubating and use the built in kafka spout. > I do not see any errors in worker logs. I got bunch of worker logs and > checked couple of them and dont see problems. > > -- > Kushan Maskey > 817.403.7500 > M. Miller & Associates <http://mmillerassociates.com/> > [email protected] > > On Sun, Nov 9, 2014 at 9:34 AM, Harsha <[email protected]> wrote: > >> Kushan, >> Which KafkaSpout are you using. Do you see any errors in worker >> logs . >> -Harsha >> >> >> On Sun, Nov 9, 2014, at 07:22 AM, Kushan Maskey wrote: >> >> THanks Harsha, here they are, >> >> >> # Licensed to the Apache Software Foundation (ASF) under one >> >> # or more contributor license agreements. See the NOTICE file >> >> # distributed with this work for additional information >> >> # regarding copyright ownership. The ASF licenses this file >> >> # to you under the Apache License, Version 2.0 (the >> >> # "License"); you may not use this file except in compliance >> >> # with the License. You may obtain a copy of the License at >> >> # >> >> # http://www.apache.org/licenses/LICENSE-2.0 >> >> # >> >> # Unless required by applicable law or agreed to in writing, software >> >> # distributed under the License is distributed on an "AS IS" BASIS, >> >> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. >> >> # See the License for the specific language governing permissions and >> >> # limitations under the License. >> >> >> ########### These MUST be filled in for a storm configuration >> >> # storm.zookeeper.servers: >> >> # - "server1" >> >> # - "server2" >> >> storm.zookeeper.servers: >> >> - "nmcxstrmd001" >> >> >> storm.local.dir: "/data/disk00/storm/localdir" >> >> >> nimbus.host: "nmcxstrmd001" >> >> >> supervisor.slots.ports: >> >> - 6700 >> >> - 6701 >> >> - 6702 >> >> - 6703 >> >> - 6704 >> >> - 6705 >> >> - 6706 >> >> - 6707 >> >> - 6708 >> >> - 6709 >> >> - 6710 >> >> - 6711 >> >> - 6712 >> >> - 6713 >> >> - 6714 >> >> - 6715 >> >> - 6716 >> >> - 6717 >> >> - 6718 >> >> - 6719 >> >> - 6720 >> >> - 6721 >> >> - 6722 >> >> - 6723 >> >> - 6724 >> >> - 6725 >> >> - 6726 >> >> - 6727 >> >> - 6728 >> >> supervisor.childopts: "-Xmx256m" >> >> worker.childopts: "-Xmx1024m" >> >> #how long supervisor will wait to ensure that a worker process is started >> >> supervisor.worker.start.timeout.secs: 120 >> >> #how long between heartbeats until supervisor considers that worker dead >> and tries to restart it >> >> supervisor.worker.timeout.secs: 30 >> >> #how frequently the supervisor checks on the status of the processes it's >> monitoring and restarts if necessary >> >> supervisor.monitor.frequency.secs: 3 >> >> #how frequently the supervisor heartbeats to the cluster state (for >> nimbus) >> >> supervisor.heartbeat.frequency.secs: 5 >> >> supervisor.enable: true >> >> >> # >> >> # nimbus.host: "nimbus" >> >> # >> >> # >> >> # ##### These may optionally be filled in: >> >> # >> >> ## List of custom serializations >> >> # topology.kryo.register: >> >> # - org.mycompany.MyType >> >> # - org.mycompany.MyType2: org.mycompany.MyType2Serializer >> >> # >> >> ## List of custom kryo decorators >> >> # topology.kryo.decorators: >> >> # - org.mycompany.MyDecorator >> >> # >> >> ## Locations of the drpc servers >> >> # drpc.servers: >> >> # - "server1" >> >> # - "server2" >> >> >> ## Metrics Consumers >> >> # topology.metrics.consumer.register: >> >> # - class: "backtype.storm.metric.LoggingMetricsConsumer" >> >> # parallelism.hint: 1 >> >> # - class: "org.mycompany.MyMetricsConsumer" >> >> # parallelism.hint: 1 >> >> # argument: >> >> # - endpoint: "metrics-collector.mycompany.org" >> >> >> >> and storm config properties taht I read from external file: >> >> >> zookeeper.server.host=zkserver123 >> >> zookeeper.server.port=2181 >> >> zookeeper.group.id=consumer_group >> >> zookeeper.session.timeout.ms=400 >> >> zookeeper.auto.commit.enable=false >> >> zookeeper.consumer.timeout.ms=500 >> >> >> zookeeper.auto.offset.reset=smallest >> >> kafka.topic.rtSalesMQ=my_topic_on_kafka >> >> >> >> storm.config.setMaxTaskParallelism=1 >> >> storm.config.setNumWorkers=4 >> >> storm.config.setMaxSpoutPending=4 >> >> storm.config.numAckers=1 >> >> storm.config.parallelism.hint.bolt=3 >> >> storm.config.parallelism.hint.spout=4 >> >> >> storm.config.receiver.buffer.size=32 >> >> storm.config.transfer.buffer.size=32 >> >> # 1024*64 = 65536 >> >> storm.config.executer.receiver.buffer.size=65536 >> >> >> storm.config.executer.transfer.buffer.size=65536 >> >> >> LMK if there is anything that is not correct. Thanks. >> >> -- >> Kushan Maskey >> 817.403.7500 >> M. Miller & Associates <http://mmillerassociates.com/> >> [email protected] >> >> On Sun, Nov 9, 2014 at 9:15 AM, Harsha <[email protected]> wrote: >> >> >> Kushan, >> Please share your storm.yaml and your KafkaSpout config. >> Thanks, >> Harsha >> >> >> On Sun, Nov 9, 2014, at 07:13 AM, Kushan Maskey wrote: >> >> I have a topology that runs and reads message from a kafka spout >> successfully. But when i deploy the same topology and run it on a server it >> does not read any message. This code use to run fine earlier like 2 days >> before. I am using same configure parameters properties as in the local. I >> am stuck and need help asap. Thank you in advance. >> >> -- >> Kushan Maskey >> 817.403.7500 >> M. Miller & Associates <http://mmillerassociates.com/> >> [email protected] >> >> >> >> >> >> >> > >
