Thank you Kosala, this is my storm data directory of one supervisor node: [root@complicated-laugh:/srv/tmpvar/storm/data]$ tree . ├── nimbus │ ├── inbox │ └── stormdist │ └── eventsStreamerv1-66-1425605777 │ ├── stormcode.ser │ ├── stormconf.ser │ └── stormjar.jar ├── supervisor │ ├── isupervisor │ │ ├── 1425642310552 │ │ ├── 1425642310552.version │ │ ├── 1425646891479 │ │ ├── 1425646891479.version │ │ ├── 1425652232940 │ │ ├── 1425652232940.version │ │ ├── 1425654754084 │ │ └── 1425654754084.version │ ├── localstate │ │ ├── 1425662664867 │ │ ├── 1425662664867.version │ │ ├── 1425662666276 │ │ ├── 1425662666276.version │ │ ├── 1425662666277 │ │ ├── 1425662666277.version │ │ ├── 1425662667867 │ │ └── 1425662667867.version │ ├── stormdist │ └── tmp └── workers
Are you saying I should delete them all (including subdirectories? Thanks AL On Thu, Mar 5, 2015 at 7:41 PM, Kosala Dissanayake <[email protected]> wrote: > Clear your storm data directory and restart your supervisors. > > On Fri, Mar 6, 2015 at 12:40 PM, Sa Li <[email protected]> wrote: > >> My storm is 0.9.3 by the way >> >> On Thu, Mar 5, 2015 at 5:39 PM, Sa Li <[email protected]> wrote: >> >>> Hi, All >>> >>> I am running a trident topo, and having such error, and it is strange >>> that I can't see the error on storm UI, but I see it on worker-6703.log of >>> one node, >>> >>> 2015-03-05T17:29:36.859-0800 b.s.m.n.Client [INFO] failed to send >>> requests to visible-alley/10.100.98.105:6703: >>> java.nio.channels.ClosedChannelException: null >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> [na:1.7.0_65] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> [na:1.7.0_65] >>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] >>> 2015-03-05T17:29:36.859-0800 b.s.m.n.Client [INFO] failed to send >>> requests to visible-alley/10.100.98.105:6703: >>> java.nio.channels.ClosedChannelException: null >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:405) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) >>> [storm-core-0.9.3.jar:0.9.3] >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> [na:1.7.0_65] >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> [na:1.7.0_65] >>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] >>> >>> >>> and such error on another node: >>> 2015-03-05T17:33:52.892-0800 b.s.d.worker [ERROR] Error on >>> initialization of server mk-worker >>> java.io.FileNotFoundException: File >>> '/srv/tmpvar/storm/data/supervisor/stormdist/eventsStreamerv1-65-1425604313/stormconf.ser' >>> does not exist >>> at >>> org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) >>> ~[commons-io-2.4.jar:2.4] >>> at >>> org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) >>> ~[commons-io-2.4.jar:2.4] >>> at >>> backtype.storm.config$read_supervisor_storm_conf.invoke(config.clj:212) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.daemon.worker$worker_data.invoke(worker.clj:182) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at >>> backtype.storm.daemon.worker$fn__3743$exec_fn__1108__auto____3744.invoke(worker.clj:363) >>> ~[storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.AFn.applyToHelper(AFn.java:185) >>> [clojure-1.5.1.jar:na] >>> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] >>> at clojure.core$apply.invoke(core.clj:617) >>> ~[clojure-1.5.1.jar:na] >>> at >>> backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.RestFn.invoke(RestFn.java:512) >>> [clojure-1.5.1.jar:na] >>> at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.AFn.applyToHelper(AFn.java:172) >>> [clojure-1.5.1.jar:na] >>> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] >>> at backtype.storm.daemon.worker.main(Unknown Source) >>> [storm-core-0.9.3.jar:0.9.3] >>> 2015-03-05T17:33:52.902-0800 b.s.util [ERROR] Halting process: ("Error >>> on initialization") >>> java.lang.RuntimeException: ("Error on initialization") >>> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.RestFn.invoke(RestFn.java:423) >>> [clojure-1.5.1.jar:na] >>> at >>> backtype.storm.daemon.worker$fn__3743$mk_worker__3799.doInvoke(worker.clj:354) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.RestFn.invoke(RestFn.java:512) >>> [clojure-1.5.1.jar:na] >>> at backtype.storm.daemon.worker$_main.invoke(worker.clj:461) >>> [storm-core-0.9.3.jar:0.9.3] >>> at clojure.lang.AFn.applyToHelper(AFn.java:172) >>> [clojure-1.5.1.jar:na] >>> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] >>> at backtype.storm.daemon.worker.main(Unknown Source) >>> [storm-core-0.9.3.jar:0.9.3] >>> >>> This is my yaml, >>> >>> storm.zookeeper.servers: >>> - "10.100.98.100" >>> - "10.100.98.101" >>> - "10.100.98.102" >>> - "10.100.98.103" >>> - "10.100.98.104" >>> - "10.100.98.105" >>> - "10.100.98.106" >>> >>> storm.zookeeper.port: 2181 >>> #nimbus.host: "localhost" >>> nimbus.thrift.port: 6627 >>> ui.port: 8080 >>> nimbus.host: "10.100.98.103" >>> storm.local.dir: "/srv/tmpvar/storm/data" >>> #java.library.path: "/usr/local/lib" >>> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64" >>> supervisor.slots.ports: >>> - 6700 >>> - 6701 >>> - 6702 >>> - 6703 >>> nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" >>> ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >>> supervisor.childopts: "-Djava.net.preferIPv4Stack=true" >>> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >>> >>> >>> Do I need to add netty stuff? >>> >>> thanks >>> >>> AL >>> >>> >> >
