It looks like a networking thing. Is there anything interesting in the network config ? Are the nodes using a broadcast_address ? Can you telnet from the machine running sstableloader to port 7000 on 10.10.10.101 ?
I *think* sstableload will use the log4j-server.properties log config. Can you turn the logging up to debug and see what it days ? it looks like the sstableloader as connected to the cluster on the thrift / listen port 9160. It's then tried to run the transfer, which is when this happens…. > WARN 10:53:18,575 Failed attempt 1 to connect to /10.10.10.101 to stream > MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - > 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or > cannot assign requested address) Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 10/05/2012, at 9:01 PM, Pieter Callewaert wrote: > Firstly I disabled ipv6 on the server to be sure it wasn't trying to use the > ipv6, but no effect. > > I've tried using sstableloader on one of the cassandra nodes, no probem here, > worked perfectly! > So was doubting if the first server was corrupt or something, so I tried on > another server, CentOS 5.7 x64 with java 7 p04, which is not running a > Cassandra instance, and again I'm having problems streaming: > > [root@bms-web2 ~]# ./apache-cassandra-1.1.0/bin/sstableloader --debug -d > 10.10.10.100 MapData024/HOS/ > Streaming revelant part of MapData024/HOS/MapData024-HOS-hc-1-Data.db to > [/10.10.10.102, /10.10.10.100, /10.10.10.101] > > progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 > (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 10:53:18,575 Failed attempt 1 to > connect to /10.10.10.101 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db > sections=2 progress=0/6566400 - 0%. Retrying in 4000 ms. > (java.net.SocketException: Invalid argument or cannot assign requested > address) > WARN 10:53:18,577 Failed attempt 1 to connect to /10.10.10.102 to stream > MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6557280 - > 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or > cannot assign requested address) > WARN 10:53:18,594 Failed attempt 1 to connect to /10.10.10.100 to stream > MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6551840 - > 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or > cannot assign requested address) > progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 > (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 10:53:22,598 Failed attempt 2 to > connect to /10.10.10.101 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db > sections=2 progress=0/6566400 - 0%. Retrying in 8000 ms. > (java.net.SocketException: Invalid argument or cannot assign requested > address) > WARN 10:53:22,601 Failed attempt 2 to connect to /10.10.10.102 to stream > MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6557280 - > 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or > cannot assign requested address) > WARN 10:53:22,611 Failed attempt 2 to connect to /10.10.10.100 to stream > MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6551840 - > 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or > cannot assign requested address) > progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 > (0)] [total: 0 - 0MB/s (avg: 0MB/s)] > [root@bms-web2 ~]# java -version > java version "1.7.0_04" > Java(TM) SE Runtime Environment (build 1.7.0_04-b20) > Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode) > [root@bms-web2 ~]# cat /etc/redhat-release > CentOS release 5.7 (Final) > > Is it possible the sstableloader only works now if a Cassandra instance is > also running on the same server? The only other difference I see is CentOS > 6.2 vs CentOS 5.x > The new sstableloader, does it still use the Cassandra.yaml or is it > completely independent? > > Kind regards > > -----Original Message----- > From: Pieter Callewaert [mailto:pieter.callewa...@be-mobile.be] > Sent: woensdag 9 mei 2012 17:41 > To: user@cassandra.apache.org > Subject: RE: sstableloader 1.1 won't stream > > I don't see any entries in the logs of the nodes. > > I've disabled SELinux, to be sure this wasn't a blocking factor, and tried > adding -Djava.net.preferIPv4Stack=true to bin/sstableloader, but no change > unfortunately. > > To summarize, I'm trying to use sstableloader from a server (CentOS release > 5.8 (Final)) not running Cassandra to a 3-node Cassandra cluster. All running > 1.1. > My next step will be to try to use sstableloader on one of the nodes from the > cluster, to see if that works... > > If anyone has any other ideas, please share. > > Kind regards, > Pieter Callewaert > > -----Original Message----- > From: Sylvain Lebresne [mailto:sylv...@datastax.com] > Sent: woensdag 9 mei 2012 10:45 > To: user@cassandra.apache.org > Subject: Re: sstableloader 1.1 won't stream > > Have you checked for errors in the servers' logs? > > -- > Sylvain > > On Tue, May 8, 2012 at 1:24 PM, Pieter Callewaert > <pieter.callewa...@be-mobile.be> wrote: >> I've updated all nodes to 1.1 but I keep getting the same problem... >> Any other thoughts about this? >> >> Kind regards, >> Pieter >> >> -----Original Message----- >> From: Benoit Perroud [mailto:ben...@noisette.ch] >> Sent: maandag 7 mei 2012 22:21 >> To: user@cassandra.apache.org >> Subject: Re: sstableloader 1.1 won't stream >> >> You may want to upgrade all your nodes to 1.1. >> >> The streaming process connect to every living nodes of the cluster (you can >> explicitely diable some nodes), so all nodes need to speak 1.1. >> >> >> >> 2012/5/7 Pieter Callewaert <pieter.callewa...@be-mobile.be>: >>> Hi, >>> >>> >>> >>> I'm trying to upgrade our bulk load process in our testing env. >>> >>> We use the SSTableSimpleUnsortedWriter to write tables, and use >>> sstableloader to stream it into our cluster. >>> >>> I've changed the writer program to fit to the 1.1 api, but now I'm >>> having troubles to load them to our cluster. The cluster exists out >>> of one 1.1 node and two 1.0.9 nodes. >>> >>> >>> >>> I've enabled debug as parameter and in the log4j conf. >>> >>> >>> >>> [root@bms-app1 ~]# ./apache-cassandra/bin/sstableloader --debug -d >>> 10.10.10.100 /tmp/201205071234/MapData024/HOS/ >>> >>> INFO 16:25:40,735 Opening >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1 (1588949 bytes) >>> >>> INFO 16:25:40,755 JNA not found. Native methods will be disabled. >>> >>> DEBUG 16:25:41,060 INDEX LOAD TIME for >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1: 327 ms. >>> >>> Streaming revelant part of >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to >>> [/10.10.10.102, /10.10.10.100, /10.10.10.101] >>> >>> INFO 16:25:41,083 Stream context metadata >>> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6557280 - 0%], 1 sstables. >>> >>> DEBUG 16:25:41,084 Adding file >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed. >>> >>> INFO 16:25:41,087 Streaming to /10.10.10.102 >>> >>> DEBUG 16:25:41,092 Files are >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6557280 - 0% >>> >>> INFO 16:25:41,099 Stream context metadata >>> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6551840 - 0%], 1 sstables. >>> >>> DEBUG 16:25:41,100 Adding file >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed. >>> >>> INFO 16:25:41,100 Streaming to /10.10.10.100 >>> >>> DEBUG 16:25:41,100 Files are >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6551840 - 0% >>> >>> INFO 16:25:41,102 Stream context metadata >>> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=2 >>> progress=0/6566400 - 0%], 1 sstables. >>> >>> DEBUG 16:25:41,102 Adding file >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed. >>> >>> INFO 16:25:41,102 Streaming to /10.10.10.101 >>> >>> DEBUG 16:25:41,102 Files are >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=2 >>> progress=0/6566400 - 0% >>> >>> >>> >>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] >>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN >>> 16:25:41,107 Failed attempt 1 to connect to /10.10.10.101 to stream >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=2 >>> progress=0/6566400 - 0%. Retrying in 4000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.102 to >>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6557280 - 0%. Retrying in 4000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.100 to >>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6551840 - 0%. Retrying in 4000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] >>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN >>> 16:25:45,109 Failed attempt 2 to connect to /10.10.10.101 to stream >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=2 >>> progress=0/6566400 - 0%. Retrying in 8000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.102 to >>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6557280 - 0%. Retrying in 8000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.100 to >>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6551840 - 0%. Retrying in 8000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] >>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN >>> 16:25:53,113 Failed attempt 3 to connect to /10.10.10.101 to stream >>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=2 >>> progress=0/6566400 - 0%. Retrying in 16000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> WARN 16:25:53,114 Failed attempt 3 to connect to /10.10.10.102 to >>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6557280 - 0%. Retrying in 16000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> WARN 16:25:53,115 Failed attempt 3 to connect to /10.10.10.100 to >>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db >>> sections=1 >>> progress=0/6551840 - 0%. Retrying in 16000 ms. (java.net.SocketException: >>> Invalid argument or cannot assign requested address) >>> >>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] >>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] >>> >>> ... >>> >>> >>> >>> Anyone any idea what I'm doing wrong? >>> >>> >>> >>> Kind regards, >>> >>> Pieter Callewaert >> >> >> >> -- >> sent from my Nokia 3210 >> > > > >
