Similar tests with Spark and cassandra connector (which uses the native driver too) work well ... I was wondering if Yarn was restricting the network access but even executing spark (using spark-shell) with --master yarn-client or --master yarn-server worked without any issue. You could also see in the logs that it succeeded to connect to node006.internal.net
[cluster1-nio-worker-0]: DEBUG core.Connection: Connection[node006.internal.net/192.168.12.22:9042-1, inFlight=0, closed=false] Connection opened successfully [cluster1-nio-worker-1]: DEBUG Host.STATES: [node006.internal.net/192.168.12.22:9042] new connection created, total = 2 then later got [node006.internal.net/192.168.12.22:9042] Pool is shutdown However I've seen that before that, there is an error : [cluster1-blocking-task-worker-0]: DEBUG Host.STATES: Defuncting Connection[node006.internal.net/192.168.12.22:9042-3, inFlight=0, closed=true] because: [node006.internal.net/192.168.12.22:9042] Error while setting keyspace and after that error the node is marked as down : DEBUG Host.STATES: [node006.internal.net/192.168.12.22:9042] marking host DOWN still investigating ... > On Nov 23, 2015, at 19:31, Patrick McFadin <pmcfa...@gmail.com> wrote: > > That's a pretty basic error from the driver. I would look at the target node > and make sure that the IP/port combination is actually bound and free of any > firewall shenanigans. > > Patrick > > On Mon, Nov 23, 2015 at 4:00 PM, Cyril Scetbon <cyril.scet...@free.fr > <mailto:cyril.scet...@free.fr>> wrote: > Let me also add that I use Cassandra 2.1.11 with Hadoop 2.7.1 (HDP 2.3.2). > > > Trying to execute an Hadoop job on Yarn, I got errors from Cassandra's > > internal code. It seems that connections are closed but I can't understand > > why ... > > > > Here is a subtract of the errors : > > > > 15/11/22 20:05:54 [main]: DEBUG core.RequestHandler: Error querying > > node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>, trying next host (error > > is: com.datastax.driver.core.ConnectionException: > > [node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown) > > Failed with exception java.io.IOException:java.io.IOException: > > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > > tried > > for > > query failed (tried: node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042> > > (com.datastax.driver.core.ConnectionException: > > [node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown)) > > 15/11/22 20:05:54 [main]: ERROR CliDriver: Failed with exception > > java.io.IOException:java.io.IOException: > > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > > tried > > for > > query failed (tried: node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042> > > (com.datastax.driver.core.ConnectionException: > > [node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown)) > > java.io.IOException: java.io.IOException: > > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > > tried > > for > > query failed (tried: node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042> > > (com.datastax.driver.core.ConnectionException: > > [node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown)) > > at > > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508) > > at > > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415) > > at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140) > > at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672) > > at > > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) > > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165) > > at > > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) > > at > > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736) > > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) > > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > > Caused by: java.io.IOException: > > com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) > > tried > > for > > query failed (tried: node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042> > > (com.datastax.driver.core.ConnectionException: > > [node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown)) > > at > > org.apache.hadoop.hive.cassandra.input.cql.HiveCqlInputFormat.getRecordReader(HiveCqlInputFormat.java:132) > > at > > org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674) > > at > > org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324) > > at > > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446) > > ... 15 more > > Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: > > All host(s) tried > > for > > query failed (tried: node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042> > > (com.datastax.driver.core.ConnectionException: > > [node006.internal.net/192.168.12.22:9042 > > <http://node006.internal.net/192.168.12.22:9042>] Pool is shutdown)) > > at > > com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84) > > at > > com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37) > > at > > com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:214) > > at > > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) > > at > > com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) > > at > > org.apache.cassandra.hadoop.cql3.CqlRecordReader.fetchKeys(CqlRecordReader.java:578) > > at > > org.apache.cassandra.hadoop.cql3.CqlRecordReader.buildQuery(CqlRecordReader.java:526) > > at > > org.apache.cassandra.hadoop.cql3.CqlRecordReader.initialize(CqlRecordReader.java:148) > > I also add a file with the complete debug logs. > > > > You can also find the complete log at > > https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0 > > <https://www.dropbox.com/s/thwugpw8amv5x9q/output.log?dl=0> > >
