Re: bulk load problem
I couldn't get the same-host sstableloader to work either. But it's easier to
use the JMX bulk-load hook that's built
into Cassandra anyway. The following is what I implemented to do this:
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import javax.management.JMX;
import javax.management.MBeanServerConnection;
import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import org.apache.cassandra.service.StorageServiceMBean;
public class JmxBulkLoader {
private JMXConnector connector;
private StorageServiceMBean storageBean;
public JmxBulkLoader(String host, int port) throws Exception
{
connect(host, port);
}
private void connect(String host, int port) throws IOException,
MalformedObjectNameException
{
JMXServiceURL jmxUrl = new
JMXServiceURL(String.format("service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi", host,
port));
Map env = new HashMap();
connector = JMXConnectorFactory.connect(jmxUrl, env);
MBeanServerConnection mbeanServerConn =
connector.getMBeanServerConnection();
ObjectName name = new
ObjectName("org.apache.cassandra.db:type=StorageService");
storageBean = JMX.newMBeanProxy(mbeanServerConn, name,
StorageServiceMBean.class);
}
public void close() throws IOException
{
connector.close();
}
public void bulkLoad(String path) {
storageBean.bulkLoad(path);
}
public static void main(String[] args) throws Exception {
if (args.length == 0) {
throw new IllegalArgumentException("usage: paths to bulk files");
}
JmxBulkLoader np = new JmxBulkLoader("localhost", 7199);
for (String arg : args) {
np.bulkLoad(arg);
}
np.close();
}
}
On Jul 9, 2012, at 5:16 AM, Pushpalanka Jayawardhana wrote:
> Hi all,
>
> I am facing the same problem when trying to load Cassandra using
> sstableloader. I am running a Cassandra instance in my own machine and
> sstableloader is also called from the same machine. Following are the steps I
> followed.
>
> get a copy of the running Cassandra instance
> set another loopback address with "sudo ifconfig lo:0 127.0.0.2 netmask
> 255.0.0.0 up"
> set listen address and rpc address of the copied Cassandra's cassandra.yaml
> to 127.0.0.2
> ran "./sstableloader -d 127.0.0.2 "
> But this give me an error 'Could not retrieve endpoint ranges: ' and just
> that.
>
> I am so grateful for any hints to get over this.
> What I want to get done is actually running the sstableloader via a java
> code. But I couldn't get over it either and trying to understand the required
> args with this. It is great if someone can help me in either cases.
>
> Thanks in advance!
>
>
>
> On Tue, Jul 3, 2012 at 5:16 AM, aaron morton wrote:
> Do you have the full stack ? It will include a cause.
>
> Cheers
>
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/06/2012, at 12:07 PM, James Pirz wrote:
>
>> Dear all,
>>
>> I am trying to use "sstableloader" in cassandra 1.1.1, to bulk load some
>> data into a single node cluster.
>> I am running the following command:
>>
>> bin/sstableloader -d 192.168.100.1 /data/ssTable/tpch/tpch/
>>
>> from "another" node (other than the node on which cassandra is running),
>> while the data should be loaded into a keyspace named "tpch". I made sure
>> that the 2nd node, from which I run sstableloader, have the same copy of
>> cassandra.yaml as the destination node.
>> I have put
>>
>> tpch-cf0-hd-1-Data.db
>> tpch-cf0-hd-1-Index.db
>>
>> under the path, I have passed to sstableloader.
>>
>> But I am getting the following error:
>>
>> Could not retrieve endpoint ranges:
>>
>> Any hint ?
>>
>> Thanks in advance,
>>
>> James
>>
>>
>>
>
>
>
>
> --
> Pushpalanka Jayawardhana | Undergraduate | Computer Science and Engineering
> University of Moratuwa
> +94779716248 | http://pushpalankajaya.blogspot.com
> Twitter: http://twitter.com/Pushpalanka | Slideshare:
> http://www.slideshare.net/Pushpalanka
>
>
hadoop inserts blow out heap
I'm a fairly novice Cassandra/Hadoop guy. I have written a Hadoop job (using the Cassandra/Hadoop integration API) that performs a full table scan and attempts to populate a new table from the results of the map/reduce. The read works fine and is fast, but the table insertion is failing with OOM errors (in the Cassandra VM). The resulting heap dump from one node shows that 2.9G of the heap is consumed by a JMXConfigurableThreadPoolExecutor that appears to be full of batch mutations. I'm using a 6-node cluster, 32G per node, 8G heap, RF=3, if any of that matters. Any suggestions would be appreciated regarding configuration changes or additional information I might capture to understand this problem. Thanks Brian J
Re: hadoop inserts blow out heap
I didn't get a response to this, so I'll give it another shot. I tweaked some parameters and cleaned up my schema. My Hadoop/Cassandra job got further, but still dies with an OOM error. This time, the heap dump displays a JMXConfigurableThradPoolExecutor with a retained heap of 7.5G. I presume this means that the Hadoop job is writing to Cassandra faster than Cassandra can write to disk. Is there anything I can do to throttle the job? The Cassandra cluster is set up with default configuration values except for a reduced memtable size. Forgot to mention this is Cassandra 1.1.2 Thanks in advance. Brian On Sep 12, 2012, at 7:52 AM, Brian Jeltema wrote: > I'm a fairly novice Cassandra/Hadoop guy. I have written a Hadoop job (using > the Cassandra/Hadoop integration API) > that performs a full table scan and attempts to populate a new table from the > results of the map/reduce. The read > works fine and is fast, but the table insertion is failing with OOM errors > (in the Cassandra VM). The resulting heap dump from one node shows that > 2.9G of the heap is consumed by a JMXConfigurableThreadPoolExecutor that > appears to be full of batch mutations. > > I'm using a 6-node cluster, 32G per node, 8G heap, RF=3, if any of that > matters. > > Any suggestions would be appreciated regarding configuration changes or > additional information I might > capture to understand this problem. > > Thanks > > Brian J
cassandra/hadoop BulkOutputFormat failures
I'm trying to do a bulk load from a Cassandra/Hadoop job using the BulkOutputFormat class. It appears that the reducers are generating the SSTables, but is failing to load them into the cluster: 12/09/14 14:08:13 INFO mapred.JobClient: Task Id : attempt_201208201337_0184_r_04_0, Status : FAILED java.io.IOException: Too many hosts failed: [/10.4.0.6, /10.4.0.5, /10.4.0.2, /10.4.0.1, /10.4.0.3, /10.4.0.4] at org.apache.cassandra.hadoop.BulkRecordWriter.close(BulkRecordWriter.java:242) at org.apache.cassandra.hadoop.BulkRecordWriter.close(BulkRecordWriter.java:207) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:579) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) A brief look at the BulkOutputFormat class shows that it depends on SSTableLoader. My Hadoop cluster and my Cassandra cluster are co-located on the same set of machines. I haven't found any stated restrictions, but does this technique only work if the Hadoop cluster is distinct from the Cassandra cluster? Any suggestions on how to get past this problem? Thanks in advance. Brian
Re: cassandra/hadoop BulkOutputFormat failures
As suggested, it was a version-skew problem. Thanks. Brian On Sep 14, 2012, at 11:34 PM, Jeremy Hanna wrote: > A couple of guesses: > - are you mixing versions of Cassandra? Streaming differences between > versions might throw this error. That is, are you bulk loading with one > version of Cassandra into a cluster that's a different version? > - (shot in the dark) is your cluster overwhelmed for some reason? > > If the temp dir hasn't been cleaned up yet, you are able to retry, fwiw. > > Jeremy > > On Sep 14, 2012, at 1:34 PM, Brian Jeltema > wrote: > >> I'm trying to do a bulk load from a Cassandra/Hadoop job using the >> BulkOutputFormat class. >> It appears that the reducers are generating the SSTables, but is failing to >> load them into the cluster: >> >> 12/09/14 14:08:13 INFO mapred.JobClient: Task Id : >> attempt_201208201337_0184_r_04_0, Status : FAILED >> java.io.IOException: Too many hosts failed: [/10.4.0.6, /10.4.0.5, >> /10.4.0.2, /10.4.0.1, /10.4.0.3, /10.4.0.4] >> at >> org.apache.cassandra.hadoop.BulkRecordWriter.close(BulkRecordWriter.java:242) >> at >> org.apache.cassandra.hadoop.BulkRecordWriter.close(BulkRecordWriter.java:207) >> at >> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:579) >> at >> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:650) >> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) >> at org.apache.hadoop.mapred.Child.main(Child.java:249) >> >> A brief look at the BulkOutputFormat class shows that it depends on >> SSTableLoader. My Hadoop cluster >> and my Cassandra cluster are co-located on the same set of machines. I >> haven't found any stated restrictions, >> but does this technique only work if the Hadoop cluster is distinct from the >> Cassandra cluster? Any suggestions >> on how to get past this problem? >> >> Thanks in advance. >> >> Brian > >
updating CF from a mapper-only Hadoop job
I wrote a Hadoop mapper-only job that uses BulkOutputFormat to load a Cassandra table. That job would consistently fail with a flurry of exceptions (primary cause looks like EOFExceptions streaming between nodes). I restructured the job to use an identity mapper and perform the updates in the reducer. Same code other be moving to the reducer. That job runs correctly. Is this expected behavior? TIA Brian
Re: inconsistent hadoop/cassandra results
Sorry if this is a duplicate - I was having mailer problems last night: > Assuming their were no further writes, running repair or using CL all should > have fixed it. > > Can you describe the inconsistency between runs? Sure. The job output is generated by a single reducer and consists of a list of key/value pairs where the key is the row key of the original table, and the value is the total count of all columns in the row. Each run produces a file with a different size, and running a diff against various output file pairs displays rows that only appear in one file, or rows with the same key but different counts. What seems particularly hard to explain is the behavior after setting CL to ALL, where the results eventually become reproducible (making it hard to place the blame on my trivial mapper/reducer implementations) but only after about half a dozen runs. And once reaching this state, setting CL to QUORUM results in additional inconsistent results. I can say with certainty that there were no other writes. I'm the sole developer working with the CF in question. I haven't seen behavior like this before, though I don't have a tremendous amount of experience. But this is the first time I've tried to use the wide-row support, which makes me a little suspicious. The wide-row support is not very well documented, so maybe I'm doing something wrong there in ignorance. Brian > > Cheers > > - > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 8/01/2013, at 2:16 AM, Brian Jeltema > wrote: > >> I need some help understanding unexpected behavior I saw in some recent >> experiments with Cassandra 1.1.5 and Hadoop 1.0.3: >> >> I've written a small map/reduce job that simply counts the number of columns >> in each row of a static CF (call it Foo) >> and generates a list of every row and column count. A relatively small >> fraction of the rows have a large number >> of columns; worst case is approximately 36 million. So when I set up the >> job, I used wide-row support: >> >> ConfigHelper.setInputColumnFamily(job.getConfiguration(), "fooKS", >> "Foo", WIDE_ROWS); // where WIDE_ROWS == true >> >> When I ran this job using the default CL (1) I noticed that the results >> varied from run to run, which I attributed to inconsistent >> replicas, since Foo was generated with CL == 1 and the RF == 3. >> >> So I ran repair for that CF on every node. The cassandra log on every node >> contains lines similar to: >> >> INFO [AntiEntropyStage:1] 2013-01-05 20:38:48,605 AntiEntropyService.java >> (line 778) [repair #e4a1d7f0-579d-11e2--d64e0a75e6df] Foo is fully synced >> >> However, repeated runs were still inconsistent. Then I set CL to ALL, which >> I presumed would always result in identical >> output, but repeated runs initially continued to be inconsistent. However, I >> noticed that the results seemed to >> be converging, and after several runs (somewhere between 4 and 6) I finally >> was producing identical results on every run. >> Then I set CL to QUORUM, and again generated inconsistent results. >> >> Does this behavior make sense? >> >> Brian >
Re: problem with Cassandra map-reduce support
In hadoop-0.20.2, org.apache.hadoop.mapreduce.JobContext is a class. Looks like in hadoop-0.21+ JobContext has morphed into an interface. I'd guess that Hadoop support in Cassandra is based on the older Hadoop. Brian On Jan 29, 2013, at 3:42 AM, Tejas Patil wrote: > I am trying to run a map-reduce job to read data from Cassandra v1.2.0. > I started off with the code here: > https://svn.apache.org/repos/asf/cassandra/trunk/examples/hadoop_word_count/src/WordCount.java > > While running it over hadoop-0.22.0, I get this: > > Exception in thread "main" java.lang.IncompatibleClassChangeError: Found > interface org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.cassandra.hadoop.ColumnFamilyInputFormat.getSplits(ColumnFamilyInputFormat.java:103) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:445) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:462) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1045) > at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1042) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1042) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1062) > at MyHadoopApp.run(MyHadoopApp.java:163) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at MyHadoopApp.main(MyHadoopApp.java:82) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) > > Does anyone knows about this ? > > Thanks, > Tejas Patil
Re: cryptic exception in Hadoop/Cassandra job
Cassandra 1.1.5, using BulkOutputFormat Brian On Jan 30, 2013, at 7:39 AM, Pieter Callewaert wrote: > Hi Brian, > > Which version of cassandra are you using? And are you using the BOF to write > to Cassandra? > > Kind regards, > Pieter > > -Original Message- > From: Brian Jeltema [mailto:brian.jelt...@digitalenvoy.net] > Sent: woensdag 30 januari 2013 13:20 > To: user@cassandra.apache.org > Subject: cryptic exception in Hadoop/Cassandra job > > > I have a Hadoop/Cassandra map/reduce job that performs a simple > transformation on a table with very roughly 1 billion columns spread across > roughly 4 million rows. During reduction, I see a relative handful of the > following: > > Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: > java.io.EOFException > at > org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.EOFException > at java.io.DataInputStream.readInt(DataInputStream.java:375) > at > org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194) > at > org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 3 more > > which ultimately leads to job failure. I can't tell if this is a bug in my > code or in the underlying framework. > Does anyone have suggestions on how to debug this? > > TIA > > Brian > >
Re: cryptic exception in Hadoop/Cassandra job
I'm not sure this is the same problem. I'm getting these even when using a single reducer for the entire job. Brian On Jan 30, 2013, at 9:26 AM, Pieter Callewaert wrote: > I have the same issue (but with sstableloaders). > Should be fixed in 1.2 release > (https://issues.apache.org/jira/browse/CASSANDRA-4813) > > Kind regards, > Pieter > > > -----Original Message- > From: Brian Jeltema [mailto:brian.jelt...@digitalenvoy.net] > Sent: woensdag 30 januari 2013 13:58 > To: user@cassandra.apache.org > Subject: Re: cryptic exception in Hadoop/Cassandra job > > Cassandra 1.1.5, using BulkOutputFormat > > Brian > > On Jan 30, 2013, at 7:39 AM, Pieter Callewaert wrote: > >> Hi Brian, >> >> Which version of cassandra are you using? And are you using the BOF to write >> to Cassandra? >> >> Kind regards, >> Pieter >> >> -Original Message- >> From: Brian Jeltema [mailto:brian.jelt...@digitalenvoy.net] >> Sent: woensdag 30 januari 2013 13:20 >> To: user@cassandra.apache.org >> Subject: cryptic exception in Hadoop/Cassandra job >> >> >> I have a Hadoop/Cassandra map/reduce job that performs a simple >> transformation on a table with very roughly 1 billion columns spread across >> roughly 4 million rows. During reduction, I see a relative handful of the >> following: >> >> Exception in thread "Streaming to /10.4.0.3:1" java.lang.RuntimeException: >> java.io.EOFException >> at >> org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:662) >> Caused by: java.io.EOFException >> at java.io.DataInputStream.readInt(DataInputStream.java:375) >> at >> org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194) >> at >> org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:104) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> ... 3 more >> >> which ultimately leads to job failure. I can't tell if this is a bug in my >> code or in the underlying framework. >> Does anyone have suggestions on how to debug this? >> >> TIA >> >> Brian >> >> > > > >
Re: cassandra cqlsh error
I had this problem using a rather old version of Open JDK. I downloaded the Sun JDK and its working now. Brian On Feb 4, 2013, at 1:04 PM, Kumar, Anjani wrote: > Thank you Aaron! I uninstalled older version of Cassandra and then brought > 1.2.1 version of apache Cassandra as per your mail below. However, I am > receiving the following error while starting Cassandra. > anjani@anjani-laptop:~/apache-cassandra-1.2.1/bin$ ./cassandra > xss = -ea -javaagent:./../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 -Xms1005M -Xmx1005M -Xmn200M > -XX:+HeapDumpOnOutOfMemoryError -Xss180k > > > Thanks, > Anjani > > From: aaron morton [mailto:aa...@thelastpickle.com] > Sent: Monday, February 04, 2013 11:34 AM > To: user@cassandra.apache.org > Subject: Re: cassandra cqlsh error > > Grab 1.2.1, it's fixed there http://cassandra.apache.org/download/ > > Cheers > > - > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 5/02/2013, at 4:37 AM, "Kumar, Anjani" wrote: > > > > I am facing problem while trying to run cqlsh. Here is what I did: > > 1. I brought the tar ball files for both 1.1.7 and 1.2.0 version. > 2. Unzipped and untarred it > 3. Started Cassandra > 4. And then tried starting cqlsh but I am getting the following error > in both the versions: > Connection error: Invalid method name: ‘set_cql_version’ > > Before installing Datastax 1.1.7 and 1.2.0 cassandra, I had installed > Cassandra through “sudo apt-get install Cassandra” on my ubuntu. Since it > doesn’t have CQL support(at least I cant find it) so I thought of installing > Datastax version of Cassandra but still no luck starting cqlsh so far. Any > suggestion? > > Thanks, > Anjani > >
are CFs consistent after a repair
I'm confused about consistency. I have a 6-node group (RF=3) and I have a table that was known to be inconsistent across replicas (a Hadoop app was sensitive to this). So a did a 'nodetool -pr repair' on every node in the cluster. After the repairs were complete, the Hadoop app still indicated inconsistencies. Is this to be expected? Brian
Re: are CFs consistent after a repair
I'm afraid I can't investigate further; enough time has elapsed that it appears the inconsistencies have worked their way out of the cluster. On Feb 7, 2013, at 9:25 PM, aaron morton wrote: >> 'nodetool -pr repair' > Assuming nodetool repair -pr > > If there is no write activity all reads (at any CL level) will return the > same value after a successful repair. > > If there is write activity there is always a possibility of inconsistencies, > and so only access where R + W >N (e.g. QUORUM + QUROUM ) will be consistent. > > Can you drill down into the consistency problem? > > Cheers > > - > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 8/02/2013, at 7:01 AM, Brian Jeltema > wrote: > >> I'm confused about consistency. I have a 6-node group (RF=3) and I have a >> table that >> was known to be inconsistent across replicas (a Hadoop app was sensitive to >> this). >> >> So a did a 'nodetool -pr repair' on every node in the cluster. After the >> repairs were >> complete, the Hadoop app still indicated inconsistencies. Is this to be >> expected? >> >> Brian >
bulk load glitch
I'm attempting to perform a bulk load by calling the jmx:bulkLoad method on
several nodes in parallel. In a Casssandra log
file I see a few occurrences of the following:
INFO [GossipTasks:1] 2012-07-02 10:12:33,626 Gossiper.java (line 759)
InetAddress /10.4.0.3 is now dead.
ERROR [GossipTasks:1] 2012-07-02 10:12:53,630 StreamOutSession.java (line 232)
StreamOutSession /10.4.0.3 failed because {} died or was restarted/removed
INFO [GossipStage:1] 2012-07-02 10:18:40,008 Gossiper.java (line 745)
InetAddress /10.4.0.3 is now UP
I can't tell whether the bulk load process recovered from the transient dead
node, or whether I need to start over.
Does anybody know?
Brian
This is Cassanrda 1.0.5, by the way.
Re: Node won't start (faillout from AWS outage) - HELP
Just the opposite, I think. The property value exists in the yaml file but does not have a corresponding definition in the Config class. Typically caused by a version mismatch in my experience. On Jul 2, 2012, at 1:20 PM, Robin Verlangen wrote: > Your missing the "sliced_buffer_size_in_kb" property in your cassandra.yaml > file. > > 2012/7/2 Deno Vichas > i'm seeing the following exception in my log now. this is even after > re-creating my config file. > > > INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) > JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 > INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java (line 122) > Heap size: 1936719872/1937768448 > INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java (line 123) > Classpath: > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share > > /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar > INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA mlockall > successful > INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java (line 127) > Loading settings from file:/etc/cassandra/cassandra.yaml > ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java (line 481) Fatal > configuration error error > Can't construct a java object for > tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create > property=sliced_buffer_size_in_kb for > JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property > 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config > in "", line 4, column 1: > cluster_name: 'StockTouchCluster' > ^ > > at > org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) > at > org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) > at > org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136) > at > org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) > at org.yaml.snakeyaml.Loader.load(Loader.java:52) > at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) > at > org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:143) > at > org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:118) > at > org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) > at > org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) > Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create > property=sliced_buffer_size_in_kb for > JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property > 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config > at > org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) > at > org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) > at > org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java
Hadoop/Cassandra 1.2 timeouts
I'm having problems with Hadoop job failures on a Cassandra 1.2 cluster due to Caused by: TimedOutException() 2013-06-24 11:29:11,953 INFO Driver -at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12932) This is running on a 6-node cluster, RF=3. If I run the job with CL=ONE, it usually runs pretty well, with an occasional timeout. But if I run at CL=QUORUM, the number of timeouts is often enough to kill the job. The table being read is effectively read-only when this job runs. It has from 5 to 10 million rows, with each row having no more than 256 columns. Each column typically only has a few hundred bytes of data at most. I've fiddled with the batch-range size and increasing the timeout, without a lot of luck. I see some evidence of GC activity in the Cassandra logs, but it's hard to see a clear correlation with the timeouts. I could use some suggestions on an approach to pin down the root cause. TIA Brian
