That does look like a bug http://www.mail-archive.com/d...@cassandra.apache.org/msg01167.html
It's certainly looks suspect. I've had a look at the code around SSTableImport and SSTableExport and the isDeleted value for the col is based on IColumn.isMarkedForDelete read when the data was exported. I'll try to have a look tonight, or if someone is still up in the states they may help. The current 0.7 branch has the same code.Are there any other errors in the log?The code sample you've got there uses the TimestampClocks that were once a part of the 0.7 development, but have been removed. So your data sample may not be usable going forwards. Can you jump to beta3?Aaron
On 17 Nov, 2010,at 02:03 PM, CassUser CassUser <cassu...@gmail.com> wrote:Looking at this closer. I noticed the following in the SSTableImport Class:
if (col.isDeleted) {
cfamily.addColumn(path, hexToBytes(col.value), new TimestampClock(col.timestamp));
} else {
cfamily.addTombstone(path, hexToBytes(col.value), new TimestampClock(col.timestamp));
}
This appears to be backwards.On Tue, Nov 16, 2010 at 4:03 PM, CassUser CassUser <cassu...@gmail.com> wrote:
Looked at how DatabaseDescriptor is loading the yaml file. Using that approach solves the problem with the column_families mapping exception.
The problems we are running into currently is regarding a known dataset not being loaded into our test instance correctly.
Steps:
1. Create temp directory to host cassandra test instance
2. Create keyspace directories
3 Update yaml file, and copy to test location
4. Add schema from yaml using DatabaseDescriptor.readTablesFromYaml() & DatabaseDescriptor.4. Use SSTableImport.importJson to setup a known dataset. We have JSON files converted to 0.7 compliant (byte [] rows)setTableDefinition
5. Start Cassandra instance using the EmbeddedCassandraService class
Everything appears to work from the log messages. I get the message " Sampling index for ..", and finally "Listening for thrift clients...". When i use a client to query the data in test instance, I notice keys with no columns/values stored (via keyrange slice). Using a open ended column slice for a known key I'm getting zero results.
I'm currently running cassandra 0.7. beta2. Is the steps I've outlined above suppose to work in 0.7? We did something similar in 0.6.4 without any problems. Is there known bugs I can look into?
Thanks.
On Tue, Nov 16, 2010 at 12:24 PM, Aaron Morton <aa...@thelastpickle.com> wrote:I've not used the embedded service.The code in o.a.c.service.EmbeddedCassandraService says it will read the yaml file. If the cluster does not have a schema stored I think it will load the one from yaml.Have you tried starting it up with an empty system data dir ? Does it pickup the schema from the yaml?AaronLoading yaml file like so:
FileInputStream yamlInputStream = new FileInputStream(
configTemplateFile);
Constructor constructor = new Constructor(Config.class);
Yaml yaml = new Yaml(new Loader(constructor));
Config conf = (Config) yaml.load(yamlInputStream);
Fails on the last line.
Although if i have the CFs defined like this:
keyspaces:
- column_families:
- !!org.apache.cassandra.config.RawColumnFamily
column_metadata: []
column_type: null
comment: null
compare_subcolumns_with: null
compare_with: BytesType
default_validation_class: null
gc_grace_seconds: 864000
keys_cached: 200000.0
max_compaction_threshold: 32
min_compaction_threshold: 4
name: Similarity
preload_row_cache: false
read_repair_chance: 1.0
rows_cached: 0.0
it appears to startup. Any idea whats going on here?
On Tue, Nov 16, 2010 at 11:58 AM, CassUser CassUser <cassu...@gmail.com> wrote:
This is embedded for testing cassandra 0.7 beta2. using EmbeddedCassandraService.
and manually adding schema programmatically using:
for (KSMetaData table : DatabaseDescriptor.readTablesFromYaml()) {
for (CFMetaData cfm : table.cfMetaData().values()) {
CFMetaData.map(cfm);
}
DatabaseDescriptor.setTableDefinition(table,
DatabaseDescriptorgetDefsVersion());
}
Is this the correct way to start up a test server, with the schema loaded?On Tue, Nov 16, 2010 at 11:41 AM, Aaron Morton <aa...@thelastpickle.com> wrote:
AFAIK the ArrayStoreException is similar to a type mismatch Is it possible you have something mixed up in your class path or source code if you built from source?It looks like the column family info was deserialised into a o.a.c.config.RawColumnFamily but when that object was added to the RawColumnFamily[] array on o.a.c.config.RawKeyspace if was the wrong type.Have you tried a clean build ? There are some things in the call stack which look custom, are you starting from the command line or is this embedded for testing?Hope that helps.Aaronauthenticator: org.apache.cassandraauth.AllowAllAuthenticatorHere is the yaml:
# Cassandra YAML generated from previous config
# Configuration wiki: http://wiki.apache.org/cassandra/StorageConfiguration
auto_bootstrap: false
binary_memtable_throughput_in_mb: 256
cluster_name: Test Cluster
column_index_size_in_kb: 64
commitlog_rotation_threshold_in_mb: 128
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
compaction_thread_priority: 1
concurrent_reads: 8
concurrent_writes: 32
disk_access_mode: auto
dynamic_snitch: false
endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
request_scheduler: org.apache.cassandra.scheduler.RoundRobinScheduler
request_scheduler_id: keyspace
hinted_handoff_enabled: true
in_memory_compaction_limit_in_mb: 128
index_interval: 128
keyspaces:
- name: myKeyspace
at org.yaml.snakeyamlconstructor.BaseConstructorconstructDocument(BaseConstructor.java:136)replica_placement_strategy: orgapache.cassandra.locator.SimpleStrategy
replication_factor: 1
column_families:
- name: Standard1
rows_cached: 100
keys_cached: 1
compare_with: UTF8Type
memtable_flush_after_mins: 60
memtable_operations_in_millions: 0.3
memtable_throughput_in_mb: 64
partitioner: org.apache.cassandra.dht.RandomPartitioner
phi_convict_threshold: 8
rpc_keepalive: true
rpc_port: 9160
rpc_timeout_in_ms: 10000
sliced_buffer_size_in_kb: 64
snapshot_before_compaction: false
storage_port: 7000
thrift_framed_transport_size_in_mb: 15
thrift_max_message_length_in_mb: 16
and here is the exception i'm receiving:
Caused by: Can't construct a java object for tag:yaml.org,2002:org.apachecassandra.config.Config; exception=Cannot create property=keyspaces for Java
bean=org.apache.cassandra.config.con...@12e43f1; Cannot create property=column_families for javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74;
null
in "<reader>", line 3, column 1:
authenticator: org.apache.cassan ...
^
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructorjava:177)
at org.yaml.snakeyaml.constructorConstructor$ConstructSequence.construct(Constructor.java:543)
at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122)
at org.yaml.snakeyaml.Loader.load(Loader.java:52)
at org.yaml.snakeyaml.Yaml.load(Yaml.java:166)
at com.atsid.cassandra.testutils.CassandraTestRunner.updateYamlConfig(CassandraTestRunner.java:134)
at com.atsid.cassandra.testutils.CassandraTestRunner.init(CassandraTestRunner.java:119)
at com.atsid.cassandra.testutils.CassandraTestRunner.main(CassandraTestRunner.java:272)
... 6 more
Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=keyspaces for javabean=org.apache.cassandraconfig.con...@12e43f1; Cannot cr
eate property=column_families for javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74; null
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370)
... 14 more
Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=column_families for javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74
; null
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructSequenceStep2(BaseConstructor.java:239)
at org.yaml.snakeyaml.constructor.BaseConstructor.constructSequence(BaseConstructor.java:233)
at org.yaml.snakeyaml.constructor.BaseConstructorconstructObject(BaseConstructor.java:177)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:298)
... 16 more
Caused by: java.lang.ArrayStoreException
at java.lang.System.arraycopy(Native Method)
at java.util.Arrays.copyOf(Arrays.java:2763)
at java.util.ArrayList.toArray(ArrayList.java:305)
at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:301)
... 23 more
