Bryan added a patch. Can someone please review? Thanks. On Wed, Nov 17, 2010 at 12:10 PM, CassUser CassUser <cassu...@gmail.com>wrote:
> We are putting together a patch for that. Also updating the unit test to > capture this case. It wasn't failing as a result of it being backwards. > > > On Wed, Nov 17, 2010 at 11:51 AM, CassUser CassUser <cassu...@gmail.com>wrote: > >> Thanks for looking into that. I created a jira issue for it: >> >> https://issues.apache.org/jira/browse/CASSANDRA-1753 >> >> >> >> On Wed, Nov 17, 2010 at 11:28 AM, Aaron Morton >> <aa...@thelastpickle.com>wrote: >> >>> That does look like a bug >>> http://www.mail-archive.com/d...@cassandra.apache.org/msg01167.html >>> >>> <http://www.mail-archive.com/d...@cassandra.apache.org/msg01167.html> >>> Aaron >>> >>> >>> On 17 Nov, 2010,at 03:55 PM, Aaron Morton <aa...@thelastpickle.com> >>> wrote: >>> >>> It's certainly looks suspect. I've had a look at the code around >>> SSTableImport and SSTableExport and the isDeleted value for the col is based >>> on IColumn.isMarkedForDelete read when the data was exported. I'll try to >>> have a look tonight, or if someone is still up in the states they may help. >>> The current 0.7 branch has the same code. >>> >>> Are there any other errors in the log? >>> >>> The code sample you've got there uses the TimestampClocks that were once >>> a part of the 0.7 development, but have been removed. So your data sample >>> may not be usable going forwards. Can you jump to beta3? >>> >>> Aaron >>> >>> >>> >>> On 17 Nov, 2010,at 02:03 PM, CassUser CassUser <cassu...@gmail.com> >>> wrote: >>> >>> Looking at this closer. I noticed the following in the SSTableImport >>> Class: >>> >>> if (col.isDeleted) { >>> cfamily.addColumn(path, hexToBytes(col.value), new >>> TimestampClock(col.timestamp)); >>> } else { >>> cfamily.addTombstone(path, hexToBytes(col.value), new >>> TimestampClock(col.timestamp)); >>> } >>> >>> This appears to be backwards. >>> >>> >>> On Tue, Nov 16, 2010 at 4:03 PM, CassUser CassUser >>> <cassu...@gmail.com>wrote: >>> >>>> Looked at how DatabaseDescriptor is loading the yaml file. Using that >>>> approach solves the problem with the column_families mapping exception. >>>> >>>> The problems we are running into currently is regarding a known dataset >>>> not being loaded into our test instance correctly. >>>> >>>> Steps: >>>> 1. Create temp directory to host cassandra test instance >>>> 2. Create keyspace directories >>>> 3 Update yaml file, and copy to test location >>>> 4. Add schema from yaml using DatabaseDescriptor. >>>> readTablesFromYaml() & DatabaseDescriptor. >>>> setTableDefinition >>>> 4. Use SSTableImport.importJson to setup a known dataset. We have JSON >>>> files converted to 0.7 compliant (byte [] rows) >>>> 5. Start Cassandra instance using the EmbeddedCassandraService class >>>> >>>> Everything appears to work from the log messages. I get the message " >>>> Sampling index for ..", and finally "Listening for thrift clients...". >>>> When >>>> i use a client to query the data in test instance, I notice keys with no >>>> columns/values stored (via keyrange slice). Using a open ended column >>>> slice >>>> for a known key I'm getting zero results. >>>> >>>> I'm currently running cassandra 0.7. beta2. Is the steps I've outlined >>>> above suppose to work in 0.7? We did something similar in 0.6.4 without >>>> any >>>> problems. Is there known bugs I can look into? >>>> >>>> Thanks. >>>> >>>> >>>> >>>> >>>> On Tue, Nov 16, 2010 at 12:24 PM, Aaron Morton <aa...@thelastpickle.com >>>> > wrote: >>>> >>>>> I've not used the embedded service. >>>>> >>>>> The code in o.a.c.service.EmbeddedCassandraService says it will read >>>>> the yaml file. If the cluster does not have a schema stored I think it >>>>> will >>>>> load the one from yaml. >>>>> >>>>> Have you tried starting it up with an empty system data dir ? Does it >>>>> pickup the schema from the yaml? >>>>> >>>>> Aaron >>>>> >>>>> >>>>> On 17 Nov, 2010,at 09:17 AM, CassUser CassUser <cassu...@gmail.com> >>>>> wrote: >>>>> >>>>> >>>>> Loading yaml file like so: >>>>> FileInputStream yamlInputStream = new FileInputStream( >>>>> configTemplateFile); >>>>> Constructor constructor = new Constructor(Config.class); >>>>> Yaml yaml = new Yaml(new Loader(constructor)); >>>>> Config conf = (Config) yaml.load(yamlInputStream); >>>>> >>>>> Fails on the last line. >>>>> >>>>> Although if i have the CFs defined like this: >>>>> keyspaces: >>>>> - column_families: >>>>> - !!org.apache.cassandra.config.RawColumnFamily >>>>> column_metadata: [] >>>>> column_type: null >>>>> comment: null >>>>> compare_subcolumns_with: null >>>>> compare_with: BytesType >>>>> default_validation_class: null >>>>> gc_grace_seconds: 864000 >>>>> keys_cached: 200000.0 >>>>> max_compaction_threshold: 32 >>>>> min_compaction_threshold: 4 >>>>> name: Similarity >>>>> preload_row_cache: false >>>>> read_repair_chance: 1.0 >>>>> rows_cached: 0.0 >>>>> >>>>> >>>>> it appears to startup. Any idea whats going on here? >>>>> >>>>> >>>>> >>>>> On Tue, Nov 16, 2010 at 11:58 AM, CassUser CassUser < >>>>> cassu...@gmail.com> wrote: >>>>> >>>>>> >>>>>> This is embedded for testing cassandra 0.7 beta2. using >>>>>> EmbeddedCassandraService. >>>>>> >>>>>> and manually adding schema programmatically using: >>>>>> for (KSMetaData table : >>>>>> DatabaseDescriptor.readTablesFromYaml()) { >>>>>> for (CFMetaData cfm : table.cfMetaData().values()) { >>>>>> CFMetaData.map(cfm); >>>>>> } >>>>>> DatabaseDescriptor.setTableDefinition(table, >>>>>> DatabaseDescriptorgetDefsVersion()); >>>>>> } >>>>>> >>>>>> Is this the correct way to start up a test server, with the schema >>>>>> loaded? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Nov 16, 2010 at 11:41 AM, Aaron Morton < >>>>>> aa...@thelastpickle.com> wrote: >>>>>> >>>>>>> >>>>>>> AFAIK the ArrayStoreException is similar to a type mismatch Is it >>>>>>> possible you have something mixed up in your class path or source code >>>>>>> if >>>>>>> you built from source? >>>>>>> >>>>>>> It looks like the column family info was deserialised into a >>>>>>> o.a.c.config.RawColumnFamily but when that object was added to the >>>>>>> RawColumnFamily[] array on o.a.c.config.RawKeyspace if was the wrong >>>>>>> type. >>>>>>> >>>>>>> Have you tried a clean build ? There are some things in the call >>>>>>> stack which look custom, are you starting from the command line or is >>>>>>> this >>>>>>> embedded for testing? >>>>>>> >>>>>>> Hope that helps. >>>>>>> Aaron >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 17 Nov, 2010,at 08:09 AM, CassUser CassUser <cassu...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Here is the yaml: >>>>>>> >>>>>>> # Cassandra YAML generated from previous config >>>>>>> # Configuration wiki: >>>>>>> http://wiki.apache.org/cassandra/StorageConfiguration >>>>>>> authenticator: org.apache.cassandraauth.AllowAllAuthenticator >>>>>>> >>>>>>> >>>>>>> auto_bootstrap: false >>>>>>> binary_memtable_throughput_in_mb: 256 >>>>>>> cluster_name: Test Cluster >>>>>>> column_index_size_in_kb: 64 >>>>>>> commitlog_rotation_threshold_in_mb: 128 >>>>>>> commitlog_sync: periodic >>>>>>> commitlog_sync_period_in_ms: 10000 >>>>>>> compaction_thread_priority: 1 >>>>>>> concurrent_reads: 8 >>>>>>> concurrent_writes: 32 >>>>>>> disk_access_mode: auto >>>>>>> dynamic_snitch: false >>>>>>> endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch >>>>>>> request_scheduler: org.apache.cassandra.scheduler.RoundRobinScheduler >>>>>>> request_scheduler_id: keyspace >>>>>>> hinted_handoff_enabled: true >>>>>>> in_memory_compaction_limit_in_mb: 128 >>>>>>> index_interval: 128 >>>>>>> keyspaces: >>>>>>> - name: myKeyspace >>>>>>> >>>>>>> replica_placement_strategy: >>>>>>> orgapache.cassandra.locator.SimpleStrategy >>>>>>> >>>>>>> >>>>>>> replication_factor: 1 >>>>>>> column_families: >>>>>>> - name: Standard1 >>>>>>> rows_cached: 100 >>>>>>> keys_cached: 1 >>>>>>> compare_with: UTF8Type >>>>>>> memtable_flush_after_mins: 60 >>>>>>> memtable_operations_in_millions: 0.3 >>>>>>> memtable_throughput_in_mb: 64 >>>>>>> partitioner: org.apache.cassandra.dht.RandomPartitioner >>>>>>> phi_convict_threshold: 8 >>>>>>> rpc_keepalive: true >>>>>>> rpc_port: 9160 >>>>>>> rpc_timeout_in_ms: 10000 >>>>>>> sliced_buffer_size_in_kb: 64 >>>>>>> snapshot_before_compaction: false >>>>>>> storage_port: 7000 >>>>>>> thrift_framed_transport_size_in_mb: 15 >>>>>>> thrift_max_message_length_in_mb: 16 >>>>>>> >>>>>>> and here is the exception i'm receiving: >>>>>>> >>>>>>> Caused by: Can't construct a java object for >>>>>>> tag:yaml.org,2002:org.apachecassandra.config.Config; >>>>>>> exception=Cannot create property=keyspaces for Java >>>>>>> bean=org.apache.cassandra.config.con...@12e43f1; Cannot create >>>>>>> property=column_families for >>>>>>> javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74; >>>>>>> null >>>>>>> in "<reader>", line 3, column 1: >>>>>>> authenticator: org.apache.cassan ... >>>>>>> ^ >>>>>>> >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructorjava:177) >>>>>>> at >>>>>>> org.yaml.snakeyamlconstructor.BaseConstructorconstructDocument(BaseConstructor.java:136) >>>>>>> >>>>>>> >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) >>>>>>> at org.yaml.snakeyaml.Loader.load(Loader.java:52) >>>>>>> at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) >>>>>>> at >>>>>>> com.atsid.cassandra.testutils.CassandraTestRunner.updateYamlConfig(CassandraTestRunner.java:134) >>>>>>> at >>>>>>> com.atsid.cassandra.testutils.CassandraTestRunner.init(CassandraTestRunner.java:119) >>>>>>> at >>>>>>> com.atsid.cassandra.testutils.CassandraTestRunner.main(CassandraTestRunner.java:272) >>>>>>> ... 6 more >>>>>>> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create >>>>>>> property=keyspaces for >>>>>>> javabean=org.apache.cassandraconfig.con...@12e43f1; >>>>>>> Cannot cr >>>>>>> eate property=column_families for >>>>>>> javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74; null >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370) >>>>>>> ... 14 more >>>>>>> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create >>>>>>> property=column_families for >>>>>>> javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74 >>>>>>> ; null >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructSequenceStep2(BaseConstructor.java:239) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructSequence(BaseConstructor.java:233) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructorConstructor$ConstructSequence.construct(Constructor.java:543) >>>>>>> >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.BaseConstructorconstructObject(BaseConstructor.java:177) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:298) >>>>>>> ... 16 more >>>>>>> Caused by: java.lang.ArrayStoreException >>>>>>> at java.lang.System.arraycopy(Native Method) >>>>>>> at java.util.Arrays.copyOf(Arrays.java:2763) >>>>>>> at java.util.ArrayList.toArray(ArrayList.java:305) >>>>>>> at >>>>>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:301) >>>>>>> ... 23 more >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
