Looked at how DatabaseDescriptor is loading the yaml file. Using that
approach solves the problem with the column_families mapping exception.
The problems we are running into currently is regarding a known dataset not
being loaded into our test instance correctly.
Steps:
1. Create temp directory to host cassandra test instance
2. Create keyspace directories
3. Update yaml file, and copy to test location
4. Add schema from yaml using DatabaseDescriptor.
readTablesFromYaml() & DatabaseDescriptor.
setTableDefinition
4. Use SSTableImport.importJson to setup a known dataset. We have JSON
files converted to 0.7 compliant (byte [] rows)
5. Start Cassandra instance using the EmbeddedCassandraService class
Everything appears to work from the log messages. I get the message "
Sampling index for ..", and finally "Listening for thrift clients...". When
i use a client to query the data in test instance, I notice keys with no
columns/values stored (via keyrange slice). Using a open ended column slice
for a known key I'm getting zero results.
I'm currently running cassandra 0.7. beta2. Is the steps I've outlined
above suppose to work in 0.7? We did something similar in 0.6.4 without any
problems. Is there known bugs I can look into?
Thanks.
On Tue, Nov 16, 2010 at 12:24 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
> I've not used the embedded service.
>
> The code in o.a.c.service.EmbeddedCassandraService says it will read the
> yaml file. If the cluster does not have a schema stored I think it will load
> the one from yaml.
>
> Have you tried starting it up with an empty system data dir ? Does it
> pickup the schema from the yaml?
>
> Aaron
>
> On 17 Nov, 2010,at 09:17 AM, CassUser CassUser <cassu...@gmail.com> wrote:
>
> Loading yaml file like so:
> FileInputStream yamlInputStream = new FileInputStream(
> configTemplateFile);
> Constructor constructor = new Constructor(Config.class);
> Yaml yaml = new Yaml(new Loader(constructor));
> Config conf = (Config) yaml.load(yamlInputStream);
>
> Fails on the last line.
>
> Although if i have the CFs defined like this:
> keyspaces:
> - column_families:
> - !!org.apache.cassandra.config.RawColumnFamily
> column_metadata: []
> column_type: null
> comment: null
> compare_subcolumns_with: null
> compare_with: BytesType
> default_validation_class: null
> gc_grace_seconds: 864000
> keys_cached: 200000.0
> max_compaction_threshold: 32
> min_compaction_threshold: 4
> name: Similarity
> preload_row_cache: false
> read_repair_chance: 1.0
> rows_cached: 0.0
>
>
> it appears to startup. Any idea whats going on here?
>
>
> On Tue, Nov 16, 2010 at 11:58 AM, CassUser CassUser <cassu...@gmail.com>wrote:
>
>> This is embedded for testing cassandra 0.7 beta2. using
>> EmbeddedCassandraService.
>>
>> and manually adding schema programmatically using:
>> for (KSMetaData table : DatabaseDescriptor.readTablesFromYaml()) {
>> for (CFMetaData cfm : table.cfMetaData().values()) {
>> CFMetaData.map(cfm);
>> }
>> DatabaseDescriptor.setTableDefinition(table,
>> DatabaseDescriptor.getDefsVersion());
>> }
>>
>> Is this the correct way to start up a test server, with the schema loaded?
>>
>>
>>
>>
>> On Tue, Nov 16, 2010 at 11:41 AM, Aaron Morton
>> <aa...@thelastpickle.com>wrote:
>>
>>> AFAIK the ArrayStoreException is similar to a type mismatch. Is it
>>> possible you have something mixed up in your class path or source code if
>>> you built from source?
>>>
>>> It looks like the column family info was deserialised into a
>>> o.a.c.config.RawColumnFamily but when that object was added to the
>>> RawColumnFamily[] array on o.a.c.config.RawKeyspace if was the wrong type.
>>>
>>> Have you tried a clean build ? There are some things in the call stack
>>> which look custom, are you starting from the command line or is this
>>> embedded for testing?
>>>
>>> Hope that helps.
>>> Aaron
>>>
>>>
>>>
>>>
>>>
>>> On 17 Nov, 2010,at 08:09 AM, CassUser CassUser <cassu...@gmail.com>
>>> wrote:
>>>
>>>
>>> Here is the yaml:
>>>
>>> # Cassandra YAML generated from previous config
>>> # Configuration wiki:
>>> http://wiki.apache.org/cassandra/StorageConfiguration
>>> authenticator: org.apache.cassandraauth.AllowAllAuthenticator
>>>
>>> auto_bootstrap: false
>>> binary_memtable_throughput_in_mb: 256
>>> cluster_name: Test Cluster
>>> column_index_size_in_kb: 64
>>> commitlog_rotation_threshold_in_mb: 128
>>> commitlog_sync: periodic
>>> commitlog_sync_period_in_ms: 10000
>>> compaction_thread_priority: 1
>>> concurrent_reads: 8
>>> concurrent_writes: 32
>>> disk_access_mode: auto
>>> dynamic_snitch: false
>>> endpoint_snitch: org.apache.cassandra.locator.SimpleSnitch
>>> request_scheduler: org.apache.cassandra.scheduler.RoundRobinScheduler
>>> request_scheduler_id: keyspace
>>> hinted_handoff_enabled: true
>>> in_memory_compaction_limit_in_mb: 128
>>> index_interval: 128
>>> keyspaces:
>>> - name: myKeyspace
>>> replica_placement_strategy:
>>> orgapache.cassandra.locator.SimpleStrategy
>>>
>>>
>>> replication_factor: 1
>>> column_families:
>>> - name: Standard1
>>> rows_cached: 100
>>> keys_cached: 1
>>> compare_with: UTF8Type
>>> memtable_flush_after_mins: 60
>>> memtable_operations_in_millions: 0.3
>>> memtable_throughput_in_mb: 64
>>> partitioner: org.apache.cassandra.dht.RandomPartitioner
>>> phi_convict_threshold: 8
>>> rpc_keepalive: true
>>> rpc_port: 9160
>>> rpc_timeout_in_ms: 10000
>>> sliced_buffer_size_in_kb: 64
>>> snapshot_before_compaction: false
>>> storage_port: 7000
>>> thrift_framed_transport_size_in_mb: 15
>>> thrift_max_message_length_in_mb: 16
>>>
>>> and here is the exception i'm receiving:
>>>
>>> Caused by: Can't construct a java object for
>>> tag:yaml.org,2002:org.apache.cassandra.config.Config;
>>> exception=Cannot create property=keyspaces for Java
>>> bean=org.apache.cassandra.config.con...@12e43f1; Cannot create
>>> property=column_families for
>>> javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74;
>>> null
>>> in "<reader>", line 3, column 1:
>>> authenticator: org.apache.cassan ...
>>> ^
>>>
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372)
>>> at
>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177)
>>> at
>>> org.yaml.snakeyamlconstructor.BaseConstructorconstructDocument(BaseConstructor.java:136)
>>>
>>> at
>>> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122)
>>> at org.yaml.snakeyaml.Loader.load(Loader.java:52)
>>> at org.yaml.snakeyaml.Yaml.load(Yaml.java:166)
>>> at
>>> com.atsid.cassandra.testutils.CassandraTestRunner.updateYamlConfig(CassandraTestRunner.java:134)
>>> at
>>> com.atsid.cassandra.testutils.CassandraTestRunner.init(CassandraTestRunner.java:119)
>>> at
>>> com.atsid.cassandra.testutils.CassandraTestRunner.main(CassandraTestRunner.java:272)
>>> ... 6 more
>>> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create
>>> property=keyspaces for javabean=org.apache.cassandra.config.con...@12e43f1;
>>> Cannot cr
>>> eate property=column_families for
>>> javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74; null
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370)
>>> ... 14 more
>>> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create
>>> property=column_families for
>>> javabean=org.apache.cassandra.config.rawkeysp...@1a8bd74
>>> ; null
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
>>> at
>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177)
>>> at
>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructSequenceStep2(BaseConstructor.java:239)
>>> at
>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructSequence(BaseConstructor.java:233)
>>> at
>>> org.yaml.snakeyaml.constructorConstructor$ConstructSequence.construct(Constructor.java:543)
>>>
>>> at
>>> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177)
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:298)
>>> ... 16 more
>>> Caused by: java.lang.ArrayStoreException
>>> at java.lang.System.arraycopy(Native Method)
>>> at java.util.Arrays.copyOf(Arrays.java:2763)
>>> at java.util.ArrayList.toArray(ArrayList.java:305)
>>> at
>>> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:301)
>>> ... 23 more
>>>
>>>
>>
>
