Hello Ben, Thanks for the willingness to help, 2013/2/27 Ben Bromhead <b...@instaclustr.com> > > Have your added the priam java agent to cassandras JVM argurments (e.g. > -javaagent:$CASS_HOME/lib/priam-cass-extensions-1.1.15.jar) and does the > web container running priam have permissions to write to the cassandra > config directory? Also what do the priam logs say? >
I put the priam log of the first node bellow. Yes, I have added priam-cass-extensions to java args and Priam IS actually writting to cassandra dir. > If you want to get up and running quickly with cassandra, AWS and priam > quickly check out > www.instaclustr.com<http://www.instaclustr.com/?cid=cass-list>you. > We deploy Cassandra under your AWS account and you have full root access > to the nodes if you want to explore and play around + there is a free tier > which is great for experimenting and trying Cassandra out. > That sounded really great. I am not sure if it would apply to our case (will consider it though), but some partners would have a great benefit from it, for sure! I will send your link to them. What priam says: 2013-02-27 14:14:58.0614 INFO pool-2-thread-1 com.netflix.priam.utils.SystemUtils Calling URL API: http://169.254.169.254/latest/meta-data/public-hostname returns: ec2-174-129-59-107.compute-1.amazon aws.com 2013-02-27 14:14:58.0615 INFO pool-2-thread-1 com.netflix.priam.utils.SystemUtils Calling URL API: http://169.254.169.254/latest/meta-data/public-ipv4 returns: 174.129.59.107 2013-02-27 14:14:58.0618 INFO pool-2-thread-1 com.netflix.priam.utils.SystemUtils Calling URL API: http://169.254.169.254/latest/meta-data/instance-id returns: i-88b32bfb 2013-02-27 14:14:58.0618 INFO pool-2-thread-1 com.netflix.priam.utils.SystemUtils Calling URL API: http://169.254.169.254/latest/meta-data/instance-type returns: c1.medium 2013-02-27 14:14:59.0614 INFO pool-2-thread-1 com.netflix.priam.defaultimpl.PriamConfiguration REGION set to us-east-1, ASG Name set to dmp_cluster-useast1b 2013-02-27 14:14:59.0746 INFO pool-2-thread-1 com.netflix.priam.defaultimpl.PriamConfiguration appid used to fetch properties is: dmp_cluster 2013-02-27 14:14:59.0843 INFO pool-2-thread-1 org.quartz.simpl.SimpleThreadPool Job execution threads will use class loader of thread: pool-2-thread-1 2013-02-27 14:14:59.0861 INFO pool-2-thread-1 org.quartz.core.SchedulerSignalerImpl Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl 2013-02-27 14:14:59.0862 INFO pool-2-thread-1 org.quartz.core.QuartzScheduler Quartz Scheduler v.1.7.3 created. 2013-02-27 14:14:59.0864 INFO pool-2-thread-1 org.quartz.simpl.RAMJobStore RAMJobStore initialized. 2013-02-27 14:14:59.0864 INFO pool-2-thread-1 org.quartz.impl.StdSchedulerFactory Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.propertie s' 2013-02-27 14:14:59.0864 INFO pool-2-thread-1 org.quartz.impl.StdSchedulerFactory Quartz scheduler version: 1.7.3 2013-02-27 14:14:59.0864 INFO pool-2-thread-1 org.quartz.core.QuartzScheduler JobFactory set to: com.netflix.priam.scheduler.GuiceJobFactory@1b6a1c4 2013-02-27 14:15:00.0239 INFO pool-2-thread-1 com.netflix.priam.aws.AWSMembership Querying Amazon returned following instance in the ASG: us-east-1b --> i-8eb32bfd,i-88b32bfb 2013-02-27 14:15:01.0470 INFO Timer-0 org.quartz.utils.UpdateChecker New update(s) found: 1.8.5 [ http://www.terracotta.org/kit/reflector?kitID=default&pageID=QuartzChangeLog ] 2013-02-27 14:15:10.0925 INFO pool-2-thread-1 com.netflix.priam.identity.InstanceIdentity Found dead instances: i-d49a0da7 2013-02-27 14:15:11.0397 ERROR pool-2-thread-1 com.netflix.priam.aws.SDBInstanceFactory Conditional check failed. Attribute (instanceId) value exists 2013-02-27 14:15:11.0398 ERROR pool-2-thread-1 com.netflix.priam.utils.RetryableCallable Retry #1 for: Status Code: 409, AWS Service: AmazonSimpleDB, AWS Request ID: 96ca7ae5-f352-b13a-febd-8801d46fe e83, AWS Error Code: ConditionalCheckFailed, AWS Error Message: Conditional check failed. Attribute (instanceId) value exists 2013-02-27 14:15:11.0686 INFO pool-2-thread-1 com.netflix.priam.aws.AWSMembership Querying Amazon returned following instance in the ASG: us-east-1b --> i-8eb32bfd,i-88b32bfb 2013-02-27 14:15:25.0258 INFO pool-2-thread-1 com.netflix.priam.identity.InstanceIdentity Found dead instances: i-d89a0dab 2013-02-27 14:15:25.0588 INFO pool-2-thread-1 com.netflix.priam.identity.InstanceIdentity Trying to grab slot 1808575601 with availability zone us-east-1b 2013-02-27 14:15:25.0732 INFO pool-2-thread-1 com.netflix.priam.identity.InstanceIdentity My token: 56713727820156410577229101240436610842 2013-02-27 14:15:25.0732 INFO pool-2-thread-1 org.quartz.core.QuartzScheduler Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started. 2013-02-27 14:15:25.0878 INFO pool-2-thread-1 org.apache.cassandra.db.HintedHandOffManager cluster_name: dmp_cluster initial_token: null hinted_handoff_enabled: true max_hint_window_in_ms: 8 hinted_handoff_throttle_in_kb: 1024 max_hints_delivery_threads: 2 authenticator: org.apache.cassandra.auth.AllowAllAuthenticator authorizer: org.apache.cassandra.auth.AllowAllAuthorizer partitioner: org.apache.cassandra.dht.RandomPartitioner data_file_directories: - /var/lib/cassandra/data commitlog_directory: /var/lib/cassandra/commitlog disk_failure_policy: stop key_cache_size_in_mb: null key_cache_save_period: 14400 row_cache_size_in_mb: 0 row_cache_save_period: 0 row_cache_provider: SerializingCacheProvider saved_caches_directory: /var/lib/cassandra/saved_caches commitlog_sync: periodic commitlog_sync_period_in_ms: 10000 commitlog_segment_size_in_mb: 32 seed_provider: - class_name: com.netflix.priam.cassandra.extensions.NFSeedProvider parameters: - seeds: 127.0.0.1 flush_largest_memtables_at: 0.75 reduce_cache_sizes_at: 0.85 reduce_cache_capacity_to: 0.6 concurrent_reads: 32 concurrent_writes: 32 memtable_flush_queue_size: 4 trickle_fsync: false trickle_fsync_interval_in_kb: 10240 storage_port: 7000 ssl_storage_port: 7001 listen_address: null start_native_transport: false native_transport_port: 9042 start_rpc: true rpc_address: null rpc_port: 9160 rpc_keepalive: true rpc_server_type: sync thrift_framed_transport_size_in_mb: 15 thrift_max_message_length_in_mb: 16 incremental_backups: true snapshot_before_compaction: false auto_snapshot: true column_index_size_in_kb: 64 in_memory_compaction_limit_in_mb: 128 multithreaded_compaction: false compaction_throughput_mb_per_sec: 8 compaction_preheat_key_cache: true read_request_timeout_in_ms: 10000 range_request_timeout_in_ms: 10000 write_request_timeout_in_ms: 10000 truncate_request_timeout_in_ms: 60000 request_timeout_in_ms: 10000 cross_node_timeout: false endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch dynamic_snitch_update_interval_in_ms: 100 dynamic_snitch_reset_interval_in_ms: 600000 dynamic_snitch_badness_threshold: 0.1 request_scheduler: org.apache.cassandra.scheduler.NoScheduler index_interval: 128 server_encryption_options: internode_encryption: none keystore: conf/.keystore keystore_password: cassandra truststore: conf/.truststore truststore_password: cassandra client_encryption_options: enabled: false keystore: conf/.keystore keystore_password: cassandra internode_compression: all inter_dc_tcp_nodelay: true auto_bootstrap: true memtable_total_space_in_mb: 1024 stream_throughput_outbound_megabits_per_sec: 400 num_tokens: 1 2013-02-27 14:15:25.0884 INFO pool-2-thread-1 com.netflix.priam.utils.SystemUtils Starting cassandra server ....Join ring=true 2013-02-27 14:15:25.0915 INFO pool-2-thread-1 com.netflix.priam.utils.SystemUtils Starting cassandra server .... 2013-02-27 14:15:30.0013 INFO http-bio-8080-exec-1 com.netflix.priam.aws.AWSMembership Query on ASG returning 3 instances 2013-02-27 14:15:31.0726 INFO http-bio-8080-exec-2 com.netflix.priam.aws.AWSMembership Query on ASG returning 3 instances 2013-02-27 14:15:37.0360 INFO DefaultQuartzScheduler_Worker-5 com.netflix.priam.aws.S3FileSystem Uploading to backup/us-east-1/dmp_cluster/56713727820156410577229101240436610842/201302271415/SST/system/local/system-local-ib-1-CompressionInfo.db with chunk size 10485760 Best regards, -- Marcelo Elias Del Valle http://mvalle.com - @mvallebr
