Re: Error while adding a new node.

[Arun]

Looks like you have too many open files issue. Increase the ulimit for the user.

 If you are starting the cassandra daemon using user cassandra, increase the 
ulimit for that user.


> On Jun 30, 2015, at 21:16, Neha Trivedi  wrote:
> 
> Hello,
> I have a 4 node cluster with SimpleSnitch.
> Cassandra :  Cassandra 2.1.3 
> 
> I am trying to add a new node (cassandra 2.1.7) and I get the following error.
> 
> ERROR [STREAM-IN-] 2015-06-30 05:13:48,516 JVMStabilityInspector.java:94 - 
> JVM state determined to be unstable.  Exiting forcefully due to:
> java.io.FileNotFoundException: /var/lib/cassandra/data/-Index.db (Too many 
> open files)
> 
> I increased the MAX_HEAP_SIZE then I get :
> ERROR [CompactionExecutor:9] 2015-06-30 23:31:44,792 CassandraDaemon.java:223 
> - Exception in thread Thread[CompactionExecutor:9,1,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: 
> /var/lib/cassandra/data/-Data.db (Too many open files)
> at 
> org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
>  ~[apache-cassandra-2.1.7.jar:2.1.7]
> 
> Is it because of the different version of Cassandra (2.1.3 and 2.17) ?
> 
> regards
> N
> 
> 
> 
> 
> 
> 
> 

Re: High Compactions Pending

[Arun]

Its constant since 4 hours. Remaining nodes have around 10 compactions. We have 
4 column families. 


> On Sep 22, 2014, at 19:39, Chris Lohfink  wrote:
> 
> 35 isn't that high really in some scenarios (ie, theres a lot of column 
> families), is it continuing to climb or does it drop down shortly after?
> 
> ---
> Chris Lohfink
> 
>> On Sep 22, 2014, at 7:57 PM, arun sirimalla  wrote:
>> 
>> I have a 6 (i2.2xlarge) node cluster on AWS with 4.5 DSE running on it. I 
>> notice high compaction pending on one of the node around 35.
>> Compaction throughput set to 64 MB and flush writes to 4. Any suggestion is  
>> much appreciated.
>> 
>> -- 
>> Arun 
>> Senior Hadoop Engineer
>> Cloudwick
>> 
>> Champion of Big Data
>> http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html
> 

Re: which snitch ?

[Arun]

Srinivas,

Use ec2multiregion snitch.


> On Oct 25, 2014, at 19:47, Srinivas Chamarthi  
> wrote:
> 
> I am using a datacenter in private network and want to replicate data to an 
> ec2 data center. I am confused which snitch to use so that my data in the dc1 
> is replicated to dc2 in ec2 ?  any help is greatly appreciated
> 
> thx
> srinivas

Need ur expertise on Cassandra issue!!

[Arun]

Hi , 

I need your help & suggestions for our production issue. 

Details: 
-- 
we have 40 CF's in cassandra cluster for each datasource like below 
MusicData--Keyspace 
spotify_1-column family-Active 
spotify_2-column family-standby 
Daily we load data into this cluster using as below process: 
1.Astyanix library to delete inactive version of CF datahere spotify_2 
2. Hadoop Bulkload JAR -pushes data from Hadoop to cassandra into spotify_2 
Data inflow rate 150GB per day . 
Datastax community version 1.1.9 with 9 nodes of 4 TB which are built on 
openstack with high end config. 

Problem: 
--- 
we're encountering the problem every week, the hadoop bulkload program is 
failing with 
java.io.IOException: Too many hosts failed: [/10.240.171.80, /10.240.171.76, 
/10.240.171.74, /10.240.171.73] 

at 
org.apache.cassandra.hadoop.BulkRecordWriter.close(BulkRecordWriter.java:243
) 
I can provide more details about the error if you need.with our initial 
analysis we came to know if we're deleting the deleted space for tombstoned 
blocks will be reclaimed in compaction process so we have increased storage 
capacity by adding new nodes but problem still persists. 
we need your expertise to comment on this production issue.please let me 
know if you need any information!! 
I will wait for your response !! 

-Arun

Cassandra monitoring

[Arun Ramakrishnan]

What are the options for a very small and nimble startup to do keep a
cassandra cluster running well oiled. We are on AWS. We are interested in a
monitoring tool and potentially also cluster management tools.

We are currently on apache cassandra 3.7. We were hoping the datastax
opscenter would be it (It is free for startups our size). But, looks like
it does not support cassandra versions greater than v2.1. It is pretty
surprising considering cassandra v2.1  came out in 2014.

We would consider downgrading to datastax cassandra 2.1 just to have robust
monitoring tools. But, I am not sure if having opscenter offsets all the
improvements that have been added to cassandra since 2.1.

Sematext has a integrations for monitoring cassandra. Does anyone have good
experience with it ?

How much work would be involved to setup Ganglia or some such option for
cassandra ?

Thanks,
Arun

Re: Cassandra monitoring

[Arun Ramakrishnan]

Thanks Jonathan.

Out of curiosity, does opscenter support some later version of cassandra
that is not OSS ?

Well, the most minimal requirement is that, I want to be able to monitor
for cluster health and hook this info to some alerting platform. We are AWS
heavy. We just really heavily on AWS cloud watch for our metrics as of now.
We prefer to not spend our time setting up additional tools if we can help
it. So, if we needed a 3rd party service we would consider an APM or
monitoring service that is on the cheaper side.




On Tue, Jun 14, 2016 at 12:20 PM, Jonathan Haddad  wrote:

> Depends what you want to monitor.  I wouldn't use a lesser version of
> Cassandra for OpsCenter, it doesn't give you a ton you can't get elsewhere
> and it's not ever going to support OSS > 2.1, so you kind of limit yourself
> to a pretty old version of Cassandra for a non-good reason.
>
> What else do you use for monitoring in your infra?  I've used a mix of OSS
> tools (nagios, statsd, graphite, ELK), and hosted solutions. The nice part
> about them is that you can monitor your whole stack in a single UI not just
> your database.
>
> On Tue, Jun 14, 2016 at 12:10 PM Arun Ramakrishnan <
> sinchronized.a...@gmail.com> wrote:
>
>> What are the options for a very small and nimble startup to do keep a
>> cassandra cluster running well oiled. We are on AWS. We are interested in a
>> monitoring tool and potentially also cluster management tools.
>>
>> We are currently on apache cassandra 3.7. We were hoping the datastax
>> opscenter would be it (It is free for startups our size). But, looks like
>> it does not support cassandra versions greater than v2.1. It is pretty
>> surprising considering cassandra v2.1  came out in 2014.
>>
>> We would consider downgrading to datastax cassandra 2.1 just to have
>> robust monitoring tools. But, I am not sure if having opscenter offsets all
>> the improvements that have been added to cassandra since 2.1.
>>
>> Sematext has a integrations for monitoring cassandra. Does anyone have
>> good experience with it ?
>>
>> How much work would be involved to setup Ganglia or some such option for
>> cassandra ?
>>
>> Thanks,
>> Arun
>>
>>
>>
>>
>>
>>
>>

Kafka(9.0.1) error : org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1164731757 larger than 104857600)

[Arun Rai]

Hello Kafka/Cassandra experts,


*I am getting below error….*

org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(
NetworkReceive.java:91)

at org.apache.kafka.common.network.NetworkReceive.
readFrom(NetworkReceive.java:71)

at org.apache.kafka.common.network.KafkaChannel.receive(
KafkaChannel.java:153)

at org.apache.kafka.common.network.KafkaChannel.read(
KafkaChannel.java:134)

at org.apache.kafka.common.network.Selector.poll(Selector.java:286)

at kafka.network.Processor.run(SocketServer.scala:413)

at java.lang.Thread.run(Thread.java:745)

*[2016-10-06 22:17:42,001] WARN Unexpected error from /10.61.48.28
<http://10.61.48.28/>; closing connection
(org.apache.kafka.common.network.Selector)*

org.apache.kafka.common.network.InvalidReceiveException: Invalid receive
(size = 1969447758 larger than 104857600)

at



Below is my case.

*Cluster configuration :*



Kafka cluster : 3 vms with each one has 4 vcpu, 8 gb RAM

Cassandra : 4 vms with each one has 8 vcpu, 16 gb RAM



1.   Producer: Producer is rdkafka client, which produce the
messages(protobuf) to kafka topic(with 12 partitions). Please message size
always vary. It’s not fixed size message. Max message size could be from 1
mb to 3 mb.

2.   Consumer client : Client is java program who reads the messages
and load into Cassandra database.

*3.   **I get this error only when both producer and consumer is
running together , after 15-20 minute later I see this error in kafka log
files.*

4.   If I let consumer or producer run at single time.. I don’t get
this error.

5.   I do not get this error if Consumer only consume the messages and
do not insert into Cassandra.



Below is my kafka server configuration props:





broker.id=1

log.dirs=/data/kafka



host.name=10.61.19.87

port=9092



advertised.host.name=10.61.19.87

advertised.port=9092

listeners=PLAINTEXT://0.0.0.0:9092



delete.topic.enable=true



# Replication configurations

num.replica.fetchers=4

replica.fetch.max.bytes=1048576

replica.fetch.wait.max.ms=500

replica.high.watermark.checkpoint.interval.ms=5000

replica.socket.timeout.ms=3

replica.socket.receive.buffer.bytes=65536

replica.lag.time.max.ms=1



controller.socket.timeout.ms=3

controller.message.queue.size=10



# Log configuration

num.partitions=12

message.max.bytes=100

auto.create.topics.enable=true

log.index.interval.bytes=4096

log.index.size.max.bytes=10485760

log.retention.hours=24

log.flush.interval.ms=1

log.flush.interval.messages=2

log.flush.scheduler.interval.ms=2000

log.roll.hours=24

log.retention.check.interval.ms=30

log.segment.bytes=1073741824



# ZK configuration

zookeeper.connect=10.61.19.84:2181,10.61.19.86:2181

zookeeper.connection.timeout.ms=18000

zookeeper.sync.time.ms=2000



# Socket server configuration

num.io.threads=8

num.network.threads=8

socket.request.max.bytes=104857600

socket.receive.buffer.bytes=1048576

socket.send.buffer.bytes=1048576

queued.max.requests=16

fetch.purgatory.purge.interval.requests=100

producer.purgatory.purge.interval.requests=100



Any help will be really appreciated.

Arun Rai

Re: Read Consistency

[arun sirimalla]

Scenario 1: Read query is fired for a key, data is found on one node and
not found on other two nodes who are responsible for the token
corresponding to key.

You read query will fail, as it expects to receive data from 2 nodes with
RF=3


Scenario 2: Read query is fired and all 3 replicas have different data with
different timestamps.

Read query will return the data with most recent timestamp and trigger a
read repair in the backend .

On Tue, Jun 23, 2015 at 10:57 AM, Anuj Wadehra 
wrote:

> Hi,
>
> Need to validate my understanding..
>
> RF=3 , Read CL = Quorum
>
> What would be returned to the client in following scenarios:
>
> Scenario 1: Read query is fired for a key, data is found on one node and
> not found on other two nodes who are responsible for the token
> corresponding to key.
>
> Options: no data is returned OR data from the only node having data is
> returned?
>
> Scenario 2: Read query is fired and all 3 replicas have different data
> with different timestamps.
>
> Options: data with latest timestamp is returned OR something else???
>
> Thanks
> Anuj
>
> Sent from Yahoo Mail on Android
> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>



-- 
Arun

Re: Read Consistency

[arun sirimalla]

So do you mean with CL set to QUORUM, if data is only on one node, the
query still succeeds.

On Tue, Jun 23, 2015 at 11:21 AM, Philip Thompson <
philip.thomp...@datastax.com> wrote:

> Anuj,
>
> In the first scenario, the data from the single node holding data is
> returned. The query will not fail if the consistency level is met, even if
> the read was inconsistent.
>
> On Tue, Jun 23, 2015 at 2:16 PM, Anuj Wadehra 
> wrote:
>
>> Why would it fail and with what Thrift error? What if the data didnt
>> exist on any of the nodes..query wont fail if doesnt find data..
>>
>> Not convinced..
>>
>> Sent from Yahoo Mail on Android
>> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>> --
>>   *From*:"arun sirimalla" 
>> *Date*:Tue, 23 Jun, 2015 at 11:39 pm
>> *Subject*:Re: Read Consistency
>>
>> Scenario 1: Read query is fired for a key, data is found on one node and
>> not found on other two nodes who are responsible for the token
>> corresponding to key.
>>
>> You read query will fail, as it expects to receive data from 2 nodes with
>> RF=3
>>
>>
>> Scenario 2: Read query is fired and all 3 replicas have different data
>> with different timestamps.
>>
>> Read query will return the data with most recent timestamp and trigger a
>> read repair in the backend .
>>
>> On Tue, Jun 23, 2015 at 10:57 AM, Anuj Wadehra 
>> wrote:
>>
>>> Hi,
>>>
>>> Need to validate my understanding..
>>>
>>> RF=3 , Read CL = Quorum
>>>
>>> What would be returned to the client in following scenarios:
>>>
>>> Scenario 1: Read query is fired for a key, data is found on one node and
>>> not found on other two nodes who are responsible for the token
>>> corresponding to key.
>>>
>>> Options: no data is returned OR data from the only node having data is
>>> returned?
>>>
>>> Scenario 2: Read query is fired and all 3 replicas have different data
>>> with different timestamps.
>>>
>>> Options: data with latest timestamp is returned OR something else???
>>>
>>> Thanks
>>> Anuj
>>>
>>> Sent from Yahoo Mail on Android
>>> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>>>
>>
>>
>>
>> --
>> Arun
>>
>
>


-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: Read Consistency

[arun sirimalla]

Thanks good to know that.

On Tue, Jun 23, 2015 at 11:27 AM, Philip Thompson <
philip.thomp...@datastax.com> wrote:

> Yes, that is what he means. CL is for how many nodes need to respond, not
> agree.
>
> On Tue, Jun 23, 2015 at 2:26 PM, arun sirimalla 
> wrote:
>
>> So do you mean with CL set to QUORUM, if data is only on one node, the
>> query still succeeds.
>>
>> On Tue, Jun 23, 2015 at 11:21 AM, Philip Thompson <
>> philip.thomp...@datastax.com> wrote:
>>
>>> Anuj,
>>>
>>> In the first scenario, the data from the single node holding data is
>>> returned. The query will not fail if the consistency level is met, even if
>>> the read was inconsistent.
>>>
>>> On Tue, Jun 23, 2015 at 2:16 PM, Anuj Wadehra 
>>> wrote:
>>>
>>>> Why would it fail and with what Thrift error? What if the data didnt
>>>> exist on any of the nodes..query wont fail if doesnt find data..
>>>>
>>>> Not convinced..
>>>>
>>>> Sent from Yahoo Mail on Android
>>>> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>>>> --
>>>>   *From*:"arun sirimalla" 
>>>> *Date*:Tue, 23 Jun, 2015 at 11:39 pm
>>>> *Subject*:Re: Read Consistency
>>>>
>>>> Scenario 1: Read query is fired for a key, data is found on one node
>>>> and not found on other two nodes who are responsible for the token
>>>> corresponding to key.
>>>>
>>>> You read query will fail, as it expects to receive data from 2 nodes
>>>> with RF=3
>>>>
>>>>
>>>> Scenario 2: Read query is fired and all 3 replicas have different data
>>>> with different timestamps.
>>>>
>>>> Read query will return the data with most recent timestamp and trigger
>>>> a read repair in the backend .
>>>>
>>>> On Tue, Jun 23, 2015 at 10:57 AM, Anuj Wadehra 
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Need to validate my understanding..
>>>>>
>>>>> RF=3 , Read CL = Quorum
>>>>>
>>>>> What would be returned to the client in following scenarios:
>>>>>
>>>>> Scenario 1: Read query is fired for a key, data is found on one node
>>>>> and not found on other two nodes who are responsible for the token
>>>>> corresponding to key.
>>>>>
>>>>> Options: no data is returned OR data from the only node having data is
>>>>> returned?
>>>>>
>>>>> Scenario 2: Read query is fired and all 3 replicas have different data
>>>>> with different timestamps.
>>>>>
>>>>> Options: data with latest timestamp is returned OR something else???
>>>>>
>>>>> Thanks
>>>>> Anuj
>>>>>
>>>>> Sent from Yahoo Mail on Android
>>>>> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Arun
>>>>
>>>
>>>
>>
>>
>> --
>> Arun
>> Senior Hadoop/Cassandra Engineer
>> Cloudwick
>>
>>
>> 2014 Data Impact Award Winner (Cloudera)
>>
>> http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
>>
>>
>


-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: 10000+ CF support from Cassandra

[Arun Chaitanya]

 any ideas or advises?

On Mon, Jun 22, 2015 at 10:55 AM, Arun Chaitanya 
wrote:

> Hello All,
>
> Now we settled on the following approach. I want to know if there are any
> problems that you foresee in the production environment.
>
> Our Approach: Use  Off Heap Memory
>
> Modifications to default cassandra.yaml and cassandra-env.sh
> 
>  * memory_allocator: JEMallocAllocator 
> (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>  * memtable_allocation_type: offheap_objects
>
>  By above two, the slab allocation 
> (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
>  1MB heap memory per table, is disabled. The memory for table metadata, 
> caches and memtable are thus
>  allocated natively and does not affect GC performance.
>
>  * tombstone_failure_threshold: 1
>
>Without this, C* throws TombstoneOverwhelmingException while in startup.
>This setting looks problematic so I want to know why just creating tables 
> makes so many tombstones ...
>
>  * -XX:+UseG1GC
>
>It is good for reducing GC time.
>Without this, full GCs > 1s are observed.
>
> We created 5000 column families with about 1000 entries per column family.
> The read/write performance seems to stable.
> The problem we saw is with startup time.
>
>  Cassandra Start Time (s) 20
>
>
>
> 349  Average CPU Usage (%) 40
>
>
>
> 49.65  GC Actitivy (%) 2.6
>
>
>
> 0.6
> Thanks a lot in advance.
>
> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson  wrote:
>
>> > I strongly advise against this approach.
>>> Jon, I think so too. But so you actually foresee any problems with this
>>> approach?
>>> I can think of a few. [I want to evaluate if we can live with this
>>> problem]
>>>
>>>
>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>> that it may be better than having 1+ CFs, which was the original
>> question (it really depends on the use case which wasn’t well defined)… map
>> size limit may be a problem, and then there is the CQL vs thrift question
>> which could start a flame war; ideally CQL maps should give you the same
>> flexibility as arbitrary thrift columns
>>
>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad  wrote:
>>
>> > Sorry for this naive question but how important is this tuning? Can
>> this have a huge impact in production?
>>
>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>> company:
>>
>>
>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>
>> About an order of magnitude difference in performance.
>>
>> Jon
>>
>> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya 
>> wrote:
>>
>>> Thanks Jon and Jack,
>>>
>>> > I strongly advise against this approach.
>>> Jon, I think so too. But so you actually foresee any problems with this
>>> approach?
>>> I can think of a few. [I want to evaluate if we can live with this
>>> problem]
>>>
>>>- No more CQL.
>>>- No data types, everything needs to be a blob.
>>>- Limited clustering Keys and default clustering order.
>>>
>>> > First off, different workloads need different tuning.
>>> Sorry for this naive question but how important is this tuning? Can
>>> this have a huge impact in production?
>>>
>>> > You might want to consider a model where you have an application layer
>>> that maps logical tenant tables into partition keys within a single large
>>> Casandra table, or at least a relatively small number of  Cassandra tables.
>>> It will depend on the typical size of your tenant tables - very small ones
>>> would make sense within a single partition, while larger ones should have
>>> separate partitions for a tenant's data. The key here is that tables are
>>> expensive, but partitions are cheap and scale very well with Cassandra.
>>> We are actually trying similar approach. But we don't want to expose
>>> this to application layer. We are attempting to hide this and provide an
>>> API.
>>>
>>> > Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>> want to consider a model where you do indeed have multiple clusters, where
>>> each handles a fraction of the tenants, since there is no need for separate
>>> tenants to be on the same cluster.
>>> I meant 10 clusters.

Re: 10000+ CF support from Cassandra

[Arun Chaitanya]

Hi Jack,

When I mean entries, I meant rows. Each column family has about 200 columns.

> Disabling of slab allocation is an expert-only feature - its use is
generally an anti-pattern, not recommended.
I understand this and have seen this recommendation at several places. I
want to understand the consequences? Is it performance, maintenance or
scalability, that is at stake.

In our use case we have about 3000 column families (ofcourse modelled in
RDBMS). If I were to limit to 250 column families, do you advise us to use
multiple clusters (the problem being cost ineffective)?

If we were to use a single cluster and support 3000 column families, the
only idea is to group few column families and store them in one column
family. In this case, grouping is a difficult task, imo. And if we want an
abstraction of grouping for developer, we need special connector for
Hadoop/Spark systems. So I do not want to enter this territory.

Sorry for such questions, but I am still wondering if I am the only one
facing this problem.

Thanks a lot,
Arun



On Wed, Jun 24, 2015 at 10:28 PM, Jack Krupansky 
wrote:

> By entries, do you mean rows or columns? Please clarify how many columns
> each of your tables has, and how many rows you are populating for each
> table.
>
> In case I didn't make it clear earlier, limit yourself to "low hundreds"
> (like 250) of tables and you should be fine. Thousands of tables is a clear
> anti-pattern for Cassandra - not recommended. If it works for you, great,
> but if not, don't say you weren't warned.
>
> Disabling of slab allocation is an expert-only feature - its use is
> generally an anti-pattern, not recommended.
>
> -- Jack Krupansky
>
> On Sun, Jun 21, 2015 at 10:55 PM, Arun Chaitanya 
> wrote:
>
>> Hello All,
>>
>> Now we settled on the following approach. I want to know if there are any
>> problems that you foresee in the production environment.
>>
>> Our Approach: Use  Off Heap Memory
>>
>> Modifications to default cassandra.yaml and cassandra-env.sh
>> 
>>  * memory_allocator: JEMallocAllocator 
>> (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>>  * memtable_allocation_type: offheap_objects
>>
>>  By above two, the slab allocation 
>> (https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
>>  1MB heap memory per table, is disabled. The memory for table metadata, 
>> caches and memtable are thus
>>  allocated natively and does not affect GC performance.
>>
>>  * tombstone_failure_threshold: 1
>>
>>Without this, C* throws TombstoneOverwhelmingException while in startup.
>>This setting looks problematic so I want to know why just creating tables 
>> makes so many tombstones ...
>>
>>  * -XX:+UseG1GC
>>
>>It is good for reducing GC time.
>>Without this, full GCs > 1s are observed.
>>
>> We created 5000 column families with about 1000 entries per column
>> family. The read/write performance seems to stable.
>> The problem we saw is with startup time.
>>
>>  Cassandra Start Time (s) 20
>>
>>
>>
>> 349  Average CPU Usage (%) 40
>>
>>
>>
>> 49.65  GC Actitivy (%) 2.6
>>
>>
>>
>> 0.6
>> Thanks a lot in advance.
>>
>> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson 
>> wrote:
>>
>>> > I strongly advise against this approach.
>>>> Jon, I think so too. But so you actually foresee any problems with this
>>>> approach?
>>>> I can think of a few. [I want to evaluate if we can live with this
>>>> problem]
>>>>
>>>>
>>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>>> that it may be better than having 1+ CFs, which was the original
>>> question (it really depends on the use case which wasn’t well defined)… map
>>> size limit may be a problem, and then there is the CQL vs thrift question
>>> which could start a flame war; ideally CQL maps should give you the same
>>> flexibility as arbitrary thrift columns
>>>
>>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad  wrote:
>>>
>>> > Sorry for this naive question but how important is this tuning? Can
>>> this have a huge impact in production?
>>>
>>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>>> company:
>>>
>>>
>>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>>
>>> About an order of ma

Re: Is it okay to use a small t2.micro instance for OpsCenter and use m3.medium instances for the actual Cassandra nodes?

[arun sirimalla]

Hi Sid,

I would recommend you to use either c3s or m3s instances for Opscenter and
for Cassandra nodes it depends on your use case.
You can go with either c3s or i2s for Cassandra nodes. But i would
recommend you to run performance tests before selecting the instance type.
If your use case requires more CPU i would recommend  c3s.

On Fri, Jun 26, 2015 at 1:20 PM, Sid Tantia 
wrote:

>  Hello, I haven’t been able to find any documentation for best practices
> on this…is it okay to set up opscenter as a smaller node than the rest of
> the cluster.
>
> For instance, on AWS can I have 3 m3.medium nodes for Cassandra and 1
> t2.micro node for OpsCenter?
>
>


-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Cassandra -stress write - Storage location

[Arun Sandu]

Hi,

I am currently working on load testing my cluster. When we write 10 to
cassandra, where does the writes data gets stored in Cassandra and the same
for read operation too.

./cassandra-stress write n=10 -rate threads=100 -node 10.34.100.13

./cassandra-stress read n=10 -node 10.34.100.13


-- 
Thanks
Arun

Re: Cassandra -stress write - Storage location

[Arun Sandu]

Thanks . Can I know, the format of the data that gets stored? Can you
please suggest me some ways to perform load testing? I need a big picture
of all the statistics.

Thanks again
Arun

On Thu, Oct 29, 2015 at 9:41 PM, Sebastian Estevez <
sebastian.este...@datastax.com> wrote:

> By default this will go in Keyspace1 Standard1.
>
> All the best,
>
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Sebastián Estévez
>
> Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com
>
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
> <http://goog_410786983>
>
>
> <http://www.datastax.com/gartner-magic-quadrant-odbms>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
> On Fri, Oct 30, 2015 at 12:07 AM, Arun Sandu 
> wrote:
>
>> Hi,
>>
>> I am currently working on load testing my cluster. When we write 10
>> to cassandra, where does the writes data gets stored in Cassandra and the
>> same for read operation too.
>>
>> ./cassandra-stress write n=10 -rate threads=100 -node 10.34.100.13
>>
>> ./cassandra-stress read n=10 -node 10.34.100.13
>>
>>
>> --
>> Thanks
>> Arun
>>
>
>


-- 
Thanks&Regards
Arun Kumar S
816-699-3039

*"This Moment Is Not Permanent...!!"*

Keyspaces missing after restarting cassandra service

[Arun Sandu]

Hello,

After restarting the cassandra, all of my keyspaces got missing. I can only
see system_traces, system, dse_system. I didn't make any changes to
cassandra.yaml.

But, I can see all the keyspaces data in my */data/ *directory. Is there
anyway to access those lost keyspaces through cqlsh?

Can someone help me with a solution?


-- 
Thanks&Regards
Arun

Hints timed out

[Arun Sandu]

Hi,

1. What would be the reason for hints getting timed out? None of the nodes
went down. using the storage_port as 7000. System.log displays the below
information:



INFO  [HintedHandoff:1] 2015-11-20 14:40:39,400
HintedHandOffManager.java:350 - Started hinted handoff for host:
e57e87c0-b8bf-4787-be00-1d48a9bf2cdb with IP: /10.34.200.55

INFO  [HANDSHAKE-/10.34.200.55] 2015-11-20 14:40:39,437
OutboundTcpConnection.java:485 - Handshaking version with /10.34.200.55

INFO  [HANDSHAKE-/10.34.200.55] 2015-11-20 14:40:39,461
OutboundTcpConnection.java:485 - Handshaking version with /10.34.200.55
INFO  [HintedHandoff:1] 2015-11-20 14:40:41,418
HintedHandOffManager.java:469 - Timed out replaying hints to /10.34.200.55;
aborting (0 delivered)


2. Repair is failing. Is this because of the hints getting timeout?

ERROR [AntiEntropyStage:2] 2015-11-20 14:10:58,824
RepairMessageVerbHandler.java:140 - Got error, removing parent repair
session
ERROR [AntiEntropyStage:2] 2015-11-20 14:10:58,825
CassandraDaemon.java:222 - Exception in thread
Thread[AntiEntropyStage:2,5,main]

Can someone help me understand the issues?

Thanks
Arun

Snitch for AWS EC2 nondefaultVPC

[Arun Sandu]

Hi all,

All our nodes are launched in AWS EC2 VPC (private). We have 2
datacenters(1 us-east , 1- asiapacific) and all communication is through
private IP's and don't have any public IPs. What is the recommended snitch
to be used? We currently have GossipingPropertyFileSnitch.

1. If Ec2MultiRegionSnitch, then what would be the broadcast_address?
2. If not Ec2MultiRegionSnitch, which snitch better fits this environment?

*Ref:*
As per the document Ec2MultiRegionSnitch
<http://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchEC2MultiRegion_c.html>,set
the listen_address
<http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__listen_address>
to the *private* IP address of the node, and the broadcast_address
<http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__broadcast_address>
to the *public* IP address of the node.

-- 
Thanks
Arun

Re: Snitch for AWS EC2 nondefaultVPC

[Arun Sandu]

Thanks Asher.

Yes, I agree. It would be better if someone can help us with clear
documentation about this.

As the cross data communication is through private IP, I would consider
updating braodcast_address to private IP and use Ec2MultiRegionSnitch.

On Tue, Mar 1, 2016 at 3:26 PM, Asher Newcomer  wrote:

> Hi Arun,
>
> This distinction has been a can of worms for me also - and I'm not sure my
> understanding is entirely correct.
>
> I use GossipingPropertyFileSnitch for my multi-region setup, which seems
> to be more flexible than the Ec2 snitches. The Ec2 snitches should work
> also, but their behavior is more opaque from my perspective.
>
> AFAIK - if all of your nodes can reach each other via private IP, and your
> anticipated clients can reach all nodes via their private IP, then using
> the private IP address as the broadcast_address is fine.
>
> If there will ever be a situation where a client or node will need to
> reach some part of the cluster using public IPs, then public IPs should be
> used as the broadcast_address.
>
> A simple flow-chart / diagram of how these various settings are used by
> Cassandra would be very helpful for people new to the project.
>
> Regards,
>
> Asher
>
> On Tue, Mar 1, 2016 at 3:12 PM, Arun Sandu  wrote:
>
>> Hi all,
>>
>> All our nodes are launched in AWS EC2 VPC (private). We have 2
>> datacenters(1 us-east , 1- asiapacific) and all communication is through
>> private IP's and don't have any public IPs. What is the recommended snitch
>> to be used? We currently have GossipingPropertyFileSnitch.
>>
>> 1. If Ec2MultiRegionSnitch, then what would be the broadcast_address?
>> 2. If not Ec2MultiRegionSnitch, which snitch better fits this environment?
>>
>> *Ref:*
>> As per the document Ec2MultiRegionSnitch
>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/architecture/architectureSnitchEC2MultiRegion_c.html>,set
>> the listen_address
>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__listen_address>
>> to the *private* IP address of the node, and the broadcast_address
>> <http://docs.datastax.com/en/cassandra/2.1/cassandra/configuration/configCassandra_yaml_r.html#reference_ds_qfg_n1r_1k__broadcast_address>
>> to the *public* IP address of the node.
>>
>> --
>> Thanks
>> Arun
>>
>
>


-- 
Thanks
Arun

Re: Snitch for AWS EC2 nondefaultVPC

[Arun Sandu]

Thanks Robert,

All the nodes in both datacenters are in DSE Search Mode(Solr). We may have
analytics datacenter as well in future. Will this have any impact in using
Ec2MultiRegionSnitch?

On Tue, Mar 1, 2016 at 7:10 PM, Robert Coli  wrote:

> On Tue, Mar 1, 2016 at 12:12 PM, Arun Sandu 
> wrote:
>
>> All our nodes are launched in AWS EC2 VPC (private). We have 2
>> datacenters(1 us-east , 1- asiapacific) and all communication is through
>> private IP's and don't have any public IPs. What is the recommended snitch
>> to be used? We currently have GossipingPropertyFileSnitch.
>>
>
> I recommend using GPFS unless you're absolutely certain you will never
> want to rely on any host but Amazon, and you will never want to (for
> example) have an analytics pseudo-datacenter within AWS.
>
> =Rob
>
>


-- 
Arun

High Compactions Pending

[arun sirimalla]

I have a 6 (i2.2xlarge) node cluster on AWS with 4.5 DSE running on it. I
notice high compaction pending on one of the node around 35.
Compaction throughput set to 64 MB and flush writes to 4. Any suggestion is
 much appreciated.

-- 
Arun
Senior Hadoop Engineer
Cloudwick

Champion of Big Data
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html

Re: Best Practice to add a node in a Cluster

[arun sirimalla]

Hi Neha,


After you add the node to the cluster, run nodetool cleanup on all nodes.
Next running repair on each node will replicate the data. Make sure you run
the repair on one node at a time, because repair is an expensive process
(Utilizes high CPU).




On Mon, Apr 27, 2015 at 8:36 PM, Neha Trivedi 
wrote:

> Thanks Eric and Matt :) !!
>
> Yes the purpose is to improve reliability.
> Right now, from our driver we are querying using degradePolicy for
> reliability.
>
>
>
> *For changing the keyspace for RF=3, the procedure is as under:*
> 1. Add a new node to the cluster (new node is not in seed list)
>
> 2. ALTER KEYSPACE system_auth WITH REPLICATION =
>   {'class' : 'NetworkTopologyStrategy', 'dc1' : 3};
>
>
>1. On each affected node, run nodetool repair
>
> <http://docs.datastax.com/en/cassandra/1.2/cassandra/tools/toolsNodetool_r.html>.
>
>2. Wait until repair completes on a node, then move to the next node.
>
>
> Any other things to take care?
>
> Thanks
> Regards
> neha
>
>
> On Mon, Apr 27, 2015 at 9:45 PM, Eric Stevens  wrote:
>
>> It depends on why you're adding a new node.  If you're running out of
>> disk space or IO capacity in your 2 node cluster, then changing RF to 3
>> will not improve either condition - you'd still be writing all data to all
>> three nodes.
>>
>> However if you're looking to improve reliability, a 2 node RF=2 cluster
>> cannot have either node offline without losing quorum, while a 3 node RF=3
>> cluster can have one node offline and still be able to achieve quorum.
>> RF=3 is a common replication factor because of this characteristic.
>>
>> Make sure your new node is not in its own seeds list, or it will not
>> bootstrap (it will come online immediately and start serving requests).
>>
>> On Mon, Apr 27, 2015 at 8:46 AM, Neha Trivedi 
>> wrote:
>>
>>> Hi
>>> We have a 2 Cluster Node with RF=2. We are planing to add a new node.
>>>
>>> Should we change RF to 3 in the schema?
>>> OR Just added a new node with the same RF=2?
>>>
>>> Any other Best Practice that we need to take care?
>>>
>>> Thanks
>>> regards
>>> Neha
>>>
>>>
>>
>


-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick

Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html

2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: calculation of disk size

[arun sirimalla]

Hi Rahul,

If you are expecting 15 GB of data per day, here is the calculation.

1 Day = 15 GB, 1 Month = 450 GB, 1 Year = 5.4 TB, so your raw data size for
one year is 5.4 TB with replication factor of 3 it would be around 16.2 TB
of data for one year.

Taking compaction into consideration and your use case being write heavy,
if you go with size tiered compaction. you would need twice the space of
your raw data.

So you would need around 32-34 TB of disk space.

Reference:
http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architecturePlanningDiskCapacity_t.html

Thanks

On Wed, Apr 29, 2015 at 9:20 PM, Rahul Bhardwaj <
rahul.bhard...@indiamart.com> wrote:

> Hi All,
>
>
> We are planning to set up a cluster of 5 nodes with RF 3 for write heavy
> project, our current database size is around 500 GB. And it is growing at
> rate of 15 GB every day. We learnt that cassandra consumes space for
> compaction processes, So how can we calculate the amount of disk space we
> would require.
>
> Kindly suggest.
>
>
>
> Regards:
> Rahul Bhardwaj
>
>
> Follow IndiaMART.com <http://www.indiamart.com> for latest updates on
> this and more: <https://plus.google.com/+indiamart>
> <https://www.facebook.com/IndiaMART> <https://twitter.com/IndiaMART>
> Mobile Channel:
> <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
> <https://play.google.com/store/apps/details?id=com.indiamart.m>
> <http://m.indiamart.com/>
>
> <https://www.youtube.com/watch?v=DzORNbeSXN8&list=PL2o4J51MqpL0mbue6kzDa6eymLVUXtlR1&index=2>
> Watch how IndiaMART Maximiser helped Mr. Khanna expand his business.
> kyunki Kaam Yahin Banta Hai <https://www.youtube.com/watch?v=cy1jiNXrzxc>
> !!!




-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick

Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html

2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: Can a Cassandra node accept writes while being repaired

[arun sirimalla]

Yes, Cassandra nodes accept writes during Repair. Also Repair triggers
compactions to remove any tombstones.

On Thu, May 7, 2015 at 9:31 AM, Khaja, Raziuddin (NIH/NLM/NCBI) [C] <
raziuddin.kh...@nih.gov> wrote:

> I was not able to find a conclusive answer to this question on the
> internet so I am asking this question here.
> Is a Cassandra node able to accept insert or delete operations while the
> node is being repaired?
> Thanks
> -Razi
>



-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick

Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html

2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: Unexpected behavior after adding successffully a new node

[arun sirimalla]

Analia,

Try running repair on node 3.

On Tue, May 12, 2015 at 7:39 AM, Analia Lorenzatto <
analialorenza...@gmail.com> wrote:

> Hello guys,
>
>
> I have a cluster 2.1.0-2 comprised of 3 nodes.  The replication factor=2.
> We successfully added the third node last week.  After that, We ran clean
> ups on one node at that time.  Then we ran repairs on all the nodes, and
> finally compactions on all the CFs.
>
> Last night, I noticed the cluster started behaving in a weird way.  The
> last node (successfully added last week) were being reported up and down
> all the time.  I could see a lot of messages like this on logs:
>
> WARN  [SharedPool-Worker-33] 2015-05-11 21:31:45,125
> AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[SharedPool-Worker-33,5,main]: {}
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
> (No such file or directory)
>
> At the same time the consumption of heap used was on the top, up to the
> point the rest of the cluster saw this node as down.  After that, I just
> restarted the cassandra service with no problems on that node.
>
> Now, I can see the three nodes on the cluster Up and Normal, but this last
> node (which was rebooted) does not have data.  But it has all the structure
> of cassandra data.
>
> I can query against the new node and I get the same result as if do the
> query against the others nodes.  But, on this new node I do not have any
> SStables:
>
>
>
> root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
> Datacenter: us-east
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address Load   Tokens  Owns (effective)  Host ID
> Rack
> UN  10.0.0.a  390.28 GB  256 66.7%
> eed9e9f5-f279-4b2f-b521-c056cbf65b52  1c
> UN  10.0.0.b  382.36 GB  256 68.3%
> 19492c26-4458-4a0b-af04-72e0aab6598e  1c
> UN  10.0.0.c  40.61 MB   256 64.9%
> b8da952c-24b3-444a-a34e-7a1804eee6e6  1c
>
> What do you recommend to do? Leave this as if, remove it and try to join
> this or a new one?
> Thanks in advance!!
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life".  By Captain Jean-Luc Picard.
>



-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick

Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html

2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

10000+ CF support from Cassandra

[Arun Chaitanya]

Good Day Everyone,

I am very happy with the (almost) linear scalability offered by C*. We had
a lot of problems with RDBMS.

But, I heard that C* has a limit on number of column families that can be
created in a single cluster.
The reason being each CF stores 1-2 MB on the JVM heap.

In our use case, we have about 1+ CF and we want to support
multi-tenancy.
(i.e 1 * no of tenants)

We are new to C* and being from RDBMS background, I would like to
understand how to tackle this scenario from your advice.

Our plan is to use Off-Heap memtable approach.
http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1

Each node in the cluster has following configuration
16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
IMO, this should be able to support 1000 CF with no(very less) impact on
performance and startup time.

We tackle multi-tenancy using different keyspaces.(Solution I found on the
web)

Using this approach we can have 10 clusters doing the job. (We actually are
worried about the cost)

Can you please help us evaluate this strategy? I want to hear communities
opinion on this.

My major concerns being,

1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF
right?

2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number of
column families increase even when we use multiple keyspace.

3. I understand the complexity using multi-cluster for single application.
The code base will get tightly coupled with infrastructure. Is this the
right approach?

Any suggestion is appreciated.

Thanks,
Arun

Re: 10000+ CF support from Cassandra

[Arun Chaitanya]

Hello Graham,

> Are the CFs different, or all the same schema?
The column families are different. May be with better data modelling, we
can combine a few of them.

> Are you contractually obligated to actually separate data into separate
CFs?
No. Its just that we have several sub systems(around 100) and the data is
different.

> It seems like you’d have a lot simpler time if you could use the part of
the partition key to separate data.
I didn't understand this approach. You mean, we combine some of them using
an extra partition key?
But the columns are of different schema, isn't it? Sorry I might be
understanding it wrong.

> Note also, I don’t know what disks you are using, but disk cache can be
pretty helpful, and you haven’t allowed for any in your machine sizing. Of
course that depends on your stored data volume also.
OK. This is new information, I will consider this.

> Also hard to answer your questions without an idea of read/write load
system wide, and indeed distribution across tenants.
The read write actually depends actually on column family and tenant. Some
analytic jobs read data on some CF, some online event jobs write a lot of
data.

Thanks a lot for your insight. Anyways my previous questions still remain
unclear for me.

Arun

On Wed, May 27, 2015 at 11:40 AM, graham sanderson  wrote:

> Are the CFs different, or all the same schema? Are you contractually
> obligated to actually separate data into separate CFs? It seems like you’d
> have a lot simpler time if you could use the part of the partition key to
> separate data.
>
> Note also, I don’t know what disks you are using, but disk cache can be
> pretty helpful, and you haven’t allowed for any in your machine sizing. Of
> course that depends on your stored data volume also.
>
> Also hard to answer your questions without an idea of read/write load
> system wide, and indeed distribution across tenants.
>
>
> On May 26, 2015, at 10:32 PM, Arun Chaitanya 
> wrote:
>
> Good Day Everyone,
>
> I am very happy with the (almost) linear scalability offered by C*. We had
> a lot of problems with RDBMS.
>
> But, I heard that C* has a limit on number of column families that can be
> created in a single cluster.
> The reason being each CF stores 1-2 MB on the JVM heap.
>
> In our use case, we have about 1+ CF and we want to support
> multi-tenancy.
> (i.e 1 * no of tenants)
>
> We are new to C* and being from RDBMS background, I would like to
> understand how to tackle this scenario from your advice.
>
> Our plan is to use Off-Heap memtable approach.
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>
> Each node in the cluster has following configuration
> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
> IMO, this should be able to support 1000 CF with no(very less) impact on
> performance and startup time.
>
> We tackle multi-tenancy using different keyspaces.(Solution I found on the
> web)
>
> Using this approach we can have 10 clusters doing the job. (We actually
> are worried about the cost)
>
> Can you please help us evaluate this strategy? I want to hear communities
> opinion on this.
>
> My major concerns being,
>
> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000 CF
> right?
>
> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
> of column families increase even when we use multiple keyspace.
>
> 3. I understand the complexity using multi-cluster for single application.
> The code base will get tightly coupled with infrastructure. Is this the
> right approach?
>
> Any suggestion is appreciated.
>
> Thanks,
> Arun
>
>
>

Re: EC2snitch in AWS

[arun sirimalla]

Hi Kaushal,

Here is the reference,
http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html

On Wed, May 27, 2015 at 9:31 AM, Kaushal Shriyan 
wrote:

> Hi,
>
> Can somebody please share me details about setting up of EC2snitch in AWS
> single region which has availability zone 1a and 1b? I am using Cassandra
> version 1.2.19 in the setup.
>
> I would appreciate your help.
>
> Regards,
>
> Kaushal
>



-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: 10000+ CF support from Cassandra

[Arun Chaitanya]

Hello Jack,

> Column families? As opposed to tables? Are you using Thrift instead of
CQL3? You should be focusing on the latter, not the former.
We have an ORM developed in our company, which maps each DTO to a column
family. So, we have many column families. We are using CQL3.

> But either way, the general guidance is that there is no absolute limit
of tables per se, but "low hundreds" is the recommended limit, regardless
of whether how many key spaces they may be divided
> between. More than that is an anti-pattern for Cassandra - maybe you can
make it work for your application, but it isn't recommended.
You want to say that most cassandra users don't have more than 2-300 column
families? Is this achieved through careful data modelling?

> A successful Cassandra deployment is critically dependent on careful data
modeling - who is responsible for modeling each of these tables, you and a
single, tightly-knit team with very common interests > and very specific
goals and SLAs or many different developers with different managers with
different goals such as SLAs?
The latter.

> When you say multi-tenant, are you simply saying that each of your
organization's customers has their data segregated, or does each customer
have direct access to the cluster?
Each organization's data is in the same cluster. No customer doesn't have
access to the cluster.

Thanks,
Arun

On Wed, May 27, 2015 at 7:17 PM, Jack Krupansky 
wrote:

> Scalability of Cassandra refers primarily to number of rows and number of
> nodes - to add more data, add more nodes.
>
> Column families? As opposed to tables? Are you using Thrift instead of
> CQL3? You should be focusing on the latter, not the former.
>
> But either way, the general guidance is that there is no absolute limit of
> tables per se, but "low hundreds" is the recommended limit, regardless of
> whether how many key spaces they may be divided between. More than that is
> an anti-pattern for Cassandra - maybe you can make it work for your
> application, but it isn't recommended.
>
> A successful Cassandra deployment is critically dependent on careful data
> modeling - who is responsible for modeling each of these tables, you and a
> single, tightly-knit team with very common interests and very specific
> goals and SLAs or many different developers with different managers with
> different goals such as SLAs?
>
> When you say multi-tenant, are you simply saying that each of your
> organization's customers has their data segregated, or does each customer
> have direct access to the cluster?
>
>
>
>
>
> -- Jack Krupansky
>
> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya 
> wrote:
>
>> Good Day Everyone,
>>
>> I am very happy with the (almost) linear scalability offered by C*. We
>> had a lot of problems with RDBMS.
>>
>> But, I heard that C* has a limit on number of column families that can be
>> created in a single cluster.
>> The reason being each CF stores 1-2 MB on the JVM heap.
>>
>> In our use case, we have about 1+ CF and we want to support
>> multi-tenancy.
>> (i.e 1 * no of tenants)
>>
>> We are new to C* and being from RDBMS background, I would like to
>> understand how to tackle this scenario from your advice.
>>
>> Our plan is to use Off-Heap memtable approach.
>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>
>> Each node in the cluster has following configuration
>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>> IMO, this should be able to support 1000 CF with no(very less) impact on
>> performance and startup time.
>>
>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>> the web)
>>
>> Using this approach we can have 10 clusters doing the job. (We actually
>> are worried about the cost)
>>
>> Can you please help us evaluate this strategy? I want to hear communities
>> opinion on this.
>>
>> My major concerns being,
>>
>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>> CF right?
>>
>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
>> of column families increase even when we use multiple keyspace.
>>
>> 3. I understand the complexity using multi-cluster for single
>> application. The code base will get tightly coupled with infrastructure. Is
>> this the right approach?
>>
>> Any suggestion is appreciated.
>>
>> Thanks,
>> Arun
>>
>
>

Re: 10000+ CF support from Cassandra

[Arun Chaitanya]

Thanks Jon and Jack,

> I strongly advise against this approach.
Jon, I think so too. But so you actually foresee any problems with this
approach?
I can think of a few. [I want to evaluate if we can live with this problem]

   - No more CQL.
   - No data types, everything needs to be a blob.
   - Limited clustering Keys and default clustering order.

> First off, different workloads need different tuning.
Sorry for this naive question but how important is this tuning? Can this
have a huge impact in production?

> You might want to consider a model where you have an application layer
that maps logical tenant tables into partition keys within a single large
Casandra table, or at least a relatively small number of  Cassandra tables.
It will depend on the typical size of your tenant tables - very small ones
would make sense within a single partition, while larger ones should have
separate partitions for a tenant's data. The key here is that tables are
expensive, but partitions are cheap and scale very well with Cassandra.
We are actually trying similar approach. But we don't want to expose this
to application layer. We are attempting to hide this and provide an API.

> Finally, you said "10 clusters", but did you mean 10 nodes? You might
want to consider a model where you do indeed have multiple clusters, where
each handles a fraction of the tenants, since there is no need for separate
tenants to be on the same cluster.
I meant 10 clusters. We want to split our tables across multiple clusters
if above approach is not possible. [But it seems to be very costly]

Thanks,







On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky 
wrote:

> How big is each of the tables - are they all fairly small or fairly large?
> Small as in no more than thousands of rows or large as in tens of millions
> or hundreds of millions of rows?
>
> Small tables are are not ideal for a Cassandra cluster since the rows
> would be spread out across the nodes, even though it might make more sense
> for each small table to be on a single node.
>
> You might want to consider a model where you have an application layer
> that maps logical tenant tables into partition keys within a single large
> Casandra table, or at least a relatively small number of Cassandra tables.
> It will depend on the typical size of your tenant tables - very small ones
> would make sense within a single partition, while larger ones should have
> separate partitions for a tenant's data. The key here is that tables are
> expensive, but partitions are cheap and scale very well with Cassandra.
>
> Finally, you said "10 clusters", but did you mean 10 nodes? You might want
> to consider a model where you do indeed have multiple clusters, where each
> handles a fraction of the tenants, since there is no need for separate
> tenants to be on the same cluster.
>
>
> -- Jack Krupansky
>
> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya 
> wrote:
>
>> Good Day Everyone,
>>
>> I am very happy with the (almost) linear scalability offered by C*. We
>> had a lot of problems with RDBMS.
>>
>> But, I heard that C* has a limit on number of column families that can be
>> created in a single cluster.
>> The reason being each CF stores 1-2 MB on the JVM heap.
>>
>> In our use case, we have about 1+ CF and we want to support
>> multi-tenancy.
>> (i.e 1 * no of tenants)
>>
>> We are new to C* and being from RDBMS background, I would like to
>> understand how to tackle this scenario from your advice.
>>
>> Our plan is to use Off-Heap memtable approach.
>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>
>> Each node in the cluster has following configuration
>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>> IMO, this should be able to support 1000 CF with no(very less) impact on
>> performance and startup time.
>>
>> We tackle multi-tenancy using different keyspaces.(Solution I found on
>> the web)
>>
>> Using this approach we can have 10 clusters doing the job. (We actually
>> are worried about the cost)
>>
>> Can you please help us evaluate this strategy? I want to hear communities
>> opinion on this.
>>
>> My major concerns being,
>>
>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting 1000
>> CF right?
>>
>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO, the number
>> of column families increase even when we use multiple keyspace.
>>
>> 3. I understand the complexity using multi-cluster for single
>> application. The code base will get tightly coupled with infrastructure. Is
>> this the right approach?
>>
>> Any suggestion is appreciated.
>>
>> Thanks,
>> Arun
>>
>
>

Re: nodetool repair

[arun sirimalla]

Hi Jean,

Running nodetool repair on a node will repair only that node in the
cluster. It is recommended to run nodetool repair on one node at a time.

Few things to keep in mind while running repair
   1. Running repair will trigger compactions
   2. Increase in CPU utilization.


Run node tool repair with -pr option, so that it will repair only the range
that node is responsible for.

On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay <
jean.tremb...@zen-innovations.com> wrote:

>  Thanks Jonathan.
>
>  But I need to know the following:
>
>  If you issue a “nodetool repair” on one node will it repair all the
> nodes in the cluster or only the one on which we issue the command?
>
>If it repairs only one node, do I have to wait that the nodetool
> repair ends, and only then issue another “nodetool repair” on the next node?
>
>  Kind regards
>
>  On 18 Jun 2015, at 19:19 , Jonathan Haddad  wrote:
>
>  If you're using DSE, you can schedule it automatically using the repair
> service.  If you're open source, check out Spotify cassandra reaper, it'll
> manage it for you.
>
>  https://github.com/spotify/cassandra-reaper
>
>
>
>  On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay <
> jean.tremb...@zen-innovations.com> wrote:
>
>> Hi,
>>
>> I want to make on a regular base repairs on my cluster as suggested by
>> the documentation.
>> I want to do this in a way that the cluster is still responding to read
>> requests.
>> So I understand that I should not use the -par switch for that as it will
>> do the repair in parallel and consume all available resources.
>>
>> If you issue a “nodetool repair” on one node will it repair all the nodes
>> in the cluster or only the one on which we issue the command?
>>
>> If it repairs only one node, do I have to wait that the nodetool repair
>> ends, and only then issue another “nodetool repair” on the next node?
>>
>> If we had down time periods I would issue a nodetool -par, but we don’t
>> have down time periods.
>>
>> Sorry for the stupid questions.
>> Thanks for your help.
>
>
>


-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: nodetool repair

[arun sirimalla]

Yes compactions will remove tombstones

On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay <
jean.tremb...@zen-innovations.com> wrote:

>  Perfect thank you.
> So making a weekly "nodetool repair -pr”  on all nodes one after the other
> will repair my cluster. That is great.
>
>  If it does a compaction, does it mean that it would also clean up my
> tombstone from my LeveledCompactionStrategy tables at the same time?
>
>  Thanks for your help.
>
>  On 19 Jun 2015, at 07:56 , arun sirimalla  wrote:
>
>  Hi Jean,
>
>  Running nodetool repair on a node will repair only that node in the
> cluster. It is recommended to run nodetool repair on one node at a time.
>
>  Few things to keep in mind while running repair
>1. Running repair will trigger compactions
>2. Increase in CPU utilization.
>
>
>  Run node tool repair with -pr option, so that it will repair only the
> range that node is responsible for.
>
> On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay <
> jean.tremb...@zen-innovations.com> wrote:
>
>> Thanks Jonathan.
>>
>>  But I need to know the following:
>>
>>  If you issue a “nodetool repair” on one node will it repair all the
>> nodes in the cluster or only the one on which we issue the command?
>>
>>If it repairs only one node, do I have to wait that the nodetool
>> repair ends, and only then issue another “nodetool repair” on the next node?
>>
>>  Kind regards
>>
>>  On 18 Jun 2015, at 19:19 , Jonathan Haddad  wrote:
>>
>>  If you're using DSE, you can schedule it automatically using the repair
>> service.  If you're open source, check out Spotify cassandra reaper, it'll
>> manage it for you.
>>
>>  https://github.com/spotify/cassandra-reaper
>>
>>
>>
>>  On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay <
>> jean.tremb...@zen-innovations.com> wrote:
>>
>>> Hi,
>>>
>>> I want to make on a regular base repairs on my cluster as suggested by
>>> the documentation.
>>> I want to do this in a way that the cluster is still responding to read
>>> requests.
>>> So I understand that I should not use the -par switch for that as it
>>> will do the repair in parallel and consume all available resources.
>>>
>>> If you issue a “nodetool repair” on one node will it repair all the
>>> nodes in the cluster or only the one on which we issue the command?
>>>
>>> If it repairs only one node, do I have to wait that the nodetool repair
>>> ends, and only then issue another “nodetool repair” on the next node?
>>>
>>> If we had down time periods I would issue a nodetool -par, but we don’t
>>> have down time periods.
>>>
>>> Sorry for the stupid questions.
>>> Thanks for your help.
>>
>>
>>
>
>
>  --
> Arun
> Senior Hadoop/Cassandra Engineer
> Cloudwick
>
>
>  2014 Data Impact Award Winner (Cloudera)
>
> http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
>
>
>


-- 
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick


2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html

Re: 10000+ CF support from Cassandra

[Arun Chaitanya]

Hello All,

Now we settled on the following approach. I want to know if there are any
problems that you foresee in the production environment.

Our Approach: Use  Off Heap Memory

Modifications to default cassandra.yaml and cassandra-env.sh

 * memory_allocator: JEMallocAllocator
(https://issues.apache.org/jira/browse/CASSANDRA-7883)
 * memtable_allocation_type: offheap_objects

 By above two, the slab allocation
(https://issues.apache.org/jira/browse/CASSANDRA-5935), which requires
 1MB heap memory per table, is disabled. The memory for table
metadata, caches and memtable are thus
 allocated natively and does not affect GC performance.

 * tombstone_failure_threshold: 1

   Without this, C* throws TombstoneOverwhelmingException while in startup.
   This setting looks problematic so I want to know why just creating
tables makes so many tombstones ...

 * -XX:+UseG1GC

   It is good for reducing GC time.
   Without this, full GCs > 1s are observed.

We created 5000 column families with about 1000 entries per column family.
The read/write performance seems to stable.
The problem we saw is with startup time.

 Cassandra Start Time (s) 20



349  Average CPU Usage (%) 40



49.65  GC Actitivy (%) 2.6



0.6
Thanks a lot in advance.

On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson  wrote:

> > I strongly advise against this approach.
>> Jon, I think so too. But so you actually foresee any problems with this
>> approach?
>> I can think of a few. [I want to evaluate if we can live with this
>> problem]
>>
>>
>> Just to be clear, I’m not saying this is a great approach, I AM saying
> that it may be better than having 1+ CFs, which was the original
> question (it really depends on the use case which wasn’t well defined)… map
> size limit may be a problem, and then there is the CQL vs thrift question
> which could start a flame war; ideally CQL maps should give you the same
> flexibility as arbitrary thrift columns
>
> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad  wrote:
>
> > Sorry for this naive question but how important is this tuning? Can
> this have a huge impact in production?
>
> Massive.  Here's a graph of when we did some JVM tuning at my previous
> company:
>
>
> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>
> About an order of magnitude difference in performance.
>
> Jon
>
> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya 
> wrote:
>
>> Thanks Jon and Jack,
>>
>> > I strongly advise against this approach.
>> Jon, I think so too. But so you actually foresee any problems with this
>> approach?
>> I can think of a few. [I want to evaluate if we can live with this
>> problem]
>>
>>- No more CQL.
>>- No data types, everything needs to be a blob.
>>- Limited clustering Keys and default clustering order.
>>
>> > First off, different workloads need different tuning.
>> Sorry for this naive question but how important is this tuning? Can this
>> have a huge impact in production?
>>
>> > You might want to consider a model where you have an application layer
>> that maps logical tenant tables into partition keys within a single large
>> Casandra table, or at least a relatively small number of  Cassandra tables.
>> It will depend on the typical size of your tenant tables - very small ones
>> would make sense within a single partition, while larger ones should have
>> separate partitions for a tenant's data. The key here is that tables are
>> expensive, but partitions are cheap and scale very well with Cassandra.
>> We are actually trying similar approach. But we don't want to expose this
>> to application layer. We are attempting to hide this and provide an API.
>>
>> > Finally, you said "10 clusters", but did you mean 10 nodes? You might
>> want to consider a model where you do indeed have multiple clusters, where
>> each handles a fraction of the tenants, since there is no need for separate
>> tenants to be on the same cluster.
>> I meant 10 clusters. We want to split our tables across multiple clusters
>> if above approach is not possible. [But it seems to be very costly]
>>
>> Thanks,
>>
>>
>>
>>
>>
>>
>>
>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky > > wrote:
>>
>>> How big is each of the tables - are they all fairly small or fairly
>>> large? Small as in no more than thousands of rows or large as in tens of
>>> millions or hundreds of millions of rows?
>>>
>>> Small tables are are not ideal for a Cas

question on capacity planning

[Jacob, Arun]

if I'm planning to store 20TB of new data per week, and expire all data every 2 
weeks, with a replication factor of 3, do I only need approximately 120 TB of 
disk? I'm going to use ttl in my column values to automatically expire data. Or 
would I need more capacity to handle sstable merges? Given this amount of data, 
would you recommend node storage at 2TB per node or more? This application will 
have a heavy write /moderate read use profile.

-- Arun

Cassandra Node Requirements

[Jacob, Arun]

I'm trying to determine a node configuration for Cassandra. From what I've been 
able to determine from reading around:


 1.  we need to cap data size at 50% of total node storage capacity for 
compaction
 2.  with RF=3, that means that I need to effectively assume that I have 1/6th 
of total storage capacity.
 3.  SSDs are preferred, but of course  reduce storage capacity
 4.  using standard storage means you bump up your RAM to keep as much in 
memory as possible.

Right now we are looking at storage requirements of 42 – 60TB, assuming a 
baseline of 3TB/day and expiring data after 14-20 days (depending on use case), 
 I would assume based on above that we need 252- 360TB total storage max.

My questions:

 1.  is 8TB (meaning 1.33 actual TB storage/node) a reasonable per node storage 
size for Cassandra? I don’t want to use SSDs due to reduced storage capacity -- 
I don't want to buy 100s of nodes to support that reduced storage capacity of 
SSDs.  Given that I will be using standard drives, what is a 
reasonable/effective per node storage capacity?
 2.  other than splitting the commit log onto a separate drive, is there any 
other drive allocation I should be doing?
 3.  Assuming I'm not using SSDs, what would be a good memory size for a node? 
I've heard anything from 32-48 GB, but need more guidance.

Anything else that anyone has run into? What are common configurations being 
used by others?

Thanks in advance,

-- Arun

Re: Cassandra Node Requirements

[Jacob, Arun]

Thanks for the links and the answers. The vagueness of my initial questions 
reflects the fact that I'm trying to configure for a general case — I will 
clarify below:

I need to account for a variety of use cases.
(1) they will be both read and write heavy.  I was assuming that SSDs would be 
really good to handle the heavy read load, but with the amount of data I need 
to store, SSDs arent economical.
(2) I should have clarified, the main use case has  95% of writes going to a 
single column family. The other CFs are going to be much smaller in relation to 
the primary CF, which will be sized to sizes below. Given that this is the 
case, are my assumptions about storage correct for that use case?
(3) In  that use case, the majority of reads will actually come from the most 
recently inserted 7% of the data. In other use cases, reads will be random. 
Another use case uses Solandra, and I am assuming that use case results in 
random reads.

Assuming 250-360TB storage, need for the primary use case, I'm still trying to 
determine  how many nodes  I need to stand up to service that much data. What 
is a reasonable amount of storage per node? You mentioned memory to storage 
ratio: I'm assuming that ratio trends higher with the more random reads you do. 
Could you provide an example ratio for a heavy read use case?

Thanks,

-- Arun

From: Edward Capriolo mailto:edlinuxg...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Wed, 24 Aug 2011 14:54:56 -0700
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: Cassandra Node Requirements



On Wed, Aug 24, 2011 at 2:54 PM, Jacob, Arun 
mailto:arun.ja...@disney.com>> wrote:
I'm trying to determine a node configuration for Cassandra. From what I've been 
able to determine from reading around:


 1.  we need to cap data size at 50% of total node storage capacity for 
compaction
 2.  with RF=3, that means that I need to effectively assume that I have 1/6th 
of total storage capacity.
 3.  SSDs are preferred, but of course  reduce storage capacity
 4.  using standard storage means you bump up your RAM to keep as much in 
memory as possible.

Right now we are looking at storage requirements of 42 – 60TB, assuming a 
baseline of 3TB/day and expiring data after 14-20 days (depending on use case), 
 I would assume based on above that we need 252- 360TB total storage max.

My questions:

 1.  is 8TB (meaning 1.33 actual TB storage/node) a reasonable per node storage 
size for Cassandra? I don’t want to use SSDs due to reduced storage capacity -- 
I don't want to buy 100s of nodes to support that reduced storage capacity of 
SSDs.  Given that I will be using standard drives, what is a 
reasonable/effective per node storage capacity?
 2.  other than splitting the commit log onto a separate drive, is there any 
other drive allocation I should be doing?
 3.  Assuming I'm not using SSDs, what would be a good memory size for a node? 
I've heard anything from 32-48 GB, but need more guidance.

Anything else that anyone has run into? What are common configurations being 
used by others?

Thanks in advance,

-- Arun




I would suggest checking out:
http://wiki.apache.org/cassandra/CassandraHardware
http://wiki.apache.org/cassandra/LargeDataSetConsiderations
http://www.slideshare.net/edwardcapriolo/real-world-capacity

1. we need to cap data size at 50% of total node storage capacity for compaction

False. You need 50% the capacity of your largest column family free with some 
other room for overhead. This changes all your numbers.

3. SSDs are preferred, but of course  reduce storage capacity

Avoid generalizations. Many use cases may get little benefit from SSD disks.

4. using standard storage means you bump up your RAM to keep as much in memory 
as possible.

In most cases you want to maintain some RAM / Hard disk ratio. SSD setups still 
likely need sizable RAM.

Your 3 questions are hard to answer because what hardware you need workload 
dependent. If really depends on active set, what percent of the data is active 
at any time. It also depends on your latency requirements, if you are modeling 
something like the way-back machine, that has different usage profile then a 
stock ticker application, that is again different from the usage patterns of an 
email system.

Generally people come to Cassandra because they are looking for low latency 
access to read and write data. This is hard to achieve on 8TB of disk. The size 
of the bloom filters and index files are themselves substantial with 8TB of 
data. You will also require a large amount of RAM on this disk to minimize disk 
seeks (or a super like SSD raid-0 (does this sound like a bad idea to you? It 
does to me :))

The only way to answer the question of how much hardware your need is wit

Re: Solandra distributed search

[Jacob, Arun]

does the Solandra specific partitioner distribute data relatively equally 
across nodes? Is this influenced by the shards.at.once property? If I'm writing 
to 3 nodes, how would the default setting of 4 for this property affect the 
distribution of data across my nodes?




From: Jake Luciani mailto:jak...@gmail.com>>
Reply-To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Date: Mon, 15 Aug 2011 12:03:22 -0700
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: Re: Solandra distributed search

Solandra manages the "shard" parameters for you. you don't need to specify 
anything.

On Mon, Aug 15, 2011 at 3:00 PM, Jeremiah Jordan 
mailto:jeremiah.jor...@morningstar.com>> wrote:
When using Solandra, do I need to use the Solr sharding synxtax in my queries? 
I don't think I do because Cassandra is handling the "sharding", not Solr, but 
just want to make sure.  The Solandra wiki references the distributed search 
limitations, which talks about the shard syntax further down the page.
>From what I see with how it is implemented I should just be able to pick a 
>random Solandra node and do my query, since they are all backed by the same 
>Cassandra data store. Correct?

Thanks!
-Jeremiah



--
http://twitter.com/tjake

Solandra: connection refused errors

[Jacob, Arun]

I'm seeing this error when trying to insert data into a core I've defined in 
Solandra

INFO [pool-7-thread-319] 2011-10-06 16:21:34,328 HttpMethodDirector.java (line 
445) Retrying request
INFO [pool-7-thread-1070] 2011-10-06 16:21:34,328 HttpMethodDirector.java (line 
445) Retrying request
INFO [pool-7-thread-335] 2011-10-06 16:21:34,327 HttpMethodDirector.java (line 
439) I/O exception (java.net.ConnectException) caught when processing request: 
Connection
refused
INFO [pool-7-thread-335] 2011-10-06 16:21:34,329 HttpMethodDirector.java (line 
445) Retrying request
ERROR [1926426205@qtp-673795938-11] 2011-10-06 16:21:34,327 SolrException.java 
(line 151) org.apache.solr.common.SolrException: 
org.apache.solr.client.solrj.SolrServerException: java.net.ConnectException: 
Connection refused


Has anyone seen this behavior before, the problem is that it seems to be 
intermittent (if it were failing all of the time, I would suspect a port or IP 
misconfiguration).

invalid shard name encountered

[Jacob, Arun]

I'm seeing this in my logs:

WARN [1832199239@qtp-673795938-0] 2011-10-06 16:15:46,424 
CassandraIndexManager.java (line 364) invalid shard name encountered: 
WDPRO-NGELOG-DEV 1

WDPRO-NGELOG-DEV  is the name of the index I'm creating. Is there a restriction 
on characters in the name?

After deleting some data from the cluster under Solandra, we keep seeing this assertion.

[Jacob, Arun]

After removing some data from Solandra via a Solr query, we are getting 
DecoratedKey assertions.

Our setup:
latest version of Solandra (I think it supports 0.8.6, please correct if wrong)

3 solandra nodes, with replication set to 2 and sharding set to 3.

No systems are currently running (ingest or read) other than a simple


http://tn7cldsolandra01/solandra/schema/myschemaquery.

The delete command I ran was:

http://tn7cldsolandra01/solandra/myschema/update
  -H”Content-Type: text/xml” –data-binary “time:[0 TO 
131814360] 

This is what’s getting dumped into one node (solandra01).   I’ve ran nodetool 
scrub and nodetool –repair on this node and nodetool –repair on our solandra02 
box (solandra03 is still doing the nodetool –repair).

ERROR [ReadStage:3709] 2011-11-11 16:54:06,185 AbstractCassandraDaemon.java 
(line 139) Fatal exception in thread Thread[ReadStage:3709,5,main]
java.lang.AssertionError: DecoratedKey(144225997531208877267913204104447190682, 
44434c4f55442d52414e44592d444556) != 
DecoratedKey(144225997531208877267913204104447190682, 
313434323235393937353331323038383737323637393133323034313034343437313930363832efbfbf736861726473)
 in /data/dcloud-querysvc/data/L/SI-g-94-Data.db
at 
org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:59)
at 
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at 
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at 
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1407)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1304)
at 
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1261)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at 
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
at 
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:668)
at 
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1133)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Cassandra for Ad-hoc Aggregation and formula calculation

[Arun Cherian]

Hi,

I have been reading up on Cassandra for the past few weeks and I am
highly impressed by the features it offers. At work, we are starting
work on a product that will handle several million CDR (Call Data
Record, basically can be thought of as a .CSV file) per day. We will
have to store the data, and perform aggregations and calculations on
them. A few veteran RDBMS admin friends (we are a small .NET shop, we
don't have any in-house DB talent) recommended Infobright and noSQL to
us, and hence my search. I was wondering if Cassandra is a good fit
for

1. Storing several million data records per day (each record will be a
few KB in size) without any data loss.
2. Aggregation of certain fields in the stored records, like Avg
across time period.
3. Using certain existing fields to calculate new values on the fly
and store it too.
4. We were wondering if pre-aggregation was a good choice (calculating
aggregation per 1 min, 5 min, 15 min etc ahead of time) but in case we
need ad-hoc aggregation, does Cassandra support that over this amount
of data?

Thanks,
Arun

Re: Scala Dataframe - spark.cassandra.sql.pushdown.additionalClasses not working

[Arun Vigesh]

On Wed, Jul 10, 2024 at 8:49 PM Arun Vigesh 
wrote:

> Hi
>
> I am encountering a Singleton object not available error when trying to
> add the parameter *spark.cassandra.sql.pushdown.additionalClasses* to
> push all filters to Cassandra. Please find the code and error message below:
>
> *Code:*
> package com.common.reader
>
> import org.apache.spark.sql.{DataFrame, SparkSession}
> import com.datastax.spark.connector.cql.TableDef
> import org.apache.spark.SparkConf
> import org.apache.spark.sql.cassandra.{AnalyzedPredicates,
> CassandraPredicateRules, CassandraSourceRelation}
>
> class CassandraReader(spark: SparkSession) {
>   def read(table: String, keyspace: String, path: String, inc_field:
> String): DataFrame =
>   {
> var df = spark
>   .read
>   .format("org.apache.spark.sql.cassandra")
>   .option( "table",table)
>   .option( "keyspace",keyspace)
>
>  
> .option("spark.cassandra.sql.pushdown.additionalClasses","com.common.reader.PushdownEverything")
> //  .options(Map(
> //"keyspace" -> keyspace,
> //"table" -> table,
> //"pushdown" -> "true",
> //
> CassandraSourceRelation.AdditionalCassandraPushDownRulesParam.name ->
> "com.common.reader.PushdownEverything"))
>   .load
> if (inc_field.nonEmpty)
> {
>   val max_inc_value= spark.sql(s"select max(${inc_field}) from
> delta.`${path}`").first.get(0)
>   println(max_inc_value)
>   df = df.filter(s"${inc_field}>'${max_inc_value}'")
> }
> df
>   }
> }
> object PushdownEverything extends CassandraPredicateRules {
>   override def apply(
>   predicates: AnalyzedPredicates,
>   tableDef: TableDef,
>   sparkConf: SparkConf): AnalyzedPredicates = {
> AnalyzedPredicates(predicates.handledByCassandra ++
> predicates.handledBySpark, Set.empty)
>   }
> }
>
> *Error:*
>
> IllegalArgumentException: Singleton object not available:
> com.common.reader.PushdownEverything
> Caused by: ClassNotFoundException: com.common.reader.PushdownEverything
> at
> com.datastax.spark.connector.util.ReflectionUtil$.findGlobalObject(ReflectionUtil.scala:54)
> at
> com.datastax.spark.connector.datasource.CassandraScanBuilder.$anonfun$additionalRules$1(CassandraScanBuilder.scala:102)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
> at
> scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
> at
> scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
> at scala.collection.TraversableLike.map(TraversableLike.scala:286)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
> at
> com.datastax.spark.connector.datasource.CassandraScanBuilder.additionalRules(CassandraScanBuilder.scala:102)
> at
> com.datastax.spark.connector.datasource.CassandraScanBuilder.pushFilters(CassandraScanBuilder.scala:76)
> at
> org.apache.spark.sql.execution.datasources.v2.PushDownUtils$.pushFilters(PushDownUtils.scala:66)
> at
> org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$pushDownFilters$1.applyOrElse(V2ScanRelationPushDown.scala:73)
> at
> org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$pushDownFilters$1.applyOrElse(V2ScanRelationPushDown.scala:60)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:465)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:69)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:465)
> at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org
> $apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:39)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:339)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:335)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:39)
> at
> org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:39)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:470)
> at
> org.apache.spark.sql.catalyst.trees.UnaryLike.mapChild

Error - Scala Dataframe - spark.cassandra.sql.pushdown.additionalClasses

[Arun Vigesh]

Hi

I am encountering a *Singleton object not available *error when trying to
add the parameter *spark.cassandra.sql.pushdown.additionalClasses* to push
all filters to Cassandra. Please find the code and error message below:

*Code:*

package com.common.reader

import org.apache.spark.sql.{DataFrame, SparkSession}
import com.datastax.spark.connector.cql.TableDef
import org.apache.spark.SparkConf
import org.apache.spark.sql.cassandra.{AnalyzedPredicates,
CassandraPredicateRules, CassandraSourceRelation}

class CassandraReader(spark: SparkSession) {
  def read(table: String, keyspace: String, path: String, inc_field:
String): DataFrame =
  {
var df = spark
  .read
  .format("org.apache.spark.sql.cassandra")
  .option( "table",table)
  .option( "keyspace",keyspace)
 
.option("spark.cassandra.sql.pushdown.additionalClasses","com.common.reader.PushdownEverything")
//  .options(Map(
//"keyspace" -> keyspace,
//"table" -> table,
//"pushdown" -> "true",
//CassandraSourceRelation.AdditionalCassandraPushDownRulesParam.name
-> "com.common.reader.PushdownEverything"))
  .load
if (inc_field.nonEmpty)
{
  val max_inc_value= spark.sql(s"select max(${inc_field}) from
delta.`${path}`").first.get(0)
  println(max_inc_value)
  df = df.filter(s"${inc_field}>'${max_inc_value}'")
}
df
  }
}
object PushdownEverything extends CassandraPredicateRules {
  override def apply(
  predicates: AnalyzedPredicates,
  tableDef: TableDef,
  sparkConf: SparkConf): AnalyzedPredicates = {
AnalyzedPredicates(predicates.handledByCassandra ++
predicates.handledBySpark, Set.empty)
  }
}

*Error:*

IllegalArgumentException: Singleton object not available:
com.common.reader.PushdownEverything
Caused by: ClassNotFoundException: com.common.reader.PushdownEverything
at
com.datastax.spark.connector.util.ReflectionUtil$.findGlobalObject(ReflectionUtil.scala:54)
at
com.datastax.spark.connector.datasource.CassandraScanBuilder.$anonfun$additionalRules$1(CassandraScanBuilder.scala:102)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at
scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at
scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198)
at
com.datastax.spark.connector.datasource.CassandraScanBuilder.additionalRules(CassandraScanBuilder.scala:102)
at
com.datastax.spark.connector.datasource.CassandraScanBuilder.pushFilters(CassandraScanBuilder.scala:76)
at
org.apache.spark.sql.execution.datasources.v2.PushDownUtils$.pushFilters(PushDownUtils.scala:66)
at
org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$pushDownFilters$1.applyOrElse(V2ScanRelationPushDown.scala:73)
at
org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown$$anonfun$pushDownFilters$1.applyOrElse(V2ScanRelationPushDown.scala:60)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:465)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:69)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:465)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org
$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:39)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:339)
at
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:335)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:39)
at
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:39)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$3(TreeNode.scala:470)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren(TreeNode.scala:1267)
at
org.apache.spark.sql.catalyst.trees.UnaryLike.mapChildren$(TreeNode.scala:1266)
at
org.apache.spark.sql.execution.datasources.WriteFiles.mapChildren(WriteFiles.scala:58)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:470)
at 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:39)


-- 
**The content of this e-mail is confidential and is intended solely for the 
use of the individual or entity to whom it is addressed. If you have 
received this e-mail by mistake, please reply to this e-mail and follow 
with its deletion.

Opscenter's Meaning of 'Requests'

[Arun Kumar K]

Hi Guys,

I have started understanding Cassandra and am working with it recently.

I have created two Column Families. For CF1, a write is an insert into a
unique row with all column values. Eg:

  Key Col1  Col2   Col3
  k1  c11   c12   c13
  k2  c21   c22   c23

For CF2. a write is an insert into a time stamped column of a row. Eg:

 Key  timeCol1  timeCol2
 k1   ct11
 k1  ct12
 k2ct21
 k2   ct22

I am using YCSB and using thrift based *client.batch_mutate()* call. For
CF1, i send all column vals for a row through the call. For CF2, i send the
new column vals for a row.

Now say opscenter reports the "write requests" as say 1000 *operations*/sec
when a record count is say 1 records.

OpsCenter API docs say 'Write Requests" as "requests per second".

What does an operation/request mean from opscenter perspective? Does it
mean unique row inserts across all column families ? Does it mean count of
each mutations for a row ?

How does opscenter identify a unique operation/request ? Is it related or
dependent on the row count or mutation count of batch_mutate() call ?

>From application perspective an operation means differently for both column
families.

Can some one guide me ?

Thanks,

Arun

Opscenter Metrics

[Arun Kumar K]

Hi guys,

I am using YCSB and using thrift based *client.batch_mutate()* call.

Now say opscenter reports the "write requests" as say 1000 *operations*/sec
when a record count is say 1 records.

OpsCenter API docs say 'Write Requests" as "requests per second"

1> What does an 'operation or request' mean from opscenter perspective?

2> Does it mean unique row inserts across all column families ? or  Does it
   mean count of each mutations for a row ?

 Can some one guide me ?


Thanks,

 Arun

Re: Opscenter Metrics

[Arun Kumar K]

Thanks Nick for the pointer !


On Thu, Jan 2, 2014 at 10:31 PM, Nick Bailey  wrote:

> I believe the answer to your similar question on server fault should
> answer this:
>
>
> http://serverfault.com/questions/564107/what-does-opscenters-write-requests-count-shown-as-ops-sec-exactly-mean
>
> On Tue, Dec 31, 2013 at 12:55 AM, Arun Kumar K  wrote:
>
>> Hi guys,
>>
>> I am using YCSB and using thrift based *client.batch_mutate()* call.
>>
>> Now say opscenter reports the "write requests" as say 1000 *operations*/sec
>> when a record count is say 1 records.
>>
>> OpsCenter API docs say 'Write Requests" as "requests per second"
>>
>> 1> What does an 'operation or request' mean from opscenter perspective?
>>
>> 2> Does it mean unique row inserts across all column families ? or  Does
>> itmean count of each mutations for a row ?
>>
>>  Can some one guide me ?
>>
>>
>> Thanks,
>>
>>  Arun
>>
>
>