Re: Cassandra backup via snapshots in production

[DuyHai Doan]

True

Delete in CQL just create tombstone so from the storage engine pov it's
just adding some physical columns

Truncate does trigger snapshot creation though
Le 21 nov. 2014 19:29, "Robert Coli"  a écrit :

> On Fri, Nov 21, 2014 at 8:40 AM, Jens Rantil  wrote:
>
>> > The main purpose is to protect us from human errors (eg. unexpected
>> manipulations: delete, drop tables, …).
>>
>> If that is the main purpose, having "auto_snapshot: true” in
>> cassandra.yaml will be enough to protect you.
>>
>
> OP includes "delete" in their list of "unexpected manipulations", and
> auto_snapshot: true will not protect you in any way from DELETE.
>
> =Rob
> http://twitter.com/rcolidba
>

Re: Compaction Strategy guidance

[Jean-Armel Luce]

Hi Andrei, Hi Nicolai,

Which version of C* are you using ?

There are some recommendations about the max storage per node :
http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2

"For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
handle 10x
(3-5TB)".

I have the feeling that those recommendations are sensitive according many
criteria such as :
- your hardware
- the compaction strategy
- ...

It looks that LCS lower those limitations.

Increasing the size of sstables might help if you have enough CPU and you
can put more load on your I/O system (@Andrei, I am interested by the
results of your  experimentation about large sstable files)

>From my point of view, there are some usage patterns where it is better to
have many small servers than a few large servers. Probably, it is better to
have many small servers if you need LCS for large tables.

Just my 2 cents.

Jean-Armel

2014-11-24 19:56 GMT+01:00 Robert Coli :

> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev 
> wrote:
>
>> One of the obvious recommendations I have received was to run more than
>> one instance of C* per host. Makes sense - it will reduce the amount of
>> data per node and will make better use of the resources.
>>
>
> This is usually a Bad Idea to do in production.
>
> =Rob
>
>

Re: Cassandra backup via snapshots in production

[Jens Rantil]

> Truncate does trigger snapshot creation though




Doesn’t it? With “auto_snapshot: true” it should.




———
Jens Rantil
Backend engineer
Tink AB

Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

On Tue, Nov 25, 2014 at 9:21 AM, DuyHai Doan  wrote:

> True
> Delete in CQL just create tombstone so from the storage engine pov it's
> just adding some physical columns
> Truncate does trigger snapshot creation though
> Le 21 nov. 2014 19:29, "Robert Coli"  a écrit :
>> On Fri, Nov 21, 2014 at 8:40 AM, Jens Rantil  wrote:
>>
>>> > The main purpose is to protect us from human errors (eg. unexpected
>>> manipulations: delete, drop tables, …).
>>>
>>> If that is the main purpose, having "auto_snapshot: true” in
>>> cassandra.yaml will be enough to protect you.
>>>
>>
>> OP includes "delete" in their list of "unexpected manipulations", and
>> auto_snapshot: true will not protect you in any way from DELETE.
>>
>> =Rob
>> http://twitter.com/rcolidba
>>

Fwd: Issues in moving data from cassandra to elasticsearch in java.

[Vinod Joseph]

Hi,

I am working on a java plugin which moves data from cassandra
to elasticsearch. This plugin must run in the server for every 5 seconds.
The data is getting moved, but the issue is that every time the plugin
runs(ie after every 5 seconds) all the data, including data which has been
already moved into elasticsearch in the previous iteration is moving to it.
So we are having duplicate values in the elastic search. How to avoid this
problem.

We are using this plugin to manage logs which are generated during any
online transaction. So we will be having millions of transactions.
Following is the table schema.

CREATE TABLE logs (
  txn_id text,
  logged_at timestamp,
  des text,
  key_name text,
  params text,
  PRIMARY KEY (txn_id, logged_at)
)

The txn_id is a 16 digit number and is not unique. It is a combination of 6
random numbers generated using a random function, followed by the epoch
timestamp in millisec(10 digits).

I want to move only the data which has been generated after the previous
transaction and not the data which was already moved in the previous
transaction.
I tried to do it with static values, counter variables, comparing the
write_time of each row and order by. Still its not working . Please suggest
me any ideas.


Thanks and regards
vinod joseph
8050136948

Cassandra schema migrator

[Jens Rantil]

Hi,


Anyone who is using, or could recommend, a tool for versioning 
schemas/migrating in Cassandra? My list of requirements is:
 * Support for adding tables.
 * Support for versioning of table properties. All our tables are to be 
defaulted to LeveledCompactionStrategy.
 * Support for adding non-existing columns.
 * Optional: Support for removing columns.
 * Optional: Support for removing tables.


We are preferably a Java shop, but could potentially integrate something 
non-Java. I understand I could write a tool that would make these decisions 
using system.schema_columnfamilies and system.schema_columns, but as always 
reusing a proven tool would be preferable.


So far I only know of Spring Data Cassandra that handles creating tables and 
adding columns. However, it does not handle table properties in any way.


Thanks,
Jens

———
Jens Rantil
Backend engineer
Tink AB

Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

Re: Compaction Strategy guidance

[Andrei Ivanov]

Hi Jean-Armel, Nikolai,

1. Increasing sstable size doesn't work (well, I think, unless we
"overscale" - add more nodes than really necessary, which is
prohibitive for us in a way). Essentially there is no change.  I gave
up and will go for STCS;-(
2. We use 2.0.11 as of now
3. We are running on EC2 c3.8xlarge instances with EBS volumes for data (GP SSD)

Jean-Armel, I believe that what you say about many small instances is
absolutely true. But, is not good in our case - we write a lot and
almost never read what we've written. That is, we want to be able to
read everything, but in reality we hardly read 1%, I think. This
implies that smaller instances are of no use in terms of read
performance for us. And generally nstances/cpu/ram is more expensive
than storage. So, we really would like to have instances with large
storage.

Andrei.





On Tue, Nov 25, 2014 at 11:23 AM, Jean-Armel Luce  wrote:
> Hi Andrei, Hi Nicolai,
>
> Which version of C* are you using ?
>
> There are some recommendations about the max storage per node :
> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
>
> "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to handle
> 10x
> (3-5TB)".
>
> I have the feeling that those recommendations are sensitive according many
> criteria such as :
> - your hardware
> - the compaction strategy
> - ...
>
> It looks that LCS lower those limitations.
>
> Increasing the size of sstables might help if you have enough CPU and you
> can put more load on your I/O system (@Andrei, I am interested by the
> results of your  experimentation about large sstable files)
>
> From my point of view, there are some usage patterns where it is better to
> have many small servers than a few large servers. Probably, it is better to
> have many small servers if you need LCS for large tables.
>
> Just my 2 cents.
>
> Jean-Armel
>
> 2014-11-24 19:56 GMT+01:00 Robert Coli :
>>
>> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev 
>> wrote:
>>>
>>> One of the obvious recommendations I have received was to run more than
>>> one instance of C* per host. Makes sense - it will reduce the amount of data
>>> per node and will make better use of the resources.
>>
>>
>> This is usually a Bad Idea to do in production.
>>
>> =Rob
>>
>
>

Re: Compaction Strategy guidance

[Marcus Eriksson]

If you are that write-heavy you should definitely go with STCS, LCS
optimizes for reads by doing more compactions

/Marcus

On Tue, Nov 25, 2014 at 11:22 AM, Andrei Ivanov  wrote:

> Hi Jean-Armel, Nikolai,
>
> 1. Increasing sstable size doesn't work (well, I think, unless we
> "overscale" - add more nodes than really necessary, which is
> prohibitive for us in a way). Essentially there is no change.  I gave
> up and will go for STCS;-(
> 2. We use 2.0.11 as of now
> 3. We are running on EC2 c3.8xlarge instances with EBS volumes for data
> (GP SSD)
>
> Jean-Armel, I believe that what you say about many small instances is
> absolutely true. But, is not good in our case - we write a lot and
> almost never read what we've written. That is, we want to be able to
> read everything, but in reality we hardly read 1%, I think. This
> implies that smaller instances are of no use in terms of read
> performance for us. And generally nstances/cpu/ram is more expensive
> than storage. So, we really would like to have instances with large
> storage.
>
> Andrei.
>
>
>
>
>
> On Tue, Nov 25, 2014 at 11:23 AM, Jean-Armel Luce 
> wrote:
> > Hi Andrei, Hi Nicolai,
> >
> > Which version of C* are you using ?
> >
> > There are some recommendations about the max storage per node :
> >
> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
> >
> > "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
> handle
> > 10x
> > (3-5TB)".
> >
> > I have the feeling that those recommendations are sensitive according
> many
> > criteria such as :
> > - your hardware
> > - the compaction strategy
> > - ...
> >
> > It looks that LCS lower those limitations.
> >
> > Increasing the size of sstables might help if you have enough CPU and you
> > can put more load on your I/O system (@Andrei, I am interested by the
> > results of your  experimentation about large sstable files)
> >
> > From my point of view, there are some usage patterns where it is better
> to
> > have many small servers than a few large servers. Probably, it is better
> to
> > have many small servers if you need LCS for large tables.
> >
> > Just my 2 cents.
> >
> > Jean-Armel
> >
> > 2014-11-24 19:56 GMT+01:00 Robert Coli :
> >>
> >> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev <
> ngrigor...@gmail.com>
> >> wrote:
> >>>
> >>> One of the obvious recommendations I have received was to run more than
> >>> one instance of C* per host. Makes sense - it will reduce the amount
> of data
> >>> per node and will make better use of the resources.
> >>
> >>
> >> This is usually a Bad Idea to do in production.
> >>
> >> =Rob
> >>
> >
> >
>

Re: Cassandra schema migrator

[Jan Kesten]

Hi Jens,

maybe you should have a look at mutagen for cassandra:

https://github.com/toddfast/mutagen-cassandra

It is a litte quiet around this for some months, but maybe still worth it.

Cheers,
Jan

Am 25.11.2014 um 10:22 schrieb Jens Rantil:

Hi,

Anyone who is using, or could recommend, a tool for versioning 
schemas/migrating in Cassandra? My list of requirements is:

 * Support for adding tables.
 * Support for versioning of table properties. All our tables are to 
be defaulted to LeveledCompactionStrategy.

 * Support for adding non-existing columns.
 * Optional: Support for removing columns.
 * Optional: Support for removing tables.

We are preferably a Java shop, but could potentially integrate 
something non-Java. I understand I could write a tool that would make 
these decisions using system.schema_columnfamilies and 
system.schema_columns, but as always reusing a proven tool would be 
preferable.


So far I only know of Spring Data Cassandra that handles creating 
tables and adding columns. However, it does not handle table 
properties in any way.


Thanks,
Jens

——— Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se 
Phone: +46 708 84 18 32 Web: www.tink.se Facebook Linkedin Twitter

Re: Compaction Strategy guidance

[Andrei Ivanov]

Yep, Marcus, I know. It's mainly a question of cost of those extra x2
disks, you know. Our "final" setup will be more like 30TB, so doubling
it is still some cost. But i guess, we will have to live with it

On Tue, Nov 25, 2014 at 1:26 PM, Marcus Eriksson  wrote:
> If you are that write-heavy you should definitely go with STCS, LCS
> optimizes for reads by doing more compactions
>
> /Marcus
>
> On Tue, Nov 25, 2014 at 11:22 AM, Andrei Ivanov  wrote:
>>
>> Hi Jean-Armel, Nikolai,
>>
>> 1. Increasing sstable size doesn't work (well, I think, unless we
>> "overscale" - add more nodes than really necessary, which is
>> prohibitive for us in a way). Essentially there is no change.  I gave
>> up and will go for STCS;-(
>> 2. We use 2.0.11 as of now
>> 3. We are running on EC2 c3.8xlarge instances with EBS volumes for data
>> (GP SSD)
>>
>> Jean-Armel, I believe that what you say about many small instances is
>> absolutely true. But, is not good in our case - we write a lot and
>> almost never read what we've written. That is, we want to be able to
>> read everything, but in reality we hardly read 1%, I think. This
>> implies that smaller instances are of no use in terms of read
>> performance for us. And generally nstances/cpu/ram is more expensive
>> than storage. So, we really would like to have instances with large
>> storage.
>>
>> Andrei.
>>
>>
>>
>>
>>
>> On Tue, Nov 25, 2014 at 11:23 AM, Jean-Armel Luce 
>> wrote:
>> > Hi Andrei, Hi Nicolai,
>> >
>> > Which version of C* are you using ?
>> >
>> > There are some recommendations about the max storage per node :
>> >
>> > http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
>> >
>> > "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
>> > handle
>> > 10x
>> > (3-5TB)".
>> >
>> > I have the feeling that those recommendations are sensitive according
>> > many
>> > criteria such as :
>> > - your hardware
>> > - the compaction strategy
>> > - ...
>> >
>> > It looks that LCS lower those limitations.
>> >
>> > Increasing the size of sstables might help if you have enough CPU and
>> > you
>> > can put more load on your I/O system (@Andrei, I am interested by the
>> > results of your  experimentation about large sstable files)
>> >
>> > From my point of view, there are some usage patterns where it is better
>> > to
>> > have many small servers than a few large servers. Probably, it is better
>> > to
>> > have many small servers if you need LCS for large tables.
>> >
>> > Just my 2 cents.
>> >
>> > Jean-Armel
>> >
>> > 2014-11-24 19:56 GMT+01:00 Robert Coli :
>> >>
>> >> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev
>> >> 
>> >> wrote:
>> >>>
>> >>> One of the obvious recommendations I have received was to run more
>> >>> than
>> >>> one instance of C* per host. Makes sense - it will reduce the amount
>> >>> of data
>> >>> per node and will make better use of the resources.
>> >>
>> >>
>> >> This is usually a Bad Idea to do in production.
>> >>
>> >> =Rob
>> >>
>> >
>> >
>
>

Re: Cassandra schema migrator

[Phil Wise]

On 25.11.2014 10:22, Jens Rantil wrote:

> Anyone who is using, or could recommend, a tool for versioning 
> schemas/migrating in Cassandra?

I've recently written a tool to solve schema migration at our company
which may be useful:

https://github.com/advancedtelematic/cql-migrate

> My list of requirements is: * Support for adding tables.

Yep, this works.

> * Support for versioning of table properties. All our tables are
> to be defaulted to LeveledCompactionStrategy.

My parser/statement splitter doesn't currently recognise the WITH ...
syntax on CREATE tables and ALTER TABLE, but it could easily be added.

Do you need to modify the properties after the table has been created?
If so the changes would need to be expressed as ALTER TABLE statements.

> * Support for adding non-existing columns.

Supported. cql-migrate works by getting the user to express the
upgrade as an ALTER TABLE ... ADD  statement.

> * Optional: Support for removing columns. * Optional: Support for 
> removing tables.

Not supported, although it wouldn't be hard to add.

> We are preferably a Java shop, but could potentially integrate 
> something non-Java. I understand I could write a tool that would
> make these decisions using system.schema_columnfamilies and 
> system.schema_columns, but as always reusing a proven tool would
> be preferable.

It is implemented in python, and can be built into a Debian package
for deployment.

Rather than try and fully parse the CQL, cql-migrate works by
splitting a .CQL script into individual statements, executing them in
turn and ignoring errors that relate to tables etc already exiting.
Doing it that way means there is very little chance that the schema
created will differ from what running the same script through cqlsh
would have produced.

Let me know if you need a hand getting going with it.

Cheers,

Phil

Re: Getting the counters with the highest values

[Eric Stevens]

> We have too many documents per day to materialize in memory, so querying
per day and aggregating the results isn’t really possible.

You don't really need to, that's part of the point.  You can paginate
across a partition with most client drivers, and materializing this view is
just copying data from one table to another with a different layout.  So
you end up just having to read then write a few thousand records at a shot.

doc_id as the partitioning key and day as the clustering key means that you
have to iterate over documents from some outside knowledge (a definitive
source on what the set of doc_id's is), and reading so many separate
partitions (one per doc_id) will produce memory pressure in your cluster.
Compared against ((day), doc_id) where you can SELECT * WHERE day=?.  Your
approach would give you a very nice time series view of views per document
over time, which itself might be useful elsewhere in your application.

That said, there are physical and practical limits on the number of columns
you can have in a partition (2 billion physical, and depending on the data
sometimes just on the order of a few hundred thousand practical without
causing some troubles in areas such as compaction, repair, and bootstrap).

However, I still suspect you may benefit by keying the counters table
primarily by date, but maybe add another key rotator in there, like ((day,
subpartition), doc_id).  Compute your sub partition deterministically but
in an evenly distributed manner from your doc_id (eg, doc_id mod 16, or
md5(doc_id).take(2), etc., depending on what the data type is for your
doc_id).

This will split your single logical partition up across n physical
partitions, opens you up to parallel processing of those partitions
(materializing can get a lot faster as you can very easily have a single
worker working on each physical partition without reduced hotspotting).
Each worker can be assigned a day and subpartition and work exclusively on
that data set for materialization (or a single worker can iterate the
subpartitions for the same effect).

Now to make a too-long email even longer, if you're approaching practical
limits on using doc_id as part of a clustering key, your materialized view
is going to have similar issues.  So you may have to either only
materialize documents with a view count over a certain threshold, or engage
in a similar sub partitioning scheme there.

On Mon Nov 24 2014 at 10:33:50 AM Robert Wille  wrote:

>  We do get a large number of documents getting counts each day, which is
> why I’m thinking the running totals table be ((doc_id), day) rather than
> ((day), doc_id). We have too many documents per day to materialize in
> memory, so querying per day and aggregating the results isn’t really
> possible.
>
>  I’m planning on bucketing the materialized ordering because we get
> enough unique document views per day that the rows will be quite large. Not
> so large as to be unmanageable, but pushing the limits. If we were so lucky
> as to get a significant increase in traffic, I might start having issues. I
> didn’t include bucketing in my post because I didn’t want to complicate my
> question. I hadn’t considered that I could bucket by hour and then use a
> local midnight instead of a global midnight. Interesting idea.
>
>  Thanks for your response.
>
>  Robert
>
>  On Nov 24, 2014, at 9:40 AM, Eric Stevens  wrote:
>
> You're right that there's no way to use the counter data type to
> materialize a view ordered by the counter.  Computing this post hoc is
> the way to go if your needs allow for it (if not, something like
> Summingbird or vanilla Storm may be necessary).
>
>  I might suggest that you make your primary key for your running totals
> by day table be ((day), doc_id) because it will make it easy to compute the
> materialized ordered view (SELECT doc_id, count FROM running_totals WHERE
> day=?) unless you expect to have a really large number of documents getting
> counts each day.
>
>  For your materialized ordering, I'd suggest a primary key of ((day),
> count) as then for a given day you'll be able to select top by count
> (SELECT count, doc_id FROM doc_counts WHERE day=? ORDER BY count DESC).
>
>  One more thing to consider if your users are not all in a single
> timezone is having your time bucket be hour instead of day so that you can
> answer by day goal posted by local midnight (except the handful of
> locations that use half hour timezone offsets) instead of a single global
> midnight.  You can then either include either just each hour in each row
> (and aggregate at read time), or you can make each row a rolling 24 hours
> (aggregating at write time), depending on which use case fits your needs
> better.
>
> On Sun Nov 23 2014 at 8:42:11 AM Robert Wille  wrote:
>
>> I’m working on moving a bunch of counters out of our relational database
>> to Cassandra. For the most part, Cassandra is a very nice fit, except for
>> one feature on our website. We manage a time series of 

Re: Cassandra version 1.0.10 Data Loss upon restart

[Ankit Patel]

Rob,

The JIRA https://issues.apache.org/jira/browse/CASSANDRA-4446 refers to the
problem after we've invoked drain. However, we did not invoke drain or
flush. We are running one node cassandra within one data center and it is
being replicated with another node in another data center. We are using the
thrift API in java application to retrieve and modify the data from the
primary node only. We are not using the other node in the other data center
for any operations. The data loss well exceeds the commit log sync period.

Thanks,
Ankit

On Mon, Nov 24, 2014 at 8:53 PM, Robert Coli  wrote:

> On Mon, Nov 24, 2014 at 5:51 PM, Robert Coli  wrote:
>
>> What is your replication factor? What CL are you using to read?
>>
>
> Ah, I see from OP that RF is 1.
>
> As a general statement, RF=1 is an edge case which very, very few people
> have ever operated in production. It is relatively likely that there are
> some undiscovered edge cases which relate to it.
>
> That said, this would be a particularly glaring one, which I would expect
> to be discovered in other contexts.
>
> =Rob
>
>

Re: Compaction Strategy guidance

[Nikolai Grigoriev]

Hi Jean-Armel,

I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but
there are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra
2.0.10.

I have about 1,8Tb of data per node now in total, which falls into that
range.

As I said, it is really a problem with large amount of data in a single CF,
not total amount of data. Quite often the nodes are idle yet having quite a
bit of pending compactions. I have discussed it with other members of C*
community and DataStax guys and, they have confirmed my observation.

I believe that increasing the sstable size won't help at all and probably
will make the things worse - everything else being equal, of course. But I
would like to hear from Andrei when he is done with his test.

Regarding the last statement - yes, C* clearly likes many small servers
more than fewer large ones. But it is all relative - and can be all
recalculated to $$$ :) C* is all about partitioning of everything -
storage, traffic...Less data per node and more nodes give you lower
latency, lower heap usage etc, etc. I think I have learned this with my
project. Somewhat hard way but still, nothing is better than the personal
experience :)

On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce  wrote:

> Hi Andrei, Hi Nicolai,
>
> Which version of C* are you using ?
>
> There are some recommendations about the max storage per node :
> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
>
> "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
> handle 10x
> (3-5TB)".
>
> I have the feeling that those recommendations are sensitive according many
> criteria such as :
> - your hardware
> - the compaction strategy
> - ...
>
> It looks that LCS lower those limitations.
>
> Increasing the size of sstables might help if you have enough CPU and you
> can put more load on your I/O system (@Andrei, I am interested by the
> results of your  experimentation about large sstable files)
>
> From my point of view, there are some usage patterns where it is better to
> have many small servers than a few large servers. Probably, it is better to
> have many small servers if you need LCS for large tables.
>
> Just my 2 cents.
>
> Jean-Armel
>
> 2014-11-24 19:56 GMT+01:00 Robert Coli :
>
>> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev 
>> wrote:
>>
>>> One of the obvious recommendations I have received was to run more than
>>> one instance of C* per host. Makes sense - it will reduce the amount of
>>> data per node and will make better use of the resources.
>>>
>>
>> This is usually a Bad Idea to do in production.
>>
>> =Rob
>>
>>
>
>


-- 
Nikolai Grigoriev
(514) 772-5178

Data synchronization between 2 running clusters on different availability zone

[Spico Florin]

Hello!
   I have the following scenario:
1. For ensuring high availability I would like to install one Cassandra
cluster on one availability zone
(on Amazon EC2 US-east) and one Cassandra cluster on other AZ (Amazon EC2
US-west).
2.I have pipeline that is running on Amazon EC2-EAST and is feeding the
Cassandra installed on this AZ.
Here are my questions:
1. Is this scenario feasible?
 2. Is the architecture correct regarding the availability of Cassandra?
3. If the architecture is fine, how do you keep data synchronized between
the two instances?

I look forward for your answers.
 Regards,
  Florin

Re: Issues in moving data from cassandra to elasticsearch in java.

[Eric Stevens]

Consider adding log_bucket timestamp, and then indexing that column.  Your
data loader can SELECT * FROM logs WHERE log_bucket = ?.  The value you
supply there would be the timestamp log bucket you're processing - in your
case logged_at % 5.

However, I'll caution against writing data to Cassandra and then trying to
reliably read it back immediately after.  You're likely to miss values this
way due to eventual consistency unless you read at CL_ALL.  But then your
data loader will break whenever you have any node offline.  Writing then
immediately reading data is a typical antipattern in any eventually
consistent system.

If using DataStax Java Driver you can use CL_ALL with
DowngradingConsistencyRetryPolicy and you would at least strike a nice
balance between reasonably strong consistency and loss of resiliency from
CL_ALL (but when you have a node offline, your load process may get
significantly slower).  This would mitigate but not eliminate the
antipattern.


On Tue Nov 25 2014 at 2:11:36 AM Vinod Joseph 
wrote:

> Hi,
>
> I am working on a java plugin which moves data from cassandra
> to elasticsearch. This plugin must run in the server for every 5 seconds.
> The data is getting moved, but the issue is that every time the plugin
> runs(ie after every 5 seconds) all the data, including data which has been
> already moved into elasticsearch in the previous iteration is moving to it.
> So we are having duplicate values in the elastic search. How to avoid this
> problem.
>
> We are using this plugin to manage logs which are generated during any
> online transaction. So we will be having millions of transactions.
> Following is the table schema.
>
> CREATE TABLE logs (
>   txn_id text,
>   logged_at timestamp,
>   des text,
>   key_name text,
>   params text,
>   PRIMARY KEY (txn_id, logged_at)
> )
>
> The txn_id is a 16 digit number and is not unique. It is a combination of
> 6 random numbers generated using a random function, followed by the epoch
> timestamp in millisec(10 digits).
>
> I want to move only the data which has been generated after the previous
> transaction and not the data which was already moved in the previous
> transaction.
> I tried to do it with static values, counter variables, comparing the
> write_time of each row and order by. Still its not working . Please suggest
> me any ideas.
>
>
> Thanks and regards
> vinod joseph
> 8050136948
>
>
>

Re: Issues in moving data from cassandra to elasticsearch in java.

[William Arbaugh]

Sounds like you're trying to use C* as a message broker. Perhaps try using 
Kafka or RabbitMQ as a front-end. Then have two subscribers - one pulls and 
places into elasticsearch and the other inserts into C*.

Yes it is more complex front-end, but it will give you the functionality you 
want.

> On Nov 25, 2014, at 10:14 AM, Eric Stevens  wrote:
> 
> Consider adding log_bucket timestamp, and then indexing that column.  Your 
> data loader can SELECT * FROM logs WHERE log_bucket = ?.  The value you 
> supply there would be the timestamp log bucket you're processing - in your 
> case logged_at % 5.
> 
> However, I'll caution against writing data to Cassandra and then trying to 
> reliably read it back immediately after.  You're likely to miss values this 
> way due to eventual consistency unless you read at CL_ALL.  But then your 
> data loader will break whenever you have any node offline.  Writing then 
> immediately reading data is a typical antipattern in any eventually 
> consistent system.
> 
> If using DataStax Java Driver you can use CL_ALL with 
> DowngradingConsistencyRetryPolicy and you would at least strike a nice 
> balance between reasonably strong consistency and loss of resiliency from 
> CL_ALL (but when you have a node offline, your load process may get 
> significantly slower).  This would mitigate but not eliminate the antipattern.
> 
> 
> On Tue Nov 25 2014 at 2:11:36 AM Vinod Joseph  wrote:
> Hi,
> 
> I am working on a java plugin which moves data from cassandra to 
> elasticsearch. This plugin must run in the server for every 5 seconds. The 
> data is getting moved, but the issue is that every time the plugin runs(ie 
> after every 5 seconds) all the data, including data which has been already 
> moved into elasticsearch in the previous iteration is moving to it. So we are 
> having duplicate values in the elastic search. How to avoid this problem. 
> 
> We are using this plugin to manage logs which are generated during any online 
> transaction. So we will be having millions of transactions. 
> Following is the table schema.
> 
> CREATE TABLE logs (
>   txn_id text,
>   logged_at timestamp,
>   des text,
>   key_name text,
>   params text,
>   PRIMARY KEY (txn_id, logged_at)
> )
> 
> The txn_id is a 16 digit number and is not unique. It is a combination of 6 
> random numbers generated using a random function, followed by the epoch 
> timestamp in millisec(10 digits).
> 
> I want to move only the data which has been generated after the previous 
> transaction and not the data which was already moved in the previous 
> transaction.
> I tried to do it with static values, counter variables, comparing the 
> write_time of each row and order by. Still its not working . Please suggest 
> me any ideas. 
> 
> 
> Thanks and regards
> vinod joseph
> 8050136948
> 
>  

Rule of thumb for concurrent asynchronous queries?

[Robert Wille]

Suppose I have the primary keys for 10,000 rows and I want them all. Is there a 
rule of thumb for the maximum number of concurrent asynchronous queries I 
should execute?

Re: Rule of thumb for concurrent asynchronous queries?

[Jack Krupansky]

Great question. The safe answer is to do a proof of concept implementation 
and try various rates to determine where the bottleneck is. It will also 
depend on the row size. Hard to say if you will be limited by the cluster 
load or network bandwidth.


Is there only one client talking to your cluster? Or are you asking what 
each of, say, one million clients can be simultaneously requesting?


The rate of requests will matter as well, particularly if the cluster has a 
non-trivial load.


My ultimate rule of thumb is simple: Moderation. Not too many threads, not 
too frequent request rate.


It would be nice if we had a way to calculate this number (both numbers) for 
you so that a client (driver) could ping for it from the cluster, as well as 
for the cluster to return a suggested wait interval before sending another 
request based on actual load.


-- Jack Krupansky

-Original Message- 
From: Robert Wille

Sent: Tuesday, November 25, 2014 10:57 AM
To: user@cassandra.apache.org
Subject: Rule of thumb for concurrent asynchronous queries?

Suppose I have the primary keys for 10,000 rows and I want them all. Is 
there a rule of thumb for the maximum number of concurrent asynchronous 
queries I should execute?= 

Re: Rule of thumb for concurrent asynchronous queries?

[Nikolai Grigoriev]

I think it all depends on how many machines will be involved in the query
(read consistency is also a factor) and how long is a typical response in
bytes. Large responses will put more pressure on the GC, which will result
in more time spent in GC and possibly long(er) GC pauses.

Cassandra can tolerate many things - but at the cost for other queries and
all the way up to the heal of the individual node.

>From the original question it is not clear if all these rows are coming
from the same or few nodes (token range) or these are really 10K primary
keys - so they are spread more or less evenly across the cluster.

Also the node disk I/O may be a concern - especially if the data is not in
OS cache (or row cache if applicable).

I think it is a tough question to get a precise answer. If I had such a
problem I would try to determine the peak speed I can achieve first. I.e.
find the limiting factor (CPU or disk I/O most likely), then shoot as many
requests in as many threads as practical for the client app. Measure the
load to prove that you've determined the limiting factor correctly (either
CPU or I/O, I doubt it will be network). Then measure the latency and
decide what kind of latency you can tolerate for your use case. And then go
down from that peak load you've created by certain factor (i.e. limit
yourself to XX% of the peak load you have achieved).

On Tue, Nov 25, 2014 at 11:34 AM, Jack Krupansky 
wrote:

> Great question. The safe answer is to do a proof of concept implementation
> and try various rates to determine where the bottleneck is. It will also
> depend on the row size. Hard to say if you will be limited by the cluster
> load or network bandwidth.
>
> Is there only one client talking to your cluster? Or are you asking what
> each of, say, one million clients can be simultaneously requesting?
>
> The rate of requests will matter as well, particularly if the cluster has
> a non-trivial load.
>
> My ultimate rule of thumb is simple: Moderation. Not too many threads, not
> too frequent request rate.
>
> It would be nice if we had a way to calculate this number (both numbers)
> for you so that a client (driver) could ping for it from the cluster, as
> well as for the cluster to return a suggested wait interval before sending
> another request based on actual load.
>
> -- Jack Krupansky
>
> -Original Message- From: Robert Wille
> Sent: Tuesday, November 25, 2014 10:57 AM
> To: user@cassandra.apache.org
> Subject: Rule of thumb for concurrent asynchronous queries?
>
> Suppose I have the primary keys for 10,000 rows and I want them all. Is
> there a rule of thumb for the maximum number of concurrent asynchronous
> queries I should execute?=
>



-- 
Nikolai Grigoriev
(514) 772-5178

Keyspace and table/cf limits

[Raj N]

What's the latest on the maximum number of keyspaces and/or tables that one
can have in Cassandra 2.1.x?

-Raj

Re: max ttl for column

[Rajanish GJ]

Mark / Philip - Thanks a lot. This is really helpful.

BTW It was my bad, i was mistaken that ttl was in miliseconds rather than
seconds..

Regards
Rajanish GJ
apigee | rajan...@apigee.com

On Fri, Nov 21, 2014 at 9:42 AM, Philip Thompson <
philip.thomp...@datastax.com> wrote:

> With the newest versions of Cassandra, cql is not hanging, but returns the
> same Invalid Query Exception you are seeing through hector. I would assume
> from the exception that 63072 is in fact that largest TTL you can use.
> What are you doing that you need to set a TTL of approximately 30 years?
>
> On Fri, Nov 21, 2014 at 11:29 AM, Rajanish GJ  wrote:
>
>>
>> Does hector or cassandra imposes a limit on max ttl value for a column?
>>
>> I am trying to insert record into one of the column family and seeing the
>> following error..
>> Cassandra version : 1.1.12
>> Hector  : 1.1-4
>>
>> Any pointers appreciated.
>>
>> me.prettyprint.hector.api.exceptions.HInvalidRequestException:
>> InvalidRequestException(why:ttl is too large. requested (951027277) maximum
>> (63072))
>> at
>> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:52)
>> ~[hector-core-1.1-4.jar:na]
>> at
>> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:260)
>> ~[hector-core-1.1-4.jar:na]
>> at
>> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:113)
>> ~[hector-core-1.1-4.jar:na]
>> at
>> me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
>> ~[hector-core-1.1-4.jar:na]
>> at
>> me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeBatch(AbstractColumnFamilyTemplate.java:115)
>> ~[hector-core-1.1-4.jar:na]
>> at
>> me.prettyprint.cassandra.service.template.AbstractColumnFamilyTemplate.executeIfNotBatched(AbstractColumnFamilyTemplate.java:163)
>> ~[hector-core-1.1-4.jar:na]
>> at
>> me.prettyprint.cassandra.service.template.ColumnFamilyTemplate.update(ColumnFamilyTemplate.java:69)
>> ~[hector-core-1.1-4.jar:na]
>>
>> =
>> *Also tried using cql, and it seems to hangs and not responding.. trying
>> again with few combinations*
>>
>> *INSERT INTO users (key,id) values ('test6','951027277 secs') USING
>> TTL 951027277 ; *
>>
>>
>>
>> Regards
>> Rajanish GJ
>> apigee | rajan...@apigee.com
>>
>
>

Re: Compaction Strategy guidance

[Andrei Ivanov]

Nikolai,

Just in case you've missed my comment in the thread (guess you have) -
increasing sstable size does nothing (in our case at least). That is,
it's not worse but the load pattern is still the same - doing nothing
most of the time. So, I switched to STCS and we will have to live with
extra storage cost - storage is way cheaper than cpu etc anyhow:-)

On Tue, Nov 25, 2014 at 5:53 PM, Nikolai Grigoriev  wrote:
> Hi Jean-Armel,
>
> I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but there
> are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra 2.0.10.
>
> I have about 1,8Tb of data per node now in total, which falls into that
> range.
>
> As I said, it is really a problem with large amount of data in a single CF,
> not total amount of data. Quite often the nodes are idle yet having quite a
> bit of pending compactions. I have discussed it with other members of C*
> community and DataStax guys and, they have confirmed my observation.
>
> I believe that increasing the sstable size won't help at all and probably
> will make the things worse - everything else being equal, of course. But I
> would like to hear from Andrei when he is done with his test.
>
> Regarding the last statement - yes, C* clearly likes many small servers more
> than fewer large ones. But it is all relative - and can be all recalculated
> to $$$ :) C* is all about partitioning of everything - storage,
> traffic...Less data per node and more nodes give you lower latency, lower
> heap usage etc, etc. I think I have learned this with my project. Somewhat
> hard way but still, nothing is better than the personal experience :)
>
> On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce  wrote:
>>
>> Hi Andrei, Hi Nicolai,
>>
>> Which version of C* are you using ?
>>
>> There are some recommendations about the max storage per node :
>> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
>>
>> "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
>> handle 10x
>> (3-5TB)".
>>
>> I have the feeling that those recommendations are sensitive according many
>> criteria such as :
>> - your hardware
>> - the compaction strategy
>> - ...
>>
>> It looks that LCS lower those limitations.
>>
>> Increasing the size of sstables might help if you have enough CPU and you
>> can put more load on your I/O system (@Andrei, I am interested by the
>> results of your  experimentation about large sstable files)
>>
>> From my point of view, there are some usage patterns where it is better to
>> have many small servers than a few large servers. Probably, it is better to
>> have many small servers if you need LCS for large tables.
>>
>> Just my 2 cents.
>>
>> Jean-Armel
>>
>> 2014-11-24 19:56 GMT+01:00 Robert Coli :
>>>
>>> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev 
>>> wrote:

 One of the obvious recommendations I have received was to run more than
 one instance of C* per host. Makes sense - it will reduce the amount of 
 data
 per node and will make better use of the resources.
>>>
>>>
>>> This is usually a Bad Idea to do in production.
>>>
>>> =Rob
>>>
>>
>>
>
>
>
> --
> Nikolai Grigoriev
> (514) 772-5178

Re: large range read in Cassandra

[Dan Kinder]

Thanks Rob.

To be clear, I expect this range query to take a long time and perform
relatively heavy I/O. What I expected Cassandra to do was use auto-paging (
https://issues.apache.org/jira/browse/CASSANDRA-4415,
http://stackoverflow.com/questions/17664438/iterating-through-cassandra-wide-row-with-cql3)
so that we aren't literally pulling the entire thing in. Am I
misunderstanding this use case? Could you clarify why exactly it would slow
way down? It seems like with each read it should be doing a simple range
read from one or two sstables.

If this won't work then it may me we need to start using Hive/Spark/Pig
etc. sooner, or page it manually using LIMIT and WHERE > [the last returned
result].

On Mon, Nov 24, 2014 at 5:49 PM, Robert Coli  wrote:

> On Mon, Nov 24, 2014 at 4:26 PM, Dan Kinder  wrote:
>
>> We have a web crawler project currently based on Cassandra (
>> https://github.com/iParadigms/walker, written in Go and using the gocql
>> driver), with the following relevant usage pattern:
>>
>> - Big range reads over a CF to grab potentially millions of rows and
>> dispatch new links to crawl
>>
>
> If you really mean millions of storage rows, this is just about the worst
> case for Cassandra. The problem you're having is probably that you
> shouldn't try to do this in Cassandra.
>
> Your timeouts are either from the read actually taking longer than the
> timeout or from the reads provoking heap pressure and resulting GC.
>
> =Rob
>
>

Re: Compaction Strategy guidance

[Nikolai Grigoriev]

Andrei,

Oh, yes, I have scanned the top of your previous email but overlooked the
last part.

I am using SSDs so I prefer to put extra work to keep my system performing
and save expensive disk space. So far I've been able to size the system
more or less correctly so these LCS limitations do not cause too much
troubles. But I do keep the CF "sharding" option as backup - for me it will
be relatively easy to implement it.

On Tue, Nov 25, 2014 at 1:25 PM, Andrei Ivanov  wrote:

> Nikolai,
>
> Just in case you've missed my comment in the thread (guess you have) -
> increasing sstable size does nothing (in our case at least). That is,
> it's not worse but the load pattern is still the same - doing nothing
> most of the time. So, I switched to STCS and we will have to live with
> extra storage cost - storage is way cheaper than cpu etc anyhow:-)
>
> On Tue, Nov 25, 2014 at 5:53 PM, Nikolai Grigoriev 
> wrote:
> > Hi Jean-Armel,
> >
> > I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but
> there
> > are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra
> 2.0.10.
> >
> > I have about 1,8Tb of data per node now in total, which falls into that
> > range.
> >
> > As I said, it is really a problem with large amount of data in a single
> CF,
> > not total amount of data. Quite often the nodes are idle yet having
> quite a
> > bit of pending compactions. I have discussed it with other members of C*
> > community and DataStax guys and, they have confirmed my observation.
> >
> > I believe that increasing the sstable size won't help at all and probably
> > will make the things worse - everything else being equal, of course. But
> I
> > would like to hear from Andrei when he is done with his test.
> >
> > Regarding the last statement - yes, C* clearly likes many small servers
> more
> > than fewer large ones. But it is all relative - and can be all
> recalculated
> > to $$$ :) C* is all about partitioning of everything - storage,
> > traffic...Less data per node and more nodes give you lower latency, lower
> > heap usage etc, etc. I think I have learned this with my project.
> Somewhat
> > hard way but still, nothing is better than the personal experience :)
> >
> > On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce 
> wrote:
> >>
> >> Hi Andrei, Hi Nicolai,
> >>
> >> Which version of C* are you using ?
> >>
> >> There are some recommendations about the max storage per node :
> >>
> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
> >>
> >> "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
> >> handle 10x
> >> (3-5TB)".
> >>
> >> I have the feeling that those recommendations are sensitive according
> many
> >> criteria such as :
> >> - your hardware
> >> - the compaction strategy
> >> - ...
> >>
> >> It looks that LCS lower those limitations.
> >>
> >> Increasing the size of sstables might help if you have enough CPU and
> you
> >> can put more load on your I/O system (@Andrei, I am interested by the
> >> results of your  experimentation about large sstable files)
> >>
> >> From my point of view, there are some usage patterns where it is better
> to
> >> have many small servers than a few large servers. Probably, it is
> better to
> >> have many small servers if you need LCS for large tables.
> >>
> >> Just my 2 cents.
> >>
> >> Jean-Armel
> >>
> >> 2014-11-24 19:56 GMT+01:00 Robert Coli :
> >>>
> >>> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev <
> ngrigor...@gmail.com>
> >>> wrote:
> 
>  One of the obvious recommendations I have received was to run more
> than
>  one instance of C* per host. Makes sense - it will reduce the amount
> of data
>  per node and will make better use of the resources.
> >>>
> >>>
> >>> This is usually a Bad Idea to do in production.
> >>>
> >>> =Rob
> >>>
> >>
> >>
> >
> >
> >
> > --
> > Nikolai Grigoriev
> > (514) 772-5178
>



-- 
Nikolai Grigoriev
(514) 772-5178

Re: Compaction Strategy guidance

[Andrei Ivanov]

Ah, clear then. SSD usage imposes a different bias in terms of costs;-)

On Tue, Nov 25, 2014 at 9:48 PM, Nikolai Grigoriev  wrote:
> Andrei,
>
> Oh, yes, I have scanned the top of your previous email but overlooked the
> last part.
>
> I am using SSDs so I prefer to put extra work to keep my system performing
> and save expensive disk space. So far I've been able to size the system more
> or less correctly so these LCS limitations do not cause too much troubles.
> But I do keep the CF "sharding" option as backup - for me it will be
> relatively easy to implement it.
>
>
> On Tue, Nov 25, 2014 at 1:25 PM, Andrei Ivanov  wrote:
>>
>> Nikolai,
>>
>> Just in case you've missed my comment in the thread (guess you have) -
>> increasing sstable size does nothing (in our case at least). That is,
>> it's not worse but the load pattern is still the same - doing nothing
>> most of the time. So, I switched to STCS and we will have to live with
>> extra storage cost - storage is way cheaper than cpu etc anyhow:-)
>>
>> On Tue, Nov 25, 2014 at 5:53 PM, Nikolai Grigoriev 
>> wrote:
>> > Hi Jean-Armel,
>> >
>> > I am using latest and greatest DSE 4.5.2 (4.5.3 in another cluster but
>> > there
>> > are no relevant changes between 4.5.2 and 4.5.3) - thus, Cassandra
>> > 2.0.10.
>> >
>> > I have about 1,8Tb of data per node now in total, which falls into that
>> > range.
>> >
>> > As I said, it is really a problem with large amount of data in a single
>> > CF,
>> > not total amount of data. Quite often the nodes are idle yet having
>> > quite a
>> > bit of pending compactions. I have discussed it with other members of C*
>> > community and DataStax guys and, they have confirmed my observation.
>> >
>> > I believe that increasing the sstable size won't help at all and
>> > probably
>> > will make the things worse - everything else being equal, of course. But
>> > I
>> > would like to hear from Andrei when he is done with his test.
>> >
>> > Regarding the last statement - yes, C* clearly likes many small servers
>> > more
>> > than fewer large ones. But it is all relative - and can be all
>> > recalculated
>> > to $$$ :) C* is all about partitioning of everything - storage,
>> > traffic...Less data per node and more nodes give you lower latency,
>> > lower
>> > heap usage etc, etc. I think I have learned this with my project.
>> > Somewhat
>> > hard way but still, nothing is better than the personal experience :)
>> >
>> > On Tue, Nov 25, 2014 at 3:23 AM, Jean-Armel Luce 
>> > wrote:
>> >>
>> >> Hi Andrei, Hi Nicolai,
>> >>
>> >> Which version of C* are you using ?
>> >>
>> >> There are some recommendations about the max storage per node :
>> >>
>> >> http://www.datastax.com/dev/blog/performance-improvements-in-cassandra-1-2
>> >>
>> >> "For 1.0 we recommend 300-500GB. For 1.2 we are looking to be able to
>> >> handle 10x
>> >> (3-5TB)".
>> >>
>> >> I have the feeling that those recommendations are sensitive according
>> >> many
>> >> criteria such as :
>> >> - your hardware
>> >> - the compaction strategy
>> >> - ...
>> >>
>> >> It looks that LCS lower those limitations.
>> >>
>> >> Increasing the size of sstables might help if you have enough CPU and
>> >> you
>> >> can put more load on your I/O system (@Andrei, I am interested by the
>> >> results of your  experimentation about large sstable files)
>> >>
>> >> From my point of view, there are some usage patterns where it is better
>> >> to
>> >> have many small servers than a few large servers. Probably, it is
>> >> better to
>> >> have many small servers if you need LCS for large tables.
>> >>
>> >> Just my 2 cents.
>> >>
>> >> Jean-Armel
>> >>
>> >> 2014-11-24 19:56 GMT+01:00 Robert Coli :
>> >>>
>> >>> On Mon, Nov 24, 2014 at 6:48 AM, Nikolai Grigoriev
>> >>> 
>> >>> wrote:
>> 
>>  One of the obvious recommendations I have received was to run more
>>  than
>>  one instance of C* per host. Makes sense - it will reduce the amount
>>  of data
>>  per node and will make better use of the resources.
>> >>>
>> >>>
>> >>> This is usually a Bad Idea to do in production.
>> >>>
>> >>> =Rob
>> >>>
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Nikolai Grigoriev
>> > (514) 772-5178
>
>
>
>
> --
> Nikolai Grigoriev
> (514) 772-5178

Re: large range read in Cassandra

[Robert Coli]

On Tue, Nov 25, 2014 at 10:45 AM, Dan Kinder  wrote:

> To be clear, I expect this range query to take a long time and perform
> relatively heavy I/O. What I expected Cassandra to do was use auto-paging (
> https://issues.apache.org/jira/browse/CASSANDRA-4415,
> http://stackoverflow.com/questions/17664438/iterating-through-cassandra-wide-row-with-cql3)
> so that we aren't literally pulling the entire thing in. Am I
> misunderstanding this use case? Could you clarify why exactly it would slow
> way down? It seems like with each read it should be doing a simple range
> read from one or two sstables.
>

If you're paging through a single partition, that's likely to be fine. When
you said "range reads ... over rows" my impression was you were talking
about attempting to page through millions of partitions.

With that confusion cleared up, the likely explanation for lack of
availability in your case is heap pressure/GC time. Look for GCs around
that time. Also, if you're using authentication, make sure that your
authentication keyspace has a replication factor greater than 1.

=Rob

Unsubscribe

[Kevin Daly]

Only BlueCat IPAM delivers true network intelligence, a smarter way to 
manage your network and devices.
Read more at www.bluecatnetworks.com/networkintelligence 
.


This e-mail and any attachments are for the sole use of the intended 
recipient and may contain confidential information.

Re: Cassandra version 1.0.10 Data Loss upon restart

[Robert Coli]

On Tue, Nov 25, 2014 at 6:40 AM, Ankit Patel  wrote:

> The JIRA https://issues.apache.org/jira/browse/CASSANDRA-4446 refers to
> the problem after we've invoked drain. However, we did not invoke drain or
> flush. We are running one node cassandra within one data center and it is
> being replicated with another node in another data center. We are using the
> thrift API in java application to retrieve and modify the data from the
> primary node only. We are not using the other node in the other data center
> for any operations. The data loss well exceeds the commit log sync period.
>

Right, what I meant by "the era of CASSANDRA-4446" is that around that time
there were reports of weird behavior around the commit log. Not your
particular weird behavior, but similar enough that I am not *overly*
surprised to hear it. Unfortunately no one is likely to want to fix your by
now quite old version, especially as a re-write of the commit log occurred
in (IIRC) 1.1.x series.

I recommend upgrading at least to the HEAD of 1.1.x ASAP.

=Rob
PS - I understand and sympathize, the above is not a terribly satisfying
answer in the face of data loss. :/

Re: Keyspace and table/cf limits

[Robert Coli]

On Tue, Nov 25, 2014 at 9:07 AM, Raj N  wrote:

> What's the latest on the maximum number of keyspaces and/or tables that
> one can have in Cassandra 2.1.x?
>

Most relevant changes lately would be :

https://issues.apache.org/jira/browse/CASSANDRA-6689
and
https://issues.apache.org/jira/browse/CASSANDRA-6694

Which should meaningfully reduce the amount of heap memtables consume. That
heap can then be used to support more heap-persistent structures associated
with many CFs. I have no idea how to estimate the scale of the improvement.

As a general/meta statement, Cassandra is very multi-threaded, and consumes
file handles like crazy. How many different query cases do you really want
to put on one cluster/node? ;D

=Rob

Re: Data synchronization between 2 running clusters on different availability zone

[Robert Coli]

On Tue, Nov 25, 2014 at 7:09 AM, Spico Florin  wrote:

> 1. For ensuring high availability I would like to install one Cassandra
> cluster on one availability zone
> (on Amazon EC2 US-east) and one Cassandra cluster on other AZ (Amazon EC2
> US-west).
>

One cluster, replication factor of 2, cluster configured with a rack aware
snitch is how this is usually done. Well, more accurately, people usually
deploy with at least RF=3 and across 3 AZs. A RF of at least 3 is also
required to use QUORUM Consistency Level.

If you will always operate only out of EC2, you probably want to look into
the EC2Snitch. If you plan to ultimately go multi-region,
EC2MultiRegionSnitch. If maybe hybrid in the future,
GossipingPropertyFileSnitch.

http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchEC2MultiRegion_c.html
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architectureSnitchGossipPF_c.html

For some good meta on the internals here :

https://issues.apache.org/jira/browse/CASSANDRA-3810

=Rob
http://twitter.com/rcolidba

Re: large range read in Cassandra

[Dan Kinder]

Thanks, very helpful Rob, I'll watch for that.

On Tue, Nov 25, 2014 at 11:45 AM, Robert Coli  wrote:

> On Tue, Nov 25, 2014 at 10:45 AM, Dan Kinder  wrote:
>
>> To be clear, I expect this range query to take a long time and perform
>> relatively heavy I/O. What I expected Cassandra to do was use auto-paging (
>> https://issues.apache.org/jira/browse/CASSANDRA-4415,
>> http://stackoverflow.com/questions/17664438/iterating-through-cassandra-wide-row-with-cql3)
>> so that we aren't literally pulling the entire thing in. Am I
>> misunderstanding this use case? Could you clarify why exactly it would slow
>> way down? It seems like with each read it should be doing a simple range
>> read from one or two sstables.
>>
>
> If you're paging through a single partition, that's likely to be fine.
> When you said "range reads ... over rows" my impression was you were
> talking about attempting to page through millions of partitions.
>
> With that confusion cleared up, the likely explanation for lack of
> availability in your case is heap pressure/GC time. Look for GCs around
> that time. Also, if you're using authentication, make sure that your
> authentication keyspace has a replication factor greater than 1.
>
> =Rob
>
>
>


-- 
Dan Kinder
Senior Software Engineer
Turnitin – www.turnitin.com
dkin...@turnitin.com

RAM vs SSD for real world performance?

[Kevin Burton]

The new SSDs that we have (as well as Fusion IO) in theory can saturate the
gigabit ethernet port.

The 4k random read and write IOs they’re doing now can easily add up quick
and they’re faster than gigabit and even two gigabit.

However, not all of that 4k is actually used.  I suspect that on average
half is wasted.

But the question is how much.  Of course YMMV.

I’m thinking of getting our servers with a moderate amount of RAM.  Say
24GB.  Then allocate 8GB to Cassandra, another 8GB to random daemons we
run, then another 8GB to page cache.

Curious what other people have seen here in practice.  Are they getting
comparable performance to RAM in practice? Latencies would be higher of
course but we’re fine with that.

-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: RAM vs SSD for real world performance?

[Robert Coli]

On Tue, Nov 25, 2014 at 5:31 PM, Kevin Burton  wrote:

> Curious what other people have seen here in practice.  Are they getting
> comparable performance to RAM in practice? Latencies would be higher of
> course but we’re fine with that.
>

My understanding is that when one runs Cassandra with SSDs, one replaces
the typical i/o bound with a CPU bound. Cassandra also has various internal
assumptions that do not make best use of the spare i/o available;
SSD+Cassandra has only been deployed at scale for a few years, so this
makes sense.

=Rob

High cpu usage & segfaulting

[Stan Lemon]

We are using v2.0.11 and have seen several instances in our 24 node cluster
where the node becomes unresponsive, when we look into it we find that
there is a cassandra process chewing up a lot of CPU. There are no other
indications in logs or anything as to what might be happening, however if
we strace the process that is chewing up CPU we see a segmental fault:

--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0x7fd61110f862)= 30618997712
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27333, NULL) = -1 EAGAIN
(Resource temporarily unavailable)
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27335, NULL) = 0
futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1

And this happens over and over again while running strafe.

Has anyone seen this? Does anyone have any ideas what might be happening,
or how we could debug it further?

Thanks for your help,

Stan

Re: RAM vs SSD for real world performance?

[Kevin Burton]

I imagine I’d generally be happy if we were CPU bound :-) … as long as the
number of transactions per second is generally reasonable.

On Tue, Nov 25, 2014 at 7:35 PM, Robert Coli  wrote:

> On Tue, Nov 25, 2014 at 5:31 PM, Kevin Burton  wrote:
>
>> Curious what other people have seen here in practice.  Are they getting
>> comparable performance to RAM in practice? Latencies would be higher of
>> course but we’re fine with that.
>>
>
> My understanding is that when one runs Cassandra with SSDs, one replaces
> the typical i/o bound with a CPU bound. Cassandra also has various internal
> assumptions that do not make best use of the spare i/o available;
> SSD+Cassandra has only been deployed at scale for a few years, so this
> makes sense.
>
> =Rob
>
>



-- 

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile

Re: High cpu usage & segfaulting

[Robert Coli]

On Tue, Nov 25, 2014 at 8:07 PM, Stan Lemon  wrote:

> We are using v2.0.11 and have seen several instances in our 24 node
> cluster where the node becomes unresponsive, when we look into it we find
> that there is a cassandra process chewing up a lot of CPU. There are no
> other indications in logs or anything as to what might be happening,
> however if we strace the process that is chewing up CPU we see a segmental
> fault:
>


> Has anyone seen this? Does anyone have any ideas what might be happening,
> or how we could debug it further?
>

Does it go away when you restart the node?

First, you should do the standard checks for if this is GC pre-fail, which
looks like a flattop on heap consumption graphs combined with a spike in GC
duration.

If you don't find that or OOM log messages, your version is new enough that
I would file a JIRA at http://issues.apache.org

=Rob

Re: High cpu usage & segfaulting

[Otis Gospodnetic]

Hi Stan,

Put some monitoring on this.  The first thing I think of when I hear
"chewing up CPU" for Java apps is GC.  In SPM 
you can easily see individual JVM memory pools and see if any of them are
at (close to) 100%.  You can typically correlate that to increased GC times
and counts.  I'd look at that before looking at strace and such.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Tue, Nov 25, 2014 at 11:07 PM, Stan Lemon  wrote:

> We are using v2.0.11 and have seen several instances in our 24 node
> cluster where the node becomes unresponsive, when we look into it we find
> that there is a cassandra process chewing up a lot of CPU. There are no
> other indications in logs or anything as to what might be happening,
> however if we strace the process that is chewing up CPU we see a segmental
> fault:
>
> --- SIGSEGV (Segmentation fault) @ 0 (0) ---
> rt_sigreturn(0x7fd61110f862)= 30618997712
> futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27333, NULL) = -1 EAGAIN
> (Resource temporarily unavailable)
> futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fd614844054, FUTEX_WAIT_PRIVATE, 27335, NULL) = 0
> futex(0x7fd614844028, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fd6148e2e54, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fd6148e2e50,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fd6148e2e28, FUTEX_WAKE_PRIVATE, 1) = 1
>
> And this happens over and over again while running strafe.
>
> Has anyone seen this? Does anyone have any ideas what might be happening,
> or how we could debug it further?
>
> Thanks for your help,
>
> Stan
>
>

Partial replication to a DC

[Robert Wille]

Is it possible to replicate a subset of the keyspaces to a data center? For 
example, if I want to run reports without impacting my production nodes, can I 
put the relevant column families in a keyspace and create a DC for reporting 
that replicates only that keyspace?

Robert

Re: Partial replication to a DC

[Robert Coli]

On Tue, Nov 25, 2014 at 8:52 PM, Robert Wille  wrote:

> Is it possible to replicate a subset of the keyspaces to a data center?
> For example, if I want to run reports without impacting my production
> nodes, can I put the relevant column families in a keyspace and create a DC
> for reporting that replicates only that keyspace?
>

Yes, that's one of the things that Datastax DSE does to enable the SOLR
special sauce. Be sure to have a version with LOCAL_X CLs, and use them.

=Rob
http://twitter.com/rcolidba

Re: mysql based columnar DB to Cassandra DB - Migration

[Akshay Ballarpure]

Hello Folks,
I have one mysql based columnar DB, i want to migrate it to Cassandra. How 
its possible ? 

Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty.   IT Services
Business Solutions
Consulting




From:   Akshay Ballarpure/HYD/TCS
To: user@cassandra.apache.org
Date:   11/18/2014 09:00 PM
Subject:mysql based columnar DB to Cassandra DB - Migration



I have one mysql based columnar DB, i want to migrate it to Cassandra. How 
its possible ? 

Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty. IT Services
Business Solutions
Consulting

=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you

RE: mysql based columnar DB to Cassandra DB - Migration

[Andreas Finke]

Hi Akshay,

this heavily depends on your data model. There is no general way to do it. It 
includes several steps:

1) Migration of applications using Mysql to Cassandra
2) Migration of the Mysql Database to Cassandra itself

Keep in mind that there are no such things like relations or joins in 
Cassandra. In general

Actually Datastax published some documents that could be of interest for you:

http://www.datastax.com/2012/03/how-to-move-data-from-relational-databases-to-datastax-enterprise-cassandra-using-sqoop

http://www.datastax.com/resources/whitepapers/mysql-to-cassandra

Also I found this from EbayTech:

http://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-practices-part-1/#.VHV2BlXF_Eg

Regards
Andi


From: Akshay Ballarpure [akshay.ballarp...@tcs.com]
Sent: 26 November 2014 07:15
To: user@cassandra.apache.org
Subject: Re: mysql based columnar DB to Cassandra DB - Migration

Hello Folks,
I have one mysql based columnar DB, i want to migrate it to Cassandra. How its 
possible ?

Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty.IT Services
   Business Solutions
   Consulting




From:Akshay Ballarpure/HYD/TCS
To:user@cassandra.apache.org
Date:11/18/2014 09:00 PM
Subject:mysql based columnar DB to Cassandra DB - Migration




I have one mysql based columnar DB, i want to migrate it to Cassandra. How its 
possible ?

Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty. IT Services
Business Solutions
Consulting


=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you

RE: mysql based columnar DB to Cassandra DB - Migration

[Akshay Ballarpure]

Thanks Andy for quick reply. 
I will have a look at the below link and get back. 


Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty.   IT Services
Business Solutions
Consulting




From:   Andreas Finke 
To: "user@cassandra.apache.org" 
Date:   11/26/2014 12:12 PM
Subject:RE: mysql based columnar DB to Cassandra DB - Migration



Hi Akshay, 

this heavily depends on your data model. There is no general way to do it. 
It includes several steps:

1) Migration of applications using Mysql to Cassandra
2) Migration of the Mysql Database to Cassandra itself

Keep in mind that there are no such things like relations or joins in 
Cassandra. In general 

Actually Datastax published some documents that could be of interest for 
you:

http://www.datastax.com/2012/03/how-to-move-data-from-relational-databases-to-datastax-enterprise-cassandra-using-sqoop

http://www.datastax.com/resources/whitepapers/mysql-to-cassandra

Also I found this from EbayTech:

http://www.ebaytechblog.com/2012/07/16/cassandra-data-modeling-best-practices-part-1/#.VHV2BlXF_Eg

Regards
Andi


From: Akshay Ballarpure [akshay.ballarp...@tcs.com] 
Sent: 26 November 2014 07:15
To: user@cassandra.apache.org
Subject: Re: mysql based columnar DB to Cassandra DB - Migration

Hello Folks, 
I have one mysql based columnar DB, i want to migrate it to Cassandra. How 
its possible ? 

Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty.IT Services
   Business Solutions
   Consulting
 



From:Akshay Ballarpure/HYD/TCS 
To:user@cassandra.apache.org 
Date:11/18/2014 09:00 PM 
Subject:mysql based columnar DB to Cassandra DB - Migration 



I have one mysql based columnar DB, i want to migrate it to Cassandra. How 
its possible ? 

Best Regards
Akshay Ballarpure
Tata Consultancy Services
Cell:- 9985084075
Mailto: akshay.ballarp...@tcs.com
Website: http://www.tcs.com

Experience certainty. IT Services
Business Solutions
Consulting
 
=-=-=
Notice: The information contained in this e-mail
message and/or attachments to it may contain 
confidential or privileged information. If you are 
not the intended recipient, any dissemination, use, 
review, distribution, printing or copying of the 
information contained in this e-mail message 
and/or attachments to it are strictly prohibited. If 
you have received this communication in error, 
please notify us by reply e-mail or telephone and 
immediately and permanently delete the message 
and any attachments. Thank you