Re: two dimensional slicing
It depends a bit on the data and the query patterns. * How many versions do you have ? * How many names in each version ? * When querying do you know the versions numbers you want to query from ? How many are there normally? * How frequent are the updates and the reads ? I would lean towards using two standard CF's, one to list all the version numbers (in a single row probably) and one to hold the names in a particular version. To do your query slice the first CF and then run multi gets to the second. Thats probably not the best solution, if you can add some more info it may get better. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 21/01/2012, at 6:20 AM, Bryce Allen wrote: > I'm storing very large versioned lists of names, and I'd like to > query a range of names within a given range of versions, which is a two > dimensional slice, in a single query. This is easy to do using > ByteOrderedPartitioner, but seems to require multiple (non parallel) > queries and extra CFs when using RandomPartitioner. > > I see two approaches when using RP: > > 1) Data is stored in a super column family, with one dimension being > the super column names and the other the sub column names. Since > slicing on sub columns requires a list of super column names, a > second standard CF is needed to get a range of names before doing a > query on the main super CF. With CASSANDRA-2710, the same is possible > using a standard CF with composite types instead of a super CF. > > 2) If one of the dimensions is small, a two dimensional slice isn't > required. The data can be stored in a standard CF with linear ordering > on a composite type (large_dimension, small_dimension). Data is queried > based on the large dimension, and the client throws out the extra data > in the other dimension. > > Neither of the above solutions are ideal. Does anyone else have a use > case where two dimensional slicing is useful? Given the disadvantages of > BOP, is it practical to make the composite column query model richer to > support this sort of use case? > > Thanks, > Bryce
Re: delay in data deleting in cassadra
also, *please* upgrade to the latest 0.8 release.
Cheers
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 21/01/2012, at 4:31 PM, Maxim Potekhin wrote:
> Did you run repairs withing GC_GRACE all the time?
>
>
>
> On 1/20/2012 3:42 AM, Shammi Jayasinghe wrote:
>>
>> Hi,
>> I am experiencing a delay in delete operations in cassandra. Its as
>> follows. I am running a thread which contains following three steps.
>>
>> Step 01: Read data from column family "foo"[1]
>> Step 02: Process received data eg: bar1,bar2,bar3,bar4,bar5
>> Step 03: Remove those processed data from "foo".[2]
>>
>> The problem occurs when this thread is invoked for the second time.
>> In that step , it returns some of data that i already deleted in the third
>> step of the previous cycle.
>>
>> Eg: it returns bar2,bar3,bar4,bar5
>>
>> It seems though i called the remove operation as follows [2], it takes time
>> to replicate it to
>> the file system. If i make a thread sleep of 5 secs between the thread
>> cycles, it does not
>> give me any data that i deleted in the third step.
>>
>> [1] . SliceQuery sliceQuery =
>> HFactory.createSliceQuery(keyspace, stringSerializer,
>> stringSerializer, bs);
>> sliceQuery.setKey(queueName);
>> sliceQuery.setRange("", "", false, messageCount);
>> sliceQuery.setColumnFamily(USER_QUEUES_COLUMN_FAMILY);
>>
>> [2]. Mutator mutator = HFactory.createMutator(keyspace,
>> stringSerializer);
>> mutator.addDeletion(queueName, USER_QUEUES_COLUMN_FAMILY, messageId,
>> stringSerializer);
>> mutator.execute();
>>
>>
>>
>> Is there a solution for this.
>>
>> Cassadra version : 0.8.0
>> Libthrift version : 0.6.1
>>
>>
>> Thanks
>> Shammi
>> --
>> Best Regards,
>>
>> Shammi Jayasinghe
>> Senior Software Engineer; WSO2, Inc.; http://wso2.com,
>> mobile: +94 71 4493085
>>
>>
>
Re: Get all keys from the cluster
If you want to keep the load out of the cassandra process and do the join to sql off line, take a look at the bin/sstablekeys utility. This will let you output the keys in an sstable. You will need to do it for every sstable on every node, create the unique list and then check in your SQL db thingy. The output will be hex encoded, and the process scans through the index file. So while it will take some IO load but it does ask the OS to skip using the IO cache. Alternatively just write a script that iterates over all the rows in the CF and does the lookup. hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/01/2012, at 1:11 AM, Eric Czech wrote: > Great! I'm glad at least one of those ideas was helpful for you. > > That's a road we've travelled before and as one last suggestion that might > help, you could alter all client writers to cassandra beforehand so that they > write to BOTH keyspaces BEFORE beginning the SQL based transfer. This might > help keep you from having to make multiple passes unless I'm not missing > something. > > On Sat, Jan 21, 2012 at 4:53 AM, Marcel Steinbach > wrote: > Thanks for your suggestions, Eric! > >> One of the application uses 1.5TB out of 1.8TB > > I'm sorry, maybe that statment was slightly ambiguous. I meant to say, that > one application uses 1.5TB, while the others use 300GB, totalling in 1.8TB of > data. Our total disk capacity, however, is at about 7 TB, so we're still far > from running out of disk space. > >> Is there any way that you could do that lookup in reverse where you pull the >> records from your SQL database, figure out which keys aren't necessary, and >> then delete any unnecessary keys that may or may not exist in cassandra? > > Unfortunately, that won't work since the SQL db does only contain the keys, > that we want to _keep_ in cassandra. > >> If that's not a possibility, then what about creating the same Cassandra >> schema in a different keyspace and copying all the relevant records from the >> current keyspace to the new keyspace using the SQL database records as a >> basis for what is actually "relevant" within the new keyspace. > > I like that idea. So instead of iterating over all cassandra rows, I would > iterate over the SQL DB, which would indeed save me a lot of IO. However, > rows inserted into my CF during iterating over the SQL DB might not be copied > into the new keyspace. But maybe we could arrange to do that > during low-demand-hours to minimize the amount of new inserts and > additionally run the "copy" a second time with a select on newly inserted sql > rows. So we'll probably go with that. > > Thanks again for your help! > > Cheers > Marcel > > On 21.01.2012, at 11:52, Eric Czech wrote: > >> Is there any way that you could do that lookup in reverse where you pull the >> records from your SQL database, figure out which keys aren't necessary, and >> then delete any unnecessary keys that may or may not exist in cassandra? >> >> If that's not a possibility, then what about creating the same Cassandra >> schema in a different keyspace and copying all the relevant records from the >> current keyspace to the new keyspace using the SQL database records as a >> basis for what is actually "relevant" within the new keyspace. If you could >> perform that transfer, then you could just delete the old 1.5TB keyspace >> altogether, leaving only the data you need. If that sort of duplication >> would put you over the 1.8TB limit during the transfer, then maybe you could >> consider CF compression upfront. >> >> Short of that, I can tell from experience that doing these sort of "left >> join" deletes from cassandra to SQL really suck. We have had to resort to >> using hadoop to do this but since our hadoop/cassandra clusters are much >> larger than our single SQL instances, keeping all the hadoop processes from >> basically "DDoS"ing our SQL servers while still making the process faster >> than thrift iterations over all the rows (via custom programs) in cassandra >> hasn't been a convincing solution. >> >> I'd say that the first solution I proposed is definitely the best, but also >> the most unrealistic. If that's really not a possibility for you, then I'd >> seriously look at trying to make my second suggestion work even if it means >> brining up new hardware or increasing the capacity of existing resources. >> That second suggestion also has the added benefit of likely minimizing I/O >> since it's the only solution that doesn't require reading or deleting any of >> the unnecessary data (beyond wholesale keyspace or CF deletions) assuming >> that the actually relevant portion of your data is significantly less than >> 1.5TB. >> >> I hope that helps! >> >> And in the future, you should really try to avoid letting your data size get >> beyond 40 - 50 % of your actual on-disk capacity. Let me kn
Re: delay in data deleting in cassadra
On Fri, Jan 20, 2012 at 11:02 PM, Peter Schuller < peter.schul...@infidyne.com> wrote: > > The problem occurs when this thread is invoked for the second time. > > In that step , it returns some of data that i already deleted in the > third > > step of the previous cycle. > > In order to get a guarantee about a subsequent read seeing a write, > you must read and write at QUORUM (or LOCAL_QUORUM if it's only within > a DC). > > Hi , I tried with changing the consistency level to QUORUM. But unfortunately it behaves same as previous. Thanks Shammi > -- > / Peter Schuller (@scode, http://worldmodscode.wordpress.com) > -- Best Regards,* Shammi Jayasinghe* Senior Software Engineer; WSO2, Inc.; http://wso2.com, mobile: +94 71 4493085
Re: Unbalanced cluster with RandomPartitioner
Setting a token outside of the partitioner range sounds like a bug. It's mostly
an issue with the RP, but I guess a custom partitioner may also want to
validate tokens are within a range.
Can you report it to https://issues.apache.org/jira/browse/CASSANDRA
Thanks
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/01/2012, at 5:58 AM, Marcel Steinbach wrote:
> I thought about our issue again and was thinking, maybe the describeOwnership
> should take into account, if a token is outside the partitioners maximum
> token range?
>
> To recap our problem: we had tokens, that were apart by 12.5% of the token
> range 2**127, however, we had an offset on each token, which moved the
> cluster's token range above 2**127. That resulted in two nodes getting almost
> none or none primary replicas.
>
> Afaik, the partitioner itself describes the key ownership in the ring, but it
> didn't take into account that we left its maximum key range.
>
> Of course, it is silly and not very likely that users make that mistake,
> however, we did it, and it took me quite some time to figure that out (maybe
> also because it wasn't me that setup the cluster).
>
> To carry it to the extreme, you could construct a cluster of n nodes with
> all tokens greater than 2**127, the ownership description would show a
> ownership of 1/n each but all data would go to the node with the lowest token
> (given RP and RF=1).
>
> I think it is wrong to calculate the ownership by subtracting the previous
> token from the current token and divide it by the maximum token without
> acknowledging we already might be "out of bounds".
>
> Cheers
> Marcel
>
> On 20.01.2012, at 16:28, Marcel Steinbach wrote:
>
>> Thanks for all the responses!
>>
>> I found our problem:
>> Using the Random Partitioner, the key range is from 0..2**127.When we added
>> nodes, we generated the keys and out of convenience, we added an offset to
>> the tokens because the move was easier like that.
>>
>> However, we did not execute the modulo 2**127 for the last two tokens, so
>> they were outside the RP's key range.
>> moving the last two tokens to their mod 2**127 will resolve the problem.
>>
>> Cheers,
>> Marcel
>>
>> On 20.01.2012, at 10:32, Marcel Steinbach wrote:
>>
>>> On 19.01.2012, at 20:15, Narendra Sharma wrote:
I believe you need to move the nodes on the ring. What was the load on the
nodes before you added 5 new nodes? Its just that you are getting data in
certain token range more than others.
>>> With three nodes, it was also imbalanced.
>>>
>>> What I don't understand is, why the md5 sums would generate such massive
>>> hot spots.
>>>
>>> Most of our keys look like that:
>>> 00013270494972450001234567
>>> with the first 16 digits being a timestamp of one of our application
>>> server's startup times, and the last 10 digits being sequentially generated
>>> per user.
>>>
>>> There may be a lot of keys that start with e.g. "0001327049497245" (or
>>> some other time stamp). But I was under the impression that md5 doesn't
>>> bother and generates uniform distribution?
>>> But then again, I know next to nothing about md5. Maybe someone else has a
>>> better insight to the algorithm?
>>>
>>> However, we also use cfs with a date ("mmdd") as key, as well as cfs
>>> with uuids as keys. And those cfs in itself are not balanced either. E.g.
>>> node 5 has 12 GB live space used in the cf the uuid as key, and node 8 only
>>> 428MB.
>>>
>>> Cheers,
>>> Marcel
>>>
On Thu, Jan 19, 2012 at 3:22 AM, Marcel Steinbach
wrote:
On 18.01.2012, at 02:19, Maki Watanabe wrote:
> Are there any significant difference of number of sstables on each nodes?
No, no significant difference there. Actually, node 8 is among those with
more sstables but with the least load (20GB)
On 17.01.2012, at 20:14, Jeremiah Jordan wrote:
> Are you deleting data or using TTL's? Expired/deleted data won't go away
> until the sstable holding it is compacted. So if compaction has happened
> on some nodes, but not on others, you will see this. The disparity is
> pretty big 400Gb to 20GB, so this probably isn't the issue, but with our
> data using TTL's if I run major compactions a couple times on that column
> family it can shrink ~30%-40%.
Yes, we do delete data. But I agree, the disparity is too big to blame
only the deletions.
Also, initially, we started out with 3 nodes and upgraded to 8 a few weeks
ago. After adding the node, we did
compactions and cleanups and didn't have a balanced cluster. So that
should have removed outdated data, right?
> 2012/1/18 Marcel Steinbach :
>> We are running regular repairs, so I don't think that's the problem.
>> And the data dir sizes match approx. the load from the nodetool.
>> Thanks for the advise, though.
Re: ideal cluster size
I second Peters point, big servers are not always the best. My experience (using spinning disks) is that 200 to 300 GB of live data load per node (including replicated data) is a sweet spot. Above this the time taken for compaction, repair, off node backups, node moves etc starts to be a pain. Also, suffering catastrophic failure of 1 node in 100 is a better situation that 1 node in 16. Finally, when you have more servers with less high performance disks you also get more memory and more CPU cores. (I'm obviously ignoring all the ops side here, automate with chef or http://www.datastax.com/products/opscenter ). wrt failure modes I wrote this last year, it's about single DC deployments but you can probably work it out for multi-dc http://thelastpickle.com/2011/06/13/Down-For-Me/ Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 22/01/2012, at 1:18 PM, Thorsten von Eicken wrote: > Good point. One thing I'm wondering about cassandra is what happens when > there is a massive failure. For example, if 1/3 of the nodes go down or > become unreachable. This could happen in EC2 if an AZ has a failure, or > in a datacenter if a whole rack or UPS goes dark. I'm not so concerned > about the time where the nodes are down. If I understand replication, > consistency, ring, and such I can architect things such that what must > continue running does continue. > > What I'm concerned about is when these nodes all come back up or > reconnect. I have a hard time figuring out what exactly happens other > than the fact that hinted handoffs get processed. Are the restarted > nodes handling reads during that time? If so, they could serve up > massive amounts of stale data, no? Do they then all start a repair, or > is this something that needs to be run manually? If many do a repair at > the same time, do I effectively end up with a down cluster due to the > repair load? If no node was lost, is a repair required or are the hinted > handoffs sufficient? > > Is there a manual or wiki section that discusses some of this and I just > missed it? > > On 1/21/2012 2:25 PM, Peter Schuller wrote: >>> Thanks for the responses! We'll definitely go for powerful servers to >>> reduce the total count. Beyond a dozen servers there really doesn't seem >>> to be much point in trying to increase count anymore for >> Just be aware that if "big" servers imply *lots* of data (especially >> in relation to memory size), it's not necessarily the best trade-off. >> Consider the time it takes to do repairs, streaming, node start-up, >> etc. >> >> If it's only about CPU resources then bigger nodes probably make more >> sense if the h/w is cost effective. >>
Re: Data Model Question
> 1. regarding time slicing, if at any point of time I am interested in what > happened in the last T minutes, then I will need to query more than one row > of the DimentionUpdates, right? Yerp. Sometimes that's is what's needed. > 2. What did you mean by "You will also want to partition the list of actions > in an actiongroup" In the ActionGroup CF in your model you have a column for every action in the group. You may want to partition that so the row does not grow for ever. > 3. For insert of a new ActionGroup, I will need to Sounds about right. Don't forget you want to round the current time to the start of the current time partition. So if it's 08:38 and you have 5 minute partitions it will be written to the 08:35 row. > Is there any problem of synchronization there? You can send them as a batch, but only the mutations for the same rows are atomic. And the changes are not isolated, i.e. a read can get a partially updated row. So your client needs to handle cases where referential integrity is somehow broken. See this presentation from Matt Dennis about how you can implement an application level transaction log if that's something you need http://www.slideshare.net/mattdennis/cassandra-nyc-2011-data-modeling > Does that makes sense? Do you think your solution is still better? > You may get poor read performance with that option. Overwriting the same row over a long period can result in poor read performance, this is better in 1.0 with leveled compaction but I cannot tell you how much better off the top of my head. Also TTL columns must be discarded during a read, and are only purged on compaction. A high number of dead columns in a row can reduce read performance. So you will want to reverse the comparator (http://thelastpickle.com/2011/10/03/Reverse-Comparators/) so you can avoid reading them. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 23/01/2012, at 8:43 PM, Tamar Fraenkel wrote: > Hi! > Thank you very much for your response! > > I have couple of questions regarding it, some are just to make sure I > understood you: > > 1. regarding time slicing, if at any point of time I am interested in what > happened in the last T minutes, then I will need to query more than one row > of the DimentionUpdates, right? > > 2. What did you mean by "You will also want to partition the list of actions > in an actiongroup" > > 3. For insert of a new ActionGroup, I will need to > > a) add (in some cases overwrite) columns for all the new ActionGroup > catetories and tags in the time_partition row of DimensionUpdates. > b) add to all the rows in DimentionFacts that are relevant (time, categories > and tags) a column for the ActionGroupId. > c) add all the information to the ActionGroup CF and Action CF, including the > tags\categories as column names and their scores as values. > Is there any problem of synchronization there? > > 4. regarding the row size, wouldn't that be solved by giving TTL to the > columns > > > 5. After reading a bit I also thought that built in secondary indices won't > work. There is also another complication, that having tagA with CategoryA is > different than having tagA with CategoryB. > > But I still thought that something like > > > CF: ActionGroupByCategoryTag > Stores which action groups are tagged by each tag. > key: > col_names: > col_value: > > And then I will get categories*tags rows from this index table, and get > sliced columns by the time range i am interested in. I can define some TTL on > the columns to get only ActionGroups created in the last T minutes. > > > Does that makes sense? Do you think your solution is still better? > > > Tamar > > > > On January 22, 2012 at 9:53 PM aaron morton wrote: > > In general if you are collecting data over time you should consider > partitioning the row's to avoid creating very large rows. Also if you have a > common request you want to support consider modeling it directly rather than > using secondary indexes. > > Assuming my understanding of the problem is in some one way correct I would > consider this for the Action Groups… > > Pick a time partition, this is the minimum time resolution you are interested > in (your T Minutes). > > CF: DimensionUpdates > Stores which dimensions (tags, categories) were updated in the time > partition. > > key: is the start of the partition e.g. 2011-01-23T08:30 > col_names: where is "tag" > or "category" and is a value from that domain. e.g. > > col_value: empty > > > CF: DimensionFacts > Stores the facts that included the dimension in a time partition. > > key: definitions as above. > col_names: ActionGroupID. > col_values: empty > > So to… > > > Find all the recent ActionGroups (those who were updated with actions > performed during the last T minutes), who has at list one of the new action’s > categories AND at list one of the
Re: get all columns for a row
The columns are stored at the intersection of the row and the CF. So if you
read all the columns for a row in a CF you are only getting those ones.
Your hector code (using the range) looks correct to me.
Have fun.
Aaron
-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 23/01/2012, at 9:24 PM, Tamar Fraenkel wrote:
> Hi!
> I am getting started with Cassandra and Hector and have a question.
> If I have CF with many columns, only a small portion of them having values
> for any given key. I want to read all columns for a given key, but I don't
> know which columns were populated.
> I managed to do that with the code below (assumes that column names are
> incremental numbers).
> Is there any better way?
> Thanks,
> Tamar
>
> private static void readAllColunsForRowKey(String key) {
> MultigetSliceQuery multigetSlicesQuery =
> HFactory.createMultigetSliceQuery(tutorialKeyspace,
> StringSerializer.get(), LongSerializer.get(),
>
>StringSerializer.get());
> multigetSlicesQuery.setColumnFamily(CF);
> multigetSlicesQuery.setKeys(key);
> multigetSlicesQuery.setRange(null, null, false, COL_COUNT);
> QueryResult> results =
> multigetSlicesQuery.execute();
> Long lastColName;
> while ((lastColName = printColsAndGetLastColName(results, key)) != null) {
> multigetSlicesQuery.setRange(lastColName + 1, null, false, COL_COUNT);
> results = multigetSlicesQuery.execute();
> }
> }
>
> private static Long printColsAndGetLastColName(QueryResult String>> results, String key) {
> Rows rows = results.get();
> if (rows.getCount() != 1) {
> System.out.println("No such key");
> return null;
> }
> Row row = rows.getByKey(key);
> ColumnSlice columnSlice = row.getColumnSlice();
> List> columns = columnSlice.getColumns();
> for (HColumn column : columns) {
> System.out.println("Column Name: " + column.getName()
> + ", Column Val: " + column.getValue());
> }
> if (columns.size() == COL_COUNT) {
> return columns.get(COL_COUNT - 1).getName();
> }
> return null;
> }
>
Re: delay in data deleting in cassadra
On Mon, Jan 23, 2012 at 2:10 PM, Shammi Jayasinghe wrote: > > > On Fri, Jan 20, 2012 at 11:02 PM, Peter Schuller < > peter.schul...@infidyne.com> wrote: > >> > The problem occurs when this thread is invoked for the second time. >> > In that step , it returns some of data that i already deleted in the >> third >> > step of the previous cycle. >> >> In order to get a guarantee about a subsequent read seeing a write, >> you must read and write at QUORUM (or LOCAL_QUORUM if it's only within >> a DC). >> >> Hi , > I tried with changing the consistency level to QUORUM. But unfortunately > it behaves same as previous. > Hi, I am sorry, it worked fine when the Consistency Level Set to QUORUM. Thank you very very much for help. Thanks Shammi > > Thanks > Shammi > >> -- >> / Peter Schuller (@scode, http://worldmodscode.wordpress.com) >> > > > > -- > Best Regards,* > > Shammi Jayasinghe* > Senior Software Engineer; WSO2, Inc.; http://wso2.com, > mobile: +94 71 4493085 > > > -- Best Regards,* Shammi Jayasinghe* Senior Software Engineer; WSO2, Inc.; http://wso2.com, mobile: +94 71 4493085
Tips for using OrderedPartitioner
Hi, We use Cassandra in a way we always want to range slice queries. Because, of the tendency to create hotspots with OrderedPartioner we decided to use RandomPartitioner. Then we would use, a row as an index row, holding values of the other row keys of the CF. I feel this has become a burden and would like to move to an OrderedPartioner to avoid this work around. The index row workaround which has become cumbersome when we query the data store. Is there any tips we can follow to allow for lesser amount of hot spots? -- Regards, Tharindu blog: http://mackiemathew.com/
Re: CQL jdbc
If I understand correctly this is due in Cassandra 1.1. Does anyone know when it
is planned to be released?
Thanks
Tamar
On January 23, 2012 at 11:13 AM Jawahar Prasad wrote:
> Hi..
> Yes there is. But just 2 days back, they have released a patch:
>
> https://issues.apache.org/jira/browse/CASSANDRA-3761
>
> I am in the discussion, you can join as well
>
> Regards
> Prasad
>
>
> On Mon, Jan 23, 2012 at 2:37 PM, Tamar Fraenkel [mailto:ta...@tok-media.com] >wrote:
>
> >
> > Hi!
> > Is there something that is a real limitation of CQL currently. For example
> > composite keys \ column names?
> > Tamar
> >
> >
> >
> >
> >
> > On January 23, 2012 at 10:29 AM Jawahar Prasad > [mailto:w3engine...@gmail.com] > wrote:
> >
> >
> > > Hi..
> > > I am using CQL for the below reasons:
> > >
> > > 1. Hector is better than Thrift, but CQL is better than Hector in terms of
> > > understanding and quickly getting things done
> > > 2. Hector is third party, CQL is from cassandra developers, So there will
> > > be a good support.
> > > 3. As per Eric, CQL will be the future and will replace all third party
> > > clients.
> > >
> > > And as i got used to SQL, CQL makes more sense to me.
> > >
> > > Regards
> > > Prasad
> > >
> > >
> > > On Mon, Jan 23, 2012 at 1:46 PM, Tamar Fraenkel > > [mailto:ta...@tok-media.com] >wrote:
> > >
> > > >
> > > > Thanks, will give it a try. By the way, I had the same issue when trying
> > > > to work with Hector and I just took all the jars that Hector tutorial
> > > > brings using Maven. Most are in the list below.
> > > >
> > > > Another question for that matter, what do you recommend working with
> > > > Hector or CQL?
> > > >
> > > > Tamar
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On January 23, 2012 at 8:38 AM Jawahar Prasad > > > [mailto:w3engine...@gmail.com] > wrote:
> > > >
> > > >
> > > > > Hello
> > > > > I just thought I will tell you how I solved it:
> > > > >
> > > > > When I generated a jar from the jdbc code, it generated the following
> > > > > jars:
> > > > >
> > > > > cassandra-clientutil.jar
> > > > > cassandra-thrift.jar
> > > > > commons-codec.jar
> > > > > commons-lang.jar
> > > > > commons-logging.jar
> > > > > guava.jar
> > > > > httpclient.jar
> > > > > httpcore.jar
> > > > > libthrift.jar
> > > > > servlet-api.jar
> > > > > cassandra-jdbc-1.0.5-SNAPSHOT.jar
> > > > > slf4j-api-1.6.1.jar
> > > > > slf4j-log4j12-1.6.1.jar
> > > > >
> > > > >
> > > > > I included all the above in my current project and the problem got
> > > > > solved. If you dont include sf4j, you might get logging errors. So
> > > > > just include all of them.
> > > > >
> > > > > Regards
> > > > > Prasad
> > > > >
> > > > >
> > > > >
Hi!
I have cassandra-clientutil, cassandra-jdbc and
cassandra-thrift in my libs, but
I get
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.cassandra.cql.jdbc.CassandraDriver
when running
Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver").newInstance();
CassandraDriver is in my classpath.
Any idea?
Tamar
> > > > >
> > > > >
> > > >
> >
datastax opscenter authentication
I am trying to integrate opscenter in our environment and I was wondering if we can use PAM authentication instead of a password file for opscenter authentication? thanks Ramesh
Re: CQL jdbc
Hey,
Based on current discussions it looks like it will be in C* 1.1, but won't
be in the default cql package - you'll need to opt into cql3 driver as
there are some incompatible BC changes and they want to give an easier
migration. It will be in the default standard distribution in 1.2.
Alex.
On Mon, Jan 23, 2012 at 1:56 PM, Tamar Fraenkel wrote:
> **
>
> If I understand correctly this is due in Cassandra 1.1. Does anyone
> know when it is planned to be released?
>
> Thanks
>
> Tamar
>
>
> On January 23, 2012 at 11:13 AM Jawahar Prasad
> wrote:
>
> Hi..
> Yes there is. But just 2 days back, they have released a patch:
>
> https://issues.apache.org/jira/browse/CASSANDRA-3761
>
> I am in the discussion, you can join as well
>
> Regards
> Prasad
>
> On Mon, Jan 23, 2012 at 2:37 PM, Tamar Fraenkel < ta...@tok-media.com >
> wrote:
>
>
> Hi!
>
> Is there something that is a real limitation of CQL currently. For example
> composite keys \ column names?
>
> Tamar
>
>
>
> On January 23, 2012 at 10:29 AM Jawahar Prasad < w3engine...@gmail.com >
> wrote:
>
> Hi..
> I am using CQL for the below reasons:
>
> 1. Hector is better than Thrift, but CQL is better than Hector in terms of
> understanding and quickly getting things done
> 2. Hector is third party, CQL is from cassandra developers, So there will
> be a good support.
> 3. As per Eric, CQL will be the future and will replace all third party
> clients.
>
> And as i got used to SQL, CQL makes more sense to me.
>
> *Regards*
> Prasad
>
> On Mon, Jan 23, 2012 at 1:46 PM, Tamar Fraenkel < ta...@tok-media.com >
> wrote:
>
>
> Thanks, will give it a try. By the way, I had the same issue when trying
> to work with Hector and I just took all the jars that Hector tutorial
> brings using Maven. Most are in the list below.
>
>
>
> Another question for that matter, what do you recommend working with
> Hector or CQL?
>
>
>
> Tamar
>
>
>
> On January 23, 2012 at 8:38 AM Jawahar Prasad < w3engine...@gmail.com >
> wrote:
>
> Hello
> I just thought I will tell you how I solved it:
>
> When I generated a jar from the jdbc code, it generated the following
> jars:
>
> cassandra-clientutil.jar
> cassandra-thrift.jar
> commons-codec.jar
> commons-lang.jar
> commons-logging.jar
> guava.jar
> httpclient.jar
> httpcore.jar
> libthrift.jar
> servlet-api.jar
> cassandra-jdbc-1.0.5-SNAPSHOT.jar
> slf4j-api-1.6.1.jar
> slf4j-log4j12-1.6.1.jar
>
>
> I included all the above in my current project and the problem got solved.
> If you dont include sf4j, you might get logging errors. So just include all
> of them.
>
> Regards
> Prasad
>
>
>Hi!
>I have cassandra-clientutil, cassandra-jdbc and
> cassandra-thrift in my libs, but
>I get
>
>java.lang.NoClassDefFoundError: Could not initialize class
>org.apache.cassandra.cql.jdbc.CassandraDriver
>when running
>
> Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver").newInstance();
>
>CassandraDriver is in my classpath.
>
>Any idea?
>Tamar
>
>
>
>
>
>
>
>
>
>
>
Re: CQL jdbc
On Mon, Jan 23, 2012 at 8:40 AM, Alex Major wrote:
> Based on current discussions it looks like it will be in C* 1.1, but won't
> be in the default cql package - you'll need to opt into cql3 driver as there
> are some incompatible BC changes and they want to give an easier migration.
> It will be in the default standard distribution in 1.2.
No. To get the JDBC driver, you need to install it from its project page.
http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/
> On Mon, Jan 23, 2012 at 1:56 PM, Tamar Fraenkel wrote:
>>
>> If I understand correctly this is due in Cassandra 1.1. Does anyone know
>> when it is planned to be released?
>>
>> Thanks
>>
>> Tamar
>>
>>
>> On January 23, 2012 at 11:13 AM Jawahar Prasad
>> wrote:
>>
>> Hi..
>> Yes there is. But just 2 days back, they have released a patch:
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-3761
>>
>> I am in the discussion, you can join as well
>>
>> Regards
>> Prasad
>>
>> On Mon, Jan 23, 2012 at 2:37 PM, Tamar Fraenkel < ta...@tok-media.com >
>> wrote:
>>
>> Hi!
>>
>> Is there something that is a real limitation of CQL currently. For example
>> composite keys \ column names?
>>
>> Tamar
>>
>>
>>
>>
>> On January 23, 2012 at 10:29 AM Jawahar Prasad < w3engine...@gmail.com >
>> wrote:
>>
>> Hi..
>> I am using CQL for the below reasons:
>>
>> 1. Hector is better than Thrift, but CQL is better than Hector in terms of
>> understanding and quickly getting things done
>> 2. Hector is third party, CQL is from cassandra developers, So there will
>> be a good support.
>> 3. As per Eric, CQL will be the future and will replace all third party
>> clients.
>>
>> And as i got used to SQL, CQL makes more sense to me.
>>
>> Regards
>> Prasad
>>
>> On Mon, Jan 23, 2012 at 1:46 PM, Tamar Fraenkel < ta...@tok-media.com >
>> wrote:
>>
>> Thanks, will give it a try. By the way, I had the same issue when trying
>> to work with Hector and I just took all the jars that Hector tutorial brings
>> using Maven. Most are in the list below.
>>
>>
>>
>> Another question for that matter, what do you recommend working with
>> Hector or CQL?
>>
>>
>>
>> Tamar
>>
>>
>>
>>
>> On January 23, 2012 at 8:38 AM Jawahar Prasad < w3engine...@gmail.com >
>> wrote:
>>
>> Hello
>> I just thought I will tell you how I solved it:
>>
>> When I generated a jar from the jdbc code, it generated the following
>> jars:
>>
>> cassandra-clientutil.jar
>> cassandra-thrift.jar
>> commons-codec.jar
>> commons-lang.jar
>> commons-logging.jar
>> guava.jar
>> httpclient.jar
>> httpcore.jar
>> libthrift.jar
>> servlet-api.jar
>> cassandra-jdbc-1.0.5-SNAPSHOT.jar
>> slf4j-api-1.6.1.jar
>> slf4j-log4j12-1.6.1.jar
>>
>>
>> I included all the above in my current project and the problem got solved.
>> If you dont include sf4j, you might get logging errors. So just include all
>> of them.
>>
>> Regards
>> Prasad
>>
>>
>>Hi!
>>I have cassandra-clientutil, cassandra-jdbc and
>> cassandra-thrift in my libs, but
>>I get
>>
>>java.lang.NoClassDefFoundError: Could not initialize
>> class
>>org.apache.cassandra.cql.jdbc.CassandraDriver
>>when running
>>
>> Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver").newInstance();
>>
>>CassandraDriver is in my classpath.
>>
>>Any idea?
>>Tamar
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
--
Eric Evans
Acunu | http://www.acunu.com | @acunu
architectural understanding of write operation node flow
Hi guys, I got an architectural question about how a write operation flows through the nodes. As far as I understand now, a client sends its write operation to whatever node it was set to use and if that node does not contain the data for this key K, then this node forwards the operation to the first node given by the hash function. This first node having key K then contacts the replication nodes depending on the selected consistency level. This means that in the unlucky event you always have a network call sequence depth of 2 (consistency level one), or 3 (assumed that the replication nodes are contacted in parallel) This is more than I expected, so I am not sure whether this is correct? can someone help me out? At first I thought that the receiver was the coordinator, and thus doing all further calls in parallel, the depth as described above would always be 2. But I just discovered that I was wrong and that it should be something like above. Another possibility would be that the client learnt the layout of the cluster at connection time and thereby tries per request to contact the coordinator directly, but I never read or see something like this happening. Remembering the picture of Dean about network and hard disk latencies, is this 3-sequential-network-call still faster? Thanks for any thoughts :) Peter -- Peter Dijkshoorn Adyen - Payments Made Easy www.adyen.com Visiting Address: Mail Address: Stationsplein 57 - 4th floor P.O. Box 10095 1012 AB Amsterdam 1001 EB Amsterdam The Netherlands The Netherlands Office +31.20.240.1240 Email peter.dijksho...@adyen.com
Re: CQL jdbc
Think there's some confusion as Tamar has two emails in the same thread
addressing two separate concerns.
I was referring to the discussion over Composite Key support that Tamar
quoted in his second email (the one that I replied to and quoted), not his
first/original question about JDBC.
Unless I've misread the very lengthy discussions over Composite Key support
in CQL then it is targeted for a 1.1 release?
Alex.
On Mon, Jan 23, 2012 at 3:43 PM, Eric Evans wrote:
> On Mon, Jan 23, 2012 at 8:40 AM, Alex Major wrote:
> > Based on current discussions it looks like it will be in C* 1.1, but
> won't
> > be in the default cql package - you'll need to opt into cql3 driver as
> there
> > are some incompatible BC changes and they want to give an easier
> migration.
> > It will be in the default standard distribution in 1.2.
>
> No. To get the JDBC driver, you need to install it from its project page.
>
> http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/
>
> > On Mon, Jan 23, 2012 at 1:56 PM, Tamar Fraenkel
> wrote:
> >>
> >> If I understand correctly this is due in Cassandra 1.1. Does anyone
> know
> >> when it is planned to be released?
> >>
> >> Thanks
> >>
> >> Tamar
> >>
> >>
> >> On January 23, 2012 at 11:13 AM Jawahar Prasad
> >> wrote:
> >>
> >> Hi..
> >> Yes there is. But just 2 days back, they have released a patch:
> >>
> >> https://issues.apache.org/jira/browse/CASSANDRA-3761
> >>
> >> I am in the discussion, you can join as well
> >>
> >> Regards
> >> Prasad
> >>
> >> On Mon, Jan 23, 2012 at 2:37 PM, Tamar Fraenkel < ta...@tok-media.com >
> >> wrote:
> >>
> >> Hi!
> >>
> >> Is there something that is a real limitation of CQL currently. For
> example
> >> composite keys \ column names?
> >>
> >> Tamar
> >>
> >>
> >>
> >>
> >> On January 23, 2012 at 10:29 AM Jawahar Prasad < w3engine...@gmail.com>
> >> wrote:
> >>
> >> Hi..
> >> I am using CQL for the below reasons:
> >>
> >> 1. Hector is better than Thrift, but CQL is better than Hector in terms
> of
> >> understanding and quickly getting things done
> >> 2. Hector is third party, CQL is from cassandra developers, So there
> will
> >> be a good support.
> >> 3. As per Eric, CQL will be the future and will replace all third party
> >> clients.
> >>
> >> And as i got used to SQL, CQL makes more sense to me.
> >>
> >> Regards
> >> Prasad
> >>
> >> On Mon, Jan 23, 2012 at 1:46 PM, Tamar Fraenkel < ta...@tok-media.com >
> >> wrote:
> >>
> >> Thanks, will give it a try. By the way, I had the same issue when trying
> >> to work with Hector and I just took all the jars that Hector tutorial
> brings
> >> using Maven. Most are in the list below.
> >>
> >>
> >>
> >> Another question for that matter, what do you recommend working with
> >> Hector or CQL?
> >>
> >>
> >>
> >> Tamar
> >>
> >>
> >>
> >>
> >> On January 23, 2012 at 8:38 AM Jawahar Prasad < w3engine...@gmail.com >
> >> wrote:
> >>
> >> Hello
> >> I just thought I will tell you how I solved it:
> >>
> >> When I generated a jar from the jdbc code, it generated the following
> >> jars:
> >>
> >> cassandra-clientutil.jar
> >> cassandra-thrift.jar
> >> commons-codec.jar
> >> commons-lang.jar
> >> commons-logging.jar
> >> guava.jar
> >> httpclient.jar
> >> httpcore.jar
> >> libthrift.jar
> >> servlet-api.jar
> >> cassandra-jdbc-1.0.5-SNAPSHOT.jar
> >> slf4j-api-1.6.1.jar
> >> slf4j-log4j12-1.6.1.jar
> >>
> >>
> >> I included all the above in my current project and the problem got
> solved.
> >> If you dont include sf4j, you might get logging errors. So just include
> all
> >> of them.
> >>
> >> Regards
> >> Prasad
> >>
> >>
> >>Hi!
> >>I have cassandra-clientutil, cassandra-jdbc and
> >> cassandra-thrift in my libs, but
> >>I get
> >>
> >>java.lang.NoClassDefFoundError: Could not initialize
> >> class
> >>org.apache.cassandra.cql.jdbc.CassandraDriver
> >>when running
> >>
> >>
> Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver").newInstance();
> >>
> >>CassandraDriver is in my classpath.
> >>
> >>Any idea?
> >>Tamar
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
>
> --
> Eric Evans
> Acunu | http://www.acunu.com | @acunu
>
Re: get all columns for a row
Thanks.
Tamar
On January 23, 2012 at 11:24 AM aaron morton wrote:
> The columns are stored at the intersection of the row and the CF. So if you
> read all the columns for a row in a CF you are only getting those ones.
>
> Your hector code (using the range) looks correct to me.
>
> Have fun.
> Aaron
>
>
>
>
>
>
>
> -
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
>
> On 23/01/2012, at 9:24 PM, Tamar Fraenkel wrote:
>
> >
> >
> > Hi!
> >
> > I am getting started with Cassandra and Hector and have a question.
> > If I have CF with many columns, only a small portion of them having values
> > for any given key. I want to read all columns for a given key, but I don't
> > know which columns were populated.
> > I managed to do that with the code below (assumes that column names are
> > incremental numbers).
> > Is there any better way?
> > Thanks,
> > Tamar
> > private static void readAllColunsForRowKey(String key) {
> > MultigetSliceQuery multigetSlicesQuery =
> > HFactory.createMultigetSliceQuery(tutorialKeyspace,
> > StringSerializer.get(), LongSerializer.get(),
> > StringSerializer.get());
> > multigetSlicesQuery.setColumnFamily(CF);
> > multigetSlicesQuery.setKeys(key);
> > multigetSlicesQuery.setRange(null, null, false, COL_COUNT);
> > QueryResult> results =
> > multigetSlicesQuery.execute();
> > Long lastColName;
> > while ((lastColName = printColsAndGetLastColName(results, key)) != null) {
> > multigetSlicesQuery.setRange(lastColName + 1, null, false, COL_COUNT);
> > results = multigetSlicesQuery.execute();
> > }
> > }
> >
> > private static Long printColsAndGetLastColName(QueryResult > Long, String>> results, String key) {
> > Rows rows = results.get();
> > if (rows.getCount() != 1) {
> > System.out.println("No such key");
> > return null;
> > }
> > Row row = rows.getByKey(key);
> > ColumnSlice columnSlice = row.getColumnSlice();
> > List> columns = columnSlice.getColumns();
> > for (HColumn column : columns) {
> > System.out.println("Column Name: " + column.getName()
> > + ", Column Val: " + column.getValue());
> > }
> > if (columns.size() == COL_COUNT) {
> > return columns.get(COL_COUNT - 1).getName();
> > }
> > return null;
> > }
> >
Re: architectural understanding of write operation node flow
Your first thought was pretty much correct: 1. The node which is called by the client is the coordinator 2. The coordinator determines the nodes in the ring which can handle the request ordered by expected latency (via snitch). The coordinator may or may not be part of these nodes 3. Given the consistency level and read repair chance the coordinator calculates the min amount of node to ask and sends read requests to them 4. As soon as the minimum count (according to consistency) of responses is collected the coordinator will respond to the request. Mismatches will lead to repair write requests to the corresponding nodes Thus the minimal depth is one (CL = 1 and coordinator can handle the request itself) or two otherwise. Hope that helps On Jan 23, 2012, at 4:47 PM, Peter Dijkshoorn wrote: > > Hi guys, > > I got an architectural question about how a write operation flows > through the nodes. > > As far as I understand now, a client sends its write operation to > whatever node it was set to use and if that node does not contain the > data for this key K, then this node forwards the operation to the first > node given by the hash function. This first node having key K then > contacts the replication nodes depending on the selected consistency level. > > This means that in the unlucky event you always have a network call > sequence depth of 2 (consistency level one), or 3 (assumed that the > replication nodes are contacted in parallel) > > This is more than I expected, so I am not sure whether this is correct? > can someone help me out? > > At first I thought that the receiver was the coordinator, and thus doing > all further calls in parallel, the depth as described above would always > be 2. But I just discovered that I was wrong and that it should be > something like above. > > Another possibility would be that the client learnt the layout of the > cluster at connection time and thereby tries per request to contact the > coordinator directly, but I never read or see something like this happening. > > Remembering the picture of Dean about network and hard disk latencies, > is this 3-sequential-network-call still faster? > > Thanks for any thoughts :) > > Peter > > -- > Peter Dijkshoorn > Adyen - Payments Made Easy > www.adyen.com > > Visiting Address: Mail Address: > Stationsplein 57 - 4th floor P.O. Box 10095 > 1012 AB Amsterdam 1001 EB Amsterdam > The Netherlands The Netherlands > > Office +31.20.240.1240 > Email peter.dijksho...@adyen.com >
Re: architectural understanding of write operation node flow
Ouch :-) you were asking write ... Well kind of similar 1. Coordinator calculates all nodes 2. If not enough (according to CL) nodes are alive it throughs unavailable 3. If nodes are down it writes and hh is enabled it writes a hint for that row 4. It sends write request to all nodes (including itself / shortcutting messaging) 5. If it receives enough (according to CL) acks before timeout everything is fine otherwise it throughs unavailable errm .. I'm more confident in the read path though especially concerning hh handling so I'm happy to be corrected here. I.e. I'm not sure if hints are written when request time out but CL is reached. On Jan 23, 2012, at 6:47 PM, Daniel Doubleday wrote: > Your first thought was pretty much correct: > > 1. The node which is called by the client is the coordinator > 2. The coordinator determines the nodes in the ring which can handle the > request ordered by expected latency (via snitch). The coordinator may or may > not be part of these nodes > 3. Given the consistency level and read repair chance the coordinator > calculates the min amount of node to ask and sends read requests to them > 4. As soon as the minimum count (according to consistency) of responses is > collected the coordinator will respond to the request. Mismatches will lead > to repair write requests to the corresponding nodes > > Thus the minimal depth is one (CL = 1 and coordinator can handle the request > itself) or two otherwise. > > Hope that helps > > On Jan 23, 2012, at 4:47 PM, Peter Dijkshoorn wrote: > >> >> Hi guys, >> >> I got an architectural question about how a write operation flows >> through the nodes. >> >> As far as I understand now, a client sends its write operation to >> whatever node it was set to use and if that node does not contain the >> data for this key K, then this node forwards the operation to the first >> node given by the hash function. This first node having key K then >> contacts the replication nodes depending on the selected consistency level. >> >> This means that in the unlucky event you always have a network call >> sequence depth of 2 (consistency level one), or 3 (assumed that the >> replication nodes are contacted in parallel) >> >> This is more than I expected, so I am not sure whether this is correct? >> can someone help me out? >> >> At first I thought that the receiver was the coordinator, and thus doing >> all further calls in parallel, the depth as described above would always >> be 2. But I just discovered that I was wrong and that it should be >> something like above. >> >> Another possibility would be that the client learnt the layout of the >> cluster at connection time and thereby tries per request to contact the >> coordinator directly, but I never read or see something like this happening. >> >> Remembering the picture of Dean about network and hard disk latencies, >> is this 3-sequential-network-call still faster? >> >> Thanks for any thoughts :) >> >> Peter >> >> -- >> Peter Dijkshoorn >> Adyen - Payments Made Easy >> www.adyen.com >> >> Visiting Address: Mail Address: >> Stationsplein 57 - 4th floor P.O. Box 10095 >> 1012 AB Amsterdam 1001 EB Amsterdam >> The Netherlands The Netherlands >> >> Office +31.20.240.1240 >> Email peter.dijksho...@adyen.com >> >
Re: datastax opscenter authentication
Unfortunately the current method using a password file is the only option for authentication in OpsCenter at the moment. I've noted PAM authentication as a feature request though. On Mon, Jan 23, 2012 at 8:16 AM, Ramesh Natarajan wrote: > I am trying to integrate opscenter in our environment and I was > wondering if we can use PAM authentication instead of a password file > for opscenter authentication? > > thanks > Ramesh
Hive + Cassandra tutorial
Hi, I'm trying to experiment with Hive using Data in Cassandra. Brisk looks good, but I'm interested in running the map reduce jobs on HDFS not on CFS. I'm taking a look at [1], but couldn't figure out how to run it with a Cassandra cluster. I was wondering is there a simple word count example similar to the Pig case? Thanks in advance. [1] - https://github.com/riptano/hive -- Regards, Tharindu blog: http://mackiemathew.com/
Re: Hive + Cassandra tutorial
Take a look at http://wiki.apache.org/cassandra/HadoopSupport and in the source download of cassandra there's a contrib/pig section that has a wordcount example. On Jan 23, 2012, at 1:16 PM, Tharindu Mathew wrote: > Hi, > > I'm trying to experiment with Hive using Data in Cassandra. Brisk looks good, > but I'm interested in running the map reduce jobs on HDFS not on CFS. > > I'm taking a look at [1], but couldn't figure out how to run it with a > Cassandra cluster. I was wondering is there a simple word count example > similar to the Pig case? > > Thanks in advance. > > [1] - https://github.com/riptano/hive > > -- > Regards, > > Tharindu > > blog: http://mackiemathew.com/ >
Re: Hive + Cassandra tutorial
Hi Jeremy, Thanks for the reply. I was looking for a similar sample for Hive. I've already gone through the Pig sample. I probably wasn't clear about this in my initial mail. On Tue, Jan 24, 2012 at 12:55 AM, Jeremy Hanna wrote: > Take a look at http://wiki.apache.org/cassandra/HadoopSupport and in the > source download of cassandra there's a contrib/pig section that has a > wordcount example. > > On Jan 23, 2012, at 1:16 PM, Tharindu Mathew wrote: > > > Hi, > > > > I'm trying to experiment with Hive using Data in Cassandra. Brisk looks > good, but I'm interested in running the map reduce jobs on HDFS not on CFS. > > > > I'm taking a look at [1], but couldn't figure out how to run it with a > Cassandra cluster. I was wondering is there a simple word count example > similar to the Pig case? > > > > Thanks in advance. > > > > [1] - https://github.com/riptano/hive > > > > -- > > Regards, > > > > Tharindu > > > > blog: http://mackiemathew.com/ > > > > -- Regards, Tharindu blog: http://mackiemathew.com/
atomicity of a row write
hi all, having read: http://wiki.apache.org/cassandra/FAQ#batch_mutate_atomic i would like some clarification: is a write to a single row key in a single column family atomic in the sense that i can do a batch mutate where i 1) write col 'A' to key 'B' 2) write 'col 'C' to key 'B' and either both column writes will succeed, or both will fail? i won't get the situation where eg col 'A' is written and col 'B' fails, my client receives an error, but col 'A' is actually persisted and becomes visible to other clients? does this hold if i write key 'B' across two different column families? (i assume not, but the faq doesn't seem to explicitly exclude this). PS i'm not worried about isolation per se, i'm interested in what the 'eventually consistent' state is.
Return list by limit 1 which is NOT null
HI, I am trying to create a keys list which I will fetch the key and then delete the same key in the subsequent call. When I use the CLI list, It always returns first row due to tombstone. Is there a way I can specify to use the limit 1 and return NOT null value. Please let me now. Thanks and Regards. [default@test_keyspace] list test_kms_keys; Using default limit of 100 --- RowKey: 33 --- RowKey: 04 => (column=key_id, value=4, timestamp=132734935120) --- RowKey: 32 => (column=key_id, value=2, timestamp=1326231507383000) --- RowKey: 31 --- RowKey: 311313 5 Rows Returned. Elapsed time: 6 msec(s). [default@test_keyspace] list test_kms_keys limit 1; --- RowKey: 33 1 Row Returned. Elapsed time: 2 msec(s).
Re: CQL jdbc
On Mon, Jan 23, 2012 at 10:49 AM, Alex Major wrote:
> Think there's some confusion as Tamar has two emails in the same thread
> addressing two separate concerns.
>
> I was referring to the discussion over Composite Key support that Tamar
> quoted in his second email (the one that I replied to and quoted), not his
> first/original question about JDBC.
Sorry, I meant to reply to the first message in the thread.
> On Mon, Jan 23, 2012 at 3:43 PM, Eric Evans wrote:
>>
>> On Mon, Jan 23, 2012 at 8:40 AM, Alex Major wrote:
>> > Based on current discussions it looks like it will be in C* 1.1, but
>> > won't
>> > be in the default cql package - you'll need to opt into cql3 driver as
>> > there
>> > are some incompatible BC changes and they want to give an easier
>> > migration.
>> > It will be in the default standard distribution in 1.2.
>>
>> No. To get the JDBC driver, you need to install it from its project page.
>>
>> http://code.google.com/a/apache-extras.org/p/cassandra-jdbc/
>>
>> > On Mon, Jan 23, 2012 at 1:56 PM, Tamar Fraenkel
>> > wrote:
>> >>
>> >> If I understand correctly this is due in Cassandra 1.1. Does anyone
>> >> know
>> >> when it is planned to be released?
>> >>
>> >> Thanks
>> >>
>> >> Tamar
>> >>
>> >>
>> >> On January 23, 2012 at 11:13 AM Jawahar Prasad
>> >> wrote:
>> >>
>> >> Hi..
>> >> Yes there is. But just 2 days back, they have released a patch:
>> >>
>> >> https://issues.apache.org/jira/browse/CASSANDRA-3761
>> >>
>> >> I am in the discussion, you can join as well
>> >>
>> >> Regards
>> >> Prasad
>> >>
>> >> On Mon, Jan 23, 2012 at 2:37 PM, Tamar Fraenkel < ta...@tok-media.com >
>> >> wrote:
>> >>
>> >> Hi!
>> >>
>> >> Is there something that is a real limitation of CQL currently. For
>> >> example
>> >> composite keys \ column names?
>> >>
>> >> Tamar
>> >>
>> >>
>> >>
>> >>
>> >> On January 23, 2012 at 10:29 AM Jawahar Prasad < w3engine...@gmail.com
>> >> >
>> >> wrote:
>> >>
>> >> Hi..
>> >> I am using CQL for the below reasons:
>> >>
>> >> 1. Hector is better than Thrift, but CQL is better than Hector in terms
>> >> of
>> >> understanding and quickly getting things done
>> >> 2. Hector is third party, CQL is from cassandra developers, So there
>> >> will
>> >> be a good support.
>> >> 3. As per Eric, CQL will be the future and will replace all third party
>> >> clients.
>> >>
>> >> And as i got used to SQL, CQL makes more sense to me.
>> >>
>> >> Regards
>> >> Prasad
>> >>
>> >> On Mon, Jan 23, 2012 at 1:46 PM, Tamar Fraenkel < ta...@tok-media.com >
>> >> wrote:
>> >>
>> >> Thanks, will give it a try. By the way, I had the same issue when
>> >> trying
>> >> to work with Hector and I just took all the jars that Hector tutorial
>> >> brings
>> >> using Maven. Most are in the list below.
>> >>
>> >>
>> >>
>> >> Another question for that matter, what do you recommend working with
>> >> Hector or CQL?
>> >>
>> >>
>> >>
>> >> Tamar
>> >>
>> >>
>> >>
>> >>
>> >> On January 23, 2012 at 8:38 AM Jawahar Prasad < w3engine...@gmail.com >
>> >> wrote:
>> >>
>> >> Hello
>> >> I just thought I will tell you how I solved it:
>> >>
>> >> When I generated a jar from the jdbc code, it generated the following
>> >> jars:
>> >>
>> >> cassandra-clientutil.jar
>> >> cassandra-thrift.jar
>> >> commons-codec.jar
>> >> commons-lang.jar
>> >> commons-logging.jar
>> >> guava.jar
>> >> httpclient.jar
>> >> httpcore.jar
>> >> libthrift.jar
>> >> servlet-api.jar
>> >> cassandra-jdbc-1.0.5-SNAPSHOT.jar
>> >> slf4j-api-1.6.1.jar
>> >> slf4j-log4j12-1.6.1.jar
>> >>
>> >>
>> >> I included all the above in my current project and the problem got
>> >> solved.
>> >> If you dont include sf4j, you might get logging errors. So just include
>> >> all
>> >> of them.
>> >>
>> >> Regards
>> >> Prasad
>> >>
>> >>
>> >> Hi!
>> >> I have cassandra-clientutil, cassandra-jdbc and
>> >> cassandra-thrift in my libs, but
>> >> I get
>> >>
>> >> java.lang.NoClassDefFoundError: Could not initialize
>> >> class
>> >> org.apache.cassandra.cql.jdbc.CassandraDriver
>> >> when running
>> >>
>> >>
>> >> Class.forName("org.apache.cassandra.cql.jdbc.CassandraDriver").newInstance();
>> >>
>> >> CassandraDriver is in my classpath.
>> >>
>> >> Any idea?
>> >> Tamar
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>>
>>
>>
>> --
>> Eric Evans
>> Acunu | http://www.acunu.com | @acunu
>
>
--
Eric Evans
Acunu | http://www.acunu.com | @acunu
Re: Cassandra performance question
Can you elaborate on to what exactly you were testing on the Cassandra side? It sounds like what this post refers to as "node" encryption corresponds to enabling "internode_encryption: all", but I couldn't guess what your client encryption is since Cassandra doesn't support that out of the box yet. (Which is highly relevant since that's where most of the slowdown you observed comes from.) On Fri, Dec 30, 2011 at 2:11 PM, Chris Marino wrote: > We did some benchmarking as well. > > http://blog.vcider.com/2011/09/virtual-networks-can-run-cassandra-up-to-60-faster/ > > Although we were primarily interested in the networking issues > > CM > > > On Fri, Dec 30, 2011 at 12:08 PM, Jeremy Hanna > wrote: >> >> This might be helpful: >> http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html >> >> On Dec 30, 2011, at 1:59 PM, Dom Wong wrote: >> >> > Hi, could anyone tell me whether this is possible with Cassandra using >> > an appropriately sized EC2 cluster. >> > >> > 100,000 clients writing 50k each to their own specific row at 5 second >> > intervals? >> > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Cassandra performance question
Hi Jonathan, yes, when I say 'node encryption' I mean inter-Cassandra node encryption. When I say 'client encryption' I mean encrypted traffic from the Cassandra nodes to the clients. For these benchmarks we used the stress test client load generator. We ran test with no encryption, then with 'node only' and then again with both node and client encryption. Looking at the post again right now I realize that I never mention the client encryption technique. My mistake. We just used OpenVPN. We set it up on the client stress test machine and set up individual VPNs to each Cassandra node in the cluster. I tired to make it clear in my post where the performance gains were coming from. Didn't mean to confuse anyone. That said, if you're running in a public cloud and you want to encrypt, you've got to tackle the client traffic encryption problem as well so it seemed like performance gains that most could expect. CM On Mon, Jan 23, 2012 at 7:46 PM, Jonathan Ellis wrote: > Can you elaborate on to what exactly you were testing on the Cassandra > side? It sounds like what this post refers to as "node" encryption > corresponds to enabling "internode_encryption: all", but I couldn't > guess what your client encryption is since Cassandra doesn't support > that out of the box yet. (Which is highly relevant since that's where > most of the slowdown you observed comes from.) > > On Fri, Dec 30, 2011 at 2:11 PM, Chris Marino wrote: > > We did some benchmarking as well. > > > > > http://blog.vcider.com/2011/09/virtual-networks-can-run-cassandra-up-to-60-faster/ > > > > Although we were primarily interested in the networking issues > > > > CM > > > > > > On Fri, Dec 30, 2011 at 12:08 PM, Jeremy Hanna < > jeremy.hanna1...@gmail.com> > > wrote: > >> > >> This might be helpful: > >> > http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html > >> > >> On Dec 30, 2011, at 1:59 PM, Dom Wong wrote: > >> > >> > Hi, could anyone tell me whether this is possible with Cassandra using > >> > an appropriately sized EC2 cluster. > >> > > >> > 100,000 clients writing 50k each to their own specific row at 5 second > >> > intervals? > >> > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
Re: Hive + Cassandra tutorial
Any idea whether the hive functionality will be merged to the Cassandra source? On Tue, Jan 24, 2012 at 1:00 AM, Tharindu Mathew wrote: > Hi Jeremy, > > Thanks for the reply. > > I was looking for a similar sample for Hive. I've already gone through the > Pig sample. I probably wasn't clear about this in my initial mail. > > > On Tue, Jan 24, 2012 at 12:55 AM, Jeremy Hanna > wrote: > >> Take a look at http://wiki.apache.org/cassandra/HadoopSupport and in the >> source download of cassandra there's a contrib/pig section that has a >> wordcount example. >> >> On Jan 23, 2012, at 1:16 PM, Tharindu Mathew wrote: >> >> > Hi, >> > >> > I'm trying to experiment with Hive using Data in Cassandra. Brisk looks >> good, but I'm interested in running the map reduce jobs on HDFS not on CFS. >> > >> > I'm taking a look at [1], but couldn't figure out how to run it with a >> Cassandra cluster. I was wondering is there a simple word count example >> similar to the Pig case? >> > >> > Thanks in advance. >> > >> > [1] - https://github.com/riptano/hive >> > >> > -- >> > Regards, >> > >> > Tharindu >> > >> > blog: http://mackiemathew.com/ >> > >> >> > > > -- > Regards, > > Tharindu > > blog: http://mackiemathew.com/ > > -- Regards, Tharindu blog: http://mackiemathew.com/
