Thanks Eric.

Happy new year 2015 for all Cassandra developers and Users :). This group
seems the most active of apache big data projects.

Will come back with more questions :)

Thanks
Ajay
On Dec 31, 2014 8:02 PM, "Eric Stevens" <migh...@gmail.com> wrote:

> You can totally avoid the impact of tombstones by rotating your partition
> key in the exact counts table, and only deleting whole partitions once
> you've counted them.  Once you've counted them you never have cause to read
> that partition key again.
>
> You can totally store the final counts in Cassandra as a standard
> (non-counter) column, and you can even use counters to keep track of the
> time slices which haven't been formally counted yet so that you can get
> reasonably accurate information about time slices that haven't been trued
> up yet.
>
> This is basically what's called a Lambda architecture - use efficient real
> time processing to get pretty close to accurate values when real time
> performance matters, then use a cleanup process to get perfectly accurate
> values when you can afford non-real-time processing times, and store that
> final computation so that you can continue to access it quickly.
>
> > is there any technical reason behind it (just out of curiosity)?
>
> Distributed counting is a fundamentally hard problem if you wish to do so
> in a manner that avoids bottlenecks (i.e. not distributed) and also
> provides for perfect accuracy.  There's plenty of research in this area,
> and there isn't a single algorithm that provides for all the properties we
> would hope for.  Instead there are different algorithms that make different
> tradeoffs.
>
> The way that Cassandra's counters can fail is that most operations in
> Cassandra are idempotent - if we're not sure whether an update has been
> applied correctly or not, we can simply apply it again, because it's safe
> to do twice.  Counters are not idempotent.  If you try to increment a
> counter, and you're not certain whether the increment was successful or
> not, it is *not* safe to try again (if it was successful the previous
> time, you've now incremented twice when it should have been once).
>
> Most of the time counters are reasonable and accurate, but in failure
> scenarios you may get some changes applied more than once, or not at all.
> With that in mind, you might find that being perfectly accurate most of the
> time, and being within a fraction of a percent the other times is
> acceptable.  If so, counters are your friend, and if not, a more complex
> lambda style approach as we've been advocating here is best.
>
> On Tue, Dec 30, 2014 at 10:54 PM, Ajay <ajay.ga...@gmail.com> wrote:
>
>> Thanks Janne and Rob.
>>
>> The idea is like this : To store the User clicks on Cassandra and a
>> scheduler to count/aggregate the  clicks per link or ad
>> hourly/daily/monthly and store in My SQL (or may be in Cassandra itself).
>> Since tombstones will be deleted only after some days (as per
>> configuration), could the subsequent queries to count the rows get affected
>> (I mean say thousands of tombstones will affect the performance of the
>> query) ?
>>
>> Secondly as I understand from this mail thread, the counter is not
>> correct for this use case, is there any technical reason behind it (just
>> out of curiosity)?
>>
>> Thanks
>> Ajay
>>
>> On Tue, Dec 30, 2014 at 10:37 PM, Janne Jalkanen <
>> janne.jalka...@ecyrd.com> wrote:
>>
>>>
>>> Hi!
>>>
>>> Yes, since all the writes for a partition (or row if you speak Thrift)
>>> always go to the same replicas, you will need to design to avoid hotspots -
>>> a pure day row will cause all the writes for a single day to go to the same
>>> replicas, so those nodes will have to work really hard for a day, and then
>>> the next day it’s again hard work for some other nodes.  If you have an
>>> user id there in front, then it would distribute better.
>>>
>>> For tombstone purposes think of your access patterns; if you have a
>>> date-based system, it probably does not matter since you will scan those
>>> UUIDs once, and then they will be tombstoned away.  It’s cleaner if you can
>>> delete the entire row with a single command, but as long as you never read
>>> it again, I don’t think this matters much.
>>>
>>> The real problems with wide rows come with compaction, and you shouldn’t
>>> have much problems with compaction because this is an append-only row, so
>>> it should be fine as a fairly wide row.  Make some back-of-the-envelope
>>> calculations and if it looks like you’re going to be hitting tens of
>>> millions of columns per day, then store per hour.
>>>
>>> One important thing: in order not to lose clicks, always use timeuuids
>>> instead of timestamps (or else two clicks coming in for the same id would
>>> overwrite itself and count as one).
>>>
>>> /Janne
>>>
>>> On 30 Dec 2014, at 06:28, Ajay <ajay.ga...@gmail.com> wrote:
>>>
>>> Thanks Janne, Alain and Eric.
>>>
>>> Now say I go with counters (hourly, daily, monthly) and also store UUID
>>> as below:
>>>
>>> user Id : yyyy/mm/dd as row key and dynamic columns for each click with
>>> column key as timestamp and value as empty. Periodically count the columns
>>> and rows and correct the counters. Now in this case, there will be one row
>>> per day but as many columns as user click.
>>>
>>> Other way is to store row per hour
>>> user id : yyyy/mm/dd/hh as row key and dynamic columns for each click
>>> with column key as timestamp and value as empty.
>>>
>>> Is there any difference (in performance or any known issues) between
>>> more rows Vs more columns as Cassandra deletes them through tombstones (say
>>> by default 20 days).
>>>
>>> Thanks
>>> Ajay
>>>
>>> On Mon, Dec 29, 2014 at 7:47 PM, Eric Stevens <migh...@gmail.com> wrote:
>>>
>>>> > If the counters get incorrect, it could't be corrected
>>>>
>>>> You'd have to store something that allowed you to correct it.  For
>>>> example, the TimeUUID approach to keep true counts, which are slow to read
>>>> but accurate, and a background process that trues up your counter columns
>>>> periodically.
>>>>
>>>> On Mon, Dec 29, 2014 at 7:05 AM, Ajay <ajay.ga...@gmail.com> wrote:
>>>>
>>>>> Thanks for the clarification.
>>>>>
>>>>> In my case, Cassandra is the only storage. If the counters get
>>>>> incorrect, it could't be corrected. For that if we store raw data, we can
>>>>> as well go that approach. But the granularity has to be as seconds level 
>>>>> as
>>>>> more than one user can click the same link. So the data will be huge with
>>>>> more writes and more rows to count for reads right?
>>>>>
>>>>> Thanks
>>>>> Ajay
>>>>>
>>>>>
>>>>> On Mon, Dec 29, 2014 at 7:10 PM, Alain RODRIGUEZ <arodr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Ajay,
>>>>>>
>>>>>> Here is a good explanation you might want to read.
>>>>>>
>>>>>>
>>>>>> http://www.datastax.com/dev/blog/whats-new-in-cassandra-2-1-a-better-implementation-of-counters
>>>>>>
>>>>>> Though we use counters for 3 years now, we used them from start C*
>>>>>> 0.8 and we are happy with them. Limits I can see in both ways are:
>>>>>>
>>>>>> Counters:
>>>>>>
>>>>>> - accuracy indeed (Tend to be small in our use case < 5% - when the
>>>>>> business allow 10%, so fair enough for us) + we recount them through a
>>>>>> batch processing tool (spark / hadoop - Kind of lambda architecture). So
>>>>>> our real-time stats are inaccurate and after a few minutes or hours we 
>>>>>> have
>>>>>> the real value.
>>>>>> - Read-Before-Write model, which is an anti-pattern. Makes you use
>>>>>> more machine due to the pressure involved, affordable for us too.
>>>>>>
>>>>>> Raw data (counted)
>>>>>>
>>>>>> - Space used (can become quite impressive very fast, depending on
>>>>>> your business) !
>>>>>> - Time to answer a request (we expose the data to customer, they
>>>>>> don't want to wait 10 sec for Cassandra to read 1 000 000 + columns)
>>>>>> - Performances in o(n) (linear) instead of o(1) (constant). Customer
>>>>>> won't always understand that for you it is harder to read 1 than 1 000 
>>>>>> 000,
>>>>>> since it should be reading 1 number in both case, and your interface will
>>>>>> have very unstable read time.
>>>>>>
>>>>>> Pick the best solution (or combination) for your use case. Those
>>>>>> disadvantages lists are not exhaustive, just things that came to my mind
>>>>>> right now.
>>>>>>
>>>>>> C*heers
>>>>>>
>>>>>> Alain
>>>>>>
>>>>>> 2014-12-29 13:33 GMT+01:00 Ajay <ajay.ga...@gmail.com>:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> So you mean to say counters are not accurate? (It is highly likely
>>>>>>> that multiple parallel threads trying to increment the counter as users
>>>>>>> click the links).
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ajay
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Dec 29, 2014 at 4:49 PM, Janne Jalkanen <
>>>>>>> janne.jalka...@ecyrd.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> It’s really a tradeoff between accurate and fast and your read
>>>>>>>> access patterns; if you need it to be fairly fast, use counters by all
>>>>>>>> means, but accept the fact that they will (especially in older 
>>>>>>>> versions of
>>>>>>>> cassandra or adverse network conditions) drift off from the true click
>>>>>>>> count.  If you need accurate, use a timeuuid and count the rows (this 
>>>>>>>> is
>>>>>>>> fairly safe for replays too).  However, if using timeuuids your storage
>>>>>>>> will need lots of space; and your reads will be slow if the click 
>>>>>>>> counts
>>>>>>>> are huge (because Cassandra will need to read every item).  Using 
>>>>>>>> counters
>>>>>>>> makes it easy to just grab a slice of the time series data and shove 
>>>>>>>> it to
>>>>>>>> a client for visualization.
>>>>>>>>
>>>>>>>> You could of course do a hybrid system; use timeuuids and then
>>>>>>>> periodically count and add the result to a regular column, and then 
>>>>>>>> remove
>>>>>>>> the columns.  Note that you might want to optimize this so that you 
>>>>>>>> don’t
>>>>>>>> end up with a lot of tombstones, e.g. by bucketing the writes so that 
>>>>>>>> you
>>>>>>>> can delete everything with just a single partition delete.
>>>>>>>>
>>>>>>>> At Thinglink some of the more important counters that we use are
>>>>>>>> backed up by the actual data. So for speed purposes we use always 
>>>>>>>> counters
>>>>>>>> for reads, but there’s a repair process that fixes the counter value 
>>>>>>>> if we
>>>>>>>> suspect it starts drifting off the real data too much.  (You might be 
>>>>>>>> able
>>>>>>>> to tell that we’ve been using counters for quite some time :-P)
>>>>>>>>
>>>>>>>> /Janne
>>>>>>>>
>>>>>>>> On 29 Dec 2014, at 13:00, Ajay <ajay.ga...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > Is it better to use Counter to User click count than maintaining
>>>>>>>> creating new row as user id : timestamp and count it.
>>>>>>>> >
>>>>>>>> > Basically we want to track the user clicks and use the same for
>>>>>>>> hourly/daily/monthly report.
>>>>>>>> >
>>>>>>>> > Thanks
>>>>>>>> > Ajay
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>

Reply via email to