Re: Manual Indexing With Buckets

[Anuj Wadehra]

Can anyone take this one?


Thanks

Anuj

Sent from Yahoo Mail on Android

From:"Anuj Wadehra" 
Date:Thu, 23 Jul, 2015 at 10:57 pm
Subject:Manual Indexing With Buckets

We have a primary table and we need search capability by batchid column. So we 
are creating a manual index for search by batch id. We are using buckets to 
restrict a row size in batch id index table to 50mb. As batch size may vary 
drastically ( ie one batch id may be associated to 100k row keys in primary 
table while other may be associated with 100million row keys), we are creating 
a metadata table to track the approximate data while insertions for a batch in 
primary table, so that batch id index table has dynamic no of buckets/rows. As 
more data is inserted for a batch in primary table, new set of 10 buckets are 
added. At any point in time, clients will write to latest 10 buckets created 
for a batch od index in round robin  to avoid hotspots.


Comments required on the following:

1. I want to know any suggestios on above design?


2. Whats the best approach for updating/deleting from index table. When a row 
is manually purged from primary table, we dont know where that row key exists 
in x number of buckets created for its batch id? 


Thanks

Anuj

Sent from Yahoo Mail on Android

RE: Manual Indexing With Buckets

[SEAN_R_DURITY]

It is a bit hard to follow. Perhaps you could include your proposed schema 
(annotated with your size predictions) to spur more discussion. To me, it 
sounds a bit convoluted. Why is a "batch" so big (up to 100 million rows)? Is a 
row in the primary only associated with one batch?


Sean Durity - Cassandra Admin, Big Data Team
To engage the team, create a 
request

From: Anuj Wadehra [mailto:anujw_2...@yahoo.co.in]
Sent: Friday, July 24, 2015 3:57 AM
To: user@cassandra.apache.org
Subject: Re: Manual Indexing With Buckets

Can anyone take this one?

Thanks
Anuj

Sent from Yahoo Mail on 
Android


From:"Anuj Wadehra" mailto:anujw_2...@yahoo.co.in>>
Date:Thu, 23 Jul, 2015 at 10:57 pm
Subject:Manual Indexing With Buckets
We have a primary table and we need search capability by batchid column. So we 
are creating a manual index for search by batch id. We are using buckets to 
restrict a row size in batch id index table to 50mb. As batch size may vary 
drastically ( ie one batch id may be associated to 100k row keys in primary 
table while other may be associated with 100million row keys), we are creating 
a metadata table to track the approximate data while insertions for a batch in 
primary table, so that batch id index table has dynamic no of buckets/rows. As 
more data is inserted for a batch in primary table, new set of 10 buckets are 
added. At any point in time, clients will write to latest 10 buckets created 
for a batch od index in round robin  to avoid hotspots.

Comments required on the following:
1. I want to know any suggestios on above design?

2. Whats the best approach for updating/deleting from index table. When a row 
is manually purged from primary table, we dont know where that row key exists 
in x number of buckets created for its batch id?

Thanks
Anuj

Sent from Yahoo Mail on 
Android







The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Compaction no longer working properly

[Robert Wille]

I have a database which has a fair amount of churn. When I need to update a 
data structure, I create a new one, and when it is complete, I delete the old 
one. I have gc_grace_seconds=0, so the space for the old data structures should 
be reclaimed on the next compaction. This has been working fine for months. 
Unfortunately, compaction has stopped doing its job, and about a week ago disk 
space started increasing steadily. Disk space usage has increased almost 50% in 
the last week.

I do see compaction tasks get kicked off, but they complete very quickly, 
generally in well less than a minute. Compaction of the bigger tables used to 
take more than an hour. Admittedly, its been quite a while since I’ve watched 
compactions run, and I probably haven’t watched them since we last upgraded, so 
perhaps there’s been a change in how LeveledCompaction works. Our last upgrade 
was well more than a week ago, so an upgrade isn’t responsible for this change 
in behavior. We’re running 2.0.14 currently.

Looking at the logs, I see stuff like this:

INFO [CompactionExecutor:49180] 2015-07-24 09:31:45,960 CompactionTask.java 
(line 296) Compacted 2 sstables to 
[/var/lib/cassandra/data/fold31_browse/node/fold31_browse-node-jb-0,].  
117,066,921 bytes to 117,346,402 (~100% of original) in 33,766ms = 
3.314288MB/s.  1,444,907 total partitions merged to 1,400,563.  Partition merge 
counts were {1:1356219, 2:44344, }

So, compaction seems to be doing something, just not very much. I’ve checked 
the process that deletes stale data, and it is doing its job. Plenty of 
deletion going on, just no space reclamation.

Any thoughts?

Thanks in advance

Robert

Re: Schema questions for data structures with recently-modified access patterns

[Robert Wille]

When performing an update, the following needs to happen:

1. Read document.last_modified
2. Get the current timestamp
3. Update document with last_modified=current timestamp
4. Insert into doc_by_last_modified with last_modified=current timestamp
5. Delete from doc_by_last_modified with last_modified=the timestamp from step 1

If two parties do the above at roughly the same time, such that in step 1 they 
both read the same last_modified timestamp, then when they do step 5, they’ll 
both delete the same old record from doc_by_last_modified, and you’ll get two 
records for the same document in doc_by_last_modified.

Would it work to put steps 3-5 into an atomic batch and use a lightweight 
transaction for step 3? (e.g. UPDATE document SET doc = :doc, last_modified = 
:cur_ts WHERE docid = :docid IF last_modified = :prev_ts) If a lightweight 
transaction is batched with other statements on other tables, will the other 
statements get cancelled if the lightweight transaction is cancelled?

Robert

On Jul 23, 2015, at 9:49 PM, Jack Krupansky 
mailto:jack.krupan...@gmail.com>> wrote:

Concurrent update should not be problematic. Duplicate entries should not be 
created. If it appears to be, explain your apparent issue so we can see whether 
it is a real issue.

But at least from all of the details you have disclosed so far, there does not 
appear to be any indication that this type of time series would be anything 
other than a good fit for Cassandra.

Besides, the new materialized view feature of Cassandra 3.0 would make it an 
even easier fit.

-- Jack Krupansky

On Thu, Jul 23, 2015 at 6:30 PM, Robert Wille 
mailto:rwi...@fold3.com>> wrote:
I obviously worded my original email poorly. I guess that’s what happens when 
you post at the end of the day just before quitting.

I want to get a list of documents, ordered from most-recently modified to 
least-recently modified, with each document appearing exactly once.

Jack, your schema does exactly that, and is essentially the same as mine (with 
exception of my missing the DESC clause, and I have a partitioning column and 
you only have clustering columns).

The problem I have with my schema (or Jack’s) is that it is very easy for a 
document to get in the list multiple times. Concurrent updates to the document, 
for example. Also, a consistency issue could cause the document to appear in 
the list more than once.

I think that Alec Collier’s comment is probably accurate, that this kind of a 
pattern just isn’t a good fit for Cassandra.

On Jul 23, 2015, at 1:54 PM, Jack Krupansky 
mailto:jack.krupan...@gmail.com>> wrote:

Maybe you could explain in more detail what you mean by recently modified 
documents, since that is precisely what I thought I suggested with descending 
ordering.

-- Jack Krupansky

On Thu, Jul 23, 2015 at 3:40 PM, Robert Wille 
mailto:rwi...@fold3.com>> wrote:
Carlos’ suggestion (nor yours) didn’t didn’t provide a way to query 
recently-modified documents.

His updated suggestion provides a way to get recently-modified documents, but 
not ordered.

On Jul 22, 2015, at 4:19 PM, Jack Krupansky 
mailto:jack.krupan...@gmail.com>> wrote:

"No way to query recently-modified documents."

I don't follow why you say that. I mean, that was the point of the data model 
suggestion I proposed. Maybe you could clarify.

I also wanted to mention that the new materialized view feature of Cassandra 
3.0 might handle this use case, including taking care of the delete, 
automatically.


-- Jack Krupansky

On Tue, Jul 21, 2015 at 12:37 PM, Robert Wille 
mailto:rwi...@fold3.com>> wrote:
The time series doesn’t provide the access pattern I’m looking for. No way to 
query recently-modified documents.

On Jul 21, 2015, at 9:13 AM, Carlos Alonso 
mailto:i...@mrcalonso.com>> wrote:

Hi Robert,

What about modelling it as a time serie?

CREATE TABLE document (
  docId UUID,
  doc TEXT,
  last_modified TIMESTAMP
  PRIMARY KEY(docId, last_modified)
) WITH CLUSTERING ORDER BY (last_modified DESC);

This way, you the lastest modification will always be the first record in the 
row, therefore accessing it should be as easy as:

SELECT * FROM document WHERE docId ==  LIMIT 1;

And, if you experience diskspace issues due to very long rows, then you can 
always expire old ones using TTL or on a batch job. Tombstones will never be a 
problem in this case as, due to the specified clustering order, the latest 
modification will always be first record in the row.

Hope it helps.

Carlos Alonso | Software Engineer | @calonso

On 21 July 2015 at 05:59, Robert Wille 
mailto:rwi...@fold3.com>> wrote:
Data structures that have a recently-modified access pattern seem to be a poor 
fit for Cassandra. I’m wondering if any of you smart guys can provide 
suggestions.

For the sake of discussion, lets assume I have the following tables:

CREATE TABLE document (
docId UUID,
doc TEXT,
last_modified TIMEUUID,
PRIMARY KEY ((docid))
)

Re: Reduced write performance when reading

[Soerian Lieve]

I was on CFQ so I changed it to noop. The problem still persisted however.
Do you have any other ideas?

On Thu, Jul 23, 2015 at 5:00 PM, Jeff Ferland  wrote:

> Imbalanced disk use is ok in itself. It’s only saturated throughput that’s
> harmful. RAID 0 does give more consistent throughput and balancing, but
> that’s another story.
>
> As for your situation with SSD drive, you can probably tweak this by
> changing the scheduler is set to noop, or read up on
> https://www.kernel.org/doc/Documentation/block/deadline-iosched.txt for
> the deadline scheduler (lower writes_starved value). If you’re one CFQ,
> definitely ditch it.
>
> -Jeff
>
> On Jul 23, 2015, at 4:17 PM, Soerian Lieve  wrote:
>
> I set up RAID0 after experiencing highly imbalanced disk usage with a JBOD
> setup so my transaction logs are indeed on the same media as the sstables.
> Is there any alternative to setting up RAID0 that doesn't have this issue?
>
> On Thu, Jul 23, 2015 at 4:03 PM, Jeff Ferland  wrote:
>
>> My immediate guess: your transaction logs are on the same media as your
>> sstables and your OS prioritizes read requests.
>>
>> -Jeff
>>
>> > On Jul 23, 2015, at 2:51 PM, Soerian Lieve  wrote:
>> >
>> > Hi,
>> >
>> > I am currently performing benchmarks on Cassandra. Independently from
>> each other I am seeing ~100k writes/sec and ~50k reads/sec. When I read and
>> write at the same time, writing drops down to ~1000 writes/sec and reading
>> stays roughly the same.
>> >
>> > The heap used is the same as when only reading, as is the disk
>> utilization. Replication factor is 3, consistency level on both reads and
>> writes is ONE. Using Cassandra 2.1.6. All cassandra.yaml settings set up
>> according to the Datastax guide. All nodes are running on SSDs.
>> >
>> > Any ideas what could cause this?
>> >
>> > Thanks,
>> > Soerian
>>
>>
>
>

Nodetool cleanup takes long time and no progress

[rock zhang]

Hi All,

After I added node, I run node tool cleanup on the old notes , but it takes 
forever, no error message,  and I don't see space are freed. 

What should I do ? Repair first ?

Thanks
Rock 

Re: Nodetool cleanup takes long time and no progress

[Jeff Jirsa]

You can check for progress using `nodetool compactionstats` (which will show 
Cleanup tasks), or check for ‘Cleaned up’ messages in the log 
(/var/log/cassandra/system.log).

However, `nodetool cleanup` has a very specific and limited task - it deletes 
data no longer owned by the node, typically after adding nodes or moving 
tokens. Have you done that recently?  Or are you trying to clean up space after 
issuing a lot of deletes?





On 7/24/15, 3:55 PM, "rock zhang"  wrote:

>Hi All,
>
>After I added node, I run node tool cleanup on the old notes , but it takes 
>forever, no error message,  and I don't see space are freed. 
>
>What should I do ? Repair first ?
>
>Thanks
>Rock 

smime.p7s
Description: S/MIME cryptographic signature

Re: Nodetool cleanup takes long time and no progress

[rock zhang]

Thank you Jeff. I just added one more node, so i want to delete moved tokens.

ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1413
   compaction type   keyspace  table completed  totalunit   
progress
   Cleanuprawdata   raw_data   25817918778   500823247199   bytes   
   5.16%
   Cleanuprawdata   raw_data  27060645   45183439   bytes   
  59.89%
Active compaction remaining time :   0h00m00s
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1410
   compaction type   keyspace  table completed  totalunit   
progress
   Cleanuprawdata   raw_data   25911965212   500823247199   bytes   
   5.17%
   Cleanuprawdata   raw_data  11783629   43660120   bytes   
  26.99%
Active compaction remaining time :   0h00m00s
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1406
   compaction type   keyspace  table completed  totalunit   
progress
   Cleanuprawdata   raw_data   5767168   45595353   bytes   
  12.65%
   Cleanuprawdata   raw_data   26091690659   500823247199   bytes   
   5.21%
Active compaction remaining time :   0h00m00s
ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
pending tasks: 1406
   compaction type   keyspace  table completed  totalunit   
progress
   Cleanuprawdata   raw_data  35078225   45595353   bytes   
  76.93%
   Cleanuprawdata   raw_data   26119307264   500823247199   bytes   
   5.22%
Active compaction remaining time :   0h00m00s




On Jul 24, 2015, at 4:04 PM, Jeff Jirsa  wrote:

> You can check for progress using `nodetool compactionstats` (which will show 
> Cleanup tasks), or check for ‘Cleaned up’ messages in the log 
> (/var/log/cassandra/system.log).
> 
> However, `nodetool cleanup` has a very specific and limited task - it deletes 
> data no longer owned by the node, typically after adding nodes or moving 
> tokens. Have you done that recently?  Or are you trying to clean up space 
> after issuing a lot of deletes?
> 
> 
> 
> 
> 
> On 7/24/15, 3:55 PM, "rock zhang"  wrote:
> 
>> Hi All,
>> 
>> After I added node, I run node tool cleanup on the old notes , but it takes 
>> forever, no error message,  and I don't see space are freed. 
>> 
>> What should I do ? Repair first ?
>> 
>> Thanks
>> Rock

Re: Nodetool cleanup takes long time and no progress

[rock zhang]

Hi Jeff,

It already 2 hours, only progress is 6%, seems it is very slow.  Is there any 
way to speedup ?
 
If I interrupted the process, what gonna happen ?  Next time it just compact 
again, right ? I think by default setting is  the compaction occurs every day . 

Thanks
Rock 

On Jul 24, 2015, at 4:58 PM, rock zhang  wrote:

> Thank you Jeff. I just added one more node, so i want to delete moved tokens.
> 
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1413
>compaction type   keyspace  table completed  totalunit 
>   progress
>Cleanuprawdata   raw_data   25817918778   500823247199   bytes 
>  5.16%
>Cleanuprawdata   raw_data  27060645   45183439   bytes 
> 59.89%
> Active compaction remaining time :   0h00m00s
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1410
>compaction type   keyspace  table completed  totalunit 
>   progress
>Cleanuprawdata   raw_data   25911965212   500823247199   bytes 
>  5.17%
>Cleanuprawdata   raw_data  11783629   43660120   bytes 
> 26.99%
> Active compaction remaining time :   0h00m00s
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1406
>compaction type   keyspace  table completed  totalunit 
>   progress
>Cleanuprawdata   raw_data   5767168   45595353   bytes 
> 12.65%
>Cleanuprawdata   raw_data   26091690659   500823247199   bytes 
>  5.21%
> Active compaction remaining time :   0h00m00s
> ubuntu@ip-172-31-30-145:~$ nodetool compactionstats
> pending tasks: 1406
>compaction type   keyspace  table completed  totalunit 
>   progress
>Cleanuprawdata   raw_data  35078225   45595353   bytes 
> 76.93%
>Cleanuprawdata   raw_data   26119307264   500823247199   bytes 
>  5.22%
> Active compaction remaining time :   0h00m00s
> 
> 
> 
> 
> On Jul 24, 2015, at 4:04 PM, Jeff Jirsa  wrote:
> 
>> You can check for progress using `nodetool compactionstats` (which will show 
>> Cleanup tasks), or check for ‘Cleaned up’ messages in the log 
>> (/var/log/cassandra/system.log).
>> 
>> However, `nodetool cleanup` has a very specific and limited task - it 
>> deletes data no longer owned by the node, typically after adding nodes or 
>> moving tokens. Have you done that recently?  Or are you trying to clean up 
>> space after issuing a lot of deletes?
>> 
>> 
>> 
>> 
>> 
>> On 7/24/15, 3:55 PM, "rock zhang"  wrote:
>> 
>>> Hi All,
>>> 
>>> After I added node, I run node tool cleanup on the old notes , but it takes 
>>> forever, no error message,  and I don't see space are freed. 
>>> 
>>> What should I do ? Repair first ?
>>> 
>>> Thanks
>>> Rock
> 

Re: Nodetool cleanup takes long time and no progress

[Robert Coli]

On Fri, Jul 24, 2015 at 5:03 PM, rock zhang  wrote:

> It already 2 hours, only progress is 6%, seems it is very slow.  Is there
> any way to speedup ?
>

Cleanup is a type of compaction; it obeys the compaction throttle.


> If I interrupted the process, what gonna happen ?  Next time it just
> compact again, right ?
>

Cleanup is never automatically triggered. Also, I doubt that it will be
trivial for you to interrupt it.


> I think by default setting is  the compaction occurs every day .
>

This doesn't parse for me. Minor compaction is continuous if the node is
receiving write traffic.

=Rob