Ok, I reproduced without mapred. Here is my recipe:
On a single-node cassandra cluster with basic config (-Xmx:1G)
loop {
* insert 5,000 records in a single columnfamily with UUID keys and
random string values (between 1 and 1000 chars) in 5 different columns
spanning two different supercolumns
* delete all the data by iterating over the rows with
get_range_slices(ONE) and calling remove(QUORUM) on each row id
returned (path containing only columnfamily)
* count number of non-tombstone rows by iterating over the rows
with get_range_slices(ONE) and testing data. Break if not zero.
}
Here's the flakey part: while this is running, call "bin/nodetool -h
localhost -p 8081 flush KeySpace" in the background every minute or
so. When the data hits some critical size, the loop will break.
Anyone care to try this at home?
On Thu, Apr 29, 2010 at 12:51 AM, Jonathan Ellis <jbel...@gmail.com> wrote:
> Good! :)
>
> Can you reproduce w/o map/reduce, with raw get_range_slices?
>
> On Wed, Apr 28, 2010 at 3:56 PM, Joost Ouwerkerk <jo...@openplaces.org> wrote:
>> Yes! Reproduced on single-node cluster:
>>
>> 10/04/28 16:30:24 INFO mapred.JobClient: ROWS=274884
>> 10/04/28 16:30:24 INFO mapred.JobClient: TOMBSTONES=951083
>>
>> 10/04/28 16:42:49 INFO mapred.JobClient: ROWS=166580
>> 10/04/28 16:42:49 INFO mapred.JobClient: TOMBSTONES=1059387
>>
>> On Wed, Apr 28, 2010 at 10:43 AM, Jonathan Ellis <jbel...@gmail.com> wrote:
>>> It sounds like either there is a fairly obvious bug, or you're doing
>>> something wrong. :)
>>>
>>> Can you reproduce against a single node?
>>>
>>> On Tue, Apr 27, 2010 at 5:14 PM, Joost Ouwerkerk <jo...@openplaces.org>
>>> wrote:
>>>> Update: I ran a test whereby I deleted ALL the rows in a column
>>>> family, using a consistency level of ALL. To do this, I mapped the
>>>> ColumnFamily and called remove on each row id. There were 1.5 million
>>>> rows, so 1.5 million rows were deleted.
>>>>
>>>> I ran a counter job immediately after. This job maps the same column
>>>> family and tests if any data is returned. If not, it considers the
>>>> row a "tombstone". If yes, it considers the row not deleted. Below
>>>> are the hadoop counters for those jobs. Note the fluctuation in the
>>>> number of rows with data over time, and the increase in time to map
>>>> the column family after the destroy job. No other clients were
>>>> accessing cassandra during this time.
>>>>
>>>> I'm thoroughly confused.
>>>>
>>>> Count: started 13:02:30 EDT, finished 13:11:33 EDT (9 minutes 2 seconds):
>>>> ROWS: 1,542,479
>>>> TOMBSTONES: 69
>>>>
>>>> Destroy: started 16:48:45 EDT, finished 17:07:36 EDT (18 minutes 50
>>>> seconds)
>>>> DESTROYED: 1,542,548
>>>>
>>>> Count: started 17:15:42 EDT, finished 17:31:03 EDT (15 minutes 21 seconds)
>>>> ROWS 876,464
>>>> TOMBSTONES 666,084
>>>>
>>>> Count: started 17:31:32, finished 17:47:16 (15mins, 44 seconds)
>>>> ROWS 1,451,665
>>>> TOMBSTONES 90,883
>>>>
>>>> Count: started 17:52:34, finished 18:10:28 (17mins, 53 seconds)
>>>> ROWS 1,425,644
>>>> TOMBSTONES 116,904
>>>>
>>>> On Tue, Apr 27, 2010 at 5:37 PM, Joost Ouwerkerk <jo...@openplaces.org>
>>>> wrote:
>>>>> Clocks are in sync:
>>>>>
>>>>> cluster04:~/cassandra$ dsh -g development "date"
>>>>> Tue Apr 27 17:36:33 EDT 2010
>>>>> Tue Apr 27 17:36:33 EDT 2010
>>>>> Tue Apr 27 17:36:33 EDT 2010
>>>>> Tue Apr 27 17:36:33 EDT 2010
>>>>> Tue Apr 27 17:36:34 EDT 2010
>>>>> Tue Apr 27 17:36:34 EDT 2010
>>>>> Tue Apr 27 17:36:34 EDT 2010
>>>>> Tue Apr 27 17:36:34 EDT 2010
>>>>> Tue Apr 27 17:36:34 EDT 2010
>>>>> Tue Apr 27 17:36:35 EDT 2010
>>>>> Tue Apr 27 17:36:35 EDT 2010
>>>>> Tue Apr 27 17:36:35 EDT 2010
>>>>>
>>>>> On Tue, Apr 27, 2010 at 5:35 PM, Nathan McCall <n...@vervewireless.com>
>>>>> wrote:
>>>>>> Have you confirmed that your clocks are all synced in the cluster?
>>>>>> This may be the result of an unintentional read-repair occurring if
>>>>>> that were the case.
>>>>>>
>>>>>> -Nate
>>>>>>
>>>>>> On Tue, Apr 27, 2010 at 2:20 PM, Joost Ouwerkerk <jo...@openplaces.org>
>>>>>> wrote:
>>>>>>> Hmm... Even after deleting with cl.ALL, I'm getting data back for some
>>>>>>> rows after having deleted them. Which rows return data is
>>>>>>> inconsistent from one run of the job to the next.
>>>>>>>
>>>>>>> On Tue, Apr 27, 2010 at 1:44 PM, Joost Ouwerkerk <jo...@openplaces.org>
>>>>>>> wrote:
>>>>>>>> To check that rows are gone, I check that KeySlice.columns is empty.
>>>>>>>> And as
>>>>>>>> I mentioned, immediately after the delete job, this returns the
>>>>>>>> expected
>>>>>>>> number.
>>>>>>>> Unfortunately I reproduced with QUORUM this morning. No node outages.
>>>>>>>> I am
>>>>>>>> going to try ALL to see if that changes anything, but I am starting to
>>>>>>>> wonder if I'm doing something else wrong.
>>>>>>>> On Mon, Apr 26, 2010 at 9:45 PM, Jonathan Ellis <jbel...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> How are you checking that the rows are gone?
>>>>>>>>>
>>>>>>>>> Are you experiencing node outages during this?
>>>>>>>>>
>>>>>>>>> DC_QUORUM is unfinished code right now, you should avoid using it.
>>>>>>>>> Can you reproduce with normal QUORUM?
>>>>>>>>>
>>>>>>>>> On Sat, Apr 24, 2010 at 12:23 PM, Joost Ouwerkerk
>>>>>>>>> <jo...@openplaces.org>
>>>>>>>>> wrote:
>>>>>>>>> > I'm having trouble deleting rows in Cassandra. After running a job
>>>>>>>>> > that
>>>>>>>>> > deletes hundreds of rows, I run another job that verifies that the
>>>>>>>>> > rows
>>>>>>>>> > are
>>>>>>>>> > gone. Both jobs run correctly. However, when I run the
>>>>>>>>> > verification
>>>>>>>>> > job an
>>>>>>>>> > hour later, the rows have re-appeared. This is not a case of
>>>>>>>>> > "ghosting"
>>>>>>>>> > because the verification job actually checks that there is data in
>>>>>>>>> > the
>>>>>>>>> > columns.
>>>>>>>>> >
>>>>>>>>> > I am running a cluster with 12 nodes and a replication factor of
>>>>>>>>> > 3. I
>>>>>>>>> > am
>>>>>>>>> > using DC_QUORUM consistency when deleting.
>>>>>>>>> >
>>>>>>>>> > Any ideas?
>>>>>>>>> > Joost.
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jonathan Ellis
>>>>>>>>> Project Chair, Apache Cassandra
>>>>>>>>> co-founder of Riptano, the source for professional Cassandra support
>>>>>>>>> http://riptano.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
