Re: Help! Cassandra Data Loader threads are getting stuck

[Aaron Morton]

Try running it without threading to see if it's a cassandra problem or an issue with your threading.

Perhaps split the file and run many single threaded processes to load the data.

Aaron

On 27 Jul, 2010,at 07:14 AM, Rana Aich <aichr...@gmail.com> wrote:

Hi All,

I have to load huge quantity of data into Cassandra (~10Billion rows). 

I'm trying to load the Data from files using multithreading.

The idea is each thread will read the TAB delimited file and process chunk of records.

For example Thread1 reads line 1-1000 lines

Thread 2 reads line 1001-2000 and insert into Cassandra.

Thread 3 reads line 2001-3000 and insert into Cassandra.

Thread 10 reads line 9001-10000 and insert into Cassandra.

Thread 1  reads line 10001-11000 and insert into Cassandra.

Thread 2 reads line 11001-12000 and insert into Cassandra.

and so on...

I'm testing with a small file size with 200000 records.

But somehow the process gets stuck and doesn't proceed any further after processing say 16,000 records.

I've attached my working file.

Any help will be very much appreciated.

Regards

raich