Re: is this correct, thrift unportable to CQL3Š.

[Sylvain Lebresne]

On Tue, Sep 24, 2013 at 7:24 PM, Hiller, Dean  wrote:

> Java has Integer as opposed to int which is what I represent "golf scores"
> with so to speak in my example.  In this case, Integer can be null but of
> course maps to "empty" just fine.  What about querying for all golf scores
> that are "empty" then?
>

The DataStax java driver already "maps" a null Integer to a null CQL3
column, and so it *cannot* also map it to an empty value. And even though
you cannot have null CQL3 columns in the primary key, mapping a null
Integer to an empty value would be highly confusing/inconsistent (from the
point of view of the java driver) because:
1) having a null Integer map differently depending on whether the receiving
column is in the PK or not would be messy/surprising.
2) it's not really technically possible because internally the driver does
not always have the information of whether the receiving column is in the
PK or not
3) what about strings? Should we also map a null String to an empty value
when the column is in the PK? Because that would mean mapping a null java
String to an empty CQL3 string value and that just feel wrong. And if we
don't do it, then it means we treat Integer and String differently with
respect to null and that's too would be pretty confusing.

But anyway, all of that is pretty specific to the java driver, so maybe if
you guys want to debate that further we should move to the driver mailing
list:
https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user

--
Sylvain


>
> Ie. This sounds like this solution would be perfect if we can get rid of
> the exception piece.
>
> Thanks much!!!
> Dean
>
> From: Sylvain Lebresne mailto:sylv...@datastax.com>>
> Reply-To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Date: Tuesday, September 24, 2013 10:58 AM
> To: "user@cassandra.apache.org" <
> user@cassandra.apache.org>
> Subject: Re: is this correct, thrift unportable to CQL3Š.
>
> On Tue, Sep 24, 2013 at 6:17 PM, Vikas Goyal  vi...@easility.com>> wrote:
> Ok. Great. It works for String and Decimal/Float but not for integer data
> type..
> i.e,, if I am passing "" to the composite key column which is either text
> or float, it works..
>
> session.execute(boundStatement.bind(rowkey, "", ByteBuffer.wrap(value)));
>
> But not working with bigint, int or varint..and getting following
> exception;
>
> Exception:Invalid type for value 1 of CQL type varint, expecting class
> java.math.BigInteger but class java.lang.String provided
>
> ..I am exploring more though..
>
> We're getting into details of the java driver at this point. For int, what
> you can do is:
>   boundStatement.setString("id", rowkey);
>   boundStatement.setBytesUnsafe("columnname", ByteBuffer.wrap(new
> byte[0]));
>   boundStatement.setBytes("columnvalue", ByteBuffer.wrap(value));
>
> In other words, the shorthand BoundStatement.bind() won't let you insert
> an empty value for types
> that don't naturally have an empty value (like int; keep in mind that an
> empty value is not the same
> than null). But you can circumvent that using setBytesUnsafe if you really
> want.
>
> Long story short, my advice would be to avoid using empty values for type
> that don't naturally have
> one (anything else than text and blob really). If you do, that's going to
> be painful (but possible) to
> work with, at least with the java driver (but for good reasons, java just
> doesn't have a good to
> represent an empty int value (again, you can have a null Integer but
> that's different)).
>
> --
> Sylvain
>
>
> Thanks a ton,
> Vikas Goyal
>
>
> On Tue, Sep 24, 2013 at 9:05 PM, Sylvain Lebresne  > wrote:
>
> However,we tried missing the value but it didn't work :(
>
> Right, because not providing a value is akin to having a null value (in
> the CQL3 sense of the term, which is different from what Dean asked about)
> and null values are not allowed for primary key columns.
> You could however insert an *empty* value if you wanted, which in you case
> is just inserting an empty string since colname is a string. Thrift doesn't
> allow more or less in that case.
>
> --
> Sylvain
>
>
>
> So our code is like below where we are using 3 values if colname is not
> null..else 2 values..
>
> if (key != null) {
> PreparedStatement statement = session.prepare("INSERT INTO
> keys.StringIndice (id, colname, colvalue) VALUES (?, ?, ?)");
> BoundStatement boundStatement = new
> BoundStatement(statement);
>
> session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
> rowKey), key, ByteBuffer.wrap(value)));
> } else {
> PreparedStatement statement = session.prepare("INSERT INTO
> " + keys + "." + table + "(id, colvalue) VALUES (?, ?)");
> BoundStatement boundStatem

Mystery PIG issue with 1.2.10

[Janne Jalkanen]

Heya!

I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle 
integer values.

Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1.  Single node for 
testing this. 

First a table:

> CREATE TABLE testc (
  key text PRIMARY KEY,
  ivalue int,
  svalue text,
  value bigint
) WITH COMPACT STORAGE;

> insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
> select * from testc;

 key | ivalue | svalue | value
-+++---
 foo | 10 |bar | 65

For my Pig setup, I then use libraries from different C* versions to actually 
talk to my database (which stays on 1.2.10 all the time).

Cassandra 1.0.12 (using cassandra_storage.jar):

> testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})

Cassandra 1.1.10:

> testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> dump testc
(foo,(svalue,bar),(ivalue,10),(value,65),{})

Cassandra 1.2.10:

> (testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> dump testc
foo,{(ivalue,
),(svalue,bar),(value,A)})


To me it appears that ints and bigints are interpreted as ascii values in cass 
1.2.10.  Did something change for CassandraStorage, is there a regression, or 
am I doing something wrong?  Quick perusal of the JIRA didn't reveal anything 
that I could directly pin on this.

Note that using compact storage does not seem to affect the issue, though it 
obviously changes the resulting pig format.

In addition, trying to use Pygmalion 

> tf = foreach testc generate key, 
> flatten(FromCassandraBag('ivalue,svalue,value',columns)) as 
> (ivalue:int,svalue:chararray,lvalue:long);
> dump tf

(foo,
,bar,A)

So no help there. Explicitly casting the values to (long) or (int) just results 
in a ClassCastException.

/Janne

Re: Migration LCS from 1.2.X to 2.0.x exception

[Marcus Eriksson]

this is the issue:
https://issues.apache.org/jira/browse/CASSANDRA-5383

guess it fell between chairs, will poke around


On Tue, Sep 24, 2013 at 4:26 PM, Nate McCall  wrote:

> What version of 1.2.x?
>
> Unfortunately, you must go through 1.2.9 first. See
> https://github.com/apache/cassandra/blob/cassandra-2.0.0/NEWS.txt#L19-L24
>
>
> On Tue, Sep 24, 2013 at 8:57 AM, Desimpel, Ignace <
> ignace.desim...@nuance.com> wrote:
>
>>  Tested on WINDOWS : On startup of the 2.0.0 version from 1.2.x files I
>> get an error as listed below.
>>
>> ** **
>>
>> This is due to the code in LeveledManifest:: mutateLevel. The method
>> already has a comment saying that it is scary …
>>
>> On windows, one cannot use the File::rename if the target file name
>> already exists. 
>>
>> Also, even on Linux, I’m not sure if a rename would actually
>> ‘overwrite/implicit-delete’ the content of the target file.
>>
>> ** **
>>
>> Anyway, adding code (below) before the FileUtils.renameWithConfirm should
>> work in both cases (maybe even rename the fromFile to be able to recover…)
>> 
>>
>> File oTo = new File(filename);
>>
>> if ( oTo.exists() ) oTo.delete();
>>
>> ** **
>>
>> ** **
>>
>> java.lang.RuntimeException: Failed to rename
>> …..xxx\Function-ic-10-Statistics.db-tmp to
>> …..xxx\Function-ic-10-Statistics.db
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:136)
>> ~[main/:na]
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:125)
>> ~[main/:na]
>>
>> at
>> org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:601)
>> ~[main/:na]
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:103)
>> ~[main/:na]
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
>> ~[main/:na]
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
>> ~[main/:na]
>>
>> ** **
>>
>> Regards,
>>
>> ** **
>>
>> Ignace Desimpel
>>
>
>

Re: 1.2.10 -> 2.0.1 migration issue

[Marcus Eriksson]

this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.


On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt wrote:

> Hi,
>
> ** **
>
> Just had a go at upgrading a node to the latest stable c* 2 release and
> think I ran into some issues with manifest migration.
>
> ** **
>
> On initial start up I hit this error as it starts to load the first of my
> CF. 
>
> ** **
>
> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2
>
> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
> Exception encountered during startup
>
> FSWriteError in
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> 
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> 
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> 
>
> Caused by: java.nio.file.NoSuchFileException:
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ->
> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
> 
>
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
> 
>
> at java.nio.file.Files.createLink(Files.java:1037)
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)**
> **
>
> ... 5 more
>
> ** **
>
> I had already successful run a test migration on our dev server. Only real
> difference I can see if the number of data directories defined and the
> amount of data being held. 
>
> ** **
>
> I’ve run upgradesstables under 1.2.10. I have always been using vnodes and
> CQL3. I recently moved to using LZ4 instead of Snappy..
>
> ** **
>
> I tried to startup again and it gave me a slightly different error
>
> ** **
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
> Exception encountered during startup
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
> 
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
> 
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> 
>
> ** **
>
> Will have a go recreating this tomorrow.
>
> ** **
>
> Any insight or guesses at what the issue might be are always welcome.
>
> ** **
>
> Thanks,
>
> Chris
>

Re: is this correct, thrift unportable to CQL3Š.

[Alex Popescu]

On Wed, Sep 25, 2013 at 12:22 AM, Sylvain Lebresne wrote:

> On Tue, Sep 24, 2013 at 7:24 PM, Hiller, Dean wrote:
>
>> Java has Integer as opposed to int which is what I represent "golf
>> scores" with so to speak in my example.  In this case, Integer can be null
>> but of course maps to "empty" just fine.  What about querying for all golf
>> scores that are "empty" then?
>>
>
> The DataStax java driver already "maps" a null Integer to a null CQL3
> column, and so it *cannot* also map it to an empty value. And even though
> you cannot have null CQL3 columns in the primary key, mapping a null
> Integer to an empty value would be highly confusing/inconsistent (from the
> point of view of the java driver) because:
> 1) having a null Integer map differently depending on whether the
> receiving column is in the PK or not would be messy/surprising.
> 2) it's not really technically possible because internally the driver does
> not always have the information of whether the receiving column is in the
> PK or not
> 3) what about strings? Should we also map a null String to an empty value
> when the column is in the PK? Because that would mean mapping a null java
> String to an empty CQL3 string value and that just feel wrong. And if we
> don't do it, then it means we treat Integer and String differently with
> respect to null and that's too would be pretty confusing.
>
> But anyway, all of that is pretty specific to the java driver, so maybe if
> you guys want to debate that further we should move to the driver mailing
> list:
> https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user
>
>
Nulls should definitely stay null (that's an old DB debate), but  if this
is a generic case, we might consider adding an emptyValue() helper method.
Would that be possible Sylvain?


> --
> Sylvain
>
>
>>
>> Ie. This sounds like this solution would be perfect if we can get rid of
>> the exception piece.
>>
>> Thanks much!!!
>> Dean
>>
>> From: Sylvain Lebresne mailto:sylv...@datastax.com
>> >>
>> Reply-To: "user@cassandra.apache.org" <
>> user@cassandra.apache.org>
>> Date: Tuesday, September 24, 2013 10:58 AM
>> To: "user@cassandra.apache.org" <
>> user@cassandra.apache.org>
>> Subject: Re: is this correct, thrift unportable to CQL3Š.
>>
>> On Tue, Sep 24, 2013 at 6:17 PM, Vikas Goyal > vi...@easility.com>> wrote:
>> Ok. Great. It works for String and Decimal/Float but not for integer data
>> type..
>> i.e,, if I am passing "" to the composite key column which is either text
>> or float, it works..
>>
>> session.execute(boundStatement.bind(rowkey, "", ByteBuffer.wrap(value)));
>>
>> But not working with bigint, int or varint..and getting following
>> exception;
>>
>> Exception:Invalid type for value 1 of CQL type varint, expecting class
>> java.math.BigInteger but class java.lang.String provided
>>
>> ..I am exploring more though..
>>
>> We're getting into details of the java driver at this point. For int,
>> what you can do is:
>>   boundStatement.setString("id", rowkey);
>>   boundStatement.setBytesUnsafe("columnname", ByteBuffer.wrap(new
>> byte[0]));
>>   boundStatement.setBytes("columnvalue", ByteBuffer.wrap(value));
>>
>> In other words, the shorthand BoundStatement.bind() won't let you insert
>> an empty value for types
>> that don't naturally have an empty value (like int; keep in mind that an
>> empty value is not the same
>> than null). But you can circumvent that using setBytesUnsafe if you
>> really want.
>>
>> Long story short, my advice would be to avoid using empty values for type
>> that don't naturally have
>> one (anything else than text and blob really). If you do, that's going to
>> be painful (but possible) to
>> work with, at least with the java driver (but for good reasons, java just
>> doesn't have a good to
>> represent an empty int value (again, you can have a null Integer but
>> that's different)).
>>
>> --
>> Sylvain
>>
>>
>> Thanks a ton,
>> Vikas Goyal
>>
>>
>> On Tue, Sep 24, 2013 at 9:05 PM, Sylvain Lebresne > > wrote:
>>
>> However,we tried missing the value but it didn't work :(
>>
>> Right, because not providing a value is akin to having a null value (in
>> the CQL3 sense of the term, which is different from what Dean asked about)
>> and null values are not allowed for primary key columns.
>> You could however insert an *empty* value if you wanted, which in you
>> case is just inserting an empty string since colname is a string. Thrift
>> doesn't allow more or less in that case.
>>
>> --
>> Sylvain
>>
>>
>>
>> So our code is like below where we are using 3 values if colname is not
>> null..else 2 values..
>>
>> if (key != null) {
>> PreparedStatement statement = session.prepare("INSERT
>> INTO keys.StringIndice (id, colname, colvalue) VALUES (?, ?, ?)");
>> BoundStateme

Re: 1.2.10 -> 2.0.1 migration issue

[Marcus Eriksson]

cant really reproduce, could you update the ticket with a bit more info
about your setup?

do you have multiple .json files in your data dirs?


On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson  wrote:

> this is most likely a bug, filed
> https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have
> a look today.
>
>
> On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt wrote:
>
>> Hi,
>>
>> ** **
>>
>> Just had a go at upgrading a node to the latest stable c* 2 release and
>> think I ran into some issues with manifest migration.
>>
>> ** **
>>
>> On initial start up I hit this error as it starts to load the first of my
>> CF. 
>>
>> ** **
>>
>> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
>> Migrating manifest for struqrealtime/impressionstorev2
>>
>> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
>> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
>> 
>>
>> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
>> Exception encountered during startup
>>
>> FSWriteError in
>> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
>> 
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)*
>> ***
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
>> 
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
>> 
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
>> 
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>> 
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
>> 
>>
>> Caused by: java.nio.file.NoSuchFileException:
>> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
>> ->
>> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
>> 
>>
>> at
>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)***
>> *
>>
>> at
>> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>>
>> at
>> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
>> 
>>
>> at java.nio.file.Files.createLink(Files.java:1037)
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)*
>> ***
>>
>> ... 5 more
>>
>> ** **
>>
>> I had already successful run a test migration on our dev server. Only
>> real difference I can see if the number of data directories defined and the
>> amount of data being held. 
>>
>> ** **
>>
>> I’ve run upgradesstables under 1.2.10. I have always been using vnodes
>> and CQL3. I recently moved to using LZ4 instead of Snappy..
>>
>> ** **
>>
>> I tried to startup again and it gave me a slightly different error
>>
>> ** **
>>
>> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
>> Migrating manifest for struqrealtime/impressionstorev2
>>
>> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
>> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration
>> 
>>
>> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
>> Exception encountered during startup
>>
>> java.lang.RuntimeException: Tried to create duplicate hard link to
>> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
>> 
>>
>> at
>> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)*
>> ***
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
>> 
>>
>> at
>> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
>> 
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
>> 
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
>> 
>>
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
>> 
>>
>> ** **
>>
>> Will have a go recreating this tomorrow.
>>
>> ** **
>>
>> Any insight or guesses at what the issue might be are always welcome.
>>
>> ** **
>>
>> Thanks,
>>
>> Chris
>>
>
>

Re: is this correct, thrift unportable to CQL3Š.

[Sylvain Lebresne]

> Nulls should definitely stay null (that's an old DB debate), but  if this
> is a generic case, we might consider adding an emptyValue() helper method.
> Would that be possible Sylvain?
>

As said previously, debating this is outside the scope of this mailing
list, but to make it short, using empty values for types that don't
naturally support them like Integer is going to be painful (inserting is
one thing, handling them on the read side is another) and confusing no
matter what. So you should avoid it in the first place (and there is *tons*
of way to do that) and consequently adding an helper to do something you
should avoid is a bad idea.

--
Sylvain



>
>
>> --
>> Sylvain
>>
>>
>>>
>>> Ie. This sounds like this solution would be perfect if we can get rid of
>>> the exception piece.
>>>
>>> Thanks much!!!
>>> Dean
>>>
>>> From: Sylvain Lebresne mailto:sylv...@datastax.com
>>> >>
>>> Reply-To: "user@cassandra.apache.org"
>>> mailto:user@cassandra.apache.org>>
>>> Date: Tuesday, September 24, 2013 10:58 AM
>>> To: "user@cassandra.apache.org" <
>>> user@cassandra.apache.org>
>>> Subject: Re: is this correct, thrift unportable to CQL3Š.
>>>
>>> On Tue, Sep 24, 2013 at 6:17 PM, Vikas Goyal >> vi...@easility.com>> wrote:
>>> Ok. Great. It works for String and Decimal/Float but not for integer
>>> data type..
>>> i.e,, if I am passing "" to the composite key column which is either
>>> text or float, it works..
>>>
>>> session.execute(boundStatement.bind(rowkey, "", ByteBuffer.wrap(value)));
>>>
>>> But not working with bigint, int or varint..and getting following
>>> exception;
>>>
>>> Exception:Invalid type for value 1 of CQL type varint, expecting class
>>> java.math.BigInteger but class java.lang.String provided
>>>
>>> ..I am exploring more though..
>>>
>>> We're getting into details of the java driver at this point. For int,
>>> what you can do is:
>>>   boundStatement.setString("id", rowkey);
>>>   boundStatement.setBytesUnsafe("columnname", ByteBuffer.wrap(new
>>> byte[0]));
>>>   boundStatement.setBytes("columnvalue", ByteBuffer.wrap(value));
>>>
>>> In other words, the shorthand BoundStatement.bind() won't let you insert
>>> an empty value for types
>>> that don't naturally have an empty value (like int; keep in mind that an
>>> empty value is not the same
>>> than null). But you can circumvent that using setBytesUnsafe if you
>>> really want.
>>>
>>> Long story short, my advice would be to avoid using empty values for
>>> type that don't naturally have
>>> one (anything else than text and blob really). If you do, that's going
>>> to be painful (but possible) to
>>> work with, at least with the java driver (but for good reasons, java
>>> just doesn't have a good to
>>> represent an empty int value (again, you can have a null Integer but
>>> that's different)).
>>>
>>> --
>>> Sylvain
>>>
>>>
>>> Thanks a ton,
>>> Vikas Goyal
>>>
>>>
>>> On Tue, Sep 24, 2013 at 9:05 PM, Sylvain Lebresne >> > wrote:
>>>
>>> However,we tried missing the value but it didn't work :(
>>>
>>> Right, because not providing a value is akin to having a null value (in
>>> the CQL3 sense of the term, which is different from what Dean asked about)
>>> and null values are not allowed for primary key columns.
>>> You could however insert an *empty* value if you wanted, which in you
>>> case is just inserting an empty string since colname is a string. Thrift
>>> doesn't allow more or less in that case.
>>>
>>> --
>>> Sylvain
>>>
>>>
>>>
>>> So our code is like below where we are using 3 values if colname is not
>>> null..else 2 values..
>>>
>>> if (key != null) {
>>> PreparedStatement statement = session.prepare("INSERT
>>> INTO keys.StringIndice (id, colname, colvalue) VALUES (?, ?, ?)");
>>> BoundStatement boundStatement = new
>>> BoundStatement(statement);
>>>
>>> session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
>>> rowKey), key, ByteBuffer.wrap(value)));
>>> } else {
>>> PreparedStatement statement = session.prepare("INSERT
>>> INTO " + keys + "." + table + "(id, colvalue) VALUES (?, ?)");
>>> BoundStatement boundStatement = new
>>> BoundStatement(statement);
>>>
>>> session.execute(boundStatement.bind(StandardConverters.convertFromBytes(String.class,
>>> rowKey), ByteBuffer.wrap(value)));
>>>   }
>>>
>>> And, I did that and getting this exception:
>>>
>>> Exception:Missing PRIMARY KEY part colname since colvalue is set
>>>
>>> And just FYI. Our Column Family definition is below:
>>>
>>> CREATE TABLE keys.StringIndice (id text,
>>> colname text,
>>> colvalue blob,
>>> PRIMARY KEY (id,colname, colvalue)) WITH COMPACT STORAGE)
>>>
>>> Thanks again,
>>> Vikas Goyal
>>>
>>>
>>>
>>> On Tue, Sep 24, 2013 at 7:02 PM, Sylvain Lebresne >> > wrote:
>>> 

Re: Bad Request: Invalid null value for clustering key part

[Petter von Dolwitz (Hem)]

I am generating the CSV file by other means than the COPY ... TO command
and I thought that the NULL option did not apply to the COPY ... FROM
command but, as it turns out, it does! This solved my problem.

The documentation here (
http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/copy_r.html)
is misleading. Should I report this by other means than this post?

Thanks,
Petter





2013/9/24 Sylvain Lebresne 

> Oh. That would be a COPY thing then. I'm not extremely familiar with cqlsh
> code (which COPY is part of) but looking at the online help for it, it
> seems to have a 'NULL' option that allows to define which character is used
> to represent nulls. And by default, it does is an empty string. So you
> could try something like:
>   COPY ... TO ... WITH NULL='null'
> (assuming that if you do have nulls you use the string 'null' to encode it
> in your csv and that your are sure nothing that's not supposed to be null
> will be represented by the string 'null').
>
> --
> Sylvain
>
>
> On Tue, Sep 24, 2013 at 9:41 AM, Petter von Dolwitz (Hem) <
> petter.von.dolw...@gmail.com> wrote:
>
>> Hi Sylvain,
>>
>> I was not describing the problem correctly. I'm sorry for this. This is
>> the situation:
>>
>> 1. I'm populating the DB with the java-driver (INSERT INTO...). Some
>> fields that are part of the primary key is *empty strings*. This works fine.
>> 2. I'm trying to populate the DB from a CSV (COPY ... FROM) using cqlsh.
>> Some fields that are part of the primary key is *empty strings*. This
>> scenario gives me the "Bad Request: Invalid null value for clustering key
>> part {field_name}" message. Seems like empty strings are treated as NULL
>> when using the COPY .. FROM command?
>>
>> This can obviously be me not knowing how to encode an empty string in a
>> CSV file. A simplified row from the CSV file can look like below:
>>
>> "field1_value","","","field4_value","field5_value"
>>
>> whereas field1 through field4 is part of the primary key.
>>
>> Thanks for your time,
>> Petter
>>
>>
>>
>>
>> 2013/9/23 Sylvain Lebresne 
>>
>>>
>>> Is it not permitted to have null values in a field that is part a
 primary key?

>>>
>>> It's not.
>>>
>>>
 This seems to be ok when using the java-driver

>>>
>>> Are you sure? Because that would qualify as a bug (in the sense that
>>> it's not supported by C* so there is not reason why this would work with
>>> any driver). If you have some java driver code that show it possible, I'd
>>> be happy to have a look.
>>>
>>> --
>>> Sylvain
>>>
>>
>>
>

Extracting throughput-heavy Cassandra operations to different process

[André Cruz]

Hello.

I have been looking into the memory patterns of Cassandra (1.1.5), and I have a 
suggestion that I haven't seen discussed here.

Cassandra is configured by default with a GC tailored for pauseless operation 
instead of throughput, and this makes sense since it needs to answer client 
queries with low latency. Under "normal" operation I see no Full GCs. But when 
repairs/compactions are run the memory pattern changes and I see frequent Full 
GCs, which take too long.

Wouldn't it make sense to do repairs/compacts in a separate JVM process with a 
GC configured differently? In this case pauses wouldn't pose such a problem. 
I'm finding it difficult to configure the GC to behave correctly with and 
without repairs going on.

Best regards,
André


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Bad Request: Invalid null value for clustering key part

[Petter von Dolwitz (Hem)]

In fact, the blog post on the subject (
http://www.datastax.com/dev/blog/simple-data-importing-and-exporting-with-cassandra)
also states that the NULL option only is used by the COPY ... TO command
but it also states that this might change in the future and the future
seems to be here now.

Thanks,
Petter


2013/9/25 Petter von Dolwitz (Hem) 

> I am generating the CSV file by other means than the COPY ... TO command
> and I thought that the NULL option did not apply to the COPY ... FROM
> command but, as it turns out, it does! This solved my problem.
>
> The documentation here (
> http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/copy_r.html)
> is misleading. Should I report this by other means than this post?
>
> Thanks,
> Petter
>
>
>
>
>
> 2013/9/24 Sylvain Lebresne 
>
>> Oh. That would be a COPY thing then. I'm not extremely familiar with
>> cqlsh code (which COPY is part of) but looking at the online help for it,
>> it seems to have a 'NULL' option that allows to define which character is
>> used to represent nulls. And by default, it does is an empty string. So you
>> could try something like:
>>   COPY ... TO ... WITH NULL='null'
>> (assuming that if you do have nulls you use the string 'null' to encode
>> it in your csv and that your are sure nothing that's not supposed to be
>> null will be represented by the string 'null').
>>
>> --
>> Sylvain
>>
>>
>> On Tue, Sep 24, 2013 at 9:41 AM, Petter von Dolwitz (Hem) <
>> petter.von.dolw...@gmail.com> wrote:
>>
>>> Hi Sylvain,
>>>
>>> I was not describing the problem correctly. I'm sorry for this. This is
>>> the situation:
>>>
>>> 1. I'm populating the DB with the java-driver (INSERT INTO...). Some
>>> fields that are part of the primary key is *empty strings*. This works fine.
>>> 2. I'm trying to populate the DB from a CSV (COPY ... FROM) using cqlsh.
>>> Some fields that are part of the primary key is *empty strings*. This
>>> scenario gives me the "Bad Request: Invalid null value for clustering key
>>> part {field_name}" message. Seems like empty strings are treated as NULL
>>> when using the COPY .. FROM command?
>>>
>>> This can obviously be me not knowing how to encode an empty string in a
>>> CSV file. A simplified row from the CSV file can look like below:
>>>
>>> "field1_value","","","field4_value","field5_value"
>>>
>>> whereas field1 through field4 is part of the primary key.
>>>
>>> Thanks for your time,
>>> Petter
>>>
>>>
>>>
>>>
>>> 2013/9/23 Sylvain Lebresne 
>>>

 Is it not permitted to have null values in a field that is part a
> primary key?
>

 It's not.


> This seems to be ok when using the java-driver
>

 Are you sure? Because that would qualify as a bug (in the sense that
 it's not supported by C* so there is not reason why this would work with
 any driver). If you have some java driver code that show it possible, I'd
 be happy to have a look.

 --
 Sylvain

>>>
>>>
>>
>

RE: 1.2.10 -> 2.0.1 migration issue

[Christopher Wirt]

Hi Marcus,

Thanks for having a look at this.

 

Just noticed this in the NEWS.txt 

 

For leveled compaction users, 2.0 must be atleast started before

 upgrading to 2.1 due to the fact that the old JSON leveled

 manifest is migrated into the sstable metadata files on startup

 in 2.0 and this code is gone from 2.1.

 

Basically, my fault for skimming over this too quickly. 

 

We will move from 1.2.10 -> 2.0 -> 2.1

 

Thanks,

Chris

 

 

From: Marcus Eriksson [mailto:krum...@gmail.com] 
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

cant really reproduce, could you update the ticket with a bit more info
about your setup?

 

do you have multiple .json files in your data dirs?

 

On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson  wrote:

this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.

 

On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt 
wrote:

Hi,

 

Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.

 

On initial start up I hit this error as it starts to load the first of my
CF. 

 

INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup

FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)

Caused by: java.nio.file.NoSuchFileException:
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json ->
/disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json

at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)

at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)

at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474
)

at java.nio.file.Files.createLink(Files.java:1037)

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)

... 5 more

 

I had already successful run a test migration on our dev server. Only real
difference I can see if the number of data directories defined and the
amount of data being held. 

 

I've run upgradesstables under 1.2.10. I have always been using vnodes and
CQL3. I recently moved to using LZ4 instead of Snappy..

 

I tried to startup again and it gave me a slightly different error

 

INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
Exception encountered during startup

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:129)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
42)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)

 

Will have a go recreating this tomorrow.

 

Any insight or guesses at what the issue might be are always welcome.

 

Thanks,

Chris

 

 

FW: Migration LCS from 1.2.X to 2.0.x exception

[Desimpel, Ignace]

Same is true if started from 1.2.9 version to 2.0.x
Seems like Cassandra-5383 is the related issue.

The code in the function mutateLevel  (from 1.2.5 to 1.2.9 and later ) is 
indirectly using the java file::renameTo function. And that will not work on 
Windows if the file already exists. Don't know about Linux, but even the java 
function description is already telling that the behavior of the function can 
be machine dependent. So even on linux based systems, and even if that 
functions returns with success, still does not confirm that actually the old 
file name/content was removed and the new file name/content is used as 
replacement.


From: Marcus Eriksson [mailto:krum...@gmail.com]
Sent: woensdag 25 september 2013 10:02
To: user@cassandra.apache.org
Subject: Re: Migration LCS from 1.2.X to 2.0.x exception

this is the issue:
https://issues.apache.org/jira/browse/CASSANDRA-5383

guess it fell between chairs, will poke around

On Tue, Sep 24, 2013 at 4:26 PM, Nate McCall 
mailto:n...@thelastpickle.com>> wrote:
What version of 1.2.x?

Unfortunately, you must go through 1.2.9 first. See 
https://github.com/apache/cassandra/blob/cassandra-2.0.0/NEWS.txt#L19-L24

On Tue, Sep 24, 2013 at 8:57 AM, Desimpel, Ignace 
mailto:ignace.desim...@nuance.com>> wrote:
Tested on WINDOWS : On startup of the 2.0.0 version from 1.2.x files I get an 
error as listed below.

This is due to the code in LeveledManifest:: mutateLevel. The method already 
has a comment saying that it is scary ...
On windows, one cannot use the File::rename if the target file name already 
exists.
Also, even on Linux, I'm not sure if a rename would actually 
'overwrite/implicit-delete' the content of the target file.

Anyway, adding code (below) before the FileUtils.renameWithConfirm should work 
in both cases (maybe even rename the fromFile to be able to recover...)
File oTo = new File(filename);
if ( oTo.exists() ) oTo.delete();


java.lang.RuntimeException: Failed to rename 
.xxx\Function-ic-10-Statistics.db-tmp to 
.xxx\Function-ic-10-Statistics.db
at 
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:136) 
~[main/:na]
at 
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:125) 
~[main/:na]
at 
org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:601)
 ~[main/:na]
at 
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:103)
 ~[main/:na]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) 
~[main/:na]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) 
~[main/:na]

Regards,

Ignace Desimpel

Re: 1.2.10 -> 2.0.1 migration issue

[Marcus Eriksson]

you are probably reading trunk NEWS.txt

read the ticket for explanation of what the issue was (it is a proper bug)


On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt wrote:

> Hi Marcus,
>
> Thanks for having a look at this.
>
> ** **
>
> Just noticed this in the NEWS.txt 
>
> ** **
>
> For *leveled* compaction users, 2.0 must be *atleast* started before
>
>  upgrading to 2.1 due to the fact that the old JSON *leveled*
>
>  manifest is migrated into the *sstable* *metadata* files on startup**
> **
>
>  in 2.0 and this code is gone from 2.1.
>
> ** **
>
> Basically, my fault for skimming over this too quickly. 
>
> ** **
>
> We will move from 1.2.10 -> 2.0 -> 2.1
>
> ** **
>
> Thanks,
>
> Chris
>
> ** **
>
> ** **
>
> *From:* Marcus Eriksson [mailto:krum...@gmail.com]
> *Sent:* 25 September 2013 09:37
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue
>
> ** **
>
> cant really reproduce, could you update the ticket with a bit more info
> about your setup?
>
> ** **
>
> do you have multiple .json files in your data dirs?
>
> ** **
>
> On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson 
> wrote:
>
> this is most likely a bug, filed
> https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have
> a look today.
>
> ** **
>
> On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt 
> wrote:
>
> Hi,
>
>  
>
> Just had a go at upgrading a node to the latest stable c* 2 release and
> think I ran into some issues with manifest migration.
>
>  
>
> On initial start up I hit this error as it starts to load the first of my
> CF. 
>
>  
>
> INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2
>
> INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
> Exception encountered during startup
>
> FSWriteError in
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> 
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> 
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:246)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485)
> 
>
> Caused by: java.nio.file.NoSuchFileException:
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> ->
> /disk1/cassandra/data/struqrealtime/impressionstorev2/impressionstorev2.json
> 
>
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>
> at
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>
> at
> sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:474)
> 
>
> at java.nio.file.Files.createLink(Files.java:1037)
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:79)**
> **
>
> ... 5 more
>
>  
>
> I had already successful run a test migration on our dev server. Only real
> difference I can see if the number of data directories defined and the
> amount of data being held. 
>
>  
>
> I’ve run upgradesstables under 1.2.10. I have always been using vnodes and
> CQL3. I recently moved to using LZ4 instead of Snappy..
>
>  
>
> I tried to startup again and it gave me a slightly different error
>
>  
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 89)
> Migrating manifest for struqrealtime/impressionstorev2
>
> INFO [main] 2013-09-24 22:58:28,218 LegacyLeveledManifest.java (line 119)
> Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration*
> ***
>
> ERROR [main] 2013-09-24 22:58:28,222 CassandraDaemon.java (line 459)
> Exception encountered during startup
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk3/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/struqrealtime-impressionstorev2-ic-1030-TOC.txt
> 
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129)
> 
>
> at
> org.apache.cassandra.db.compaction.Lega

Re: Query about class org.apache.cassandra.io.sstable.SSTableSimpleWriter

[Jayadev Jayaraman]

Can someone answer this doubt reg. SSTableSimpleWriter ? I'd asked about
this earlier but it probably missed. Apologies for repeating the question
(with minor additions)  :

"""
Let's say I've initialized a *SSTableSimpleWriter* instance and a new
column with TTL set :

*org.apache.cassandra.io.sstable.SSTableSimpleWriter writer = new
SSTableSimpleWriter( ... /* params here */);*
*org.apache.cassandra.thrift.Column column; // initialize this with name,
value, timestamp, TTL*

What is the difference between calling *writer.addColumn()* on the column's
name, value and timestamp, and *writer.addExpiringColumn()* on the column's
name, value, TTL, timestamp and expiration timestamp ? Does the former
result in the column expiring still , in cassandra 1.2.x (i.e. does setting
the TTL on a Column object change the name or value in a way so as to
ensure the column will expire as required) ? If not , what is the TTL
attribute used for in the Column object ?
"""

Thanks,
Jayadev


On Tue, Sep 24, 2013 at 2:48 PM, Jayadev Jayaraman wrote:

> Let's say I've initialized a *SSTableSimpleWriter* instance and a new
> column with TTL set :
>
> *SSTableSimpleWriter writer = new SSTableSimpleWriter( ... /* params here
> */);*
> *Column column;*
>
> What is the difference between calling *writer.addColumn()* on the
> column's name and value, and *writer.addExpiringColumn()* on the column
> and its TTL ? Does the former result in the column expiring still , in
> cassandra 1.2.x ? Or does it not ?
>
>
>

RE: 1.2.10 -> 2.0.1 migration issue

[Christopher Wirt]

Hi Marcus,

 

I've seen your patch. This works with what I'm seeing. The first data
directory only contained the JSON manifest at that time.

 

As a workaround I've made sure that each of the snapshot directories now
exist before starting up.

 

I still end up with the second exception I posted regarding a duplicate hard
link. Possibly two unrelated exceptions.

 

After getting this error. Looking at the datadirs

Data1 contains 

JSON manifests

Loads of data files

Snapshot directory

Data2 contains

Just the snapshot directory

Data3 contains

Just the snapshot directory

 

INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2

INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
pre-sstablemetamigration

ERROR 12:56:22,787 Exception encountered during startup

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)

Exception encountered during startup: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

 

Thanks,

 

Chris

 

 

From: Marcus Eriksson [mailto:krum...@gmail.com] 
Sent: 25 September 2013 13:11
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

you are probably reading trunk NEWS.txt

 

read the ticket for explanation of what the issue was (it is a proper bug)

 

On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt 
wrote:

Hi Marcus,

Thanks for having a look at this.

 

Just noticed this in the NEWS.txt 

 

For leveled compaction users, 2.0 must be atleast started before

 upgrading to 2.1 due to the fact that the old JSON leveled

 manifest is migrated into the sstable metadata files on startup

 in 2.0 and this code is gone from 2.1.

 

Basically, my fault for skimming over this too quickly. 

 

We will move from 1.2.10 -> 2.0 -> 2.1

 

Thanks,

Chris

 

 

From: Marcus Eriksson [mailto:krum...@gmail.com] 
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

cant really reproduce, could you update the ticket with a bit more info
about your setup?

 

do you have multiple .json files in your data dirs?

 

On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson  wrote:

this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.

 

On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt 
wrote:

Hi,

 

Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.

 

On initial start up I hit this error as it starts to load the first of my
CF. 

 

INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup

FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:83)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(

RE: 1.2.10 -> 2.0.1 migration issue

[Christopher Wirt]

Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I'm
seeing this error on the machine I tried to migrate earlier to 2.0.1

 

Thanks

 

From: Christopher Wirt [mailto:chris.w...@struq.com] 
Sent: 25 September 2013 14:04
To: 'user@cassandra.apache.org'
Subject: RE: 1.2.10 -> 2.0.1 migration issue

 

Hi Marcus,

 

I've seen your patch. This works with what I'm seeing. The first data
directory only contained the JSON manifest at that time.

 

As a workaround I've made sure that each of the snapshot directories now
exist before starting up.

 

I still end up with the second exception I posted regarding a duplicate hard
link. Possibly two unrelated exceptions.

 

After getting this error. Looking at the datadirs

Data1 contains 

JSON manifests

Loads of data files

Snapshot directory

Data2 contains

Just the snapshot directory

Data3 contains

Just the snapshot directory

 

INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2

INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
pre-sstablemetamigration

ERROR 12:56:22,787 Exception encountered during startup

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)

java.lang.RuntimeException: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(
LegacyLeveledManifest.java:138)

at
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(Le
gacyLeveledManifest.java:91)

at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)

at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:4
43)

at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)

Exception encountered during startup: Tried to create duplicate hard link to
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etamigration/impressionstorev2.json

 

Thanks,

 

Chris

 

 

From: Marcus Eriksson [mailto:krum...@gmail.com] 
Sent: 25 September 2013 13:11
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

you are probably reading trunk NEWS.txt

 

read the ticket for explanation of what the issue was (it is a proper bug)

 

On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt 
wrote:

Hi Marcus,

Thanks for having a look at this.

 

Just noticed this in the NEWS.txt 

 

For leveled compaction users, 2.0 must be atleast started before

 upgrading to 2.1 due to the fact that the old JSON leveled

 manifest is migrated into the sstable metadata files on startup

 in 2.0 and this code is gone from 2.1.

 

Basically, my fault for skimming over this too quickly. 

 

We will move from 1.2.10 -> 2.0 -> 2.1

 

Thanks,

Chris

 

 

From: Marcus Eriksson [mailto:krum...@gmail.com] 
Sent: 25 September 2013 09:37
To: user@cassandra.apache.org
Subject: Re: 1.2.10 -> 2.0.1 migration issue

 

cant really reproduce, could you update the ticket with a bit more info
about your setup?

 

do you have multiple .json files in your data dirs?

 

On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson  wrote:

this is most likely a bug, filed
https://issues.apache.org/jira/browse/CASSANDRA-6093 and will try to have a
look today.

 

On Wed, Sep 25, 2013 at 1:48 AM, Christopher Wirt 
wrote:

Hi,

 

Just had a go at upgrading a node to the latest stable c* 2 release and
think I ran into some issues with manifest migration.

 

On initial start up I hit this error as it starts to load the first of my
CF. 

 

INFO [main] 2013-09-24 22:56:01,018 LegacyLeveledManifest.java (line 89)
Migrating manifest for struqrealtime/impressionstorev2

INFO [main] 2013-09-24 22:56:01,019 LegacyLeveledManifest.java (line 119)
Snapshotting struqrealtime, impressionstorev2 to pre-sstablemetamigration

ERROR [main] 2013-09-24 22:56:01,030 CassandraDaemon.java (line 459)
Exception encountered during startup

FSWriteError in
/disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablem
etam

Re: 1.2.10 -> 2.0.1 migration issue

[Marcus Eriksson]

you probably have to remove the old snapshots before trying to restart


On Wed, Sep 25, 2013 at 3:05 PM, Christopher Wirt wrote:

> Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m
> seeing this error on the machine I tried to migrate earlier to 2.0.1
>
> ** **
>
> Thanks
>
> ** **
>
> *From:* Christopher Wirt [mailto:chris.w...@struq.com]
> *Sent:* 25 September 2013 14:04
> *To:* 'user@cassandra.apache.org'
> *Subject:* RE: 1.2.10 -> 2.0.1 migration issue
>
> ** **
>
> Hi Marcus,
>
> ** **
>
> I’ve seen your patch. This works with what I’m seeing. The first data
> directory only contained the JSON manifest at that time.
>
> ** **
>
> As a workaround I’ve made sure that each of the snapshot directories now
> exist before starting up.
>
> ** **
>
> I still end up with the second exception I posted regarding a duplicate
> hard link. Possibly two unrelated exceptions.
>
> ** **
>
> After getting this error. Looking at the datadirs
>
> Data1 contains 
>
> JSON manifests
>
> Loads of data files
>
> Snapshot directory
>
> Data2 contains
>
> Just the snapshot directory
>
> Data3 contains
>
> Just the snapshot directory
>
> ** **
>
> INFO 12:56:22,766 Migrating manifest for struqrealtime/impressionstorev2**
> **
>
> INFO 12:56:22,767 Snapshotting struqrealtime, impressionstorev2 to
> pre-sstablemetamigration
>
> ERROR 12:56:22,787 Exception encountered during startup
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> 
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> 
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> 
>
> java.lang.RuntimeException: Tried to create duplicate hard link to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> 
>
> at
> org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:71)**
> **
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:138)
> 
>
> at
> org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443)
> 
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:486)
> 
>
> Exception encountered during startup: Tried to create duplicate hard link
> to
> /disk1/cassandra/data/struqrealtime/impressionstorev2/snapshots/pre-sstablemetamigration/impressionstorev2.json
> 
>
> ** **
>
> Thanks,
>
> ** **
>
> Chris
>
> ** **
>
> ** **
>
> *From:* Marcus Eriksson [mailto:krum...@gmail.com]
> *Sent:* 25 September 2013 13:11
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue
>
> ** **
>
> you are probably reading trunk NEWS.txt
>
> ** **
>
> read the ticket for explanation of what the issue was (it is a proper bug)
> 
>
> ** **
>
> On Wed, Sep 25, 2013 at 12:59 PM, Christopher Wirt 
> wrote:
>
> Hi Marcus,
>
> Thanks for having a look at this.
>
>  
>
> Just noticed this in the NEWS.txt 
>
>  
>
> For *leveled* compaction users, 2.0 must be *atleast* started before
>
>  upgrading to 2.1 due to the fact that the old JSON *leveled*
>
>  manifest is migrated into the *sstable* *metadata* files on startup**
> **
>
>  in 2.0 and this code is gone from 2.1.
>
>  
>
> Basically, my fault for skimming over this too quickly. 
>
>  
>
> We will move from 1.2.10 -> 2.0 -> 2.1
>
>  
>
> Thanks,
>
> Chris
>
>  
>
>  
>
> *From:* Marcus Eriksson [mailto:krum...@gmail.com]
> *Sent:* 25 September 2013 09:37
> *To:* user@cassandra.apache.org
> *Subject:* Re: 1.2.10 -> 2.0.1 migration issue
>
>  
>
> cant really reproduce, could you update the ticket with a bit more info
> about your setup?
>
>  
>
> do you have multiple .json files in your data dirs?
>
>  
>
> On Wed, Sep 25, 2013 at 10:07 AM, Marcus Eriksson 
> wrote:
>
> this is most likely a bug, filed
> https://issues.

FW: Migration LCS from 1.2.X to 2.0.x exception

[Desimpel, Ignace]

Did a test for File::rename on Ubuntu and windows.
Seems that on Ubuntu the File::renameTo(FROM, TO) is actually replacing the 
existing (same as TO) file and content with the new (same as FROM) file and 
content.
So that would make it a problem only on Windows. Adding a File::delete before 
the rename in case of Windows could do the (scary) trick.

From: Desimpel, Ignace [mailto:ignace.desim...@nuance.com]
Sent: woensdag 25 september 2013 13:13
To: user@cassandra.apache.org
Subject: FW: Migration LCS from 1.2.X to 2.0.x exception

Same is true if started from 1.2.9 version to 2.0.x
Seems like Cassandra-5383 is the related issue.

The code in the function mutateLevel  (from 1.2.5 to 1.2.9 and later ) is 
indirectly using the java file::renameTo function. And that will not work on 
Windows if the file already exists. Don't know about Linux, but even the java 
function description is already telling that the behavior of the function can 
be machine dependent. So even on linux based systems, and even if that 
functions returns with success, still does not confirm that actually the old 
file name/content was removed and the new file name/content is used as 
replacement.


From: Marcus Eriksson 
[mailto:krum...@gmail.com]
Sent: woensdag 25 september 2013 10:02
To: user@cassandra.apache.org
Subject: Re: Migration LCS from 1.2.X to 2.0.x exception

this is the issue:
https://issues.apache.org/jira/browse/CASSANDRA-5383

guess it fell between chairs, will poke around

On Tue, Sep 24, 2013 at 4:26 PM, Nate McCall 
mailto:n...@thelastpickle.com>> wrote:
What version of 1.2.x?

Unfortunately, you must go through 1.2.9 first. See 
https://github.com/apache/cassandra/blob/cassandra-2.0.0/NEWS.txt#L19-L24

On Tue, Sep 24, 2013 at 8:57 AM, Desimpel, Ignace 
mailto:ignace.desim...@nuance.com>> wrote:
Tested on WINDOWS : On startup of the 2.0.0 version from 1.2.x files I get an 
error as listed below.

This is due to the code in LeveledManifest:: mutateLevel. The method already 
has a comment saying that it is scary ...
On windows, one cannot use the File::rename if the target file name already 
exists.
Also, even on Linux, I'm not sure if a rename would actually 
'overwrite/implicit-delete' the content of the target file.

Anyway, adding code (below) before the FileUtils.renameWithConfirm should work 
in both cases (maybe even rename the fromFile to be able to recover...)
File oTo = new File(filename);
if ( oTo.exists() ) oTo.delete();


java.lang.RuntimeException: Failed to rename 
.xxx\Function-ic-10-Statistics.db-tmp to 
.xxx\Function-ic-10-Statistics.db
at 
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:136) 
~[main/:na]
at 
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:125) 
~[main/:na]
at 
org.apache.cassandra.db.compaction.LeveledManifest.mutateLevel(LeveledManifest.java:601)
 ~[main/:na]
at 
org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:103)
 ~[main/:na]
at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:247) 
~[main/:na]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:443) 
~[main/:na]

Regards,

Ignace Desimpel

Re: Mystery PIG issue with 1.2.10

[Chad Johnston]

As an FYI, creating the table without the "WITH COMPACT STORAGE" and using
CqlStorage works just fine in 1.2.10.

I know that CqlStorage and AbstractCassandraStorage got changed for 1.2.10
- maybe there's a regression with the existing CassandraStorage?

Chad


On Wed, Sep 25, 2013 at 1:51 AM, Janne Jalkanen wrote:

> Heya!
>
> I am seeing something rather strange in the way Cass 1.2 + Pig seem to
> handle integer values.
>
> Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1.  Single node
> for testing this.
>
> First a table:
>
> > CREATE TABLE testc (
>   key text PRIMARY KEY,
>   ivalue int,
>   svalue text,
>   value bigint
> ) WITH COMPACT STORAGE;
>
> > insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
> > select * from testc;
>
>  key | ivalue | svalue | value
> -+++---
>  foo | 10 |bar | 65
>
> For my Pig setup, I then use libraries from different C* versions to
> actually talk to my database (which stays on 1.2.10 all the time).
>
> Cassandra 1.0.12 (using cassandra_storage.jar):
>
> > testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> (foo,(svalue,bar),(ivalue,10),(value,65),{})
>
> Cassandra 1.1.10:
>
> > testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> (foo,(svalue,bar),(ivalue,10),(value,65),{})
>
> Cassandra 1.2.10:
>
> > (testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> foo,{(ivalue,
> ),(svalue,bar),(value,A)})
>
>
> To me it appears that ints and bigints are interpreted as ascii values in
> cass 1.2.10.  Did something change for CassandraStorage, is there a
> regression, or am I doing something wrong?  Quick perusal of the JIRA
> didn't reveal anything that I could directly pin on this.
>
> Note that using compact storage does not seem to affect the issue, though
> it obviously changes the resulting pig format.
>
> In addition, trying to use Pygmalion
>
> > tf = foreach testc generate key,
> flatten(FromCassandraBag('ivalue,svalue,value',columns)) as
> (ivalue:int,svalue:chararray,lvalue:long);
> > dump tf
>
> (foo,
> ,bar,A)
>
> So no help there. Explicitly casting the values to (long) or (int) just
> results in a ClassCastException.
>
> /Janne

Re: Bad Request: Invalid null value for clustering key part

[Tyler Hobbs]

On Wed, Sep 25, 2013 at 5:00 AM, Petter von Dolwitz (Hem) <
petter.von.dolw...@gmail.com> wrote:

> I am generating the CSV file by other means than the COPY ... TO command
> and I thought that the NULL option did not apply to the COPY ... FROM
> command but, as it turns out, it does! This solved my problem.
>
> The documentation here (
> http://www.datastax.com/documentation/cql/3.1/webhelp/index.html#cql/cql_reference/copy_r.html)
> is misleading. Should I report this by other means than this post?
>

I've opened a ticket with our docs team to get this updated.  Thanks!


-- 
Tyler Hobbs
DataStax 

Re: 1.2.10 -> 2.0.1 migration issue

[Robert Coli]

On Wed, Sep 25, 2013 at 6:05 AM, Christopher Wirt wrote:

> Should also say. I have managed to move one node from 1.2.10 to 2.0.0. I’m
> seeing this error on the machine I tried to migrate earlier to 2.0.1
>

I'm confused... for the record :

1) you tried to upgrade from 1.2.10 to 2.0.1
2) the NEWS.txt snippet you posted refers to upgrading from versions below
2.0 to 2.1
3) 2.0.1 is 2.0, not 2.1

Therefore the problem is actually
https://issues.apache.org/jira/browse/CASSANDRA-6093 ?

=Rob

Nodes not added to existing cluster

[Skye Book]

Hi all,

I have a three node cluster using the EC2 Multi-Region Snitch currently 
operating only in US-EAST.  On having a node go down this morning, I started a 
new node with an identical configuration, except for the seed list, the listen 
address and the rpc address.  The new node comes up and creates its own cluster 
rather than joining the pre-existing ring.  I've tried creating a node both 
before ad after using `nodetool remove` for the bad node, each time with the 
same result.

Does anyone have any suggestions for where to look that might put me on the 
right track?

Thanks,
-Skye

Re: Nodes not added to existing cluster

[Robert Coli]

On Wed, Sep 25, 2013 at 12:41 PM, Skye Book  wrote:

> I have a three node cluster using the EC2 Multi-Region Snitch currently
> operating only in US-EAST.  On having a node go down this morning, I
> started a new node with an identical configuration, except for the seed
> list, the listen address and the rpc address.  The new node comes up and
> creates its own cluster rather than joining the pre-existing ring.  I've
> tried creating a node both *before* ad *after* using `nodetool remove`
> for the bad node, each time with the same result.
>

What version of Cassandra?

This particular confusing behavior is fixed upstream, in a version you
should not deploy to production yet. Take some solace, however, that you
may be the last Cassandra administrator to die for a broken code path!

https://issues.apache.org/jira/browse/CASSANDRA-5768

Does anyone have any suggestions for where to look that might put me on the
> right track?
>

It must be that your seed list is wrong in some way, or your node state is
wrong. If you're trying to bootstrap a node, note that you can't bootstrap
a node when it is in its own seed list.

If you have installed Cassandra via debian package, there is a possibility
that your node has started before you explicitly started it. If so, it
might have invalid node state.

Have you tried wiping the data directory and trying again?

What is your seed list? Are you sure the new node can reach the seeds on
the network layer?

=Rob

Re: Nodes not added to existing cluster

[Laing, Michael]

Check your security groups to be sure you have appropriate access.

If in a VPC check both IN and OUT; if using ACLs check those.


On Wed, Sep 25, 2013 at 3:41 PM, Skye Book  wrote:

> Hi all,
>
> I have a three node cluster using the EC2 Multi-Region Snitch currently
> operating only in US-EAST.  On having a node go down this morning, I
> started a new node with an identical configuration, except for the seed
> list, the listen address and the rpc address.  The new node comes up and
> creates its own cluster rather than joining the pre-existing ring.  I've
> tried creating a node both *before* ad *after* using `nodetool remove`
> for the bad node, each time with the same result.
>
> Does anyone have any suggestions for where to look that might put me on
> the right track?
>
> Thanks,
> -Skye
>
>

Re: Nodes not added to existing cluster

[Skye Book]

Thank you, both Michael and Robert for your suggestions.  I actually saw 5760, 
but we were running on 2.0.0, which it seems like this was fixed in.

That said, I noticed that my Chef scripts were failing to set the 
broadcast_address correctly, which I'm guessing is the cause of the problem, 
fixing that and trying a redeploy.  I am curious, though, how any of this 
worked in the first place spread across three AZ's without that being set?

-Skye

On Sep 25, 2013, at 3:56 PM, Robert Coli  wrote:

> On Wed, Sep 25, 2013 at 12:41 PM, Skye Book  wrote:
> I have a three node cluster using the EC2 Multi-Region Snitch currently 
> operating only in US-EAST.  On having a node go down this morning, I started 
> a new node with an identical configuration, except for the seed list, the 
> listen address and the rpc address.  The new node comes up and creates its 
> own cluster rather than joining the pre-existing ring.  I've tried creating a 
> node both before ad after using `nodetool remove` for the bad node, each time 
> with the same result.
> 
> What version of Cassandra?
> 
> This particular confusing behavior is fixed upstream, in a version you should 
> not deploy to production yet. Take some solace, however, that you may be the 
> last Cassandra administrator to die for a broken code path!
> 
> https://issues.apache.org/jira/browse/CASSANDRA-5768
> 
> Does anyone have any suggestions for where to look that might put me on the 
> right track?
> 
> It must be that your seed list is wrong in some way, or your node state is 
> wrong. If you're trying to bootstrap a node, note that you can't bootstrap a 
> node when it is in its own seed list.
> 
> If you have installed Cassandra via debian package, there is a possibility 
> that your node has started before you explicitly started it. If so, it might 
> have invalid node state.
> 
> Have you tried wiping the data directory and trying again?
> 
> What is your seed list? Are you sure the new node can reach the seeds on the 
> network layer?
> 
> =Rob

[Cassandra] Initial Setup - VMs for Research

[shathawa]

Request some initial setup guidance for Cassandra deployment

I expect to mentor a project at the Oregon State University
computer science department for a senior engineering student
project.

I am trying to pre-configure one or more VMware virtual
machines to hold an initial Cassandra database for a NOSQL
project.

Any guidance on the steps for initial deployment would
be appreciated.

My VMware machines already have the necessary 3rd party
tools such as Oracle Java 7 and are running on a Debian Linux
7.1.0 release.  The Oregon State University computer science
department will eventually host these virtual machines on
their department servers if the student project is selected.

Sincerely,
Steven J. Hathaway
(Senior IT Systems Architect)

RE: [Cassandra] Initial Setup - VMs for Research

[Kanwar Sangha]

What help are u looking for ? 

http://www.datastax.com/docs/datastax_enterprise3.1/install/install_deb_pkg

-Original Message-
From: shath...@e-z.net [mailto:shath...@e-z.net] 
Sent: 25 September 2013 15:27
To: user@cassandra.apache.org
Subject: [Cassandra] Initial Setup - VMs for Research

Request some initial setup guidance for Cassandra deployment

I expect to mentor a project at the Oregon State University
computer science department for a senior engineering student
project.

I am trying to pre-configure one or more VMware virtual
machines to hold an initial Cassandra database for a NOSQL
project.

Any guidance on the steps for initial deployment would
be appreciated.

My VMware machines already have the necessary 3rd party
tools such as Oracle Java 7 and are running on a Debian Linux
7.1.0 release.  The Oregon State University computer science
department will eventually host these virtual machines on
their department servers if the student project is selected.

Sincerely,
Steven J. Hathaway
(Senior IT Systems Architect)

Re: [Cassandra] Initial Setup - VMs for Research

[Aaron Morton]

There is also Debian for the standard Apache Cassandra build 
http://wiki.apache.org/cassandra/DebianPackaging

Or you can use DS Community which is a totally free distro of Apache Cassandra 
http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/install/installDeb_t.html
(Instructions for DS Community will work for both)

Data Stax Enterprise is free for development purposes but can not be used in 
production without a licence. In addition is ships with extra installation 
companents (hadoop, mahoout, solr) that you probably want need. 

Hope that helps. 

 
-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 26/09/2013, at 2:42 PM, Kanwar Sangha  wrote:

> What help are u looking for ? 
> 
> http://www.datastax.com/docs/datastax_enterprise3.1/install/install_deb_pkg
> 
> -Original Message-
> From: shath...@e-z.net [mailto:shath...@e-z.net] 
> Sent: 25 September 2013 15:27
> To: user@cassandra.apache.org
> Subject: [Cassandra] Initial Setup - VMs for Research
> 
> Request some initial setup guidance for Cassandra deployment
> 
> I expect to mentor a project at the Oregon State University
> computer science department for a senior engineering student
> project.
> 
> I am trying to pre-configure one or more VMware virtual
> machines to hold an initial Cassandra database for a NOSQL
> project.
> 
> Any guidance on the steps for initial deployment would
> be appreciated.
> 
> My VMware machines already have the necessary 3rd party
> tools such as Oracle Java 7 and are running on a Debian Linux
> 7.1.0 release.  The Oregon State University computer science
> department will eventually host these virtual machines on
> their department servers if the student project is selected.
> 
> Sincerely,
> Steven J. Hathaway
> (Senior IT Systems Architect)
> 

Re: Nodes not added to existing cluster

[Aaron Morton]

>  I am curious, though, how any of this worked in the first place spread 
> across three AZ's without that being set?
boradcast_address is only needed when you are going cross region (IIRC it's the 
EC2MultiRegionSnitch) that sets it. 

As rob said, make sure the seed list includes on of the other nodes and that 
the cluster_name set. 

Cheers

-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 26/09/2013, at 8:12 AM, Skye Book  wrote:

> Thank you, both Michael and Robert for your suggestions.  I actually saw 
> 5760, but we were running on 2.0.0, which it seems like this was fixed in.
> 
> That said, I noticed that my Chef scripts were failing to set the 
> broadcast_address correctly, which I'm guessing is the cause of the problem, 
> fixing that and trying a redeploy.  I am curious, though, how any of this 
> worked in the first place spread across three AZ's without that being set?
> 
> -Skye
> 
> On Sep 25, 2013, at 3:56 PM, Robert Coli  wrote:
> 
>> On Wed, Sep 25, 2013 at 12:41 PM, Skye Book  wrote:
>> I have a three node cluster using the EC2 Multi-Region Snitch currently 
>> operating only in US-EAST.  On having a node go down this morning, I started 
>> a new node with an identical configuration, except for the seed list, the 
>> listen address and the rpc address.  The new node comes up and creates its 
>> own cluster rather than joining the pre-existing ring.  I've tried creating 
>> a node both before ad after using `nodetool remove` for the bad node, each 
>> time with the same result.
>> 
>> What version of Cassandra?
>> 
>> This particular confusing behavior is fixed upstream, in a version you 
>> should not deploy to production yet. Take some solace, however, that you may 
>> be the last Cassandra administrator to die for a broken code path!
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-5768
>> 
>> Does anyone have any suggestions for where to look that might put me on the 
>> right track?
>> 
>> It must be that your seed list is wrong in some way, or your node state is 
>> wrong. If you're trying to bootstrap a node, note that you can't bootstrap a 
>> node when it is in its own seed list.
>> 
>> If you have installed Cassandra via debian package, there is a possibility 
>> that your node has started before you explicitly started it. If so, it might 
>> have invalid node state.
>> 
>> Have you tried wiping the data directory and trying again?
>> 
>> What is your seed list? Are you sure the new node can reach the seeds on the 
>> network layer?
>> 
>> =Rob
> 

Re: Mystery PIG issue with 1.2.10

[Aaron Morton]

> > (testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> foo,{(ivalue,
> ),(svalue,bar),(value,A)})



If the CQL 3 data ye wish to read, CqlStorage be the driver of your success. 

(btw there is a ticket out to update the example if you get excited 
https://issues.apache.org/jira/browse/CASSANDRA-5709)

Cheers


-
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 26/09/2013, at 3:57 AM, Chad Johnston  wrote:

> As an FYI, creating the table without the "WITH COMPACT STORAGE" and using 
> CqlStorage works just fine in 1.2.10.
> 
> I know that CqlStorage and AbstractCassandraStorage got changed for 1.2.10 - 
> maybe there's a regression with the existing CassandraStorage?
> 
> Chad
> 
> 
> On Wed, Sep 25, 2013 at 1:51 AM, Janne Jalkanen  
> wrote:
> Heya!
> 
> I am seeing something rather strange in the way Cass 1.2 + Pig seem to handle 
> integer values.
> 
> Setup: Cassandra 1.2.10, OSX 10.8, JDK 1.7u40, Pig 0.11.1.  Single node for 
> testing this.
> 
> First a table:
> 
> > CREATE TABLE testc (
>   key text PRIMARY KEY,
>   ivalue int,
>   svalue text,
>   value bigint
> ) WITH COMPACT STORAGE;
> 
> > insert into testc (key,ivalue,svalue,value) values ('foo',10,'bar',65);
> > select * from testc;
> 
>  key | ivalue | svalue | value
> -+++---
>  foo | 10 |bar | 65
> 
> For my Pig setup, I then use libraries from different C* versions to actually 
> talk to my database (which stays on 1.2.10 all the time).
> 
> Cassandra 1.0.12 (using cassandra_storage.jar):
> 
> > testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> (foo,(svalue,bar),(ivalue,10),(value,65),{})
> 
> Cassandra 1.1.10:
> 
> > testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> (foo,(svalue,bar),(ivalue,10),(value,65),{})
> 
> Cassandra 1.2.10:
> 
> > (testc = LOAD 'cassandra://keyspace/testc' USING CassandraStorage();
> > dump testc
> foo,{(ivalue,
> ),(svalue,bar),(value,A)})
> 
> 
> To me it appears that ints and bigints are interpreted as ascii values in 
> cass 1.2.10.  Did something change for CassandraStorage, is there a 
> regression, or am I doing something wrong?  Quick perusal of the JIRA didn't 
> reveal anything that I could directly pin on this.
> 
> Note that using compact storage does not seem to affect the issue, though it 
> obviously changes the resulting pig format.
> 
> In addition, trying to use Pygmalion
> 
> > tf = foreach testc generate key, 
> > flatten(FromCassandraBag('ivalue,svalue,value',columns)) as 
> > (ivalue:int,svalue:chararray,lvalue:long);
> > dump tf
> 
> (foo,
> ,bar,A)
> 
> So no help there. Explicitly casting the values to (long) or (int) just 
> results in a ClassCastException.
> 
> /Janne
> 

Re: Nodes not added to existing cluster

[Skye Book]

Hi Aaron, thanks for the clarification.

As might be expected, having the broadcast_address fixed hasn't fixed anything. 
 What I did find after writing my last email is that output.log is littered 
with these:

 INFO 05:03:49,015 Cannot handshake version with /aa.bb.cc.dd
 INFO 05:03:49,017 Handshaking version with /aa.bb.cc.dd
 INFO 05:03:49,803 Cannot handshake version with /ww.xx.yy.zz
 INFO 05:03:49,805 Handshaking version with /ww.xx.yy.zz

The two addresses that it is unable to handshake with are the other two 
addresses of nodes in the cluster I'm unable to join.  I started thinking that 
maybe EC2 was having an-advertised problem communicating between AZ's but 
bringing up nodes in both of the other availability zones resulted in the same 
wrong behavior.

I've gist'd my cassandra.yaml, its pretty standard and hasn't caused an issue 
in the past for me.  https://gist.github.com/skyebook/ec9364cdcec02e803ffc

Skye Book
http://skyebook.net -- @sbook

On Sep 26, 2013, at 12:34 AM, Aaron Morton  wrote:

>>  I am curious, though, how any of this worked in the first place spread 
>> across three AZ's without that being set?
> boradcast_address is only needed when you are going cross region (IIRC it's 
> the EC2MultiRegionSnitch) that sets it. 
> 
> As rob said, make sure the seed list includes on of the other nodes and that 
> the cluster_name set. 
> 
> Cheers
> 
> -
> Aaron Morton
> New Zealand
> @aaronmorton
> 
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
> 
> On 26/09/2013, at 8:12 AM, Skye Book  wrote:
> 
>> Thank you, both Michael and Robert for your suggestions.  I actually saw 
>> 5760, but we were running on 2.0.0, which it seems like this was fixed in.
>> 
>> That said, I noticed that my Chef scripts were failing to set the 
>> broadcast_address correctly, which I'm guessing is the cause of the problem, 
>> fixing that and trying a redeploy.  I am curious, though, how any of this 
>> worked in the first place spread across three AZ's without that being set?
>> 
>> -Skye
>> 
>> On Sep 25, 2013, at 3:56 PM, Robert Coli  wrote:
>> 
>>> On Wed, Sep 25, 2013 at 12:41 PM, Skye Book  wrote:
>>> I have a three node cluster using the EC2 Multi-Region Snitch currently 
>>> operating only in US-EAST.  On having a node go down this morning, I 
>>> started a new node with an identical configuration, except for the seed 
>>> list, the listen address and the rpc address.  The new node comes up and 
>>> creates its own cluster rather than joining the pre-existing ring.  I've 
>>> tried creating a node both before ad after using `nodetool remove` for the 
>>> bad node, each time with the same result.
>>> 
>>> What version of Cassandra?
>>> 
>>> This particular confusing behavior is fixed upstream, in a version you 
>>> should not deploy to production yet. Take some solace, however, that you 
>>> may be the last Cassandra administrator to die for a broken code path!
>>> 
>>> https://issues.apache.org/jira/browse/CASSANDRA-5768
>>> 
>>> Does anyone have any suggestions for where to look that might put me on the 
>>> right track?
>>> 
>>> It must be that your seed list is wrong in some way, or your node state is 
>>> wrong. If you're trying to bootstrap a node, note that you can't bootstrap 
>>> a node when it is in its own seed list.
>>> 
>>> If you have installed Cassandra via debian package, there is a possibility 
>>> that your node has started before you explicitly started it. If so, it 
>>> might have invalid node state.
>>> 
>>> Have you tried wiping the data directory and trying again?
>>> 
>>> What is your seed list? Are you sure the new node can reach the seeds on 
>>> the network layer?
>>> 
>>> =Rob
>> 
>