You do seem to be experiencing a problem indeed, but hard to say what it is
from just this (could be anything from a configuration problem, a cassandra
bug or a bug in the java driver).
But since you seem to be able to reproduce easily, if you can provide a
script that reproduce that issue, I'd gladly have a look (feel free to send
it to me privately if you prefer).
--
Sylvain
On Tue, Mar 5, 2013 at 4:40 PM, Gabriel Ciuloaica <gciuloa...@gmail.com>wrote:
> So, I have added more logging to the test app (comments inline). For some
> reason I'm loosing updates. In a for loop I'm executing upload, read
> writetime, download blob. Executed 10 times... See iteration number 2 and
> 3....
>
> 1. initialize session
>
> 0 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /
> 10.11.1.109 added
> 1 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /
> 10.11.1.200 added
>
>
>
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:214 bytes --> uploading a blob (INSERT)
> UPLOAD:1154 ms
> Write time: 1362497519584000 ms --> reading writetime(avatar)
> DOWNLOAD LENGTH: 214 bytes --> download the blob (SELECT)
> DOWNLOAD: 134 ms
> ---> md5 of the blob on
> upload ---> md5
> of the blob on download
> Upload Digest MD5 : b1944c41a25192520d33d15d00db2718 === Download Digest
> MD5 : b1944c41a25192520d33d15d00db2718
> -------------------------------- 0
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:4031 bytes
> UPLOAD:675 ms
> Write time: 1362497521493000 ms
> DOWNLOAD LENGTH: 4031 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : 20b71b77f90b3f8ae8995a7ce7f68295 === Download Digest
> MD5 : 20b71b77f90b3f8ae8995a7ce7f68295
> -------------------------------- 1
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:3392 bytes
> UPLOAD:668 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 1158e1ea54d46a4d0bd45becc4523585 === Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
> -------------------------------- 2
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:253 bytes
> UPLOAD:668 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : fc9ce009530d6018a80c344d87b8ada4 === Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
> -------------------------------- 3
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:266 bytes
> UPLOAD:704 ms
> Write time: 1362497556815000 ms
> DOWNLOAD LENGTH: 3392 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 5726af06e91a520deed093aba6afe112 === Download Digest
> MD5 : 1158e1ea54d46a4d0bd45becc4523585
> -------------------------------- 4
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:3082 bytes
> UPLOAD:901 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : fa2ea1972992cafea1c71b6c3e718058 === Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
> -------------------------------- 5
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:1481 bytes
> UPLOAD:703 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 135 ms
> Upload Digest MD5 : f208e4d3ea133fad5f9d175052ca70cf === Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
> -------------------------------- 6
> COORDINATOR: 10.11.1.109
> UPLOAD LENGTH:5214 bytes
> UPLOAD:801 ms
> Write time: 1362497562076000 ms
> DOWNLOAD LENGTH: 3082 bytes
> DOWNLOAD: 134 ms
> Upload Digest MD5 : c58d92d8273c7c9a7db76363b0b3e4c7 === Download Digest
> MD5 : fa2ea1972992cafea1c71b6c3e718058
> -------------------------------- 7
> COORDINATOR: 10.11.1.200
> UPLOAD LENGTH:2992 bytes
> UPLOAD:665 ms
> Write time: 1362497567779000 ms
> DOWNLOAD LENGTH: 2992 bytes
> DOWNLOAD: 134 ms
> Upload Digest MD5 : 0848513c1b4214adf73c6ea5509ec294 === Download Digest
> MD5 : 0848513c1b4214adf73c6ea5509ec294
> -------------------------------- 8
> COORDINATOR: 10.11.1.108
> UPLOAD LENGTH:3670 bytes
> UPLOAD:672 ms
> Write time: 1362497567779000 ms
> DOWNLOAD LENGTH: 2992 bytes
> DOWNLOAD: 136 ms
> Upload Digest MD5 : 27e235b9a90a22004d4098a0228ee07b === Download Digest
> MD5 : 0848513c1b4214adf73c6ea5509ec294
> -------------------------------- 9
>
>
> Thanks,
> Gabi
>
> On 3/5/13 1:31 PM, Gabriel Ciuloaica wrote:
>
> Hi Sylvain,
>
> thanks for fast answer. I have updated keyspace definition and
> cassandra-topologies.properties to all 3 nodes and restarted each node.
> Both problems are still reproducible. I'm not able to read my writes and
> also the selects shows same data as in my previous email.
>
> for write and read I'm using:
> private static final String WRITE_STATEMENT = "INSERT INTO avatars (id,
> image_type, avatar) VALUES (?,?,?);";
> private static final String READ_STATEMENT = "SELECT avatar, image_type
> FROM avatars WHERE id=?";
>
> I'm using java-driver (1.0.0-beta1) with prepared statement, sync calls.
>
> Write snippet:
>
> Session session;
> try {
> session = cassandraSession.getSession();
> BoundStatement stmt = session.prepare(WRITE_STATEMENT)
>
> .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
> stmt.enableTracing();
> stmt.setLong("id", accountId);
> stmt.setString("image_type", image.getType());
> stmt.setBytes("avatar", ByteBuffer.wrap(image.getBytes()));
> ResultSet result = session.execute(stmt);
> LOG.info("UPLOAD COORDINATOR: {}", result.getQueryTrace()
> .getCoordinator().getCanonicalHostName());
>
> } catch (NoHostAvailableException e) {
> LOG.error("Could not prepare the statement.", e);
> throw new StorageUnavailableException(e);
> } finally {
> cassandraSession.releaseSession();
> }
>
> Read snippet:
>
> Session session = null;
> byte[] imageBytes = null;
> String imageType = "png";
> try {
> session = cassandraSession.getSession();
> BoundStatement stmt = session.prepare(READ_STATEMENT)
>
> .setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM).bind();
> stmt.setLong("id", accountId);
> ResultSet result = session.execute(stmt);
> Iterator<Row> it = result.iterator();
> ByteBuffer avatar = null;
>
> while (it.hasNext()) {
> Row row = it.next();
> avatar = row.getBytes("avatar");
> imageType = row.getString("image_type");
> }
> if (avatar == null) {
> throw new AvatarNotFoundException("Avatar hasn't been
> found");
> }
> int length = avatar.remaining();
> imageBytes = new byte[length];
> avatar.get(imageBytes, 0, length);
>
> } catch (NoHostAvailableException e) {
> LOG.error("Could not prepare the statement.", e);
> throw new StorageUnavailableException(e);
> } finally {
> cassandraSession.releaseSession();
> }
>
> Let me know what other information is need it.
>
> Thanks,
> Gabi
>
>
> On 3/5/13 12:52 PM, Sylvain Lebresne wrote:
>
> Without looking into details too closely, I'd say you're probably hitting
> https://issues.apache.org/jira/browse/CASSANDRA-5292 (since you use
> NTS+propertyFileSnitch+a DC name in caps).
>
> Long story short, the CREATE KEYSPACE interpret your DC-TORONTO as
> dc-toronto, which then probably don't match what you have in you property
> file. This will be fixed in 1.2.3. In the meantime, a workaround would be
> to use the cassandra-cli to create/update your keyspace definition.
>
> --
> Sylvain
>
>
> On Tue, Mar 5, 2013 at 11:24 AM, Gabriel Ciuloaica
> <gciuloa...@gmail.com>wrote:
>
>> Hello,
>>
>> I'm trying to find out what the problem is and where it is located.
>> I have a 3 nodes Cassandra cluster (1.2.1), RF=3.
>> I have a keyspace and a cf as defined (using PropertyFileSnitch):
>>
>> CREATE KEYSPACE backend WITH replication = {
>> 'class': 'NetworkTopologyStrategy',
>> 'DC-TORONTO': '3'
>> };
>>
>> USE backend;
>>
>> CREATE TABLE avatars (
>> id bigint PRIMARY KEY,
>> avatar blob,
>> image_type text
>> ) WITH
>> bloom_filter_fp_chance=0.010000 AND
>> caching='KEYS_ONLY' AND
>> comment='' AND
>> dclocal_read_repair_chance=0.000000 AND
>> gc_grace_seconds=864000 AND
>> read_repair_chance=0.100000 AND
>> replicate_on_write='true' AND
>> compaction={'class': 'SizeTieredCompactionStrategy'} AND
>> compression={'sstable_compression': 'SnappyCompressor'};
>>
>> Status of the cluster:
>> Datacenter: DC-TORONTO
>> ======================
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns Host ID
>> Rack
>> UN 10.11.1.109 44.98 MB 256 46.8%
>> 726689df-edc3-49a0-b680-370953994a8c RAC2
>> UN 10.11.1.200 6.57 MB 64 10.3%
>> d6d700d4-28aa-4722-b215-a6a7d304b8e7 RAC3
>> UN 10.11.1.108 54.32 MB 256 42.8%
>> 73cd86a9-4efb-4407-9fe8-9a1b3a277af7 RAC1
>>
>> I'm trying to read my writes, by using CQL (datastax-java-driver), using
>> LOCAL_QUORUM for reads and writes. For some reason, some of the writes are
>> lost. Not sure if it is a driver issue or cassandra issue.
>> Dinging further, using cqlsh client (1.2.1), I found a strange situation:
>>
>> select count(*) from avatars;
>>
>> count
>> -------
>> 226
>>
>> select id from avatars;
>>
>> id
>> ---------
>> 314
>> 396
>> 19
>> ......... -> 77 rows in result
>>
>> select id, image_type from avatars;
>>
>> id | image_type
>> ---------+------------
>> 332 | png
>> 314 | png
>> 396 | jpeg
>> 19 | png
>> 1250014 | jpeg
>> ........ -> 226 rows in result.
>>
>> I do not understand why for second select I'm able to retrieve just a
>> part of the rows and not all rows.
>>
>> Not sure if this is related or not to the initial problem.
>>
>> Any help is really appreciated.
>> Thanks,
>> Gabi
>>
>>
>>
>>
>>
>
>
>
