Re: heavy insert load overloads CPUs, with MutationStage pending

[Keith Freeman]

Paul-  Sorry to go off-list but I'm diving pretty far into details 
here.  Ignore if you wish.

Thanks a lot for the example, definitely very helpful.  I'm surprised 
that the Cassandra experts aren't more interested-in/alarmed-by our 
results, it seems like we've proved that insert performance for wide 
rows in CQL is enormously worse than it was before CQL.  And I have a 
feeling 2.0 won't help much -- I'm already using entirely-prepared batches.

To reproduce your example, I switched to cassandra 1.2.6  and astyanax 
1.56.42.  But anything I try to do with that version combination gives 
me an exception on the client side (e.g. execute() on a query):
13-09-13 15:42:42.511 [pool-6-thread-1] ERROR 
c.n.a.t.ThriftSyncConnectionFactoryImpl - Error creating connection
java.lang.NoSuchMethodError: 
org.apache.cassandra.thrift.TBinaryProtocol: method 
<init>(Lorg/apache/thrift/transport/TTransport;)V not found
    at 
com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection.open(ThriftSyncConnectionFactoryImpl.java:195) 
~[astyanax-thrift-1.56.37.jar:na]
    at 
com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$ThriftConnection$1.run(ThriftSyncConnectionFactoryImpl.java:232) 
[astyanax-thrift-1.56.37.jar:na]
    at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_07]
From my googling this is due to a cassandra API change in 
TBinaryProtocol, which is why I had to use cassandra 1.2.5 jars to get 
my astyanax client to work at all in my earlier experiments. Did you 
encounter this?  Also, you had 1.2.8 in the stackoverflow post, but 
1.2.6 in this email, did you have to rollback?

Thanks for any help you can offer, hope I can return the favor at some 
point.


On 09/12/2013 02:26 PM, Paul Cichonski wrote:
I'm running Cassandra 1.2.6 without compact storage on my tables. The trick is 
making your Astyanax (I'm running 1.56.42) mutation work with the CQL table 
definition (this is definitely a bit of a hack since most of the advice says 
don't mix the CQL and Thrift APIs so it is your call on how far you want to 
go). If you want to still try and test it out you need to leverage the Astyanax 
CompositeColumn construct to make it work 
(https://github.com/Netflix/astyanax/wiki/Composite-columns)

I've provided a slightly modified version of what I am doing below:

CQL table def:

CREATE TABLE standard_subscription_index
(
        subscription_type text,
        subscription_target_id text,
        entitytype text,
        entityid int,
        creationtimestamp timestamp,
        indexed_tenant_id uuid,
        deleted boolean,
     PRIMARY KEY ((subscription_type, subscription_target_id), entitytype, 
entityid)
)

ColumnFamily definition:

private static final ColumnFamily<SubscriptionIndexCompositeKey, 
SubscribingEntityCompositeColumn> COMPOSITE_ROW_COLUMN = new 
ColumnFamily<SubscriptionIndexCompositeKey, SubscribingEntityCompositeColumn>(
        SUBSCRIPTION_CF_NAME, new 
AnnotatedCompositeSerializer<SubscriptionIndexCompositeKey>(SubscriptionIndexCompositeKey.class),
        new 
AnnotatedCompositeSerializer<SubscribingEntityCompositeColumn>(SubscribingEntityCompositeColumn.class));


SubscriptionIndexCompositeKey is a class that contains the fields from the row key (e.g., 
subscription_type, subscription_target_id), and SubscribingEntityCompositeColumn contains 
the fields from the composite column (as it would look if you view your data using 
Cassandra-cli), so: entityType, entityId, columnName. The columnName field is the tricky 
part as it defines what to interpret the column value as (i.e., if it is a value for the 
creationtimestamp the column might be "someEntityType:4:creationtimestamp"

The actual mutation looks something like this:

final MutationBatch mutation = getKeyspace().prepareMutationBatch();
final ColumnListMutation<SubscribingEntityCompositeColumn> row = 
mutation.withRow(COMPOSITE_ROW_COLUMN,
                new SubscriptionIndexCompositeKey(targetEntityType.getName(), 
targetEntityId));

for (Subscription sub : subs) {
        row.putColumn(new 
SubscribingEntityCompositeColumn(sub.getEntityType().getName(), 
sub.getEntityId(),
                                "creationtimestamp"), 
sub.getCreationTimestamp());
        row.putColumn(new 
SubscribingEntityCompositeColumn(sub.getEntityType().getName(), 
sub.getEntityId(),
                                "deleted"), sub.isDeleted());
        row.putColumn(new 
SubscribingEntityCompositeColumn(sub.getEntityType().getName(), 
sub.getEntityId(),
                                "indexed_tenant_id"), tenantId);
}

Hope that helps,
Paul