[
https://issues.apache.org/jira/browse/CASSANDRA-5544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13669054#comment-13669054
]
Shamim Ahmed commented on CASSANDRA-5544:
-----------------------------------------
Alex, thank you very much for your quick response.
However, i am afraid that above patch will not solve the problem i described
"we got more than 100 maps but 2 of them (always two maps) got very large Map
input records and runs more than hours - point 4" - this behavior is
unexpected. This means Map input records is not evenly through cluster, most of
the maps getting Map input records = 10000 but only two of them getting more
than millions.
Certainly i will do some test through thrift api as you described.
One more things, would you kindly allows user to define PIG_INPUT_SPLIT_SIZE
through cassandra store URL as "STORE updated INTO
'cassandra://KEYSPACE/CF?allow_deletes=true&PIG_INPUT_SPLIT_SIZE=xxxxxx' USING
CassandraStorage()" instead of system environment.
> Hadoop jobs assigns only one mapper in task
> --------------------------------------------
>
> Key: CASSANDRA-5544
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5544
> Project: Cassandra
> Issue Type: Bug
> Components: Hadoop
> Affects Versions: 1.2.1
> Environment: Red hat linux 5.4, Hadoop 1.0.3, pig 0.11.1
> Reporter: Shamim Ahmed
> Assignee: Alex Liu
> Attachments: 5544-1.txt, 5544.txt, Screen Shot 2013-05-26 at 4.49.48
> PM.png
>
>
> We have got very strange beheviour of hadoop cluster after upgrading
> Cassandra from 1.1.5 to Cassandra 1.2.1. We have 5 nodes cluster of
> Cassandra, where three of them are hodoop slaves. Now when we are submitting
> job through Pig script, only one map assigns in task running on one of the
> hadoop slaves regardless of
> volume of data (already tried with more than million rows).
> Configure of pig as follows:
> export PIG_HOME=/oracle/pig-0.10.0
> export PIG_CONF_DIR=${HADOOP_HOME}/conf
> export PIG_INITIAL_ADDRESS=192.168.157.103
> export PIG_RPC_PORT=9160
> export PIG_PARTITIONER=org.apache.cassandra.dht.Murmur3Partitioner
> Also we have these following properties in hadoop:
> <property>
> <name>mapred.tasktracker.map.tasks.maximum</name>
> <value>10</value>
> </property>
> <property>
> <name>mapred.map.tasks</name>
> <value>4</value>
> </property>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
