[
https://issues.apache.org/jira/browse/CASSANDRA-11542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15266239#comment-15266239
]
Stefania commented on CASSANDRA-11542:
--------------------------------------
bq. Saw that we are always doing the conversion to CassandraRow with RDDs,
dataframes go directly to the internal SQL Type.
In the dataframe tests we also only retrieve the two columns for the
calculation rather than all columns . I described this above, sorry if it
wasn't clear.
bq.The code you presented looks good to me, there is the potential issue of
blocking on resultsets that take a long time to complete while other
result-sets are already on the driver but i'm not sure if this is a big deal.
Do you have any idea of the parallelization in these test? How many partitions
are the different runs generating?
Thanks for checking the code. The result set futures should not block because
the driver completes them as soon as they are transferred to the iterator's
thread. I'm actually using futures as a lazy way to also transfer error
conditions rather than just results.
In terms of parallelism, each C* node receives 256 token range queries per RDD
iteration. This should be fine since each node has 256 tokens. I've also
checked the spark tasks by connecting to the web UI at port 4040 and initially
I could see 10 tasks per Cassandra RDD operation, then they increased to 20
when I increased the number of executor cores to 4. I have 5 nodes with 2
executors each so the initial number 10 makes sense as by default there is one
core per executor, however I don't understand why I ended up with 20 rather
than 40 when I increased the number of cores to 4. {{spark-env.sh}} is
[here|https://github.com/stef1927/spark-load-perf/blob/master/bin/install_spark.sh#L34]
if you want to check it out but there's not much to it other than the number
of executor cores. I also note that the CSV and Parquet RDD operations have as
many tasks as there are HDFS partitions, so 1000 tasks. This would give them a
big advantage if we have cores idle but I don't know how to reliably increase
tasks for C* RDDs.
I've collected JFR files for both Cassandra and the Spark executors:
[^jfr_recordings.zip]. I still need to analyze them but from a quick look there
are at least two interesting things client side (plus maybe a third one): we
seem to spend a lot of time in {{CassandraRow._indexOfOrThrow()}} and in
selecting the codecs in the driver. As for the C* JFR recorder, we spend 80% in
the new bulk read code but we also still spend 15% of time in
{{ReadCommandVerbHandler}}, which I don't understand.
I will post another update when I have more details on the JFR analysis and any
optimizations that might follow.
> Create a benchmark to compare HDFS and Cassandra bulk read times
> ----------------------------------------------------------------
>
> Key: CASSANDRA-11542
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11542
> Project: Cassandra
> Issue Type: Sub-task
> Components: Testing
> Reporter: Stefania
> Assignee: Stefania
> Fix For: 3.x
>
> Attachments: jfr_recordings.zip, spark-load-perf-results-001.zip,
> spark-load-perf-results-002.zip
>
>
> I propose creating a benchmark for comparing Cassandra and HDFS bulk reading
> performance. Simple Spark queries will be performed on data stored in HDFS or
> Cassandra, and the entire duration will be measured. An example query would
> be the max or min of a column or a count\(*\).
> This benchmark should allow determining the impact of:
> * partition size
> * number of clustering columns
> * number of value columns (cells)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
