Steven Aerts created AVRO-3156:
----------------------------------
Summary: Performance degradation in SpecificRecordBuilder
introduced in 1.9.0
Key: AVRO-3156
URL: https://issues.apache.org/jira/browse/AVRO-3156
Project: Apache Avro
Issue Type: Improvement
Components: java
Affects Versions: 1.10.0, 1.9.0
Environment: Using SpecificData in environments with multiple
classloaders.
Reporter: Steven Aerts
Attachments: image-2021-06-11-14-27-16-689.png
The change introduced in spark 1.9.0 which changed:
{code:java}
SpecificData.get(){code}
into:
{code:java}
SpecificData.getForSchema(schema){code}
introduced a significant performance degradation in environments where the
class of {{schema}} is provided by a different classloader then the classloader
containing {{SpecificData}}.
A possible solution is to use the {{classCache}} of the default
{{SpecificData}} so the sometimes expensive classloader codepath is cached. (PR
coming up)
We noticed this in after trying out a spark upstep from spark 3.1.0 (avro
1.8.2) to 3.2.0 (spark 1.10.2) where 74% of the time was spend in millions of
times resolving the same class.
With this patch this resolving time was brought back from 74% to 0.70%.
JMC flamegraph showing this issue:
!image-2021-06-11-14-27-16-689.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
