https://issues.apache.org/bugzilla/show_bug.cgi?id=52086
Bug #: 52086
Summary: SSTRecord.serialize() performance improvement patch
for huge hssf output
Product: POI
Version: 3.7
Platform: PC
Status: NEW
Severity: enhancement
Priority: P2
Component: HSSF
AssignedTo: dev@poi.apache.org
ReportedBy: kuz+...@altpaper.net
Classification: Unclassified
Created attachment 27844
--> https://issues.apache.org/bugzilla/attachment.cgi?id=27844
zip file containing patch and chart image
This bug report is an extracted abstract from the mail 'SSTRecord.serialize()
performance improvement patch for huge hssf output' in the 'dev@poi.apache.org'
ML.
Please use the 'patch' program to patch the files, such as 'patch -p 0
< HOGE.patch'.
I tried to use eclipse to patch them, but I found that any trial will fail.
---
This patch contains some output performance hacks
around the packages of org.apache.poi.hssf.record.cont and
org.apache.poi.util.
The patch provides both 2~4x performance improvement and some conveniences
in the serialization of SST.
The essential feature of patch is to extend the LittleEndianOutput (
and the implementation classes )
for itself (themselves) to write out the String in the both formats of
ASCII and UTF16LE.
This extension internalizes the frequent polymorphism calls of
UnknownLengthRecordOutput#writeShort() or writeByte()
in the ContinuableRecord#writeCharacterData().
The call internalization enables the jvm to avoid the polymorphism
cost along the technique of code inlining per class .
Furthermore, the template adapters of this extension are provided by
LittleEndianOutputAdapter, LittleEndianOutputByteStreamAdatper and
LittleEndianOutputFilterAdapter
to ease to build up the implementation class of LittleEndianOutput.
By using the class tree, I implemented
LittleEndianOutputBufferedRandomAccessFile for the performance check
needs,
which uses the random access file coupled with buffers as the output
destination and
also supports the DelayableLittleEndianOutput interface.
The features of patch can be enabled and disabled by flipping the two
boolean flags of
ContinuableRecordOutput.useFasterWrite and
UnknownLengthRecordOutput.useFasterWrite.
The performances at changing these flags are investigated below.
You can verify the correctness of this patch
by running the test 'testSSTRecord_DigestCheck()' contained in the
previous 'first-sstser-verify37.patch'
for each example serialization method.
The example methods are Memory, DirectRandomAccessFile, StreamFile and
LZFCompressFile in 'SerializationFunction'.
Please be careful to use the LZFCompressFile serialization method because
it requires the compress-lzf-0.8.4.jar or upper which can be fetched
from the maven repository.
See
http://www.jarvana.com/jarvana/archive-details/com/ning/compress-lzf/0.8.4/compress-lzf-0.8.4.jar
for the more details of jar.
The performance of this patch is investigated in the table below by
invoking the newly created method
TestSSTRecord.testSSTRecordPerformance()
with the small code change from 'int N = 1<<10' to 'int N= 1<<20'
under the jdk(1.6.0_26) of option '-Xmx1224m -server'.
The elapsed time of 2^20 SSTReocrds serialization in seconds are
measured 60 times.
Then, the statistics is calculated by excluding 40 extreme
measurements for each serialization, avoiding ill measurements.
The value and plus minus sign represent the mean and the standard
deviation of serialization time.
java: oracle jdk 1.6.0_26
option: -Xmx1224m -server
cpu: Intel core i5-2400
OS: windows 7
Optimization enum_SerializeFunction Mean Time Standard
Deviation
--- --- --- -- ---
U@T/C@T Memory 0.248 +- 0.002 secs
U@T/C@T DirectRandomAccessFile 0.94 +- 0.024 secs
U@T/C@T StreamFile 0.936 +- 0.072 secs
U@T/C@T LZFCompressFile 0.362 +- 0.002 secs
--- --- --- -- ---
U@F/C@T Memory 0.213 +- 0.002 secs
U@F/C@T DirectRandomAccessFile 0.881 +- 0.046 secs
U@F/C@T StreamFile 0.827 +- 0.039 secs
U@F/C@T LZFCompressFile 0.438 +- 0.004 secs
--- --- --- -- ---
U@T/C@F Memory 0.744 +- 0.001 secs
U@T/C@F DirectRandomAccessFile 0.939 +- 0.029 secs
U@T/C@F StreamFile 0.901 +- 0.031 secs
U@T/C@F LZFCompressFile 0.658 +- 0.002 secs
--- --- --- -- ---
U@F/C@F Memory 1.011 +- 0.005 secs
U@F/C@F DirectRandomAccessFile 1.29 +- 0.003 secs
U@F/C@F StreamFile 0.837 +- 0.039 secs
U@F/C@F LZFCompressFile 0.902 +- 0.003 secs
--- --- --- -- ---
MANUAL_HACK Memory 0.237 +- 0.002 secs
MANUAL_HACK DirectRandomAccessFile 1.174 +- 0.094 secs
MANUAL_HACK StreamFile 0.806 +- 0.042 secs
MANUAL_HACK LZFCompressFile 0.384 +- 0.002 secs
Please see the 'ser_perf.png' image attached, which
contains a chart of the mean time for each serialize method in the
optimizations.
The 'U@[TF]/C@[TF]' of 'Optimization' column indicates whether the
flags of UnknownLengthRecordOutput.useFasterWrite and
ContinuableRecordOutput.useFasterWrite
are TRUE or FALSE.
In this case, the 'U@F/C@F' is identical to the original method.
The values of 'MANUAL' rows mean the ones in which the current poi
class trees around the SSTSerializer are fully refactored and
optimized as well as possible.
#Note.. The 'MANUAL' code is not included in this patch because the
class tree is too much changed.
>From the table and chart,
I concluded that the method
'ContinuableRecordOutput.useFasterWrite=true' and
'UnknownLengthRecordOutput.useFasterWrite=true'
is 2~4 times faster than the original method in the cpu dependent
cases such as the Memory and LZFCompressedFile.
Furthermore, from the result of full manual hack, the performance is
reasonable and optimal for the small source code changes of patch.
On the other hand, the worse performances in the disk dependent cases,
such as DirectRandomAccessFile and StreamFile, are required to be
fixed.
Nevertheless the patch cannot solve the cases.
In my experience, the disk resource are more limited in a server
processing than the cpu resource
although it is required to improve the disk performance in the disk
dependent cases.
These facts indicate that the compressed writer such as
LZFCompressedFile is necessary to improve the throughput of huge
'.xls's in a server.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@poi.apache.org
For additional commands, e-mail: dev-h...@poi.apache.org
