[
https://issues.apache.org/jira/browse/FLINK-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-18915.
----------------------------------
Resolution: Fixed
Merged on master with 3a9a93053903dfefc3fb651e47800afc00d13a9e
and on release-1.11 with 393b13cfd103896e0cb60c6c7230101bef9a956f
> FIXED_PATH(dummy Hadoop Path) with WriterImpl may cause ORC writer OOM
> ----------------------------------------------------------------------
>
> Key: FLINK-18915
> URL: https://issues.apache.org/jira/browse/FLINK-18915
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Affects Versions: 1.11.0, 1.11.1
> Reporter: wei
> Assignee: Yun Gao
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> # OrcBulkWriterFactory
> {code:java}
> @Override
> public BulkWriter<T> create(FSDataOutputStream out) throws IOException {
> OrcFile.WriterOptions opts = getWriterOptions();
> opts.physicalWriter(new PhysicalWriterImpl(out, opts));
> return new OrcBulkWriter<>(vectorizer, new WriterImpl(null, FIXED_PATH,
> opts));
> }{code}
>
> # MemoryManagerImpl
> {code:java}
> //
> public void addWriter(Path path, long requestedAllocation,
> Callback callback) throws IOException {
> checkOwner();
> WriterInfo oldVal = writerList.get(path);
> // this should always be null, but we handle the case where the memory
> // manager wasn't told that a writer wasn't still in use and the task
> // starts writing to the same path.
> if (oldVal == null) {
> oldVal = new WriterInfo(requestedAllocation, callback);
> writerList.put(path, oldVal);
> totalAllocation += requestedAllocation;
> } else {
> // handle a new writer that is writing to the same path
> totalAllocation += requestedAllocation - oldVal.allocation;
> oldVal.allocation = requestedAllocation;
> oldVal.callback = callback;
> }
> updateScale(true);
> }
> {code}
> SinkTask may have multi BulkWriter create, FIXED_PATH will cause overlay the
> last writer callback；Last writer's WriterImpl#checkMemory will never called;
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
