[
https://issues.apache.org/jira/browse/BEAM-6952?focusedWorklogId=221344&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-221344
]
ASF GitHub Bot logged work on BEAM-6952:
----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Apr/19 16:27
Start Date: 01/Apr/19 16:27
Worklog Time Spent: 10m
Work Description: dlesco commented on pull request #8187: [BEAM-6952]
concatenated compressed files bug with python sdk
URL: https://github.com/apache/beam/pull/8187
The Python apache_beam.io.filesystem module has a bug handling concatenated
compressed files.
This PR has two commits:
1. a new unit test that shows the problem
2. a fix to the problem.
The unit test is added to the apache_beam.io.filesystem_test module. It was
added to this module because the test:
apache_beam.io.textio_test.test_read_gzip_concat does not encounter the problem
in the Beam 2.11 and earlier code base because the test data is too small: the
data is smaller than read_size, so it goes through logic in the code that
avoids the problem in the code. So, this test sets read_size smaller and test
data bigger, in order to encounter the problem. It would be difficult to test
in the textio_test module, because you'd need very large test data because
default read_size is 16MiB, and the ReadFromText interface does not allow you
to modify the read_size.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [ ] [**Choose
reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and
mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
<br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)
| --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/)
| --- | ---
See
[.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md)
for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 221344)
Time Spent: 10m
Remaining Estimate: 0h
> concatenated compressed files bug with python sdk
> -------------------------------------------------
>
> Key: BEAM-6952
> URL: https://issues.apache.org/jira/browse/BEAM-6952
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Affects Versions: 2.11.0
> Reporter: Daniel Lescohier
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The Python apache_beam.io.filesystem module has a bug handling concatenated
> compressed files.
> The PR I will create has two commits:
> # a new unit test that shows the problem
> # a fix to the problem.
> The unit test is added to the apache_beam.io.filesystem_test module. It was
> added to this module because the test:
> apache_beam.io.textio_test.test_read_gzip_concat does not encounter the
> problem in the Beam 2.11 and earlier code base because the test data is too
> small: the data is smaller than read_size, so it goes through logic in the
> code that avoids the problem in the code. So, this test sets read_size
> smaller and test data bigger, in order to encounter the problem. It would be
> difficult to test in the textio_test module, because you'd need very large
> test data because default read_size is 16MiB, and the ReadFromText interface
> does not allow you to modify the read_size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
