Re: Improving integration test coverage of I/O transforms

2017-11-17 Thread Chamikara Jayalath
I created following JIRAs. Integration test for TextIO ReadAll and dynamic writes: https://issues.apache.org/jira/browse/BEAM-3211 Integration test for MongoDBIO: https://issues.apache.org/jira/browse/BEAM-3212 Large scale performance test for MongoDBIO: https://issues.apache.org/jira/browse/BEAM-

Re: Improving integration test coverage of I/O transforms

2017-11-15 Thread Chamikara Jayalath
Sounds good. I'll create JIRAs for both small scale and large scale versions. To be clear, large scale versions will use PerfKitBenchmarker based performance testing framework which will allow us to (1) manage Kubernetes based multi-node deployments of data stores (2) publish/track benchmark result

Re: Improving integration test coverage of I/O transforms

2017-11-14 Thread Jean-Baptiste Onofré
Hi Cham, Some are in progress on private branch (at least for MongoDbIO ;)). +1 to create the Jira, we will assign to the right persons. Regards JB On 11/15/2017 01:09 AM, Chamikara Jayalath wrote: Hi All, I went through the current list of I/O transforms offered by Beam Java SDK and it look

Improving integration test coverage of I/O transforms

2017-11-14 Thread Chamikara Jayalath
Hi All, I went through the current list of I/O transforms offered by Beam Java SDK and it looks like we currently don't have enough integration test coverage for some of the I/O transforms. For example, it might be good to improve test coverage of following (not a complete list). MongoDbIO Read/W