Kengo Seki created AVRO-2562:
--------------------------------
Summary: Expose the JavaScript BlockEncoder's piped stream
Key: AVRO-2562
URL: https://issues.apache.org/jira/browse/AVRO-2562
Project: Apache Avro
Issue Type: New Feature
Components: javascript
Reporter: Kengo Seki
Assignee: Kengo Seki
The "createFileEncoder" test for the JavaScript implementation is flaky. It
sometimes fails in docker mode, as follows:
{code}
~/repos/avro$ git checkout master
~/repos/avro$ ./build.sh docker
sekikn@13d38772b8e6:~/avro$ ./build.sh veryclean
sekikn@13d38772b8e6:~/avro$ cd lang/js
sekikn@13d38772b8e6:~/avro/lang/js$ node -v
v6.17.1
sekikn@13d38772b8e6:~/avro/lang/js$ npm i
sekikn@13d38772b8e6:~/avro/lang/js$ for i in $(seq 1 100); do echo "#${i}"; if
! npm t; then break; fi; done
#1
> [email protected] test /home/sekikn/avro/lang/js
> mocha
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
381 passing (557ms)
(snip)
#7
> [email protected] test /home/sekikn/avro/lang/js
> mocha
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
380 passing (554ms)
1 failing
1) files createFileEncoder:
Uncaught AssertionError: 0 == 2
+ expected - actual
-0
+2
at BlockDecoder.<anonymous> (test/test_files.js:574:18)
at endReadableNT (_stream_readable.js:978:12)
at _combinedTickCallback (internal/process/next_tick.js:80:11)
at process._tickCallback (internal/process/next_tick.js:104:9)
npm ERR! Test failed. See above for more details.
{code}
And it always fails with Node.js v8.10.0, which is installed by apt-get on
Ubuntu 18.04 LTS.
{code}
~/repos/avro$ git checkout master
~/repos/avro$ cd lang/js
~/repos/avro/lang/js$ ./build.sh test
> [email protected] cover /home/sekikn/repos/avro/lang/js
> istanbul cover _mocha
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․
380 passing (537ms)
1 failing
1) files createFileEncoder:
Uncaught AssertionError [ERR_ASSERTION]: 0 == 2
+ expected - actual
-0
+2
at BlockDecoder.<anonymous> (test/test_files.js:574:18)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
=============================================================================
Writing coverage object [/home/sekikn/repos/avro/lang/js/coverage/coverage.json]
Writing coverage reports at [/home/sekikn/repos/avro/lang/js/coverage]
=============================================================================
=============================== Coverage summary ===============================
Statements : 100% ( 2529/2529 )
Branches : 100% ( 996/996 )
Functions : 100% ( 414/414 )
Lines : 100% ( 2443/2443 )
================================================================================
npm ERR! Linux 5.0.0-25-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "run" "cover"
npm ERR! node v8.10.0
npm ERR! npm v3.5.2
npm ERR! code ELIFECYCLE
npm ERR! [email protected] cover: `istanbul cover _mocha`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] cover script 'istanbul cover _mocha'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the avro-js package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! istanbul cover _mocha
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs avro-js
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls avro-js
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /home/sekikn/repos/avro/lang/js/npm-debug.log
{code}
In this test, {{createFileDecoder()}} is called in the event listener for the
{{'finish'}} event emitted from the {{encoder}} object.
{code:title=test/test_files.js}
553 test('createFileEncoder', function (cb) {
554 var type = createType({
555 type: 'record',
556 name: 'Person',
557 fields: [
558 {name: 'name', type: 'string'},
559 {name: 'age', type: 'int'}
560 ]
561 });
562 var path = tmp.fileSync().name;
563 var encoder = files.createFileEncoder(path, type);
564 encoder.write({name: 'Ann', age: 32});
565 encoder.end({name: 'Bob', age: 33});
566 var n = 0;
567 encoder.on('finish', function () {
568 files.createFileDecoder(path)
569 .on('data', function (obj) {
570 n++;
571 assert(type.isValid(obj));
572 })
573 .on('end', function () {
574 assert.equal(n, 2);
575 cb();
576 });
577 });
578 });
{code}
And the {{encoder}} object has the underlying writable stream created by
{{createWriteStream()}}.
{code:title=lib/files.js}
562 function createFileEncoder(path, schema, opts) {
563 var encoder = new BlockEncoder(schema, opts);
564 encoder.pipe(fs.createWriteStream(path, {defaultEncoding: 'binary'}));
565 return encoder;
566 }
{code}
In such a situation, the caller should wait for the underlying stream's
{{'finish'}} event rather than {{encoder}}'s. Because when the {{encoder}} has
finished to write and the caller catch its {{'finish'}} event, the underlying
stream perhaps doesn't finished to write into the disk yet (cf.
https://stackoverflow.com/questions/40864496/detecting-the-end-of-a-piped-stream-operation).
I think BlockEncoder should expose a reference to the underlying piped stream,
so that users can listen its event (typically {{'finish'}}).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
