Hey Ryan,
Yes, the IPC primitives were designed explicitly for the use-case you're
describing.
Rather than building on Observables, they use on a similar fundamental
primitive native to JS, AsyncIterables. You may already be familiar with
AsyncIterables, as they're returned by async generator functions (the
`async function*` syntax), and are consumed via `for await...of` loops.
AsyncIterables are async-pull streams, i.e the consumer pulls a Promise
on demand, awaits the Promise to unwrap the value, repeat. This is in
contrast to the Observable, where the producer pushes values on demand
to the consumer without considering whether the consumer has capacity
process the value.
AsyncIterables map 1-1 to native node streams, as well as browser WhatWG
DOM streams <https://github.com/whatwg/streams/blob/main/FAQ.md>, and we
provide methods to convert to either of those if you need. It's also
possible to make an AsyncIterable from an Observable source (with a bit
of buffering at the junction), and an Observable from an AsyncIterable
(no buffering required).
There are quite a few options baked into the IPC RecordBatchReader and
RecordBatchWriter for handling advanced use-cases, but I've put together
this small example to illustrate some of the basics.
https://codepen.io/trxcllnt/pen/OJReoeW
This example uses IxJS <https://github.com/ReactiveX/IxJS>, the sister
library to RxJS for AsyncIterables. If you're familiar with Rx, Ix
should feel similar.
I also have a number of other repositories that can serve as examples
for reading/writing Arrow IPC streams:
https://github.com/trxcllnt/fastify-arrow
https://github.com/trxcllnt/arrow-to-parquet-js
https://github.com/trxcllnt/csv-to-arrow-js
If you need to go even lower-level, the Arrow repository has few
debugging utilities that use more of the IPC internals:
bin/print-buffer-alignment.js
<https://github.com/apache/arrow/blob/919980184fe2b27063adec0d0908c75cd17a8437/js/bin/print-buffer-alignment.js>
src/bin/arrow2csv.ts
<https://github.com/apache/arrow/blob/919980184fe2b27063adec0d0908c75cd17a8437/js/src/bin/arrow2csv.ts>
If you have Arrow installed locally in a project, you can use the above
script via `npx` to view a table from the command line:
$ cat ./some-table.arrow | npx arrow2csv
Feel free to reach out or @ me on GitHub if you have more questions
about the Grafana integration.
Best,
Paul
On 1/24/21 4:21 PM, Brian Hulette wrote:
+Paul Taylor <mailto:ptay...@apache.org> would your work with whatwg
streams be relevant here? Are there any examples that would be useful
for Ryan?
Brian
On Sat, Jan 23, 2021 at 4:52 PM Ryan McKinley <ryan...@gmail.com
<mailto:ryan...@gmail.com>> wrote:
Hello-
I am exploring options to support streaming in grafana. We have a
golang
websocket server and am exploring options to send data to the browser.
Are there any good examples of reading IPC data with callbacks for
each
block? I see examples for mapd, and for reading whole tables --
but am
hoping for something that lets me read initial header data, then
get each
record batch as a callback (rxjs)
https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format
Thanks for any pointers
Ryan
