ftschafer opened a new issue, #1325:
URL: https://github.com/apache/arrow-go/issues/1325
### Describe the usage question you have. Please include as many useful
details as possible.
We decode a long sequence of independent, self-contained IPC blobs (schema +
record batches) that arrive one at a time over a transport. Only the first blob
carries the schema; later blobs are record-batches-only. We reused a single
ipc.Reader, swapping the underlying io.Reader between blobs, and it worked
through v18.7.0.
On v18.8.0 the second blob onward decodes as zero rows. Reading a blob to
io.EOF sets r.done = true, and next() now early-exits on done:
```
func (r *Reader) next() bool {
...
if r.err != nil || r.done { return false }
```
done is sticky with no public reset, so the reused reader short-circuits
from the second blob on. This looks like it came from the change that started
rejecting non-dictionary initial messages
(https://github.com/apache/arrow-go/pull/1054/files).
Therefore, we have some questions:
- Was this behavior change intended to ship in a minor release (v18.7 →
v18.8)? It's source-breaking for the reuse pattern, which reads unusual for a
minor bump.
- If reuse is not supported, what's the recommended way to decode this "many
small streams, schema known once" pattern? Right now our options seem to be a
fresh Reader per blob plus re-injecting a serialized schema message in front of
every schema-less blob, or reaching into unexported state. Is there a better
approach we're missing?
### Component(s)
Release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
