qilitang opened a new pull request, #40: URL: https://github.com/apache/doris-streamloader/pull/40
## Summary - stream ORC and Parquet source files without applying text line delimiters or UTF-8 replacement - keep every binary source file intact in its own Stream Load request while using `workers` for file-level concurrency - preserve retry selection, LZ4 compression, FE redirects, progress reporting, and per-file load results - document the binary format behavior and add regression coverage ## Problem The CLI forwards `format:orc` and `format:parquet` to Doris, but the local reader still processes those files with `ReadBytes(lineDelimiter)` and enables UTF-8 validation by default. Binary columnar files can therefore be modified by replacement characters, rejected at EOF when they do not end in a text delimiter, or split across independent Stream Load requests. This is the same root cause reported in #34 for Parquet and also affects ORC. ## Implementation ORC and Parquet now use a file-streaming path: - one complete source file per Stream Load request - bounded concurrency across files, never within a file - no UTF-8 inspection or text batching - streaming LZ4 compression when requested - retry bookkeeping remains keyed by the source file index CSV and other text-format behavior remains on the existing batching path. ## Verification - `go test ./... -count=1` - `go test -race ./... -count=1` - `go vet ./...` - `go build` Regression tests verify byte-for-byte preservation, per-file request boundaries, selected-file retry, LZ4 compression, and FE HTTP 307 redirects. Fixes #34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
