hanke580 opened a new issue, #51368:
URL: https://github.com/apache/arrow/issues/51368
### Describe the bug, including details regarding any error messages,
version, and platform.
`arrow::csv` silently drops one byte when a CRLF inside a quoted field
straddles a parser
block boundary. The `\r` is kept and the `\n` disappears. No error, no
warning — the value
simply comes back one byte shorter than it was written.
This reproduces on **Arrow C++ `main`**, in C++, with no Python in the
picture, and at the
**shipped default block size** with no options tuned.
**Build under test**
```
Arrow C++ main @ 0bd8def ("GH-51343: [C++][CI][Packaging] Disable
Precompile Headers
on simdjson...", 2026-09-17)
libarrow.so.2600.0.0, 26.0.0-SNAPSHOT, Release build
cmake 4.4.3 / g++ 11.4.0 / Ubuntu 22.04 x86_64
```
Also reproduces on released **pyarrow 25.0.1** (newest on PyPI), on all
three entry points —
`csv.read_csv`, the streaming `csv.open_csv`, and the dataset API — which is
what places it
in the shared reader rather than in a wrapper.
**Minimal case.** A CSV whose row 0 holds a quoted value containing one
embedded CRLF.
`newlines_in_values = true` is required for such a file; without it the
reader rejects the
file outright (`Expected 3 columns, got 2`) rather than corrupting it.
```
id,s,p
0,"zzzz…zzzA<CR><LF>B",x
1,plain,x
2,plain,x
```
Walking that CRLF one byte at a time across a 1024-byte block boundary, 1028
offsets tried
on `main`:
```
block_size = 1024, walking one embedded CRLF across the boundary
pad=1012: CR at byte 1023, LF at byte 1024, boundary at 1024
wrote 1016B ...A\r\n read 1015B ...zA\r
1 corrupting offset(s), 0 error(s), out of 1028 tried.
```
Exactly one offset out of 1028: the one where **the CR is the last byte of a
block and the
LF is the first byte of the next**.
**It is not an artefact of a small block size.** With no `ReadOptions` at
all, so the shipped
1 MiB block applies, on `main`:
```
file bytes CR at byte verdict
1048602 1048575 CORRUPT: LF silently dropped
1048603 1048576 ok
1048604 1048577 ok
1048605 1048578 ok
1048606 1048579 ok
```
**It is specific to the CR–LF pair.** Same sweep on `main`, varying the byte
sequence placed
inside the quoted field (1032 offsets each):
| sequence inside the quoted field | corrupting offsets |
|---|---:|
| `A\r\nB` (embedded CRLF) | **1** |
| `A\r\r\nB` (CR CR LF) | **1** |
| `A\nB` (lone LF) | 0 |
| `A\rB` (lone CR) | 0 |
| `A\n\rB` (LF then CR) | 0 |
| `A€B` (3-byte UTF-8) | 0 |
A lone `\n` or a lone `\r` at exactly the same boundary survives; only the
two sequences that
contain a `\r` immediately followed by `\n` lose a byte, and the CR-CR-LF
case is the same
defect one byte over. So this is not general boundary mishandling — the
doubled-quote escape
and multi-byte UTF-8 paths are sound across the boundary.
### Reproduction
**C++ (this is the one that runs against `main`).** Build Arrow with
`-DARROW_CSV=ON`, then:
```cpp
#include <arrow/csv/api.h>
#include <arrow/io/api.h>
#include <arrow/table.h>
#include <arrow/array.h>
#include <fstream>
#include <cstdio>
int main() {
const std::string PREFIX = "id,s,p\n0,\"", VAL = "A\r\nB", PATH =
"/tmp/f.csv";
const int block = 1024;
for (int pad = block - 1020; pad < block + 8; pad++) {
{ std::ofstream f(PATH, std::ios::binary);
f << PREFIX << std::string(pad, 'z') << VAL <<
"\",x\n1,plain,x\n2,plain,x\n"; }
auto file = arrow::io::ReadableFile::Open(PATH).ValueOrDie();
auto ro = arrow::csv::ReadOptions::Defaults(); ro.block_size = block;
auto po = arrow::csv::ParseOptions::Defaults(); po.newlines_in_values =
true;
auto co = arrow::csv::ConvertOptions::Defaults();
co.column_types = {{"id", arrow::int32()}, {"s", arrow::utf8()}, {"p",
arrow::utf8()}};
auto table =
arrow::csv::TableReader::Make(arrow::io::default_io_context(), file,
ro, po,
co).ValueOrDie()->Read().ValueOrDie();
auto col = table->GetColumnByName("s");
std::string got;
for (int i = 0; i < col->num_chunks(); i++) {
auto a = std::static_pointer_cast<arrow::StringArray>(col->chunk(i));
if (a->length() > 0) { got = a->GetString(0); break; }
}
std::string want = std::string(pad, 'z') + VAL;
if (got != want)
printf("pad=%d: wrote %zuB, read %zuB -> LF dropped\n", pad,
want.size(), got.size());
}
}
```
Prints exactly one line, at `pad=1012`. Drop `ro.block_size = block` and use
`pad = (1<<20) - PREFIX.size() - 2` to see the same at the shipped default.
**Python (released pyarrow 25.0.1), for convenience:**
```python
import pyarrow as pa, pyarrow.csv as pc
PREFIX, VAL = b'id,s,p\n0,"', "A\r\nB"
def check(pad, block):
with open("/tmp/f.csv", "wb") as fh:
fh.write(PREFIX + b"z" * pad + VAL.encode() +
b'",x\n1,plain,x\n2,plain,x\n')
t = pc.read_csv("/tmp/f.csv",
read_options=pc.ReadOptions(block_size=block) if block
else pc.ReadOptions(),
parse_options=pc.ParseOptions(newlines_in_values=True),
convert_options=pc.ConvertOptions(
column_types={"id": pa.int32(), "s": pa.string(),
"p": pa.string()}))
return t.column("s").to_pylist()[0] == "z" * pad + VAL
print(check(1012, 1024)) # False <- LF dropped
print(check((1 << 20) - len(PREFIX) - 2, None)) # False, at the shipped 1
MiB block
```
`polars` and `duckdb` read the identical bytes back correctly, so the file
is well formed and
this is the reader, not the writer.
### Where it comes from (a lead, not a confirmed diagnosis)
I have not bisected this to a single line, but the shape of the failure
matches a specific
pattern that appears in both the chunker and the parser: a `\r` handler that
looks ahead one
byte for the `\n` and gives up if the byte is not in the current buffer.
`cpp/src/arrow/csv/chunker.cc`, in the lexer's unquoted-field path:
```cpp
if (ARROW_PREDICT_FALSE(c == '\r')) {
if (ARROW_PREDICT_TRUE(data != data_end) && *data == '\n') {
data++;
}
goto LineEnd; // CR at the end of the buffer => treated as a complete
terminator
}
```
`cpp/src/arrow/csv/parser.cc` has the same look-ahead guard (`data <
data_end && *data ==
'\n'`), and additionally treats a `\n` that is the *first* byte it sees as
an empty line:
```cpp
// Special case empty lines: do we start with a newline separator?
c = *data;
if (ARROW_PREDICT_FALSE(IsControlChar(c))) {
...
if (c == '\n') {
data++;
goto EmptyLine;
}
}
```
A CR that ends a buffer is therefore accepted as a whole line terminator,
and the orphaned LF
that opens the next buffer is then consumed as a separator rather than as
data. That is
consistent with every observation above: it needs the pair, it needs the
split to fall
between the two bytes, and a lone CR or lone LF at the same offset is
unaffected.
`FindFirstInternal` in `chunker.cc` runs the lexer over `partial` and then
over `block`
(`cpp/src/arrow/csv/chunker.cc`, `LexingBoundaryFinder`), which is the seam
where the
carried-over state would need to remember "the previous buffer ended on a
CR" for the pair to
be reassembled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
