GitHub user Mike-Animal-Counseling added a comment to the discussion: Implement a custom FileFormat
Arrow's **C++ Dataset API** supports this: implement a `FileFormat` backed by your own `FileWriter` and `FileWriteOptions`. The write side includes `MakeWriter()` and `DefaultWriteOptions()`; the existing formats are useful examples. Those interfaces are in [`file_base.h`](https://github.com/apache/arrow/blob/apache-arrow-25.0.0/cpp/src/arrow/dataset/file_base.h). If you're using **PyArrow**, this is a native-binding job rather than a Python callback interface. Its [`FileFormat` wrapper](https://github.com/apache/arrow/blob/apache-arrow-25.0.0/python/pyarrow/_dataset.pyx#L1295) holds a C++ format object. A plain Python subclass doesn't provide the native writer needed by `write_dataset`; you'd also need binding support for your implementation. For a Python-only pipeline, you can still use Arrow to stream the JSON input, then hand each batch to your own partition routing and binary writer: ```python import pyarrow.json as paj with paj.open_json('input.jsonl') as reader: for batch in reader: route_and_write_batch(batch) # Your application's function. ``` That function would own partition selection, file naming, and closing the custom writers. It doesn't plug the custom codec into Dataset's existing partitioned writer. If keeping that automatic Dataset machinery is the main requirement, the native `FileFormat` route is the one to investigate. I checked the 25.0.0 Python bindings and ran the [`open_json`](https://arrow.apache.org/docs/python/generated/pyarrow.json.open_json.html) batch-reading part locally. I haven't implemented your binary format or compiled a custom C++ writer. AI-assisted research and writing. GitHub link: https://github.com/apache/arrow/discussions/48069#discussioncomment-19283849 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
