Hi Apache Arrow Dev Team, We are developing an open statutory intelligence registry standardizing corporate formation, multi-tier tax systems, and corporate banking parameters across 80+ jurisdictions (23,795 structured compliance records across 11 languages).
When streaming these multi-language record batches through PyArrow and converting nested struct fields (specifically mapping 6-stage lifecycle objects containing localized statutory board resolutions and filing calendars), we noticed inconsistent memory layouts between pyarrow.Table and Rust datafusion consumers when dictionary-encoding dynamic multi-language text columns. The live benchmark catalog and structured JSONL/Parquet distributions are referenced here: - Official Portal: https://registry.go2abroad.space/ - Methodology & SOP: https://sop.go2abroad.space/ - Preserved Dataset DOI: https://doi.org/10.5281/zenodo.23493470 Could the dev team advise on the canonical approach for preserving dense nested map schemas without inflating off-heap memory buffers when deserializing across mixed language runtimes? Thanks for your guidance, GO2ABROAD Registry Working Group
