cj-zhukov opened a new issue, #27515:
URL: https://github.com/apache/datafusion/issues/27515

   ### Is your feature request related to a problem or challenge?
   
   There is no way to turn a struct, or a set of columns, into a JSON string 
column without leaving the lazy plan: today this needs `collect()`, serializing 
the batches with `arrow_json::ArrayWriter` and rebuilding the DataFrame, which 
materializes the table and discards the original plan (see the helper in 
#26185).
   
   #26185 asked for this on the DataFrame API. It was closed when the 
Spark-compatible functions moved to their own repository, and a 
**Spark-flavored** `to_json` has since landed there 
(apache/datafusion-spark#85, apache/datafusion-spark#86). That function follows 
Spark's `JacksonGenerator` rules (Java `Double.toString` numbers, Base64 
binary, millisecond timestamps in the session time zone, `"NaN"` as a string, 
...), which is right for Spark users but not what a DataFusion user expects 
from DataFusion's own JSON output.
   
   DataFusion itself still has no `to_json`: `named_struct` / `struct` build 
the value, `COPY ... TO 'file.json'` can write it, but there is no expression 
that returns the JSON text as a column.
   
   ### Describe the solution you'd like
   
   A `to_json(expression)` scalar function in `datafusion-functions` (`core`, 
next to `named_struct`) whose output is exactly what DataFusion's JSON writer 
produces for the value, i.e. arrow-json with its default options:
   
   - a struct becomes an object keyed by its field names, a list an array, a 
map an object, a scalar its JSON form; nested values are serialized recursively
   - NULL struct fields and NULL map entries are omitted (as in `COPY TO 
JSON`), NULL list elements are written as `null`, a NULL input returns NULL
   - numbers as arrow-json writes them (`0.00001`, `1.0e10`, decimals with 
their scale), non-finite floats as `null`, binary as hexadecimal, timestamps in 
RFC 3339 form in the type's time zone
   - types without a JSON form (unions, maps with non-string keys) are rejected 
when the query is planned
   
   ```sql
   SELECT id, to_json(named_struct('foo', foo, 'bar', bar)) AS json FROM t;
   ```
   
   ```rust
   use datafusion::prelude::*;
   
   let df = df.with_column("json", to_json(named_struct(vec![lit("foo"), 
col("foo"), lit("bar"), col("bar")])))?;
   // or an existing struct column
   let df = df.with_column("json", to_json(col("event")))?;
   ```
   
   The result stays a lazy projection: each batch is serialized with the 
arrow-json row encoder, nothing is collected. The shape follows the merged 
Spark implementation (apache/datafusion-spark#86), without the Spark-specific 
encoders.
   
   Not in scope for a first version: an options argument (explicit nulls, 
date/time formats).
   
   ### Describe alternatives you've considered
   
   - Keeping the `collect()`-based helper in user code: works, but executes the 
plan and loses laziness, filter pushdown and the original lineage.
   - `arrow_cast(struct_col, 'Utf8')`: produces Arrow's display format (`{a: 1, 
b: x}`), not JSON.
   - Using `datafusion-spark`'s `to_json`: gives Spark's text rules, which 
differ from what DataFusion's own JSON writer produces, and pulls in the Spark 
function set.
   
   ### Additional context
   
   - Previous issue and PR: #26185, #26187 (closed when the Spark crate moved)
   - Spark-flavored `to_json`, merged: apache/datafusion-spark#85, 
apache/datafusion-spark#86
   - arrow-json row encoder: `arrow::json::writer::make_encoder`
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to