wangzhigang1999 commented on issue #7458: URL: https://github.com/apache/kyuubi/issues/7458#issuecomment-4806491536
### Where this stands Quick update. The statement hook is working, and I prototyped the result hook too — but benchmarking it shifted my thinking on the scope, so let me lay out where I've landed. First, on batch: I floated earlier that batch could go through the same hook, but it doesn't really hold up. A batch is just an app submission — the gateway only sees the submission parameters (resource, class, args, conf), not the SQL the application actually runs on the engine, and never the result rows. So the statement/result hooks only make sense on the interactive path. The most batch could ever have is a submission-level hook over the `BatchRequest` itself, which feels like a separate thing — happy to revisit that on its own later. On the result hook: it's a general per-row transform (encryption, redaction, masking are just instances of it), sitting at `KyuubiOperation.getNextRowSetInternal`. The interesting part turned out to be the cost, which depends a lot on the result format. In a local microbenchmark (single-threaded, per fetch batch, Thrift columnar vs Arrow IPC, a wide row of 15 columns transforming 2 string columns) I saw: | rows/batch | Thrift | Arrow | slower | |---:|---:|---:|---:| | 10K | ~0.3 ms / ~2 MB | 3.56 ms / 17 MB | 11.9× / ~8× heap | | 100K | 3.55 ms / 19.5 MB | 47.2 ms / 154 MB | 13.3× / ~8× heap | Numbers are from my own setup so treat them as indicative, but the shape is what the code predicts: with Thrift the server already holds the deserialized `TRowSet`, so a transform only touches the columns you actually change — sub-millisecond, basically noise. Arrow is forwarded opaquely (the engine packs the whole batch into a single binary column the server doesn't decode), so to touch even one column you have to decode and re-encode the entire batch, and that cost scales with the **total** column count rather than what you transform. On a wide table that's 12–20× slower and ~8× the heap churn per batch (the ratio peaks ~20× at 1K rows; a narrow 1-column row is only 2.6×). Under concurrency that's not something I'd want to hide silently behind a hook. So for v1 I'd keep it tight: interactive path only, result transform on Thrift columnar, and fail-closed when a transform is configured and the operation returns Arrow — instead of quietly eating that cost. This isn't a global ban on Arrow; it only refuses the combination of an enabled transform plus an Arrow result. Arrow support can come later as an explicit opt-in, type-preserving only. The statement hook is ready, so I'll send that PR first and let the result transform follow. The one thing I'd like a sanity check on before I commit to it is that fail-closed call — refusing only when a transform meets an Arrow result. Does that feel right to everyone? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
