andygrove opened a new pull request, #6600: URL: https://github.com/apache/datafusion-comet/pull/6600
## Which issue does this PR close? N/A. Follow-up to #4971. ## Rationale for this change #4971 added two native `get_json_object` variants. One enforces Jackson's default 1000-digit number limit and nesting limit, which Jackson 2.15 added via `StreamReadConstraints`. The other, `get_json_object_spark34`, skips those limits. The Spark 3.4 shim always picks the second one because Spark 3.4 bundles Jackson 2.14. The behavior depends on the Jackson version, not the Spark version. A Spark 3.4 deployment running with Jackson 2.15+ enforces the limits, so Spark returns null for documents with an over-long number while Comet's native path returns a value. That breaks the existing large-number queries in `get_json_object.sql`. ## What changes are included in this PR? - The Spark 3.4 `CometExprShim` reads the runtime Jackson version (`com.fasterxml.jackson.core.json.PackageVersion.VERSION`). It uses `get_json_object` for Jackson 2.15+ and `get_json_object_spark34` otherwise. The version is read once, lazily. Spark 3.5+ shims are unchanged. - Updated the comment in `get_json_object.sql` to match. ## How are these changes tested? Covered by the existing large-number queries in `expressions/string/get_json_object.sql`: - `-Pspark-3.4` (Jackson 2.14): passes, same as before. - A Spark 3.4 build with Jackson 2.17: `CometSqlFileTestSuite` fails on those queries without this change and passes with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
