sam-1112 commented on issue #5812: URL: https://github.com/apache/datafusion-comet/issues/5812#issuecomment-6797861752
I have preliminary exploratory results for this issue and would like feedback on the corpus scope before opening a PR. The tooling reads commit-pinned public sources and measures admission with the unchanged `CometRegex.supportLevel()` (`RegexFlavor.RLike`). Python call-site patterns come from AST string values after receiver review, with no SQL unescaping. Spark and Hive fixture literals are decoded with the Spark and Hive APIs and checked with `java.util.regex`. Occurrences are source locations; unique patterns are exact decoded strings, with no regex canonicalization. The whitelist and runtime behavior are unchanged. The only qualified application sample is Gentropy, one genomics repository. Its patterns mostly check alleles, identifiers, or dates: - 6 source locations and 5 unique decoded patterns. - Occurrence admission: 1/6 (16.67%). - Unique-pattern admission: 1/5 (20%). Four rejected Gentropy occurrences (three patterns) contain both `^` and `$`. In analysis-only syntax controls, removing only the leading `^` still leaves all four rejected. The other rejected occurrence contains `\d`. This describes the sample; it does not propose a whitelist change. Other sources investigated: - Mozilla `python_mozetl`, Fink broker, and Fink filters: no raw predicate keywords in the scoped packages. - Cuallee: its PySpark `has_pattern()` implementation uses `regexp_extract`, so those examples do not measure RLike coverage. Two intended RLIKE predicates have fixed-looking pattern text in SQL that also interpolates an unchecked column expression, so they are classified as `UNKNOWN` and excluded from the admission denominator. - SparkUtils Quality: no whole-word RLIKE, REGEXP, REGEXP_LIKE, or has_pattern matches in the declared scan scope. - Sumeh: one manually reviewed PySpark test-rule definition, kept in a separate toolkit population and admitted 0/1. The upstream test was not executed. - Wikimedia Refinery: 17 literal candidates were audited locally and excluded from the admission denominator. Deployment/template semantics and redistribution terms remain unresolved. - CMSSpark: 10 intended RLIKE positions depend on runtime inputs. Documented defaults are not counted as query occurrences. - Databricks Labs UCX was not included because its license has service-specific use restrictions. Spark SQL and Hive fixtures were measured on separate denominators and are not pooled with the application rate. The two Hive fixture groups contain the same 19 eligible unique decoded patterns. The reports include exclusions, rejection-feature analysis, length and structure distributions, and reproduction instructions. This sample is too small and too narrowly selected to describe representative Spark workload coverage. The small number of checked-in literal patterns is a finding of this bounded survey; it does not show how often Spark applications use regex. A PR would add the dev-only tooling and aggregate reports under `dev/regex-coverage/`. It would not vendor upstream SQL, change production code, or close this issue. Would that exploratory PR be useful now, or should I broaden the application sample first? The extractor measures direct `Column.rlike` literals, verified `functions.lit` patterns, and reviewed rule constructors. Notebooks, generic `F.expr`, and runtime-supplied patterns stay outside the measured set. Are there particular redistributable public Spark workloads, with checked-in RLIKE literals outside the repositories above, that you would recommend? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
