sam-1112 opened a new pull request, #7601:
URL: https://github.com/apache/datafusion-comet/pull/7601
## Which issue does this PR close?
Related to #5812. This draft presents exploratory results for maintainer
feedback on the collection methodology and corpus scope.
## Rationale for this change
CometRegex uses a conservative whitelist to admit regex patterns for Rust
execution. Measuring its coverage on traceable public sources can help
prioritize future compatibility work.
The current application sample remains small: Gentropy contributes six
source locations and five unique patterns, with occurrence admission of 16.67%
and unique-pattern admission of 20%. Additional surveys document why other
candidates cannot yet enter the measured denominator.
Feedback is welcome on whether this exploratory scope is sufficient for
#5812 and which additional application sources would be most useful.
## What changes are included in this PR?
- Add commit-pinned source manifests, SQL/Python extraction tools, and
source/license provenance.
- Support PySpark functions.rlike calls with verified string-valued lit
arguments, with explicit validation policies for resolved constants.
- Measure admission using the actual, unchanged CometRegex.supportLevel,
Spark/Hive literal decoding, and Java regex validation.
- Report occurrence and unique-pattern admission, rejection witnesses,
overlapping feature tags, and length/structure distributions.
- Keep Application, Toolkit/Framework, and Engine Fixture populations
separate.
- Add Cuallee, SparkUtils Quality, and Sumeh surveys, plus Wikimedia literal
and CMSSpark dynamic-input audits.
The expansion adds one eligible Sumeh toolkit test rule and no eligible
application patterns. Wikimedia's 17 candidates remain excluded pending
deployment/configuration and licensing verification; CMSSpark's 10 dynamic
positions remain outside the baseline.
The internal application collection goals of three independent sources,
three domains, and 30 unique eligible patterns remain unmet. These results do
not establish ecosystem-wide representativeness or predict performance gains.
No production whitelist, routing, or Rust implementation changes are
included.
## How are these changes tested?
- All 46 Python extraction, measurement, structure, and corpus-quality tests
pass:
`python3 -m unittest discover -s dev/regex-coverage -p 'test_*.py' -v`
- JVM component checks validate actual literal decoding, Java regex
validation, analyzer loading, and Scanner inspection.
- All 139 records from the previous baseline preserve eligibility, decoded
values, and admission decisions.
- Repeated discovery produces byte-identical outputs; repeated measurement
and analysis preserve recorded hashes.
- The existing CometRegexSuite passed all 14 tests during the baseline
stage; production sources remain unchanged.
Reproduction commands, source-selection biases, exclusions, and measured
results are documented in dev/regex-coverage/README.md and the accompanying
reports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
