mbutrovich commented on code in PR #73:
URL: https://github.com/apache/datafusion-spark/pull/73#discussion_r4615337220
##########
crates/spark/src/function/string/regexp_extract.rs:
##########
@@ -0,0 +1,778 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::collections::HashMap;
+use std::hash::{Hash, Hasher};
+use std::sync::{Arc, Mutex};
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Int32Array, StringArray, StringArrayType,
StringBuilder,
+ new_null_array,
+};
+use arrow::compute::{can_cast_types, cast};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::as_int32_array;
+use datafusion_common::utils::offset_span_len;
+use datafusion_common::{
+ DataFusionError, Result, ScalarValue, exec_datafusion_err, exec_err,
+ internal_datafusion_err, internal_err, plan_err,
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+ Volatility,
+};
+use datafusion_macros::user_doc;
+use regex::{CaptureLocations, Regex};
+use regex_syntax::ast::{
+ AssertionKind, Ast, ClassSetBinaryOp, ClassSetBinaryOpKind, ClassSetItem,
+ ClassUnicode, ClassUnicodeKind, Flag, Flags, FlagsItemKind, GroupKind,
Visitor,
+};
+
+use crate::function::error_utils::invalid_arg_count_exec_err;
+
+const NAME: &str = "regexp_extract";
+
+/// Spark-compatible `regexp_extract` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#regexp_extract>
+#[user_doc(
+ doc_section(label = "Regular Expression Functions"),
+ description = "Returns the text matched by group `idx` of the first match
of `regexp` in `str`. \
+ Unlike DataFusion's `regexp_match`, which returns a list of every
capture group, this returns a single string. \
+ Returns an empty string if there is no match, or if the group did not
take part in the match. \
+ `idx` is only checked when the pattern matches: an `idx` outside
`0..=groupCount` is an error \
+ (`INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX`, as in Spark 3.5 and
later; Spark 3.4 raised \
+ `_LEGACY_ERROR_TEMP_2006`/`2007`) on a row that matches, and an empty
string on a row that does not. \
+ An invalid pattern is an error (`INVALID_PARAMETER_VALUE.PATTERN`).
Returns NULL if any argument is NULL, \
+ and a NULL row is not checked, so an invalid pattern is only reported
on a row it is applied to. \
+ As in Spark, `str` and `regexp` are cast to strings and `idx` to INT. \
+ Known differences from Spark
(https://github.com/apache/datafusion-spark/issues/255): patterns use the Rust \
+ `regex` engine, not Java's `java.util.regex`. Look-around,
backreferences, quoting with `\\Q...\\E`, \
+ `\\Z`, `\\h`, atomic groups `(?>...)`, the `(?d)` flag, and patterns
that exceed the Rust engine's \
+ compiled-size limit raise an \"unsupported regex pattern\" error,
where Spark evaluates them. Only unclosed \
+ or unopened groups and classes, invalid class ranges and repetitions,
and unknown flags are reported as \
+ `INVALID_PARAMETER_VALUE.PATTERN`; any other pattern the Rust engine
rejects is reported as unsupported, \
+ including a few that Java rejects too, such as `\\y`. \
+ Rust syntax that Java rejects is also
`INVALID_PARAMETER_VALUE.PATTERN`: the `R` flag, `\\b{start}` and \
+ the other `\\b{...}` word boundaries, `(?P<name>...)`, and group names
that are not an ASCII letter \
+ followed by ASCII letters and digits. Syntax that both engines accept
but read differently is reported as \
+ unsupported rather than evaluated to a different answer: the `U` flag
(Java's UNICODE_CHARACTER_CLASS, \
+ which swaps greediness in Rust), `[:alpha:]` and the other POSIX
classes in brackets, the class operators \
+ `--` and `~~`, `\\<` and `\\>`, and the names of Java's ASCII-only
POSIX classes, such as `\\p{Alpha}` \
+ and `\\p{Lower}`. Other Unicode property names that Java does not
know, such as `\\p{Letter}` or \
+ `\\p{Latin}`, are accepted, where Java rejects them. \
+ Possessive quantifiers are read as nested greedy repetition (`a++` as
`(a+)+`), so they can match \
+ where Java's would not. \
+ `\\d`, `\\w` and `\\s` match Unicode characters, where Java's match
only ASCII. \
+ `.` matches `\\r`, where Java's does not. `$` matches only at the very
end of the input, where Java's also \
+ matches before a final line terminator. `(?i)` folds case using
Unicode rules (so `k` matches the Kelvin \
+ sign), where Java folds only ASCII by default.",
Review Comment:
Could the list of differences also name these, each with a commented-out
Spark result in the `.slt` like the others?
- With `(?m)`, Java's `^` and `$` also treat `\r`, `\r\n`, U+0085, U+2028,
and U+2029 as line ends, where Rust's treat only `\n`. `regexp_extract('a' ||
chr(13) || 'b', '(?m)^b', 0)` is `b` in Spark 4.2.0 and an empty string here.
- `.` matches U+0085, U+2028, and U+2029 here as well as `\r`. Java's `.`
matches none of them.
- A capture group inside a repetition keeps a different value when the group
can match the empty string. `regexp_extract('aab', '(a*)+b', 1)` is an empty
string in Spark 4.2.0 and `aa` here. `regexp_extract('aa', '(a|)+', 1)` is an
empty string in Spark and `a` here.
- Rust accepts spaces inside a counted repetition, such as `a{2, 3}` and `a{
2}`. Spark rejects both with `INVALID_PARAMETER_VALUE.PATTERN`.
- `(?c)` belongs in the unsupported list next to `(?d)`, if the flag change
I suggest on line 467 goes in.
##########
crates/spark/src/function/string/regexp_extract.rs:
##########
@@ -0,0 +1,778 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::collections::HashMap;
+use std::hash::{Hash, Hasher};
+use std::sync::{Arc, Mutex};
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Int32Array, StringArray, StringArrayType,
StringBuilder,
+ new_null_array,
+};
+use arrow::compute::{can_cast_types, cast};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::as_int32_array;
+use datafusion_common::utils::offset_span_len;
+use datafusion_common::{
+ DataFusionError, Result, ScalarValue, exec_datafusion_err, exec_err,
+ internal_datafusion_err, internal_err, plan_err,
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+ Volatility,
+};
+use datafusion_macros::user_doc;
+use regex::{CaptureLocations, Regex};
+use regex_syntax::ast::{
+ AssertionKind, Ast, ClassSetBinaryOp, ClassSetBinaryOpKind, ClassSetItem,
+ ClassUnicode, ClassUnicodeKind, Flag, Flags, FlagsItemKind, GroupKind,
Visitor,
+};
+
+use crate::function::error_utils::invalid_arg_count_exec_err;
+
+const NAME: &str = "regexp_extract";
+
+/// Spark-compatible `regexp_extract` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#regexp_extract>
+#[user_doc(
+ doc_section(label = "Regular Expression Functions"),
+ description = "Returns the text matched by group `idx` of the first match
of `regexp` in `str`. \
+ Unlike DataFusion's `regexp_match`, which returns a list of every
capture group, this returns a single string. \
+ Returns an empty string if there is no match, or if the group did not
take part in the match. \
+ `idx` is only checked when the pattern matches: an `idx` outside
`0..=groupCount` is an error \
+ (`INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX`, as in Spark 3.5 and
later; Spark 3.4 raised \
+ `_LEGACY_ERROR_TEMP_2006`/`2007`) on a row that matches, and an empty
string on a row that does not. \
+ An invalid pattern is an error (`INVALID_PARAMETER_VALUE.PATTERN`).
Returns NULL if any argument is NULL, \
+ and a NULL row is not checked, so an invalid pattern is only reported
on a row it is applied to. \
+ As in Spark, `str` and `regexp` are cast to strings and `idx` to INT. \
+ Known differences from Spark
(https://github.com/apache/datafusion-spark/issues/255): patterns use the Rust \
+ `regex` engine, not Java's `java.util.regex`. Look-around,
backreferences, quoting with `\\Q...\\E`, \
+ `\\Z`, `\\h`, atomic groups `(?>...)`, the `(?d)` flag, and patterns
that exceed the Rust engine's \
+ compiled-size limit raise an \"unsupported regex pattern\" error,
where Spark evaluates them. Only unclosed \
+ or unopened groups and classes, invalid class ranges and repetitions,
and unknown flags are reported as \
+ `INVALID_PARAMETER_VALUE.PATTERN`; any other pattern the Rust engine
rejects is reported as unsupported, \
+ including a few that Java rejects too, such as `\\y`. \
+ Rust syntax that Java rejects is also
`INVALID_PARAMETER_VALUE.PATTERN`: the `R` flag, `\\b{start}` and \
+ the other `\\b{...}` word boundaries, `(?P<name>...)`, and group names
that are not an ASCII letter \
+ followed by ASCII letters and digits. Syntax that both engines accept
but read differently is reported as \
+ unsupported rather than evaluated to a different answer: the `U` flag
(Java's UNICODE_CHARACTER_CLASS, \
+ which swaps greediness in Rust), `[:alpha:]` and the other POSIX
classes in brackets, the class operators \
+ `--` and `~~`, `\\<` and `\\>`, and the names of Java's ASCII-only
POSIX classes, such as `\\p{Alpha}` \
+ and `\\p{Lower}`. Other Unicode property names that Java does not
know, such as `\\p{Letter}` or \
+ `\\p{Latin}`, are accepted, where Java rejects them. \
+ Possessive quantifiers are read as nested greedy repetition (`a++` as
`(a+)+`), so they can match \
+ where Java's would not. \
+ `\\d`, `\\w` and `\\s` match Unicode characters, where Java's match
only ASCII. \
+ `.` matches `\\r`, where Java's does not. `$` matches only at the very
end of the input, where Java's also \
+ matches before a final line terminator. `(?i)` folds case using
Unicode rules (so `k` matches the Kelvin \
+ sign), where Java folds only ASCII by default.",
+ syntax_example = "regexp_extract(str, regexp[, idx])",
+ sql_example = r#"```sql
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)');
+1
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 0);
+1a
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 2);
+a
+> SELECT regexp_extract('abc', '(\d+)', 1);
+-- empty string
+> SELECT regexp_extract('a1', '(\d+)', NULL);
+NULL
+```"#,
+ argument(
+ name = "str",
+ description = "String expression to search. Values of other types,
such as numbers and binary, are cast to a string. NULL returns NULL."
+ ),
+ argument(
+ name = "regexp",
+ description = "String regular expression in Rust `regex` syntax.
Values of other types are cast to a string. NULL returns NULL."
+ ),
+ argument(
+ name = "idx",
+ description = "Optional capture group index, cast to INT (numeric and
string values are accepted). Defaults to 1; 0 is the whole match. NULL returns
NULL."
+ )
+)]
+#[derive(Debug)]
+pub struct SparkRegexpExtract {
+ signature: Signature,
+ /// Literal patterns compiled by earlier batches. One instance serves every
+ /// query in a session, so the cache is keyed by pattern.
+ patterns: PatternCache,
+}
+
+// The cache does not affect the result, so two instances with the same
+// signature are equal.
+impl PartialEq for SparkRegexpExtract {
+ fn eq(&self, other: &Self) -> bool {
+ self.signature == other.signature
+ }
+}
+
+impl Eq for SparkRegexpExtract {}
+
+impl Hash for SparkRegexpExtract {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.signature.hash(state);
+ }
+}
+
+/// The literal patterns kept compiled. When it is full it is cleared, which
+/// bounds its size if many different literal patterns are used.
+const PATTERN_CACHE_SIZE: usize = 64;
+
+#[derive(Debug, Default)]
+struct PatternCache {
+ patterns: Mutex<HashMap<String, Regex>>,
+ /// How many patterns were compiled, so a test can tell that a later batch
+ /// read the cache rather than compiling and inserting the pattern again.
+ #[cfg(test)]
+ compiles: std::sync::atomic::AtomicUsize,
+}
+
+impl PatternCache {
+ fn get_or_compile(&self, pattern: &str) -> Result<Regex> {
+ let lock = || {
+ self.patterns
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ };
+ if let Some(regex) = lock().get(pattern) {
+ return Ok(regex.clone());
+ }
+ #[cfg(test)]
+ self.compiles
+ .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ let regex = compile(pattern)?;
+ let mut patterns = lock();
+ if patterns.len() >= PATTERN_CACHE_SIZE {
+ patterns.clear();
+ }
+ patterns.insert(pattern.to_string(), regex.clone());
+ Ok(regex)
+ }
+}
+
+impl Default for SparkRegexpExtract {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkRegexpExtract {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ patterns: PatternCache::default(),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkRegexpExtract {
+ fn name(&self) -> &str {
+ NAME
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::Utf8)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ if !(2..=3).contains(&arg_types.len()) {
+ return Err(invalid_arg_count_exec_err(NAME, (2, 3),
arg_types.len()));
+ }
+ // Like Spark's implicit casts: str and regexp are cast to strings,
idx to INT.
+ let castable_to_string = |t: &DataType| {
+ *t == DataType::Null || (!t.is_nested() && can_cast_types(t,
&DataType::Utf8))
+ };
+ for (name, arg_type) in ["str", "regexp"].iter().zip(arg_types) {
+ if !castable_to_string(arg_type) {
+ return plan_err!("{NAME} expects a string {name}, got
{arg_type}");
+ }
+ }
+ let mut coerced = vec![DataType::Utf8, DataType::Utf8];
+ if let Some(idx) = arg_types.get(2) {
+ let castable_to_int = idx.is_numeric()
+ || matches!(
+ idx,
+ DataType::Null
+ | DataType::Utf8
+ | DataType::LargeUtf8
+ | DataType::Utf8View
+ );
+ if !castable_to_int {
+ return plan_err!("{NAME} expects an INT idx, got {idx}");
+ }
+ coerced.push(DataType::Int32);
+ }
+ Ok(coerced)
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) ->
Result<ColumnarValue> {
+ let all_scalar = args
+ .args
+ .iter()
+ .all(|arg| matches!(arg, ColumnarValue::Scalar(_)));
+ let len = if all_scalar { 1 } else { args.number_rows };
+
+ // The usual call has a literal pattern and idx: keep those as scalars
so
+ // the pattern is compiled once and no per-row copies are allocated.
+ let pattern_column = match &args.args[1] {
+ ColumnarValue::Scalar(_) => None,
+ ColumnarValue::Array(array) => Some(to_utf8("regexp", array)?),
+ };
+ let pattern = match (&args.args[1], &pattern_column) {
+ (_, Some(patterns)) => Pattern::Array(patterns.as_string::<i32>()),
+ (ColumnarValue::Scalar(scalar), None) => match scalar_str(scalar)?
{
+ Some(pattern) => Pattern::Scalar(pattern),
+ None => return null_result(len, all_scalar),
+ },
+ (ColumnarValue::Array(_), None) => unreachable!("converted above"),
+ };
+ let index = match args.args.get(2) {
+ None => Index::Scalar(1),
+ Some(ColumnarValue::Scalar(ScalarValue::Int32(Some(idx)))) => {
+ Index::Scalar(*idx)
+ }
+ Some(ColumnarValue::Scalar(ScalarValue::Int32(None) |
ScalarValue::Null)) => {
+ return null_result(len, all_scalar);
+ }
+ Some(ColumnarValue::Scalar(other)) => {
+ return internal_err!(
+ "{NAME} expects an INT idx, got {}",
+ other.data_type()
+ );
+ }
+ Some(ColumnarValue::Array(array)) =>
Index::Array(as_int32_array(array)?),
+ };
+
+ // A caller that builds the physical expression directly, such as
Comet,
+ // can pass any string type. Each result is a substring of its input,
so
+ // the bytes of the visible input values bound the output.
+ let strings = args.args[0].to_array(len)?;
+ let result = match strings.data_type() {
+ DataType::Utf8 => {
+ let strings = strings.as_string::<i32>();
+ let capacity = offset_span_len(strings.offsets());
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ DataType::LargeUtf8 => {
+ let strings = strings.as_string::<i64>();
+ let capacity = offset_span_len(strings.offsets());
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ DataType::Utf8View => {
+ let strings = strings.as_string_view();
+ let capacity = strings.lengths().map(|len| len as usize).sum();
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ other => return exec_err!("{NAME} expects a string str, got
{other}"),
+ };
+ if all_scalar {
+ Ok(ColumnarValue::Scalar(ScalarValue::try_from_array(
+ &result, 0,
+ )?))
+ } else {
+ Ok(ColumnarValue::Array(result))
+ }
+ }
+}
+
+#[derive(Clone, Copy)]
+enum Pattern<'a> {
+ Scalar(&'a str),
+ Array(&'a StringArray),
+}
+
+#[derive(Clone, Copy)]
+enum Index<'a> {
+ Scalar(i32),
+ Array(&'a Int32Array),
+}
+
+/// A pattern column as `Utf8`. Other string types are cast, and anything else
+/// is an error.
+fn to_utf8(name: &str, array: &ArrayRef) -> Result<ArrayRef> {
+ match array.data_type() {
+ DataType::Utf8 => Ok(Arc::clone(array)),
+ DataType::LargeUtf8 | DataType::Utf8View => Ok(cast(array,
&DataType::Utf8)?),
+ other => exec_err!("{NAME} expects a string {name}, got {other}"),
+ }
+}
+
+fn scalar_str(scalar: &ScalarValue) -> Result<Option<&str>> {
+ match scalar {
+ ScalarValue::Null => Ok(None),
+ other => other.try_as_str().ok_or_else(|| {
+ internal_datafusion_err!(
+ "{NAME} expects a string regexp, got {}",
+ other.data_type()
+ )
+ }),
+ }
+}
+
+fn null_result(len: usize, all_scalar: bool) -> Result<ColumnarValue> {
+ Ok(if all_scalar {
+ ColumnarValue::Scalar(ScalarValue::Utf8(None))
+ } else {
+ ColumnarValue::Array(new_null_array(&DataType::Utf8, len))
+ })
+}
+
+fn extract<'a, S: StringArrayType<'a> + Copy>(
+ strings: S,
+ capacity: usize,
+ pattern: Pattern,
+ index: Index,
+ cache: &PatternCache,
+) -> Result<ArrayRef> {
+ let mut builder = StringBuilder::with_capacity(strings.len(), capacity);
+ // A literal pattern comes from the cache on the first non-NULL row, so a
NULL
+ // row is not checked. A column pattern keeps only the last one compiled,
+ // since it rarely changes per row. The capture buffer is reused across
rows
+ // instead of allocated per match.
+ let mut cached: Option<(&str, Regex, CaptureLocations)> = None;
+
+ for row in 0..strings.len() {
+ let row_pattern = match pattern {
+ Pattern::Scalar(pattern) => Some(pattern),
+ Pattern::Array(patterns) => {
+ (!patterns.is_null(row)).then(|| patterns.value(row))
+ }
+ };
+ let idx = match index {
+ Index::Scalar(idx) => Some(idx),
+ Index::Array(indexes) => (!indexes.is_null(row)).then(||
indexes.value(row)),
+ };
+ let (false, Some(pattern_str), Some(idx)) =
+ (strings.is_null(row), row_pattern, idx)
+ else {
+ builder.append_null();
+ continue;
+ };
+ let (_, regex, locations) = match &mut cached {
+ Some(entry) if entry.0 == pattern_str => entry,
+ _ => {
+ let regex = match pattern {
+ Pattern::Scalar(_) => cache.get_or_compile(pattern_str)?,
+ Pattern::Array(_) => compile(pattern_str)?,
+ };
+ let locations = regex.capture_locations();
+ cached.insert((pattern_str, regex, locations))
+ }
+ };
+ let subject = strings.value(row);
+ // Like Spark, idx is only checked when the pattern matches.
+ let text = match regex.captures_read(locations, subject) {
+ None => "",
+ Some(_) => locations
+ .get(check_group_index(regex, idx)?)
+ .map_or("", |(start, end)| &subject[start..end]),
+ };
+ builder.append_value(text);
+ }
+ Ok(Arc::new(builder.finish()))
+}
+
+fn check_group_index(regex: &Regex, idx: i32) -> Result<usize> {
+ let group_count = regex.captures_len() - 1;
+ usize::try_from(idx)
+ .ok()
+ .filter(|group| *group <= group_count)
+ .ok_or_else(|| {
+ exec_datafusion_err!(
+ "[INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX] The value of
parameter(s) \
+ `idx` in `{NAME}` is invalid: Expects group index between 0
and \
+ {group_count}, but got {idx}. SQLSTATE: 22023"
+ )
+ })
+}
+
+fn compile(pattern: &str) -> Result<Regex> {
+ let quoted = to_sql_value(pattern);
+ let unsupported = |reason: String| {
+ DataFusionError::NotImplemented(format!(
+ "Unsupported regex pattern {quoted} in `{NAME}`: {reason}. \
+ See https://github.com/apache/datafusion-spark/issues/255"
+ ))
+ };
+ match java_difference(pattern) {
+ None => {}
+ Some(JavaDifference::Invalid) => {
+ return Err(exec_datafusion_err!(
+ "[INVALID_PARAMETER_VALUE.PATTERN] The value of parameter(s)
`regexp` in \
+ `{NAME}` is invalid: {quoted}. SQLSTATE: 22023"
+ ));
+ }
+ Some(JavaDifference::Unsupported(reason)) => return
Err(unsupported(reason)),
+ }
+ // The pattern parsed, so a later stage of the Rust engine rejected it.
+ Regex::new(pattern).map_err(|err| match err {
+ regex::Error::CompiledTooBig(_) => unsupported(
+ "the compiled pattern exceeds the Rust regex size
limit".to_string(),
+ ),
+ err => unsupported(format!("the Rust regex engine rejects it
({err})")),
+ })
+}
+
+/// How `java.util.regex`, which Spark uses, would treat a pattern differently
+/// from the Rust regex engine.
+#[derive(Debug)]
+enum JavaDifference {
+ /// Java rejects the pattern, so it is reported as an invalid pattern.
+ Invalid,
+ /// Java accepts the pattern but the Rust engine rejects it, or reads it
+ /// differently, so it is reported as unsupported rather than evaluated to
+ /// a different answer than Spark's.
+ Unsupported(String),
+}
+
+/// Checks a pattern with `regex_syntax`, whose AST and error kinds are API,
+/// rather than matching on the text of [`regex::Error`].
+///
+/// A pattern that does not parse is invalid only for the kinds of syntax error
+/// that Java also raises. Anything else, such as `\Q...\E`, `\Z`, `\h`,
+/// `(?>...)` or `(?d)`, is unsupported, which also covers a few patterns that
+/// Java rejects, such as `\y`. A pattern that parses is walked for the
+/// constructs that the two engines read differently.
+fn java_difference(pattern: &str) -> Option<JavaDifference> {
+ use regex_syntax::ast::ErrorKind;
+ let ast_err = match regex_syntax::ast::parse::Parser::new().parse(pattern)
{
+ Ok(ast) => return regex_syntax::ast::visit(&ast,
JavaDifferences).err(),
+ Err(err) => err,
+ };
+ let unsupported = |reason: String|
Some(JavaDifference::Unsupported(reason));
+ match ast_err.kind() {
+ ErrorKind::GroupUnclosed
+ | ErrorKind::GroupUnopened
+ | ErrorKind::ClassUnclosed
+ | ErrorKind::ClassRangeInvalid
+ | ErrorKind::RepetitionMissing
+ | ErrorKind::RepetitionCountInvalid => Some(JavaDifference::Invalid),
+ // Java also rejects an unknown flag, except `d` (UNIX_LINES) and `>`
+ // (an atomic group).
+ ErrorKind::FlagUnrecognized => {
+ let span = ast_err.span();
+ let flag = &pattern[span.start.offset..span.end.offset];
+ if matches!(flag, "d" | ">") {
+ unsupported(format!(
+ "the Rust regex engine does not support the flag '{flag}'"
+ ))
+ } else {
+ Some(JavaDifference::Invalid)
+ }
+ }
+ ErrorKind::UnsupportedLookAround => unsupported(
+ "look-around is not supported by the Rust regex
engine".to_string(),
+ ),
+ ErrorKind::UnsupportedBackreference => unsupported(
+ "backreferences are not supported by the Rust regex
engine".to_string(),
+ ),
+ kind => unsupported(format!(
+ "the Rust regex engine does not support it ({kind})"
+ )),
+ }
+}
+
+/// Java's POSIX character classes, which match ASCII only (unless the `U` flag
+/// is set, which is unsupported). The Rust engine reads the same names as
+/// Unicode properties, so `\p{Alpha}` matches `é` there but not in Java.
+const JAVA_POSIX_CLASSES: [&str; 13] = [
+ "Lower", "Upper", "ASCII", "Alpha", "Digit", "Alnum", "Punct", "Graph",
"Print",
+ "Blank", "Cntrl", "XDigit", "Space",
+];
+
+/// Finds the constructs of a parsed pattern that Java rejects or reads
+/// differently. Each is checked by its AST node, so an escaped or quoted
+/// look-alike, such as `\[\[:alpha:\]\]`, is not reported.
+struct JavaDifferences;
+
+impl JavaDifferences {
+ fn check_flags(flags: &Flags) -> Result<(), JavaDifference> {
+ // Java has no R flag, even negated.
+ if flags
+ .items
+ .iter()
+ .any(|item| item.kind == FlagsItemKind::Flag(Flag::CRLF))
+ {
+ return Err(JavaDifference::Invalid);
+ }
+ // `(?-U)` is the default in both engines, so only setting it differs.
+ if flags.flag_state(Flag::SwapGreed) == Some(true) {
+ return Err(JavaDifference::Unsupported(
+ "Java's flag 'U' is UNICODE_CHARACTER_CLASS, where the Rust
regex \
+ engine's swaps greediness"
+ .to_string(),
+ ));
+ }
+ Ok(())
+ }
+
+ fn check_unicode_class(class: &ClassUnicode) -> Result<(), JavaDifference>
{
+ match &class.kind {
+ ClassUnicodeKind::Named(name)
+ if JAVA_POSIX_CLASSES.contains(&name.as_str()) =>
+ {
+ let p = if class.negated { 'P' } else { 'p' };
+ Err(JavaDifference::Unsupported(format!(
+ "Java's \\{p}{{{name}}} is an ASCII-only POSIX class,
where the Rust \
+ regex engine's is a Unicode property"
+ )))
+ }
+ _ => Ok(()),
+ }
+ }
+}
+
+impl Visitor for JavaDifferences {
+ type Output = ();
+ type Err = JavaDifference;
+
+ fn finish(self) -> Result<(), JavaDifference> {
+ Ok(())
+ }
+
+ fn visit_pre(&mut self, ast: &Ast) -> Result<(), JavaDifference> {
+ match ast {
+ Ast::Flags(set) => Self::check_flags(&set.flags),
+ Ast::Group(group) => match &group.kind {
+ GroupKind::NonCapturing(flags) => Self::check_flags(flags),
+ // Java rejects `(?P<name>...)`, and a name that is not an
ASCII
+ // letter followed by ASCII letters and digits.
+ GroupKind::CaptureName { starts_with_p, name } => {
+ let mut chars = name.name.chars();
+ let valid = !starts_with_p
+ && chars.next().is_some_and(|c|
c.is_ascii_alphabetic())
+ && chars.all(|c| c.is_ascii_alphanumeric());
+ if valid {
+ Ok(())
+ } else {
+ Err(JavaDifference::Invalid)
+ }
+ }
+ GroupKind::CaptureIndex(_) => Ok(()),
+ },
+ Ast::Assertion(assertion) => match assertion.kind {
+ // Java rejects `\b{start}`, `\b{end}`, `\b{start-half}` and
+ // `\b{end-half}`.
+ AssertionKind::WordBoundaryStart
+ | AssertionKind::WordBoundaryEnd
+ | AssertionKind::WordBoundaryStartHalf
+ | AssertionKind::WordBoundaryEndHalf =>
Err(JavaDifference::Invalid),
+ AssertionKind::WordBoundaryStartAngle
+ | AssertionKind::WordBoundaryEndAngle =>
Err(JavaDifference::Unsupported(
+ "Java reads \\< and \\> as literal characters, where the
Rust regex \
+ engine reads them as word boundaries"
+ .to_string(),
+ )),
+ AssertionKind::StartLine
+ | AssertionKind::EndLine
+ | AssertionKind::StartText
+ | AssertionKind::EndText
+ | AssertionKind::WordBoundary
+ | AssertionKind::NotWordBoundary => Ok(()),
+ },
+ Ast::ClassUnicode(class) => Self::check_unicode_class(class),
+ _ => Ok(()),
+ }
+ }
+
+ fn visit_class_set_item_pre(
+ &mut self,
+ item: &ClassSetItem,
+ ) -> Result<(), JavaDifference> {
+ match item {
+ ClassSetItem::Ascii(_) => Err(JavaDifference::Unsupported(
+ "Java reads [:name:] inside a class as a nested class of its
characters, \
+ where the Rust regex engine reads it as a POSIX class"
+ .to_string(),
+ )),
+ ClassSetItem::Unicode(class) => Self::check_unicode_class(class),
+ _ => Ok(()),
+ }
+ }
+
+ fn visit_class_set_binary_op_pre(
+ &mut self,
+ op: &ClassSetBinaryOp,
+ ) -> Result<(), JavaDifference> {
+ let operator = match op.kind {
+ // `&&` is intersection in both engines.
+ ClassSetBinaryOpKind::Intersection => return Ok(()),
Review Comment:
`&&` with an empty operand reads differently. Java treats `[a&&]` and
`[&&a]` as `[a]`, so both return `a` for subject `a` in Spark 4.2.0. Rust
intersects with the empty set and returns an empty string. Java also reads
`[a&&&b]` differently: Spark returns `&` for subject `&`, and this PR returns
an empty string. Could this arm report an intersection as unsupported when
either operand is `ClassSet::Item(ClassSetItem::Empty(_))`, or when the right
operand starts with an unescaped `&`? The usual forms, such as
`[a-z&&[^aeiou]]`, `[a-c&&[^b]]`, and `[^a-c&&b]`, gave the same results in
both engines.
##########
crates/spark/src/function/string/regexp_extract.rs:
##########
@@ -0,0 +1,778 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::collections::HashMap;
+use std::hash::{Hash, Hasher};
+use std::sync::{Arc, Mutex};
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Int32Array, StringArray, StringArrayType,
StringBuilder,
+ new_null_array,
+};
+use arrow::compute::{can_cast_types, cast};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::as_int32_array;
+use datafusion_common::utils::offset_span_len;
+use datafusion_common::{
+ DataFusionError, Result, ScalarValue, exec_datafusion_err, exec_err,
+ internal_datafusion_err, internal_err, plan_err,
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+ Volatility,
+};
+use datafusion_macros::user_doc;
+use regex::{CaptureLocations, Regex};
+use regex_syntax::ast::{
+ AssertionKind, Ast, ClassSetBinaryOp, ClassSetBinaryOpKind, ClassSetItem,
+ ClassUnicode, ClassUnicodeKind, Flag, Flags, FlagsItemKind, GroupKind,
Visitor,
+};
+
+use crate::function::error_utils::invalid_arg_count_exec_err;
+
+const NAME: &str = "regexp_extract";
+
+/// Spark-compatible `regexp_extract` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#regexp_extract>
+#[user_doc(
+ doc_section(label = "Regular Expression Functions"),
+ description = "Returns the text matched by group `idx` of the first match
of `regexp` in `str`. \
+ Unlike DataFusion's `regexp_match`, which returns a list of every
capture group, this returns a single string. \
+ Returns an empty string if there is no match, or if the group did not
take part in the match. \
+ `idx` is only checked when the pattern matches: an `idx` outside
`0..=groupCount` is an error \
+ (`INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX`, as in Spark 3.5 and
later; Spark 3.4 raised \
+ `_LEGACY_ERROR_TEMP_2006`/`2007`) on a row that matches, and an empty
string on a row that does not. \
+ An invalid pattern is an error (`INVALID_PARAMETER_VALUE.PATTERN`).
Returns NULL if any argument is NULL, \
+ and a NULL row is not checked, so an invalid pattern is only reported
on a row it is applied to. \
+ As in Spark, `str` and `regexp` are cast to strings and `idx` to INT. \
+ Known differences from Spark
(https://github.com/apache/datafusion-spark/issues/255): patterns use the Rust \
+ `regex` engine, not Java's `java.util.regex`. Look-around,
backreferences, quoting with `\\Q...\\E`, \
+ `\\Z`, `\\h`, atomic groups `(?>...)`, the `(?d)` flag, and patterns
that exceed the Rust engine's \
+ compiled-size limit raise an \"unsupported regex pattern\" error,
where Spark evaluates them. Only unclosed \
+ or unopened groups and classes, invalid class ranges and repetitions,
and unknown flags are reported as \
+ `INVALID_PARAMETER_VALUE.PATTERN`; any other pattern the Rust engine
rejects is reported as unsupported, \
+ including a few that Java rejects too, such as `\\y`. \
+ Rust syntax that Java rejects is also
`INVALID_PARAMETER_VALUE.PATTERN`: the `R` flag, `\\b{start}` and \
+ the other `\\b{...}` word boundaries, `(?P<name>...)`, and group names
that are not an ASCII letter \
+ followed by ASCII letters and digits. Syntax that both engines accept
but read differently is reported as \
+ unsupported rather than evaluated to a different answer: the `U` flag
(Java's UNICODE_CHARACTER_CLASS, \
+ which swaps greediness in Rust), `[:alpha:]` and the other POSIX
classes in brackets, the class operators \
+ `--` and `~~`, `\\<` and `\\>`, and the names of Java's ASCII-only
POSIX classes, such as `\\p{Alpha}` \
+ and `\\p{Lower}`. Other Unicode property names that Java does not
know, such as `\\p{Letter}` or \
+ `\\p{Latin}`, are accepted, where Java rejects them. \
+ Possessive quantifiers are read as nested greedy repetition (`a++` as
`(a+)+`), so they can match \
+ where Java's would not. \
+ `\\d`, `\\w` and `\\s` match Unicode characters, where Java's match
only ASCII. \
+ `.` matches `\\r`, where Java's does not. `$` matches only at the very
end of the input, where Java's also \
+ matches before a final line terminator. `(?i)` folds case using
Unicode rules (so `k` matches the Kelvin \
+ sign), where Java folds only ASCII by default.",
+ syntax_example = "regexp_extract(str, regexp[, idx])",
+ sql_example = r#"```sql
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)');
+1
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 0);
+1a
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 2);
+a
+> SELECT regexp_extract('abc', '(\d+)', 1);
+-- empty string
+> SELECT regexp_extract('a1', '(\d+)', NULL);
+NULL
+```"#,
+ argument(
+ name = "str",
+ description = "String expression to search. Values of other types,
such as numbers and binary, are cast to a string. NULL returns NULL."
+ ),
+ argument(
+ name = "regexp",
+ description = "String regular expression in Rust `regex` syntax.
Values of other types are cast to a string. NULL returns NULL."
+ ),
+ argument(
+ name = "idx",
+ description = "Optional capture group index, cast to INT (numeric and
string values are accepted). Defaults to 1; 0 is the whole match. NULL returns
NULL."
+ )
+)]
+#[derive(Debug)]
+pub struct SparkRegexpExtract {
+ signature: Signature,
+ /// Literal patterns compiled by earlier batches. One instance serves every
+ /// query in a session, so the cache is keyed by pattern.
+ patterns: PatternCache,
+}
+
+// The cache does not affect the result, so two instances with the same
+// signature are equal.
+impl PartialEq for SparkRegexpExtract {
+ fn eq(&self, other: &Self) -> bool {
+ self.signature == other.signature
+ }
+}
+
+impl Eq for SparkRegexpExtract {}
+
+impl Hash for SparkRegexpExtract {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.signature.hash(state);
+ }
+}
+
+/// The literal patterns kept compiled. When it is full it is cleared, which
+/// bounds its size if many different literal patterns are used.
+const PATTERN_CACHE_SIZE: usize = 64;
+
+#[derive(Debug, Default)]
+struct PatternCache {
+ patterns: Mutex<HashMap<String, Regex>>,
+ /// How many patterns were compiled, so a test can tell that a later batch
+ /// read the cache rather than compiling and inserting the pattern again.
+ #[cfg(test)]
+ compiles: std::sync::atomic::AtomicUsize,
+}
+
+impl PatternCache {
+ fn get_or_compile(&self, pattern: &str) -> Result<Regex> {
+ let lock = || {
+ self.patterns
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ };
+ if let Some(regex) = lock().get(pattern) {
+ return Ok(regex.clone());
+ }
+ #[cfg(test)]
+ self.compiles
+ .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ let regex = compile(pattern)?;
+ let mut patterns = lock();
+ if patterns.len() >= PATTERN_CACHE_SIZE {
+ patterns.clear();
+ }
+ patterns.insert(pattern.to_string(), regex.clone());
+ Ok(regex)
+ }
+}
+
+impl Default for SparkRegexpExtract {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkRegexpExtract {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ patterns: PatternCache::default(),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkRegexpExtract {
+ fn name(&self) -> &str {
+ NAME
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::Utf8)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ if !(2..=3).contains(&arg_types.len()) {
+ return Err(invalid_arg_count_exec_err(NAME, (2, 3),
arg_types.len()));
+ }
+ // Like Spark's implicit casts: str and regexp are cast to strings,
idx to INT.
+ let castable_to_string = |t: &DataType| {
+ *t == DataType::Null || (!t.is_nested() && can_cast_types(t,
&DataType::Utf8))
+ };
+ for (name, arg_type) in ["str", "regexp"].iter().zip(arg_types) {
+ if !castable_to_string(arg_type) {
+ return plan_err!("{NAME} expects a string {name}, got
{arg_type}");
+ }
+ }
+ let mut coerced = vec![DataType::Utf8, DataType::Utf8];
+ if let Some(idx) = arg_types.get(2) {
+ let castable_to_int = idx.is_numeric()
+ || matches!(
+ idx,
+ DataType::Null
+ | DataType::Utf8
+ | DataType::LargeUtf8
+ | DataType::Utf8View
+ );
+ if !castable_to_int {
+ return plan_err!("{NAME} expects an INT idx, got {idx}");
+ }
+ coerced.push(DataType::Int32);
+ }
+ Ok(coerced)
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) ->
Result<ColumnarValue> {
+ let all_scalar = args
+ .args
+ .iter()
+ .all(|arg| matches!(arg, ColumnarValue::Scalar(_)));
+ let len = if all_scalar { 1 } else { args.number_rows };
+
+ // The usual call has a literal pattern and idx: keep those as scalars
so
+ // the pattern is compiled once and no per-row copies are allocated.
+ let pattern_column = match &args.args[1] {
+ ColumnarValue::Scalar(_) => None,
+ ColumnarValue::Array(array) => Some(to_utf8("regexp", array)?),
+ };
+ let pattern = match (&args.args[1], &pattern_column) {
+ (_, Some(patterns)) => Pattern::Array(patterns.as_string::<i32>()),
+ (ColumnarValue::Scalar(scalar), None) => match scalar_str(scalar)?
{
+ Some(pattern) => Pattern::Scalar(pattern),
+ None => return null_result(len, all_scalar),
+ },
+ (ColumnarValue::Array(_), None) => unreachable!("converted above"),
+ };
+ let index = match args.args.get(2) {
+ None => Index::Scalar(1),
+ Some(ColumnarValue::Scalar(ScalarValue::Int32(Some(idx)))) => {
+ Index::Scalar(*idx)
+ }
+ Some(ColumnarValue::Scalar(ScalarValue::Int32(None) |
ScalarValue::Null)) => {
+ return null_result(len, all_scalar);
+ }
+ Some(ColumnarValue::Scalar(other)) => {
+ return internal_err!(
+ "{NAME} expects an INT idx, got {}",
+ other.data_type()
+ );
+ }
+ Some(ColumnarValue::Array(array)) =>
Index::Array(as_int32_array(array)?),
+ };
+
+ // A caller that builds the physical expression directly, such as
Comet,
+ // can pass any string type. Each result is a substring of its input,
so
+ // the bytes of the visible input values bound the output.
+ let strings = args.args[0].to_array(len)?;
+ let result = match strings.data_type() {
+ DataType::Utf8 => {
+ let strings = strings.as_string::<i32>();
+ let capacity = offset_span_len(strings.offsets());
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ DataType::LargeUtf8 => {
+ let strings = strings.as_string::<i64>();
+ let capacity = offset_span_len(strings.offsets());
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ DataType::Utf8View => {
+ let strings = strings.as_string_view();
+ let capacity = strings.lengths().map(|len| len as usize).sum();
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ other => return exec_err!("{NAME} expects a string str, got
{other}"),
+ };
+ if all_scalar {
+ Ok(ColumnarValue::Scalar(ScalarValue::try_from_array(
+ &result, 0,
+ )?))
+ } else {
+ Ok(ColumnarValue::Array(result))
+ }
+ }
+}
+
+#[derive(Clone, Copy)]
+enum Pattern<'a> {
+ Scalar(&'a str),
+ Array(&'a StringArray),
+}
+
+#[derive(Clone, Copy)]
+enum Index<'a> {
+ Scalar(i32),
+ Array(&'a Int32Array),
+}
+
+/// A pattern column as `Utf8`. Other string types are cast, and anything else
+/// is an error.
+fn to_utf8(name: &str, array: &ArrayRef) -> Result<ArrayRef> {
+ match array.data_type() {
+ DataType::Utf8 => Ok(Arc::clone(array)),
+ DataType::LargeUtf8 | DataType::Utf8View => Ok(cast(array,
&DataType::Utf8)?),
+ other => exec_err!("{NAME} expects a string {name}, got {other}"),
+ }
+}
+
+fn scalar_str(scalar: &ScalarValue) -> Result<Option<&str>> {
+ match scalar {
+ ScalarValue::Null => Ok(None),
+ other => other.try_as_str().ok_or_else(|| {
+ internal_datafusion_err!(
+ "{NAME} expects a string regexp, got {}",
+ other.data_type()
+ )
+ }),
+ }
+}
+
+fn null_result(len: usize, all_scalar: bool) -> Result<ColumnarValue> {
+ Ok(if all_scalar {
+ ColumnarValue::Scalar(ScalarValue::Utf8(None))
+ } else {
+ ColumnarValue::Array(new_null_array(&DataType::Utf8, len))
+ })
+}
+
+fn extract<'a, S: StringArrayType<'a> + Copy>(
+ strings: S,
+ capacity: usize,
+ pattern: Pattern,
+ index: Index,
+ cache: &PatternCache,
+) -> Result<ArrayRef> {
+ let mut builder = StringBuilder::with_capacity(strings.len(), capacity);
+ // A literal pattern comes from the cache on the first non-NULL row, so a
NULL
+ // row is not checked. A column pattern keeps only the last one compiled,
+ // since it rarely changes per row. The capture buffer is reused across
rows
+ // instead of allocated per match.
+ let mut cached: Option<(&str, Regex, CaptureLocations)> = None;
+
+ for row in 0..strings.len() {
+ let row_pattern = match pattern {
+ Pattern::Scalar(pattern) => Some(pattern),
+ Pattern::Array(patterns) => {
+ (!patterns.is_null(row)).then(|| patterns.value(row))
+ }
+ };
+ let idx = match index {
+ Index::Scalar(idx) => Some(idx),
+ Index::Array(indexes) => (!indexes.is_null(row)).then(||
indexes.value(row)),
+ };
+ let (false, Some(pattern_str), Some(idx)) =
+ (strings.is_null(row), row_pattern, idx)
+ else {
+ builder.append_null();
+ continue;
+ };
+ let (_, regex, locations) = match &mut cached {
+ Some(entry) if entry.0 == pattern_str => entry,
+ _ => {
+ let regex = match pattern {
+ Pattern::Scalar(_) => cache.get_or_compile(pattern_str)?,
+ Pattern::Array(_) => compile(pattern_str)?,
+ };
+ let locations = regex.capture_locations();
+ cached.insert((pattern_str, regex, locations))
+ }
+ };
+ let subject = strings.value(row);
+ // Like Spark, idx is only checked when the pattern matches.
+ let text = match regex.captures_read(locations, subject) {
+ None => "",
+ Some(_) => locations
+ .get(check_group_index(regex, idx)?)
+ .map_or("", |(start, end)| &subject[start..end]),
+ };
+ builder.append_value(text);
+ }
+ Ok(Arc::new(builder.finish()))
+}
+
+fn check_group_index(regex: &Regex, idx: i32) -> Result<usize> {
+ let group_count = regex.captures_len() - 1;
+ usize::try_from(idx)
+ .ok()
+ .filter(|group| *group <= group_count)
+ .ok_or_else(|| {
+ exec_datafusion_err!(
+ "[INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX] The value of
parameter(s) \
+ `idx` in `{NAME}` is invalid: Expects group index between 0
and \
+ {group_count}, but got {idx}. SQLSTATE: 22023"
+ )
+ })
+}
+
+fn compile(pattern: &str) -> Result<Regex> {
+ let quoted = to_sql_value(pattern);
+ let unsupported = |reason: String| {
+ DataFusionError::NotImplemented(format!(
+ "Unsupported regex pattern {quoted} in `{NAME}`: {reason}. \
+ See https://github.com/apache/datafusion-spark/issues/255"
+ ))
+ };
+ match java_difference(pattern) {
+ None => {}
+ Some(JavaDifference::Invalid) => {
+ return Err(exec_datafusion_err!(
+ "[INVALID_PARAMETER_VALUE.PATTERN] The value of parameter(s)
`regexp` in \
+ `{NAME}` is invalid: {quoted}. SQLSTATE: 22023"
+ ));
+ }
+ Some(JavaDifference::Unsupported(reason)) => return
Err(unsupported(reason)),
+ }
+ // The pattern parsed, so a later stage of the Rust engine rejected it.
+ Regex::new(pattern).map_err(|err| match err {
+ regex::Error::CompiledTooBig(_) => unsupported(
+ "the compiled pattern exceeds the Rust regex size
limit".to_string(),
+ ),
+ err => unsupported(format!("the Rust regex engine rejects it
({err})")),
+ })
+}
+
+/// How `java.util.regex`, which Spark uses, would treat a pattern differently
+/// from the Rust regex engine.
+#[derive(Debug)]
+enum JavaDifference {
+ /// Java rejects the pattern, so it is reported as an invalid pattern.
+ Invalid,
+ /// Java accepts the pattern but the Rust engine rejects it, or reads it
+ /// differently, so it is reported as unsupported rather than evaluated to
+ /// a different answer than Spark's.
+ Unsupported(String),
+}
+
+/// Checks a pattern with `regex_syntax`, whose AST and error kinds are API,
+/// rather than matching on the text of [`regex::Error`].
+///
+/// A pattern that does not parse is invalid only for the kinds of syntax error
+/// that Java also raises. Anything else, such as `\Q...\E`, `\Z`, `\h`,
+/// `(?>...)` or `(?d)`, is unsupported, which also covers a few patterns that
+/// Java rejects, such as `\y`. A pattern that parses is walked for the
+/// constructs that the two engines read differently.
+fn java_difference(pattern: &str) -> Option<JavaDifference> {
+ use regex_syntax::ast::ErrorKind;
+ let ast_err = match regex_syntax::ast::parse::Parser::new().parse(pattern)
{
+ Ok(ast) => return regex_syntax::ast::visit(&ast,
JavaDifferences).err(),
+ Err(err) => err,
+ };
+ let unsupported = |reason: String|
Some(JavaDifference::Unsupported(reason));
+ match ast_err.kind() {
+ ErrorKind::GroupUnclosed
+ | ErrorKind::GroupUnopened
+ | ErrorKind::ClassUnclosed
+ | ErrorKind::ClassRangeInvalid
+ | ErrorKind::RepetitionMissing
+ | ErrorKind::RepetitionCountInvalid => Some(JavaDifference::Invalid),
+ // Java also rejects an unknown flag, except `d` (UNIX_LINES) and `>`
+ // (an atomic group).
+ ErrorKind::FlagUnrecognized => {
+ let span = ast_err.span();
+ let flag = &pattern[span.start.offset..span.end.offset];
+ if matches!(flag, "d" | ">") {
+ unsupported(format!(
+ "the Rust regex engine does not support the flag '{flag}'"
+ ))
+ } else {
+ Some(JavaDifference::Invalid)
+ }
+ }
+ ErrorKind::UnsupportedLookAround => unsupported(
+ "look-around is not supported by the Rust regex
engine".to_string(),
+ ),
+ ErrorKind::UnsupportedBackreference => unsupported(
+ "backreferences are not supported by the Rust regex
engine".to_string(),
+ ),
+ kind => unsupported(format!(
+ "the Rust regex engine does not support it ({kind})"
+ )),
+ }
+}
+
+/// Java's POSIX character classes, which match ASCII only (unless the `U` flag
+/// is set, which is unsupported). The Rust engine reads the same names as
+/// Unicode properties, so `\p{Alpha}` matches `é` there but not in Java.
+const JAVA_POSIX_CLASSES: [&str; 13] = [
+ "Lower", "Upper", "ASCII", "Alpha", "Digit", "Alnum", "Punct", "Graph",
"Print",
+ "Blank", "Cntrl", "XDigit", "Space",
+];
Review Comment:
Rust's `\p{ASCII}` is the same class as Java's, U+0000 to U+007F. I matched
both engines against every code point that Java 17 assigns, and the two sets
are identical. So `regexp_extract('aé', '[^\p{ASCII}]', 0)`, which is `é` in
Spark 4.2.0, now gets the unsupported error. Could we drop `ASCII` from this
list?
The check also misses Java's `\p{IsHexDigit}`, which Java also accepts as
`IsHex_Digit` and in any case after `Is`. Rust reads it as Unicode's
`Hex_Digit`. Java's set is Rust's plus 630 decimal digits outside ASCII, such
as U+0663. So `regexp_extract('x' || chr(1635), '\p{IsHexDigit}', 0)` is U+0663
in Spark 4.2.0 and an empty string here. A condition like this in
`check_unicode_class` would catch it:
```rust
name.strip_prefix("Is").is_some_and(|p| {
p.eq_ignore_ascii_case("HexDigit") || p.eq_ignore_ascii_case("Hex_Digit")
})
```
I compared the other names that both engines accept the same way: general
categories with and without `Is`, `gc=`, and `general_category=`, scripts with
`Is`, `sc=`, and `script=`, and Java's `Is` binary properties. They matched,
apart from a few code points whose properties changed between the two engines'
Unicode versions.
##########
crates/spark/src/function/string/regexp_extract.rs:
##########
@@ -0,0 +1,778 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+use std::collections::HashMap;
+use std::hash::{Hash, Hasher};
+use std::sync::{Arc, Mutex};
+
+use arrow::array::{
+ Array, ArrayRef, AsArray, Int32Array, StringArray, StringArrayType,
StringBuilder,
+ new_null_array,
+};
+use arrow::compute::{can_cast_types, cast};
+use arrow::datatypes::DataType;
+use datafusion_common::cast::as_int32_array;
+use datafusion_common::utils::offset_span_len;
+use datafusion_common::{
+ DataFusionError, Result, ScalarValue, exec_datafusion_err, exec_err,
+ internal_datafusion_err, internal_err, plan_err,
+};
+use datafusion_expr::{
+ ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
+ Volatility,
+};
+use datafusion_macros::user_doc;
+use regex::{CaptureLocations, Regex};
+use regex_syntax::ast::{
+ AssertionKind, Ast, ClassSetBinaryOp, ClassSetBinaryOpKind, ClassSetItem,
+ ClassUnicode, ClassUnicodeKind, Flag, Flags, FlagsItemKind, GroupKind,
Visitor,
+};
+
+use crate::function::error_utils::invalid_arg_count_exec_err;
+
+const NAME: &str = "regexp_extract";
+
+/// Spark-compatible `regexp_extract` expression.
+///
+/// <https://spark.apache.org/docs/latest/api/sql/index.html#regexp_extract>
+#[user_doc(
+ doc_section(label = "Regular Expression Functions"),
+ description = "Returns the text matched by group `idx` of the first match
of `regexp` in `str`. \
+ Unlike DataFusion's `regexp_match`, which returns a list of every
capture group, this returns a single string. \
+ Returns an empty string if there is no match, or if the group did not
take part in the match. \
+ `idx` is only checked when the pattern matches: an `idx` outside
`0..=groupCount` is an error \
+ (`INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX`, as in Spark 3.5 and
later; Spark 3.4 raised \
+ `_LEGACY_ERROR_TEMP_2006`/`2007`) on a row that matches, and an empty
string on a row that does not. \
+ An invalid pattern is an error (`INVALID_PARAMETER_VALUE.PATTERN`).
Returns NULL if any argument is NULL, \
+ and a NULL row is not checked, so an invalid pattern is only reported
on a row it is applied to. \
+ As in Spark, `str` and `regexp` are cast to strings and `idx` to INT. \
+ Known differences from Spark
(https://github.com/apache/datafusion-spark/issues/255): patterns use the Rust \
+ `regex` engine, not Java's `java.util.regex`. Look-around,
backreferences, quoting with `\\Q...\\E`, \
+ `\\Z`, `\\h`, atomic groups `(?>...)`, the `(?d)` flag, and patterns
that exceed the Rust engine's \
+ compiled-size limit raise an \"unsupported regex pattern\" error,
where Spark evaluates them. Only unclosed \
+ or unopened groups and classes, invalid class ranges and repetitions,
and unknown flags are reported as \
+ `INVALID_PARAMETER_VALUE.PATTERN`; any other pattern the Rust engine
rejects is reported as unsupported, \
+ including a few that Java rejects too, such as `\\y`. \
+ Rust syntax that Java rejects is also
`INVALID_PARAMETER_VALUE.PATTERN`: the `R` flag, `\\b{start}` and \
+ the other `\\b{...}` word boundaries, `(?P<name>...)`, and group names
that are not an ASCII letter \
+ followed by ASCII letters and digits. Syntax that both engines accept
but read differently is reported as \
+ unsupported rather than evaluated to a different answer: the `U` flag
(Java's UNICODE_CHARACTER_CLASS, \
+ which swaps greediness in Rust), `[:alpha:]` and the other POSIX
classes in brackets, the class operators \
+ `--` and `~~`, `\\<` and `\\>`, and the names of Java's ASCII-only
POSIX classes, such as `\\p{Alpha}` \
+ and `\\p{Lower}`. Other Unicode property names that Java does not
know, such as `\\p{Letter}` or \
+ `\\p{Latin}`, are accepted, where Java rejects them. \
+ Possessive quantifiers are read as nested greedy repetition (`a++` as
`(a+)+`), so they can match \
+ where Java's would not. \
+ `\\d`, `\\w` and `\\s` match Unicode characters, where Java's match
only ASCII. \
+ `.` matches `\\r`, where Java's does not. `$` matches only at the very
end of the input, where Java's also \
+ matches before a final line terminator. `(?i)` folds case using
Unicode rules (so `k` matches the Kelvin \
+ sign), where Java folds only ASCII by default.",
+ syntax_example = "regexp_extract(str, regexp[, idx])",
+ sql_example = r#"```sql
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)');
+1
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 0);
+1a
+> SELECT regexp_extract('1a 2b 14m', '(\d+)([a-z]+)', 2);
+a
+> SELECT regexp_extract('abc', '(\d+)', 1);
+-- empty string
+> SELECT regexp_extract('a1', '(\d+)', NULL);
+NULL
+```"#,
+ argument(
+ name = "str",
+ description = "String expression to search. Values of other types,
such as numbers and binary, are cast to a string. NULL returns NULL."
+ ),
+ argument(
+ name = "regexp",
+ description = "String regular expression in Rust `regex` syntax.
Values of other types are cast to a string. NULL returns NULL."
+ ),
+ argument(
+ name = "idx",
+ description = "Optional capture group index, cast to INT (numeric and
string values are accepted). Defaults to 1; 0 is the whole match. NULL returns
NULL."
+ )
+)]
+#[derive(Debug)]
+pub struct SparkRegexpExtract {
+ signature: Signature,
+ /// Literal patterns compiled by earlier batches. One instance serves every
+ /// query in a session, so the cache is keyed by pattern.
+ patterns: PatternCache,
+}
+
+// The cache does not affect the result, so two instances with the same
+// signature are equal.
+impl PartialEq for SparkRegexpExtract {
+ fn eq(&self, other: &Self) -> bool {
+ self.signature == other.signature
+ }
+}
+
+impl Eq for SparkRegexpExtract {}
+
+impl Hash for SparkRegexpExtract {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.signature.hash(state);
+ }
+}
+
+/// The literal patterns kept compiled. When it is full it is cleared, which
+/// bounds its size if many different literal patterns are used.
+const PATTERN_CACHE_SIZE: usize = 64;
+
+#[derive(Debug, Default)]
+struct PatternCache {
+ patterns: Mutex<HashMap<String, Regex>>,
+ /// How many patterns were compiled, so a test can tell that a later batch
+ /// read the cache rather than compiling and inserting the pattern again.
+ #[cfg(test)]
+ compiles: std::sync::atomic::AtomicUsize,
+}
+
+impl PatternCache {
+ fn get_or_compile(&self, pattern: &str) -> Result<Regex> {
+ let lock = || {
+ self.patterns
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ };
+ if let Some(regex) = lock().get(pattern) {
+ return Ok(regex.clone());
+ }
+ #[cfg(test)]
+ self.compiles
+ .fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ let regex = compile(pattern)?;
+ let mut patterns = lock();
+ if patterns.len() >= PATTERN_CACHE_SIZE {
+ patterns.clear();
+ }
+ patterns.insert(pattern.to_string(), regex.clone());
+ Ok(regex)
+ }
+}
+
+impl Default for SparkRegexpExtract {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl SparkRegexpExtract {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::user_defined(Volatility::Immutable),
+ patterns: PatternCache::default(),
+ }
+ }
+}
+
+impl ScalarUDFImpl for SparkRegexpExtract {
+ fn name(&self) -> &str {
+ NAME
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
+ Ok(DataType::Utf8)
+ }
+
+ fn documentation(&self) -> Option<&Documentation> {
+ self.doc()
+ }
+
+ fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+ if !(2..=3).contains(&arg_types.len()) {
+ return Err(invalid_arg_count_exec_err(NAME, (2, 3),
arg_types.len()));
+ }
+ // Like Spark's implicit casts: str and regexp are cast to strings,
idx to INT.
+ let castable_to_string = |t: &DataType| {
+ *t == DataType::Null || (!t.is_nested() && can_cast_types(t,
&DataType::Utf8))
+ };
+ for (name, arg_type) in ["str", "regexp"].iter().zip(arg_types) {
+ if !castable_to_string(arg_type) {
+ return plan_err!("{NAME} expects a string {name}, got
{arg_type}");
+ }
+ }
+ let mut coerced = vec![DataType::Utf8, DataType::Utf8];
+ if let Some(idx) = arg_types.get(2) {
+ let castable_to_int = idx.is_numeric()
+ || matches!(
+ idx,
+ DataType::Null
+ | DataType::Utf8
+ | DataType::LargeUtf8
+ | DataType::Utf8View
+ );
+ if !castable_to_int {
+ return plan_err!("{NAME} expects an INT idx, got {idx}");
+ }
+ coerced.push(DataType::Int32);
+ }
+ Ok(coerced)
+ }
+
+ fn invoke_with_args(&self, args: ScalarFunctionArgs) ->
Result<ColumnarValue> {
+ let all_scalar = args
+ .args
+ .iter()
+ .all(|arg| matches!(arg, ColumnarValue::Scalar(_)));
+ let len = if all_scalar { 1 } else { args.number_rows };
+
+ // The usual call has a literal pattern and idx: keep those as scalars
so
+ // the pattern is compiled once and no per-row copies are allocated.
+ let pattern_column = match &args.args[1] {
+ ColumnarValue::Scalar(_) => None,
+ ColumnarValue::Array(array) => Some(to_utf8("regexp", array)?),
+ };
+ let pattern = match (&args.args[1], &pattern_column) {
+ (_, Some(patterns)) => Pattern::Array(patterns.as_string::<i32>()),
+ (ColumnarValue::Scalar(scalar), None) => match scalar_str(scalar)?
{
+ Some(pattern) => Pattern::Scalar(pattern),
+ None => return null_result(len, all_scalar),
+ },
+ (ColumnarValue::Array(_), None) => unreachable!("converted above"),
+ };
+ let index = match args.args.get(2) {
+ None => Index::Scalar(1),
+ Some(ColumnarValue::Scalar(ScalarValue::Int32(Some(idx)))) => {
+ Index::Scalar(*idx)
+ }
+ Some(ColumnarValue::Scalar(ScalarValue::Int32(None) |
ScalarValue::Null)) => {
+ return null_result(len, all_scalar);
+ }
+ Some(ColumnarValue::Scalar(other)) => {
+ return internal_err!(
+ "{NAME} expects an INT idx, got {}",
+ other.data_type()
+ );
+ }
+ Some(ColumnarValue::Array(array)) =>
Index::Array(as_int32_array(array)?),
+ };
+
+ // A caller that builds the physical expression directly, such as
Comet,
+ // can pass any string type. Each result is a substring of its input,
so
+ // the bytes of the visible input values bound the output.
+ let strings = args.args[0].to_array(len)?;
+ let result = match strings.data_type() {
+ DataType::Utf8 => {
+ let strings = strings.as_string::<i32>();
+ let capacity = offset_span_len(strings.offsets());
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ DataType::LargeUtf8 => {
+ let strings = strings.as_string::<i64>();
+ let capacity = offset_span_len(strings.offsets());
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ DataType::Utf8View => {
+ let strings = strings.as_string_view();
+ let capacity = strings.lengths().map(|len| len as usize).sum();
+ extract(strings, capacity, pattern, index, &self.patterns)?
+ }
+ other => return exec_err!("{NAME} expects a string str, got
{other}"),
+ };
+ if all_scalar {
+ Ok(ColumnarValue::Scalar(ScalarValue::try_from_array(
+ &result, 0,
+ )?))
+ } else {
+ Ok(ColumnarValue::Array(result))
+ }
+ }
+}
+
+#[derive(Clone, Copy)]
+enum Pattern<'a> {
+ Scalar(&'a str),
+ Array(&'a StringArray),
+}
+
+#[derive(Clone, Copy)]
+enum Index<'a> {
+ Scalar(i32),
+ Array(&'a Int32Array),
+}
+
+/// A pattern column as `Utf8`. Other string types are cast, and anything else
+/// is an error.
+fn to_utf8(name: &str, array: &ArrayRef) -> Result<ArrayRef> {
+ match array.data_type() {
+ DataType::Utf8 => Ok(Arc::clone(array)),
+ DataType::LargeUtf8 | DataType::Utf8View => Ok(cast(array,
&DataType::Utf8)?),
+ other => exec_err!("{NAME} expects a string {name}, got {other}"),
+ }
+}
+
+fn scalar_str(scalar: &ScalarValue) -> Result<Option<&str>> {
+ match scalar {
+ ScalarValue::Null => Ok(None),
+ other => other.try_as_str().ok_or_else(|| {
+ internal_datafusion_err!(
+ "{NAME} expects a string regexp, got {}",
+ other.data_type()
+ )
+ }),
+ }
+}
+
+fn null_result(len: usize, all_scalar: bool) -> Result<ColumnarValue> {
+ Ok(if all_scalar {
+ ColumnarValue::Scalar(ScalarValue::Utf8(None))
+ } else {
+ ColumnarValue::Array(new_null_array(&DataType::Utf8, len))
+ })
+}
+
+fn extract<'a, S: StringArrayType<'a> + Copy>(
+ strings: S,
+ capacity: usize,
+ pattern: Pattern,
+ index: Index,
+ cache: &PatternCache,
+) -> Result<ArrayRef> {
+ let mut builder = StringBuilder::with_capacity(strings.len(), capacity);
+ // A literal pattern comes from the cache on the first non-NULL row, so a
NULL
+ // row is not checked. A column pattern keeps only the last one compiled,
+ // since it rarely changes per row. The capture buffer is reused across
rows
+ // instead of allocated per match.
+ let mut cached: Option<(&str, Regex, CaptureLocations)> = None;
+
+ for row in 0..strings.len() {
+ let row_pattern = match pattern {
+ Pattern::Scalar(pattern) => Some(pattern),
+ Pattern::Array(patterns) => {
+ (!patterns.is_null(row)).then(|| patterns.value(row))
+ }
+ };
+ let idx = match index {
+ Index::Scalar(idx) => Some(idx),
+ Index::Array(indexes) => (!indexes.is_null(row)).then(||
indexes.value(row)),
+ };
+ let (false, Some(pattern_str), Some(idx)) =
+ (strings.is_null(row), row_pattern, idx)
+ else {
+ builder.append_null();
+ continue;
+ };
+ let (_, regex, locations) = match &mut cached {
+ Some(entry) if entry.0 == pattern_str => entry,
+ _ => {
+ let regex = match pattern {
+ Pattern::Scalar(_) => cache.get_or_compile(pattern_str)?,
+ Pattern::Array(_) => compile(pattern_str)?,
+ };
+ let locations = regex.capture_locations();
+ cached.insert((pattern_str, regex, locations))
+ }
+ };
+ let subject = strings.value(row);
+ // Like Spark, idx is only checked when the pattern matches.
+ let text = match regex.captures_read(locations, subject) {
+ None => "",
+ Some(_) => locations
+ .get(check_group_index(regex, idx)?)
+ .map_or("", |(start, end)| &subject[start..end]),
+ };
+ builder.append_value(text);
+ }
+ Ok(Arc::new(builder.finish()))
+}
+
+fn check_group_index(regex: &Regex, idx: i32) -> Result<usize> {
+ let group_count = regex.captures_len() - 1;
+ usize::try_from(idx)
+ .ok()
+ .filter(|group| *group <= group_count)
+ .ok_or_else(|| {
+ exec_datafusion_err!(
+ "[INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX] The value of
parameter(s) \
+ `idx` in `{NAME}` is invalid: Expects group index between 0
and \
+ {group_count}, but got {idx}. SQLSTATE: 22023"
+ )
+ })
+}
+
+fn compile(pattern: &str) -> Result<Regex> {
+ let quoted = to_sql_value(pattern);
+ let unsupported = |reason: String| {
+ DataFusionError::NotImplemented(format!(
+ "Unsupported regex pattern {quoted} in `{NAME}`: {reason}. \
+ See https://github.com/apache/datafusion-spark/issues/255"
+ ))
+ };
+ match java_difference(pattern) {
+ None => {}
+ Some(JavaDifference::Invalid) => {
+ return Err(exec_datafusion_err!(
+ "[INVALID_PARAMETER_VALUE.PATTERN] The value of parameter(s)
`regexp` in \
+ `{NAME}` is invalid: {quoted}. SQLSTATE: 22023"
+ ));
+ }
+ Some(JavaDifference::Unsupported(reason)) => return
Err(unsupported(reason)),
+ }
+ // The pattern parsed, so a later stage of the Rust engine rejected it.
+ Regex::new(pattern).map_err(|err| match err {
+ regex::Error::CompiledTooBig(_) => unsupported(
+ "the compiled pattern exceeds the Rust regex size
limit".to_string(),
+ ),
+ err => unsupported(format!("the Rust regex engine rejects it
({err})")),
+ })
+}
+
+/// How `java.util.regex`, which Spark uses, would treat a pattern differently
+/// from the Rust regex engine.
+#[derive(Debug)]
+enum JavaDifference {
+ /// Java rejects the pattern, so it is reported as an invalid pattern.
+ Invalid,
+ /// Java accepts the pattern but the Rust engine rejects it, or reads it
+ /// differently, so it is reported as unsupported rather than evaluated to
+ /// a different answer than Spark's.
+ Unsupported(String),
+}
+
+/// Checks a pattern with `regex_syntax`, whose AST and error kinds are API,
+/// rather than matching on the text of [`regex::Error`].
+///
+/// A pattern that does not parse is invalid only for the kinds of syntax error
+/// that Java also raises. Anything else, such as `\Q...\E`, `\Z`, `\h`,
+/// `(?>...)` or `(?d)`, is unsupported, which also covers a few patterns that
+/// Java rejects, such as `\y`. A pattern that parses is walked for the
+/// constructs that the two engines read differently.
+fn java_difference(pattern: &str) -> Option<JavaDifference> {
+ use regex_syntax::ast::ErrorKind;
+ let ast_err = match regex_syntax::ast::parse::Parser::new().parse(pattern)
{
+ Ok(ast) => return regex_syntax::ast::visit(&ast,
JavaDifferences).err(),
+ Err(err) => err,
+ };
+ let unsupported = |reason: String|
Some(JavaDifference::Unsupported(reason));
+ match ast_err.kind() {
+ ErrorKind::GroupUnclosed
+ | ErrorKind::GroupUnopened
+ | ErrorKind::ClassUnclosed
+ | ErrorKind::ClassRangeInvalid
+ | ErrorKind::RepetitionMissing
+ | ErrorKind::RepetitionCountInvalid => Some(JavaDifference::Invalid),
+ // Java also rejects an unknown flag, except `d` (UNIX_LINES) and `>`
+ // (an atomic group).
+ ErrorKind::FlagUnrecognized => {
+ let span = ast_err.span();
+ let flag = &pattern[span.start.offset..span.end.offset];
+ if matches!(flag, "d" | ">") {
Review Comment:
Java also accepts the `c` flag (`CANON_EQ`, canonical equivalence).
`regexp_extract('a', '(?c)a', 0)` is `a` in Spark 4.2.0, and `(?-c)a` and
`(?ic)A` work too. Here they get `INVALID_PARAMETER_VALUE.PATTERN`, which tells
the user a valid Spark pattern is invalid. The Rust engine has no canonical
equivalence, so could `c` be unsupported like `d` and `>`?
```suggestion
// Java also rejects an unknown flag, except `c` (CANON_EQ), `d`
// (UNIX_LINES) and `>` (an atomic group).
ErrorKind::FlagUnrecognized => {
let span = ast_err.span();
let flag = &pattern[span.start.offset..span.end.offset];
if matches!(flag, "c" | "d" | ">") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]