iffyio commented on code in PR #1621:
URL:
https://github.com/apache/datafusion-sqlparser-rs/pull/1621#discussion_r1898558274
##########
src/tokenizer.rs:
##########
@@ -1141,33 +1141,44 @@ impl<'a> Tokenizer<'a> {
let s2 = peeking_take_while(chars, |ch|
ch.is_ascii_hexdigit());
return Ok(Some(Token::HexStringLiteral(s2)));
}
-
// match one period
- if let Some('.') = chars.peek() {
- // Check if this actually is a float point number
- let mut char_clone = chars.peekable.clone();
- char_clone.next();
- // Next char should be a digit, otherwise, it is not a
float point number
- if char_clone
- .peek()
- .map(|c| c.is_ascii_digit())
- .unwrap_or(false)
- {
+ if self.dialect.support_unquoted_hyphenated_identifiers() {
Review Comment:
@goldmedal @ayman-sigma
I'm wondering if it would make sense to move this special case handling from
the tokenizer and into the parser. Since this is a BigQuery specific feature
and only applies in a specific context, I feel like it would be more
complicated to pull off within the tokenizer without affecting other use
cases/dialects. Thinking in the parser we have more context on when this syntax
is valid and a lot of that work was done in
https://github.com/apache/datafusion-sqlparser-rs/pull/1109
(I'm imagining since the tokenizer would be left to parse decimals as usual,
we won't have the consideration in
https://github.com/apache/datafusion-sqlparser-rs/issues/1622 ?)
Essentially I'm wondering if we can extend [the
work](https://github.com/apache/datafusion-sqlparser-rs/blob/6daa4b059cde8b77b67a3699b174ef0f8edff350/src/parser/mod.rs#L9015-L9063)
to cover this scenario doublle number scenario. Using the example from
https://github.com/apache/datafusion-sqlparser-rs/pull/1598 it would
essentially mean that e.g. the following token stream
`[Word("foo"), Minus, Number("123."), Word("bar")]`
gets combined into
`ObjectName(vec![Word("foo-123"), Word("bar")])`
would something like that be feasible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
