alamb opened a new issue, #16756: URL: https://github.com/apache/datafusion/issues/16756
### Is your feature request related to a problem or challenge? https://github.com/apache/datafusion/issues/14836 @pepijnve [asked in discord](https://discord.com/channels/885562378132000778/1166447479609376850/1393235157850259476): > The project I'm working on uses a parquet-files-on-s3 style approach for its storage. In order to tell DataFusion where all those files are located I'm implementing a catalog that's similar in spirit to duck lake. What I bumped into now is that at the moment there's no support for attaching databases using ATTACH .... CREATE EXTERNAL TABLE... is the closest thing. Has the idea of supporting this been explored already? I couldn't find anything in the issue tracker. Seems like it might be useful for Iceberg, DeltaLake, DuckLake, etc. > > I found the duckdb attach and sqlite attach variants in the sqlparser in the meantime. What I had in mind was something a bit more flexible/general purpose. Something closer to CREATE EXTERNAL CATALOG ... STORED AS ... LOCATION ... OPTIONS ... with a extension mechanism. It turns out there is an example of how to extend the SQL parser (and it is the mechanism that datafusio-cli uses): https://github.com/apache/datafusion/blob/main/datafusion-examples/examples/sql_dialect.rs However, it would be nice to make this example more visible / easy to find / understand ### Describe the solution you'd like Write a blog post with a title something like "Implementing your own SQL dialect and SQL statements with DataFusion" ### Describe alternatives you've considered I suggest following the "low key technical evangelism" recipe: teach the readers something about how databases work in general (using DataFusion as an example) and then show how to use DataFusion to do something cool An outline might be Introduce the idea at a high level: 1. You want to add some feature to SQL (maybe use the motivating example from @pepijnve above) 2. Background: How SQL is normally planned (sql text --> sqlparser AST --> LogicalPlan) 3. Explain how to insert your own parser / point to the examples 4. Show relevant excerpts from the example ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
