On Git (and other VCS for the matter) specifically, I believe respec only supports GitHub because it uses the GitHub API instead of Git. I’m only guessing, but using Git for random access would have terrible performance and is likely not an option for them.
Airflow does not have the same problems though since we are allowed to cache things on-disk and do a batch update on each DAG-parsing iteration. I believe it is possible to add custom backends to fsspec, so Airflow should be able to extend the feature to cover arbitrary repositories without too much work. > On Jun 8, 2024, at 23:25, Jed Cunningham <jedcunning...@apache.org> wrote: > > Sorry for the delayed reply here. I've been chewing on this one a bit > though. > > One concern I have is that I highly value having a provider agnostic remote > git integration. fsspec, however, has local git or github - no arbitrary > remote git support. That means Airflow, in my view, can't just rely on > fsspec alone, but will have to also deal with cloning arbitrary repos. Or > we expect a "ready to go" cloned repo on local disk, which might be a good > tradeoff. This was something we'd have to tackle for AIP-63 anyways, though. > > Another concern, as I've been considering Airflow 3, is that I see wanting > to version "more" than just the DAG folder - possibly a whole venv (or > similar). Doing that versioning file-by-file doesn't feel particularly > practical to me. Which means that, I think, we could end up using fsspec to > get zips/tarballs/"assets", which is still very helpful imo. But it does > limit the "slickness" of being sorta behind the scenes like a custom module > loader and resource loader would be. > > I actually see this as also being coupled to Ash's upcoming “Task Execution > interface”. > > I admittedly didn't quite follow the VersionedFS idea you all were > discussing. I don't see how it could easily allow Airflows goal of "this > version of the dag folder and maybe more" to a backend that is versioned by > file. Maybe you can eli5 and connect the dots for me? > > All that said, my goal this next week is to spend some quality time in this > area so we can all start nailing down options for Airflow 3. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
