This is a fantastic idea! I love it ! It also has some very far reaching possible spin-offs in the future - literally few days ago, when I discussed some of the future security related work that we might want to do, there was a concept of having a sort of CI of all CIs where we (and by we I mean wider Python ecosystem) could gather a status of pre-release versions of dependencies before they hit release stage, and some kind of interchange between those CI systems that will be machine-parseable is pretty much prerequisite for that. So we could generally try it out and sort out some issues, see how it works in our small "airflow" world, but in the future we might be able to use similar mechanisms to get alerts for a number of our dependencies - and even further than that, we could make such approach much more wide-spread (I am discussing it with people from Python Software Foundation/Packaging team / Python security, so there is a chance this might actually materialize in a long term). This would be the first step.
I think the first step for it could be rather simple and we do not have to invent our own standard - we could easily start with junit-xml style output produced by each dashboard and available under some URL that we could pull in our canary builds and have a step in our canary builds that could aggregate multiple xmlunit files coming from various dashboards, display them as the output, and fail the job in case some tests are failing (with maybe some thresholds). Pytest and a number of tools natively supports the junit-xml format, it's pretty established as machine-readable test results, and I think it has all we need to start with https://docs.pytest.org/en/latest/how-to/usage.html#creating-junitxml-format-files. There is a lot of tooling around this format - including easy ways how we could possibly integrate it with Github Actions output (think links to the tests that failed directly in GitHub UI), showing logs of failed tests etc. etc. If we can get the Astronomer, Amazon and Google team on board with it, we could likely implement a simple version quickly and iterate over it - later we could think about possibly evolving that into a more extensible approach. J. On Thu, Jun 20, 2024 at 11:27 PM Ferruzzi, Dennis <ferru...@amazon.com.invalid> wrote: > Congrats to the Google team for getting their dashboard live, it looks > great! I've been thinking of something for a while and thought I'd mention > it here. I'm wearing a few different hats here so I'll try to clarify > context on my plural pronouns the best I can. > > Now that we [Providers] have a couple of big dashboards up, I'm curious if > we [Airflow dev community] might collaborate on a community "optional > guideline" for a json (or yaml or whatever) format output on the dashboards > for any providers interested in participating. I'm not interested in (or > trying to) impose any kind of hard-line policy or standard here, but I > wonder if we [owners of the existing dashboards] might set some non-binding > precedent for future providers to join. If others don't follow suit, then > they wouldn't benefit from whatever uses folks come up with for the data, > but I personally don't think we [Airflow] can or should try to impose this > on providers. > > To my knowledge there are three provider-owned system test dashboards > currently live, and I look forward to seeing more in time: > > Astronomer (found this LLM-specific one, not sure if there is another > one): https://astronomer.github.io/llm-dags-dashboard/ > AWS: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html > and https://aws-mwaa.github.io/open-source/system-tests/dashboard.json > Google: > https://storage.googleapis.com/providers-dashboard-html/dashboard.html > > Each was developed independently, and the path/name of the Google one may > hint that there is already an alternative to the html view that I'm just > not familiar with, so maybe we [the three providers] could collaborate on > some precedent that others could follow? We [AWS] already have ours > exporting in json so discussion might start there and see where it goes? > Either way... Even if we [Airflow] don't do anything with the json, I bet a > user could find interesting things to build if we give them the tools. > Maybe aggregating a dashboard which monitors (and alerts?) the status of > the system tests which cover the operators their workflow depends on, > maybe? Who knows what someone may come up with once they have the tools to > mix and match the data from various providers. > > Is there any interest in the idea of a "standard json schema" for these > and any future system test dashboards? > > > - ferruzzi >
