Hello, Thank you for the comments! Indeed, +1 to the idea, I believe this would be a good step to increase the quality of providers. From our (Google) side, the dashboard's CI outputs the results to a database, which are then used to generate an HTML page. Yet, generating and publishing a JSON or a JUnit XML style file would be a simple task for us. The information in our database is similar to the structure of the AWS providers json file https://aws-mwaa.github.io/open-source/system-tests/dashboard.json + a field for logs. We also have an extra field that specifies the commit-id against which the CI was run, which I believe is helpful in case users want to know whether their PR was merged before or after a failure. If we want to go with the junit-xml style format (I checked this reference <https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format>), one thing I could think of is to make each "Dashboard CI run" generate an xml file where each test is represented by a testcase, which as Jarek mentioned, could be used in some way in the canary builds. Let me know what you think.
Best, Freddy On Fri, Jun 21, 2024 at 11:12 AM Jarek Potiuk <ja...@potiuk.com> wrote: > This is a fantastic idea! I love it ! > > It also has some very far reaching possible spin-offs in the future - > literally few days ago, when I discussed some of the future security > related work that we might want to do, there was a concept of having a sort > of CI of all CIs where we (and by we I mean wider Python ecosystem) could > gather a status of pre-release versions of dependencies before they hit > release stage, and some kind of interchange between those CI systems that > will be machine-parseable is pretty much prerequisite for that. So we could > generally try it out and sort out some issues, see how it works in our > small "airflow" world, but in the future we might be able to use similar > mechanisms to get alerts for a number of our dependencies - and even > further than that, we could make such approach much more wide-spread (I am > discussing it with people from Python Software Foundation/Packaging team / > Python security, so there is a chance this might actually materialize in a > long term). This would be the first step. > > I think the first step for it could be rather simple and we do not have to > invent our own standard - we could easily start with junit-xml style output > produced by each dashboard and available under some URL that we could pull > in our canary builds and have a step in our canary builds that could > aggregate multiple xmlunit files coming from various dashboards, display > them as the output, and fail the job in case some tests are failing (with > maybe some thresholds). Pytest and a number of tools natively supports the > junit-xml format, it's pretty established as machine-readable test results, > and I think it has all we need to start with > > https://docs.pytest.org/en/latest/how-to/usage.html#creating-junitxml-format-files > . > There is a lot of tooling around this format - including easy ways how we > could possibly integrate it with Github Actions output (think links to the > tests that failed directly in GitHub UI), showing logs of failed tests etc. > etc. > > If we can get the Astronomer, Amazon and Google team on board with it, we > could likely implement a simple version quickly and iterate over it - later > we could think about possibly evolving that into a more extensible > approach. > > J. > > > > > On Thu, Jun 20, 2024 at 11:27 PM Ferruzzi, Dennis > <ferru...@amazon.com.invalid> wrote: > > > Congrats to the Google team for getting their dashboard live, it looks > > great! I've been thinking of something for a while and thought I'd > mention > > it here. I'm wearing a few different hats here so I'll try to clarify > > context on my plural pronouns the best I can. > > > > Now that we [Providers] have a couple of big dashboards up, I'm curious > if > > we [Airflow dev community] might collaborate on a community "optional > > guideline" for a json (or yaml or whatever) format output on the > dashboards > > for any providers interested in participating. I'm not interested in (or > > trying to) impose any kind of hard-line policy or standard here, but I > > wonder if we [owners of the existing dashboards] might set some > non-binding > > precedent for future providers to join. If others don't follow suit, > then > > they wouldn't benefit from whatever uses folks come up with for the data, > > but I personally don't think we [Airflow] can or should try to impose > this > > on providers. > > > > To my knowledge there are three provider-owned system test dashboards > > currently live, and I look forward to seeing more in time: > > > > Astronomer (found this LLM-specific one, not sure if there is another > > one): https://astronomer.github.io/llm-dags-dashboard/ > > AWS: https://aws-mwaa.github.io/open-source/system-tests/dashboard.html > > and https://aws-mwaa.github.io/open-source/system-tests/dashboard.json > > Google: > > https://storage.googleapis.com/providers-dashboard-html/dashboard.html > > > > Each was developed independently, and the path/name of the Google one may > > hint that there is already an alternative to the html view that I'm just > > not familiar with, so maybe we [the three providers] could collaborate on > > some precedent that others could follow? We [AWS] already have ours > > exporting in json so discussion might start there and see where it goes? > > Either way... Even if we [Airflow] don't do anything with the json, I > bet a > > user could find interesting things to build if we give them the tools. > > Maybe aggregating a dashboard which monitors (and alerts?) the status of > > the system tests which cover the operators their workflow depends on, > > maybe? Who knows what someone may come up with once they have the tools > to > > mix and match the data from various providers. > > > > Is there any interest in the idea of a "standard json schema" for these > > and any future system test dashboards? > > > > > > - ferruzzi > > >
