+1. Let's do it :). I think if others will agree, a LAZY CONSENSUS thread
will do + documenting the format as PR + getting each team to implement it
is a good way to progress.
Once we have it exposed by at least two dashboards, we can add a "canary"
test that will pull them and fail in case of error (soft - non-blocking
fail).
On Mon, Aug 5, 2024 at 11:07 AM Freddy Demiane <fdemi...@google.com.invalid>
wrote:
> Hello all,
>
> I took a long time to reply to this thread, I was on a long vacation. +1 to
> Dennis' suggestion, I believe the provided JSON format contains the minimal
> information required to render a dashboard. I experimented with generating
> a JSON output from a development CI (from Google's side), I added a sample
> output of how it could look like.
> As of the Junit XML format, while I was searching online, I found multiple
> different standards, and for tools to visualize results, each tool has
> their own standard (even though the formats look similar). Again, I never
> used the JUnit XML format nor tooling for visualization, so I might be
> wrong in this regard.
> I suggest sticking with a defined JSON schema, and later if we want to
> generate some aggregated report, and decide to go with a Junit XML
> visualizer, we can transform the data.
> Nevertheless, I played a bit with generating a Junit XML result, which I
> also attached in this email.
> Let me know what you think.
>
> Best,
> Freddy
>
> JSON Sample Output:
>
> {
> "provider": "Google",
> "tests": 159,
> "errors": 0,
> "failures": 21,
> "skipped": 68,
> "timestamp": "2024-07-30 12:00:00.460429+00:00",
> "duration": 8263.065936,
> "properties": {
> "airflow_commit_id": "41508f23ad",
> "providers_commit_id": "41508f23ad"
> },
> "testcases": [
> {
> "name": "example_gcp_translate_speech",
> "file": "
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/translate_speech/example_translate_speech.py
> ",
> "duration": 0,
> "result": {
> "state": "SKIPPED",
> "message": "Skipped",
> "type": ""
> },
> "properties": {
> "logs_folder_link": "",
> "failed_tasks": []
> },
> "skipped": true
> },
> {
> "name": "example_gcp_translate",
> "file": "
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/translate/example_translate.py
> ",
> "duration": "0:00:02.443829",
> "result": {
> "state": "SUCCESS",
> "message": "",
> "type": ""
> },
> "properties": {
> "logs_folder_link":
> "
> https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-30
> 12:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=example_gcp_translate
> <https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-3012:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=example_gcp_translate>
> ",
> "failed_tasks": []
> },
> "skipped": false
> },
> {
> "name": "vertex_ai_pipeline_job_operations",
> "file": "
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_pipeline_job.py
> ",
> "duration": "0:24:35.451846",
> "result": {
> "state": "FAILURE",
> "message": "Check the logs at:
>
> https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-30
>
> 12:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_pipeline_job_operations
> <https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-3012:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_pipeline_job_operations>
> ",
> "type": "Test Failure"
> },
> "properties": {
> "logs_folder_link":
> "
> https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-30
>
> 12:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_pipeline_job_operations
> <https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-3012:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_pipeline_job_operations>
> ",
> "failed_tasks": [
> "watcher",
> "run_pipeline_job"
> ]
> },
> "skipped": false
> },
>
> XML Sample Output:
>
> <testsuite name="Google" tests="159" failures="21" time="8263.065936"
> timestamp="2024-07-30 12:00:00.460429+00:00">
> <testcase name="example_gcp_translate_speech"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/translate_speech/example_translate_speech.py
> "
> time="0">
> <skipped>Ignored</skipped>
> </testcase>
> <testcase name="example_gcp_translate"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/translate/example_translate.py
> "
> time="0:00:02.443829"/>
> <testcase name="vertex_ai_pipeline_job_operations"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_pipeline_job.py
> "
> time="0:24:35.451846">
> <failure message="Check the logs at:
>
> https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-30
>
> 12:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_pipeline_job_operations
> <https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-3012:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_pipeline_job_operations>
> "
> type="ERROR"/>
> </testcase>
> <testcase name="vertex_ai_custom_job_operations_list_custom_job"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_list_custom_jobs.py
> "
> time="0:00:19.742450"/>
> <testcase name="vertex_ai_auto_ml_operations_video_training_job"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_auto_ml_video_training.py
> "
> time="-1">
> <failure message="" type="ERROR"/>
> </testcase>
> <testcase name="vertex_ai_batch_prediction_operations"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_batch_prediction_job.py
> "
> time="2:00:16.013429">
> <failure message="Check the logs at:
>
> https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-30
>
> 12:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_batch_prediction_operations
> <https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-3012:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_batch_prediction_operations>
> "
> type="ERROR"/>
> </testcase>
> <testcase name="vertex_ai_generative_model_dag"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_generative_model.py
> "
> time="0:00:18.350772"/>
> <testcase name="vertex_ai_custom_job_operations_custom"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_custom_job.py
> "
> time="0:21:58.908473"/>
> <testcase name="vertex_ai_auto_ml_operations_list_training_job"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_auto_ml_list_training.py
> "
> time="0:00:24.284901"/>
> <testcase name="vertex_ai_auto_ml_operations_forecasting_training_job"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_auto_ml_forecasting_training.py
> "
> time="1:42:17.086311"/>
> <testcase name="vertex_ai_auto_ml_operations_text_training_job"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_auto_ml_text_training.py
> "
> time="-1">
> <failure message="" type="ERROR"/>
> </testcase>
> <testcase name="vertex_ai_hyperparameter_tuning_job_operations"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_hyperparameter_tuning_job.py
> "
> time="0:31:34.276985">
> <failure message="Check the logs at:
>
> https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-30
>
> 12:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_hyperparameter_tuning_job_operations
> <https://console.cloud.google.com/storage/browser/dashboard-system-tests-public-logs-dev/2024-07-3012:00:00.460429+00:00-41508f23ad-41508f23ad/dag_id=vertex_ai_hyperparameter_tuning_job_operations>
> "
> type="ERROR"/>
> </testcase>
> <testcase name="vertex_ai_model_service_operations"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_model_service.py
> "
> time="0:19:51.716990"/>
> <testcase name="vertex_ai_auto_ml_operations_tabular_training_job"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_auto_ml_tabular_training.py
> "
> time="1:57:02.637592"/>
> <testcase name="vertex_ai_endpoint_service_operations"
> file="
> https://github.com/apache/airflow/blob/main/tests/system/providers/google/cloud/vertex_ai/example_vertex_ai_endpoint.py
> "
> time="-1">
> <failure message="" type="ERROR"/>
> </testcase>
>
>
> On Wed, Jun 26, 2024 at 12:47 AM Ferruzzi, Dennis
> <ferru...@amazon.com.invalid> wrote:
>
> > I was unaware of the Teradata dashboard! Outstanding to see.
> >
> > I can take point on design discussion and documentation for this, but in
> > the end it'll be up to each provider to update their own infra, so there
> is
> > only so much I can do.
> >
> > I didn't really think this would catch on so enthusiastically. One
> other
> > thing I was thinking about but dropped from the initial idea would be
> > adding an optional field to the provider.yaml with a dashboard url.
> > Currently it is up to the provider to manually add a link to the list on
> > the ecosystem page. If we make it part of the yaml file, new providers
> > might see it when looking for a template and jump onboard. It would also
> > make the dashboards more programmatically-discoverable, maybe even
> > something that can be used in generating a docs page and skip the manual
> > step of adding it to the ecosystem page if someone wants to do that at
> some
> > point. Given the way this discussion caught on, maybe it should be two
> > fields: 'dashboard-html' and 'dashboard-json' (or -xml or whatever we
> > decide is the vended format).
> >
> >
> > As at least half of the existing dashboards already export some form of
> > json, I'd propose we stick to that unless someone has a compelling reason
> > to convert to XML? I looked into junit-xml and I like the way they break
> > down their schema, so maybe json-ify that with some tweaks?
> >
> > Proposed formatting:
> >
> > {
> > "testsuite": {
> > "provider": string, [REQUIRED]
> > "tests": int, [REQUIRED] // Could drop this as it's just
> > len(testcases) but by that same logic it's easy enough to add it...
> > "errors": int, [REQUIRED]
> > "failures": int, [REQUIRED]
> > "skipped": int, [REQUIRED]
> > "timestamp": string, [REQUIRED] // Standardize on UTC?
> > "duration": float, [OPTIONAL] // Seconds?
> > "properties": {}, [OPTIONAL] // Let's make all "properties"
> > blocks free-form and optional; a provider may add whatever extra values
> in
> > this block that they want.
> > "testcases": [
> > {
> > "name": string, [OPTIONAL]
> > "file": string, [REQUIRED]
> > "duration": float, [OPTIONAL] // Seconds?
> > "result": {
> > "state": "SUCCESS" | "SKIPPED" | "FAILURE", [REQUIRED]
> > "message": string, [OPTIONAL]
> > "type": string, [OPTIONAL] // Exception type in case of a
> > failure.
> > },
> > "properties": {}, [OPTIONAL] // Let's make all "properties"
> > blocks free-form and optional; a provider may add whatever extra values
> in
> > this block that they want.
> > },
> > ]
> > }
> > }
> >
> > Sample:
> >
> >
> > {
> > "testsuite": {
> > "provider": "AWS",
> > "tests": 3,
> > "errors": 0,
> > "failures": 1,
> > "skipped": 1,
> > "timestamp": "2020-01-26T13:45:02",
> > "duration": 139.89,
> > "properties": {
> > "commit": "ef7bebf",
> > "executor": "celery",
> > },
> > "testcases": [
> > // Example successful test
> > {
> > "name": "example_appflow",
> > "file": "tests/system/providers/amazon/aws/example_appflow.py",
> > "duration": 45.87,
> > "result": {
> > "state": "SUCCESS"
> > },
> > "properties": {
> > "source": "
> >
> https://github.com/apache/airflow/blob/main/tests/system/providers/amazon/aws/example_appflow.py
> > ",
> > "operators": [
> > "AppflowRunOperator",
> > "S3CreateBucketOperator",
> > "S3CreateObjectOperator",
> > "S3DeleteBucketOperator"
> > ]
> > }
> > },
> > // Example of a test case that was skipped.
> > {
> > "name": "example_athena",
> > "file": "tests/system/providers/amazon/aws/example_athena.py",
> > "duration": 0.01,
> > "result": {
> > "state": "SKIPPED",
> > "message": "Message explaining why."
> > },
> > "properties": {
> > "source": "
> >
> https://github.com/apache/airflow/blob/main/tests/system/providers/amazon/aws/example_athena.py
> > ",
> > "operators": [
> > "AthenaOperator",
> > "S3CreateBucketOperator",
> > "S3CreateObjectOperator",
> > "S3DeleteBucketOperator"
> > ]
> > }
> > },
> > // Example of a test case that failed.
> > {
> > "name": "example_batch",
> > "file": "tests/system/providers/amazon/aws/example_batch.py",
> > "duration": 94.01,
> > "result": {
> > "state": "FAILURE",
> > "message": "Some failure message, maybe a link to logs or a
> > stack trace?",
> > "type": "AssertionError",
> > },
> > "properties": {
> > "source": "
> >
> https://github.com/apache/airflow/blob/main/tests/system/providers/amazon/aws/example_batch.py
> > ",
> > "operators": [
> > "BatchCreateComputeEnvironmentOperator",
> > "BatchComputeEnvironmentSensor",
> > "BatchJobQueueSensor",
> > "BatchOperator",
> > "BatchSensor",
> > ]
> > }
> > },
> > ]
> > }
> > }
> >
> >
> > - ferruzzi
> >
> >
> > ________________________________
> > From: Eugen Kosteev <eu...@kosteev.com>
> > Sent: Tuesday, June 25, 2024 6:04 AM
> > To: dev@airflow.apache.org
> > Subject: RE: [EXT] System Test Dashboards - Phase Two??
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne
> pouvez
> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain
> que
> > le contenu ne présente aucun risque.
> >
> >
> >
> > +1 to the idea of standardizing the format of the system test results
> > output
> >
> > On Tue, Jun 25, 2024 at 10:40 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > So we have almost everyone on board!
> > >
> > > Now we also need the Teradata team to add whatever JSON/XML we come up
> > with
> > > :). In case people have not noticed, among our dashboards [1] we also
> > have
> > > Teradata dashboard [2]
> > >
> > > [1]
> > >
> > >
> >
> https://airflow.apache.org/ecosystem/#airflow-provider-system-test-dashboards
> > > [2] https://teradata.github.io/airflow/
> > >
> > > Anyone would like to take a lead on it? I am personally fine with
> either
> > > approach - Junit xml (or json version of it), or custom json is fine
> for
> > > me.
> > >
> > > J.
> > >
> > >
> > > On Tue, Jun 25, 2024 at 10:24 AM Pankaj Koti
> > > <pankaj.k...@astronomer.io.invalid> wrote:
> > >
> > > > For context, the Astronomer LLM providers dashboard operates as
> > follows:
> > > >
> > > > 1. Fetch the latest source code for providers and system
> tests/example
> > > DAGs
> > > > from the Airflow repository, deploy them to an Airflow instance, and
> > > > execute the
> > > > DAGs.
> > > > 2. Use the Airflow API to retrieve the DAG run statuses and produce a
> > > JSON
> > > > output of these statuses.
> > > > 3. The dashboard, hosted on GitHub Pages, consumes the JSON data
> > > > generated in step 2.
> > > >
> > > > We are willing to adopt and adhere to a JSON or XML specification
> and a
> > > > model HTML view if one is established.
> > > >
> > > > Best regards,
> > > >
> > > > *Pankaj Koti*
> > > > Senior Software Engineer (Airflow OSS Engineering team)
> > > > Location: Pune, Maharashtra, India
> > > > Timezone: Indian Standard Time (IST)
> > > >
> > > >
> > > > On Mon, Jun 24, 2024 at 11:40 PM Ferruzzi, Dennis
> > > > <ferru...@amazon.com.invalid> wrote:
> > > >
> > > > > > The information in our database is similar to the structure of
> the
> > > AWS
> > > > > providers json file
> > > > > >
> > https://aws-mwaa.github.io/open-source/system-tests/dashboard.json
> > > +
> > > > a
> > > > > field for logs.
> > > > > > We also have an extra field that specifies the commit-id against
> > > which
> > > > > the CI was run,
> > > > > > which I believe is helpful in case users want to know whether
> > their
> > > PR
> > > > > was merged before
> > > > > > or after a failure.
> > > > >
> > > > > The commit ID is a handy addition for sure, I may look into adding
> > that
> > > > to
> > > > > the AWS dashboard. I haven't had a chance to look into junit-xml
> > yet,
> > > > but
> > > > > I think what we could do is agree on a minimum structure and allow
> > for
> > > > > extras. For example, logs are great, but if Google provides them
> > and
> > > > AWS
> > > > > doesn't, that shouldn't break anything for the user trying to fetch
> > > logs.
> > > > > But the test name, timestamp, and success/fail state are definitely
> > > among
> > > > > the required minimum fields.
> > > > >
> > > > > > we could consider enforcing the presence of *some* dashboard that
> > > shows
> > > > > results of regular system tests executions for any new provider.
> > > > >
> > > > > The issue there is that smaller providers come and go, and are
> often
> > > > added
> > > > > by community members, not even necessarily with the provider's
> > > knowledge.
> > > > > We can't force them to provide any support. If Random Contributor
> > > adds
> > > > > support for a new provider, neither the contributor nor the
> provider
> > > can
> > > > be
> > > > > required to provide hosting for a dashboard and infrastructure to
> run
> > > the
> > > > > tests. So (for the foreseeable future) the dashboards need to be
> an
> > > > opt-in
> > > > > project by/for the providers. Maybe some day the project might be
> > > able
> > > > to
> > > > > provide hosting for the smaller dashboards or something, but I
> think
> > > the
> > > > > infrastructure to run the tests will always be optional and at the
> > > > expense
> > > > > (and effort) of some other interested party (almost certainly the
> > > > provider
> > > > > themselves, but who knows... ).
> > > > >
> > > > >
> > > > > - ferruzzi
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: Michał Modras <michalmod...@google.com.INVALID>
> > > > > Sent: Monday, June 24, 2024 5:20 AM
> > > > > To: dev@airflow.apache.org
> > > > > Subject: RE: [EXT] System Test Dashboards - Phase Two??
> > > > >
> > > > > CAUTION: This email originated from outside of the organization. Do
> > not
> > > > > click links or open attachments unless you can confirm the sender
> and
> > > > know
> > > > > the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur
> > > externe.
> > > > > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous
> ne
> > > > pouvez
> > > > > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas
> > certain
> > > > que
> > > > > le contenu ne présente aucun risque.
> > > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > +1 to this idea. I think standardizing the format of the presented
> > test
> > > > run
> > > > > results makes sense. I also agree that we don't necessarily need to
> > > > enforce
> > > > > it in any hard way. However, given that we have dashboards of these
> > > three
> > > > > major providers, we could consider enforcing the presence of *some*
> > > > > dashboard
> > > > > that shows results of regular system tests executions for any new
> > > > provider.
> > > > > WDYT?
> > > > >
> > > > > Best,
> > > > > Michal
> > > > >
> > > > > On Sun, Jun 23, 2024 at 10:09 PM Freddy Demiane
> > > > > <fdemi...@google.com.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Thank you for the comments! Indeed, +1 to the idea, I believe
> this
> > > > would
> > > > > be
> > > > > > a good step to increase the quality of providers. From our
> (Google)
> > > > side,
> > > > > > the dashboard's CI outputs the results to a database, which are
> > then
> > > > used
> > > > > > to generate an HTML page. Yet, generating and publishing a JSON
> or
> > a
> > > > > JUnit
> > > > > > XML style file would be a simple task for us. The information in
> > our
> > > > > > database is similar to the structure of the AWS providers json
> file
> > > > > >
> https://aws-mwaa.github.io/open-source/system-tests/dashboard.json
> > +
> > > > > > a field for logs. We also have an extra field that specifies the
> > > > > commit-id
> > > > > > against which the CI was run, which I believe is helpful in case
> > > users
> > > > > want
> > > > > > to know whether their PR was merged before or after a failure.
> > > > > > If we want to go with the junit-xml style format (I checked this
> > > > > reference
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.ibm.com/docs/en/developer-for-zos/16.0?topic=formats-junit-xml-format
> > > > > > >),
> > > > > > one thing I could think of is to make each "Dashboard CI run"
> > > generate
> > > > an
> > > > > > xml file where each test is represented by a testcase, which as
> > Jarek
> > > > > > mentioned, could be used in some way in the canary builds.
> > > > > > Let me know what you think.
> > > > > >
> > > > > > Best,
> > > > > > Freddy
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 21, 2024 at 11:12 AM Jarek Potiuk <ja...@potiuk.com>
> > > > wrote:
> > > > > >
> > > > > > > This is a fantastic idea! I love it !
> > > > > > >
> > > > > > > It also has some very far reaching possible spin-offs in the
> > > future -
> > > > > > > literally few days ago, when I discussed some of the future
> > > security
> > > > > > > related work that we might want to do, there was a concept of
> > > having
> > > > a
> > > > > > sort
> > > > > > > of CI of all CIs where we (and by we I mean wider Python
> > ecosystem)
> > > > > could
> > > > > > > gather a status of pre-release versions of dependencies before
> > they
> > > > hit
> > > > > > > release stage, and some kind of interchange between those CI
> > > systems
> > > > > that
> > > > > > > will be machine-parseable is pretty much prerequisite for that.
> > So
> > > we
> > > > > > could
> > > > > > > generally try it out and sort out some issues, see how it works
> > in
> > > > our
> > > > > > > small "airflow" world, but in the future we might be able to
> use
> > > > > similar
> > > > > > > mechanisms to get alerts for a number of our dependencies - and
> > > even
> > > > > > > further than that, we could make such approach much more
> > > wide-spread
> > > > (I
> > > > > > am
> > > > > > > discussing it with people from Python Software
> > Foundation/Packaging
> > > > > team
> > > > > > /
> > > > > > > Python security, so there is a chance this might actually
> > > materialize
> > > > > in
> > > > > > a
> > > > > > > long term). This would be the first step.
> > > > > > >
> > > > > > > I think the first step for it could be rather simple and we do
> > not
> > > > have
> > > > > > to
> > > > > > > invent our own standard - we could easily start with junit-xml
> > > style
> > > > > > output
> > > > > > > produced by each dashboard and available under some URL that we
> > > could
> > > > > > pull
> > > > > > > in our canary builds and have a step in our canary builds that
> > > could
> > > > > > > aggregate multiple xmlunit files coming from various
> dashboards,
> > > > > display
> > > > > > > them as the output, and fail the job in case some tests are
> > failing
> > > > > (with
> > > > > > > maybe some thresholds). Pytest and a number of tools natively
> > > > supports
> > > > > > the
> > > > > > > junit-xml format, it's pretty established as machine-readable
> > test
> > > > > > results,
> > > > > > > and I think it has all we need to start with
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.pytest.org/en/latest/how-to/usage.html#creating-junitxml-format-files
> > > > > > > .
> > > > > > > There is a lot of tooling around this format - including easy
> > ways
> > > > how
> > > > > we
> > > > > > > could possibly integrate it with Github Actions output (think
> > links
> > > > to
> > > > > > the
> > > > > > > tests that failed directly in GitHub UI), showing logs of
> failed
> > > > tests
> > > > > > etc.
> > > > > > > etc.
> > > > > > >
> > > > > > > If we can get the Astronomer, Amazon and Google team on board
> > with
> > > > it,
> > > > > we
> > > > > > > could likely implement a simple version quickly and iterate
> over
> > > it -
> > > > > > later
> > > > > > > we could think about possibly evolving that into a more
> > extensible
> > > > > > > approach.
> > > > > > >
> > > > > > > J.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jun 20, 2024 at 11:27 PM Ferruzzi, Dennis
> > > > > > > <ferru...@amazon.com.invalid> wrote:
> > > > > > >
> > > > > > > > Congrats to the Google team for getting their dashboard live,
> > it
> > > > > looks
> > > > > > > > great! I've been thinking of something for a while and
> thought
> > > I'd
> > > > > > > mention
> > > > > > > > it here. I'm wearing a few different hats here so I'll try
> to
> > > > > clarify
> > > > > > > > context on my plural pronouns the best I can.
> > > > > > > >
> > > > > > > > Now that we [Providers] have a couple of big dashboards up,
> I'm
> > > > > curious
> > > > > > > if
> > > > > > > > we [Airflow dev community] might collaborate on a community
> > > > "optional
> > > > > > > > guideline" for a json (or yaml or whatever) format output on
> > the
> > > > > > > dashboards
> > > > > > > > for any providers interested in participating. I'm not
> > > interested
> > > > in
> > > > > > (or
> > > > > > > > trying to) impose any kind of hard-line policy or standard
> > here,
> > > > but
> > > > > I
> > > > > > > > wonder if we [owners of the existing dashboards] might set
> some
> > > > > > > non-binding
> > > > > > > > precedent for future providers to join. If others don't
> follow
> > > > suit,
> > > > > > > then
> > > > > > > > they wouldn't benefit from whatever uses folks come up with
> for
> > > the
> > > > > > data,
> > > > > > > > but I personally don't think we [Airflow] can or should try
> to
> > > > impose
> > > > > > > this
> > > > > > > > on providers.
> > > > > > > >
> > > > > > > > To my knowledge there are three provider-owned system test
> > > > dashboards
> > > > > > > > currently live, and I look forward to seeing more in time:
> > > > > > > >
> > > > > > > > Astronomer (found this LLM-specific one, not sure if there is
> > > > another
> > > > > > > > one): https://astronomer.github.io/llm-dags-dashboard/
> > > > > > > > AWS:
> > > > > >
> https://aws-mwaa.github.io/open-source/system-tests/dashboard.html
> > > > > > > > and
> > > > > >
> https://aws-mwaa.github.io/open-source/system-tests/dashboard.json
> > > > > > > > Google:
> > > > > > > >
> > > > >
> > https://storage.googleapis.com/providers-dashboard-html/dashboard.html
> > > > > > > >
> > > > > > > > Each was developed independently, and the path/name of the
> > Google
> > > > one
> > > > > > may
> > > > > > > > hint that there is already an alternative to the html view
> that
> > > I'm
> > > > > > just
> > > > > > > > not familiar with, so maybe we [the three providers] could
> > > > > collaborate
> > > > > > on
> > > > > > > > some precedent that others could follow? We [AWS] already
> have
> > > > ours
> > > > > > > > exporting in json so discussion might start there and see
> where
> > > it
> > > > > > goes?
> > > > > > > > Either way... Even if we [Airflow] don't do anything with the
> > > > json, I
> > > > > > > bet a
> > > > > > > > user could find interesting things to build if we give them
> the
> > > > > tools.
> > > > > > > > Maybe aggregating a dashboard which monitors (and alerts?)
> the
> > > > > status
> > > > > > of
> > > > > > > > the system tests which cover the operators their workflow
> > depends
> > > > on,
> > > > > > > > maybe? Who knows what someone may come up with once they
> have
> > > the
> > > > > > tools
> > > > > > > to
> > > > > > > > mix and match the data from various providers.
> > > > > > > >
> > > > > > > > Is there any interest in the idea of a "standard json schema"
> > for
> > > > > these
> > > > > > > > and any future system test dashboards?
> > > > > > > >
> > > > > > > >
> > > > > > > > - ferruzzi
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Eugene
> >
>
