potiuk commented on issue #25088: URL: https://github.com/apache/datafusion/issues/25088#issuecomment-6419037582
Good ! hope others will follow - and to be 100% we are not done yet - we are still testing the "Your PR - Your CI" case, this is not yet turned on for external contributors, so I expect even more usage decrease from Airflow. On Mon, Oct 26, 2026 at 3:42 PM Andrew Lamb ***@***.***> wrote: > *alamb* left a comment (apache/datafusion#25088) > <https://github.com/apache/datafusion/issues/25088#issuecomment-6418849635> > > Anecdotally I see PRs from arrow-rs and datafusion running with shorter > queue times. > > Top 30 by running > > The "prev" column is Thursday's queued/running for the same repo. > > ┌──────────────────────┬────────┬─────────┬────────────┬────────────────────────────────────────────────────────────┐ > │ repo │ queued │ running │ prev (q/r) │ workflows │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ fineract │ 0 │ 100 │ 0/2 │ Fineract CI=100 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ cloudstack │ 6 │ 92 │ 9/45 │ Build=3; CodeQL Analysis=5; Running Copilot Code Review... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ airflow │ 2 │ 91 │ 0/120 │ Tests (AMD)=93 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ celeborn │ 4 │ 65 │ 0/0 │ Celeborn CI=24; Celeborn SBT CI=41; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ maven │ 0 │ 49 │ 0/0 │ Java CI=49 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ grails-core │ 17 │ 44 │ 19/77 │ CI=54; CodeQL; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ iotdb │ 5 │ 38 │ 0/23 │ Cluster IT - 1C1D=8; Cluster IT - 1C1D1A=6; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ iggy │ 4 │ 38 │ 2/16 │ Pre-merge=38; Codecov notify=2; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ kyuubi │ 0 │ 38 │ 0/0 │ CI=38 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ arrow │ 0 │ 29 │ 0/0 │ C++=8; C GLib & Ruby=3; Docs=2; Integration=2; Python... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ iceberg │ 0 │ 28 │ 3/5 │ CVE Scan=8; Flink CI=3; Delta Conversion CI=2; Hive... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ beam │ 0 │ 25 │ 0/53 │ CodeQL Advanced=1; PerformanceTests TFRecordIOIT=1; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ paimon │ 59 │ 23 │ 67/9 │ CI=82 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ maven-javadoc-plugin │ 57 │ 19 │ 0/0 │ Verify=76 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ ozone │ 0 │ 19 │ 2/28 │ build-branch=19 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ brpc │ 3 │ 16 │ 2/0 │ Build and Test on Linux=16; License Check=3 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ datafusion-comet │ 4 │ 15 │ 0/4 │ Comet CI=12; Check PR Title=1; CodeQL=1; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ paimon-cpp │ 3 │ 14 │ 0/0 │ Build and Test=12; Gcc 8 Test=1; License Check=1; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ superset │ 0 │ 13 │ 12/40 │ E2E=6; Python-Integration=6; Python-Unit=1 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ paimon-rust │ 37 │ 12 │ 0/0 │ CI=45; Release Go Binding=2; Release Python Binding=2 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ nuttx │ 28 │ 12 │ 141/0 │ Build=15; MemBrowse Memory Report=2; Check=1; Lint=1; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ tomcat │ 18 │ 11 │ 23/2 │ CodeQL Advanced=6; Manager2 Test=3; Diff Coverage=2; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ gluten │ 13 │ 11 │ 2/1 │ CH Backend Code style checks=2; Clickhouse Backend T... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ fluss │ 54 │ 10 │ 30/7 │ CI=29; Check License=6; Check Documentation=4; ... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ kafka │ 0 │ 10 │ 0/6 │ CI=10 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ zeppelin │ 0 │ 10 │ 0/0 │ core=8; frontend=2 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ maven-resolver │ 18 │ 8 │ 0/0 │ Verify=25; PR Signal=1 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ openjpa │ 12 │ 8 │ 0/1 │ Pull Request=16; Java CI=4 │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ ofbiz-framework │ 56 │ 7 │ 10/0 │ CodeQL=34; Build and push docker images=2; Java CI... │ > ├──────────────────────┼────────┼─────────┼────────────┼────────────────────────────────────────────────────────────┤ > │ gravitino │ 30 │ 7 │ 59/3 │ Access Control Integration Test=1; ... │ > └──────────────────────┴────────┴─────────┴────────────┴────────────────────────────────────────────────────────────┘ > > Compared to last week, Airflow usage is down to about 1/3 of where it was > (35days vs 100days) -- thank you @potiuk <https://github.com/potiuk> > > ***@***.***:~/Downloads$ python3 usage.py --time 1d apache/airflow > > github self-hosted > conclusion jobs share mins (days) share mins (days) > success 7496 96.1% 48579 (33.7) 87.1% 13388 (9.3) > failure 150 2.1% 1070 (0.7) 0.8% 118 (0.1) > in_progress 62 0.0% 0 (0.0) 0.0% 0 (0.0) > cancelled 707 1.7% 881 (0.6) 12.1% 1859 (1.3) > skipped 6006 0.0% 0 (0.0) 0.0% 0 (0.0) > total 14421 50530 (35.1) 15365 (10.7) > ``` > > It seems like `arrow-rs` and `datafusion` are consuming a very small amount of runner time (1.7 runner- days / 1.6 runner-days, respectively) > > ``` > ***@***.***:~/Downloads$ python3 usage.py --time 1d apache/arrow-rs > 496 runs of apache/arrow-rs created since 2026-10-25T12:30:46Z (24.0 h window) > jobs loaded: 0 runs from cache, 496 from the API, 0 failed (cache: gh-runner-usage-cache) > > github self-hosted > conclusion jobs share mins (days) share mins (days) > success 1136 99.7% 2372 (1.6) - 0 (0.0) > failure 2 0.0% 0 (0.0) - 0 (0.0) > queued 20 0.0% 0 (0.0) - 0 (0.0) > action_required 1 0.0% 0 (0.0) - 0 (0.0) > cancelled 11 0.3% 7 (0.0) - 0 (0.0) > neutral 33 0.0% 0 (0.0) - 0 (0.0) > skipped 799 0.0% 0 (0.0) - 0 (0.0) > total 2002 2379 (1.7) 0 (0.0) > ***@***.***:~/Downloads$ python3 usage.py --time 1d apache/datafusion > 998 runs of apache/datafusion created since 2026-10-25T12:32:02Z (24.0 h window) > jobs loaded: 0 runs from cache, 998 from the API, 0 failed (cache: gh-runner-usage-cache) > > github self-hosted > conclusion jobs share mins (days) share mins (days) > success 1533 92.2% 2081 (1.4) 99.6% 849 (0.6) > failure 15 7.1% 160 (0.1) 0.4% 3 (0.0) > in_progress 13 0.0% 0 (0.0) 0.0% 0 (0.0) > queued 18 0.0% 0 (0.0) 0.0% 0 (0.0) > cancelled 14 0.7% 17 (0.0) 0.0% 0 (0.0) > neutral 2 0.0% 0 (0.0) 0.0% 0 (0.0) > skipped 2201 0.0% 0 (0.0) 0.0% 0 (0.0) > total 3796 2258 (1.6) 852 (0.6) > ``` > > I will do some digging into high usage by other projects too > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/datafusion/issues/25088?email_source=notifications&email_token=AAERMI35R3B2IMXL4RWKA2L5V5PFZA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMNBRHA4DIOJWGM22M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6418849635>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAERMIY3SNPMY24R6P7O23T5V5PFZAVCNFSNUABFKJSXA33TNF2G64TZHMZTKOBZGE3TGMJYHNEXG43VMU5TKMZYHE2DEMBQGY4KC5QC> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
