IF YOU WOULD LIKE TO SEE THE FORMATTED VERSION OF THIS NEWSLETTER, PLEASE GO HERE <https://www.astronomer.io/community-newsletter/november-2023/>
Hey Team, In November, the Airflow community was busy finalizing the release of Airflow 2.8 (TBA), in addition to enjoying themselves at AWS re:Invent, and, as always, helping each other in our Airflow journey’s on Slack, Stack, and anywhere else builders live! Additionally this month, we: - Released Airflow 2.7.3 <https://pypi.org/project/apache-airflow/> - Welcomed new committer Jens Scheffler - Hosted events in London, Cincinnati, and New York! As always, please reach out with items for the next issue to: < briana.oky...@astronomer.io> Thanks for reading! AIRFLOW 2.7.3 Airflow 2.7.3 was released on November 5th and contains several bug fixes and relevant feature updates. - PyPI <https://pypi.org/project/apache-airflow/> - Release Notes <https://pypi.org/project/apache-airflow/> - Docs <https://airflow.apache.org/docs/apache-airflow/2.7.2/> - Sources < https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-sources.html > - Docker <https://hub.docker.com/r/apache/airflow/tags/?page=1&name=2.7.1> - Constraints <https://github.com/apache/airflow/tree/constraints-2.7.2> PROVIDERS New versions of 25 Airflow Providers packages were released. For all source releases, PyPI packages and docs, see: - Sources < https://airflow.apache.org/docs/apache-airflow-providers/installing-from-sources > - PyPI < https://airflow.apache.org/docs/apache-airflow-providers/installing-from-pypi > - Docs <https://airflow.apache.org/docs/> Ecosystem Initial releases: Astronomer Cosmos 1.2.5 <https://pypi.org/project/astronomer-cosmos/> OpenLineage 1.5.0 < https://github.com/OpenLineage/OpenLineage/releases/tag/1.5.0> UPCOMING EVENTS 12/7 Webinar: What’s New in the Astro Platform Release, Q4 2023 < https://www.astronomer.io/events/webinars/whats-new-in-the-astro-platform-release-q4-2023/ > 12/7 Meetup in Prague: First Airflow Prague Meetup with Kiwi.com & Astronomer < https://www.meetup.com/prague-airflow-meetup-group/events/296837456/> 12/12 Meetup in Hyderabad: Hyderabad Airflow Meetup @ Microsoft HQ < https://www.meetup.com/hyderabad-apache-airflow-meetup-group/events/297490529/ > 12/14 Webinar: Orchestrate next-generation AI tools: Explore six new Airflow providers < https://www.astronomer.io/events/webinars/orchestrate-next-generation-ai-tools-explore-six-new-airflow-providers/ > RECENT EVENTS 11/2 Webinar: Run LLMOps in Production with Airflow < https://www.astronomer.io/events/webinars/run-llmops-in-production-with-apache-airflow/ > 11/9 Webinar: Intro to Airflow: Get Started Writing Pipelines for Any Use Case < https://www.astronomer.io/events/webinars/intro-to-airflow-get-started-writing-pipelines-for-any-use-case/ > 11/15 Meetup in NYC: Airflow AI Meetup < https://www.astronomer.io/events/airflow-ai-meetup-nyc-111523/> 11/16 Meetup in Cincinnati: ML Ops in Airflow @ Astronomer HQ < https://www.meetup.com/cincinnati-machine-learning-meetup/events/lwnfvryfcpbvb/ > 11/16 Workshop: Get Apache Airflow Certified For Free! < https://www.astronomer.io/live-certification-workshop/> 11/22 Meetup in London: Airflow Meetup @ Monzo < https://www.meetup.com/london-apache-airflow-meetup/events/296907588/> 11/27-30th Conference in Las Vegas, NV: AWS re:Invent < https://reinvent.awsevents.com/> 11/29 Webinar: Deliver Data with Precision: Supercharge your Airflows with Apache Airflow on Astro - An Azure Native ISV Service < https://www.astronomer.io/events/webinars/astro-airflow-azure-native-isv/> 11/29 Meetup in Amsterdam: Airflow Adventures: Streamlining Demand Forecasting & Operationalizing LLM Apps < https://www.meetup.com/DataCouncil-Amsterdam/events/296855284/?_xtd=gatlbWFpbF9jbGlja9oAJGZjNmUwYzc2LWYwMjgtNGFkYS05NjhiLWVlNDM4MWI3ODNjYg%253D%253D&_af=event&_af_eid=296855284 > 11/30 Webinar: Airflow at Faire: Democratizing Machine Learning at Scale < https://www.astronomer.io/events/webinars/airflow-at-faire-democratizing-machine-learning-at-scale/ > 12/1 Webinar: #AzureHappyHours- Boost Airflows: Apache on Astro- Azure ISV < https://developer.microsoft.com/en-us/reactor/events/21303/> PR OF THE MONTH PR #32646 Add task context logging feature to allow forwarding messages to task logs < https://github.com/apache/airflow/pull/32646> By @pankajkoti <https://github.com/pankajkoti> The November PR of the Month is no surprise to anyone with a unanimous vote across the board. Congratulations to Pankaj Koti and team for this amazing feature that “makes DAG owners’ lives much easier '' according to one voter. COMMUNICATIONS DIGEST Dag Management Customizable feature using Airflow Plugins by Giseung Beak on Airflow Medium < https://medium.com/apache-airflow/customizable-feature-using-airflow-plugins-c1379796b042 > Apache Airflow TaskFlow API vs. Traditional Operators: An In-Depth Comparison for Efficient DAGs by George Yates on the Astronomer Blog The ins and outs of Airflow’s new Secrets Cache by Raphaël Vandon on Airflow Medium < https://medium.com/apache-airflow/the-ins-and-outs-of-airflows-new-secrets-cache-f7b9ec25ca1e > Orchestrating Feature Pipelines: Announcing the Tecton Airflow Provider by Nick Acosta on the Astronomer Blog < https://www.astronomer.io/blog/orchestrating-feature-pipelines-announcing-the-tecton-airflow-provider/ > Test Airflow Upgrades with the Astro CLI by David Koenitzer on the Astronomer Blog < https://www.astronomer.io/blog/test-airflow-upgrades-with-the-astro-cli/> Data Engineering with Apache Airflow, Snowflake, Snowpark, dbt & Cosmos by George Yates in Quickstart Guides < https://quickstarts.snowflake.com/guide/data_engineering_with_apache_airflow/index.html?index=..%2F..index#0 > Introducing Apache Airflow™ on Astro – an Azure Native ISV Service by Conor Beverland on the Astronomer Blog < https://www.astronomer.io/blog/introducing-apache-airflow-on-astro-an-azure-native-isv-service/ > Sensor Approach to Airflow Pipelines by Franco Nyawanga on Airflow Medium < https://medium.com/apache-airflow/sensor-approach-to-airflow-pipelines-704cba211283 > Call paginated API with Airflow’s HttpOperator by Joffrey Bienvenu on Airflow Medium < https://medium.com/apache-airflow/call-paginated-api-with-airflows-httpoperator-3ab5103472f8 > Data Engineering with Apache Airflow, Snowflake, Snowpark, dbt & Cosmos by George Yates in Quickstart Guides < https://quickstarts.snowflake.com/guide/data_engineering_with_apache_airflow/index.html?index=..%2F..index#0 > Airflow Integration and Use Cases LIVE with Astronomer: Using the new Weaviate Provider for Sentiment Analysis with Airflow on Youtube < https://www.youtube.com/watch?v=XKOFjIMEX1w> Airflow in action: providing environmental forecast services by Eloi Codina-Torras on Airflow Medium < https://medium.com/apache-airflow/airflow-in-action-providing-environmental-forecast-services-1a5a86110864 > LIVE with Astronomer: Airflow and Databricks, Better Together! On Youtube < https://www.youtube.com/watch?v=1qlTus6xv0o> MLOps Advanced Tips & Tricks for Improving Your Airflow DAGs! On Youtube < https://www.youtube.com/watch?v=627s6q5Epwo> Accelerating ML Application Development: Production-Ready Airflow Integrations with Critical AI Tools by Steven Hillion on the Astronomer Blog < https://www.astronomer.io/blog/accelerating-ml-app-development-with-ai-airflow-integrations/ > DEVLIST DIGEST Votes: [RESULT][VOTE] Add providers for Pinecone, OpenAI & Cohere to enable first-class LLMOps < https://lists.apache.org/thread/oq7h2n88zfo3dzldy5w8xlp9kyngs7x8> Release Airflow 2.7.3 from 2.7.3rc1 < https://lists.apache.org/thread/xxfzmfowr4sn8fkbmjplk5pbxg3lx776> [RESULT][VOTE] Release Airflow 2.7.3 from 2.7.3rc1 < https://lists.apache.org/thread/hx3z847w263ztz8gk3lypdvc6gycssdz> Agree on the process of removal of providers < https://lists.apache.org/thread/tsn52vk0hmrqgxy4zn009qldfddrn12y> Migrating our reference images to Debian Bookworm for Airflow 2.8 < https://lists.apache.org/thread/gcy143nqodf8dqbjxo2xt5gq4npv334p> Airflow Providers prepared on November 08, 2023 < https://lists.apache.org/thread/gwtbdbnk6ps67hsxq47bh7g3odydf9tq> [RESULT][VOTE] Airflow Providers - release of November 08, 2023 < https://lists.apache.org/thread/zpo2d5nj2bbw7qtkml0ljb7nyqyykl4p> Decouple Core and API Clients versioning < https://lists.apache.org/thread/m9jdkq7w38d95jykm0l3cm38pmvkzzl7> Release Airflow Python Client 2.7.3 from 2.7.3rc1 < https://lists.apache.org/thread/hlgwpvg2wvk945bpjok1jsjvqrt3001p> [RESULT][VOTE] Release Airflow Python Client 2.7.3 from 2.7.3rc1 < https://lists.apache.org/thread/lnbj2r82c3850z2fbdfnlv4rd712w3so> Airflow Providers prepared on November 12, 2023 < https://lists.apache.org/thread/kc885rppl423vb1yowdf71t3x1ch5lf1> [RESULT][VOTE] Airflow Providers - release of November 12, 2023 < https://lists.apache.org/thread/y6w94jzq0scqdnb83g6bg996pzlbqkp3> Remove Qubole provider < https://lists.apache.org/thread/7wwz9lwtk6dp15qv179jso63f2xx54ot> Airflow Providers prepared on November 24, 2023 < https://lists.apache.org/thread/8xc4vbsl301bf05sflfy6w12orl983wb> Add fab provider < https://lists.apache.org/thread/o457svvnsn4xdnb7j79jyqsxvldj6qmz> [RESULT][VOTE] Airflow Providers - release of November 24, 2023 < https://lists.apache.org/thread/lq1vtsd38s9nxkn0q2b0f1zsr77120pt> November PR of the Month < https://lists.apache.org/thread/jtjfrmbrro3l640m738yclzklfkopodm> Remove 'daskexecutor' provider < https://lists.apache.org/thread/fxv44cqqljrrhll3fdpdgc9h9fz5ghcy> Change default docker image to point to "latest supported" < https://lists.apache.org/thread/0oxnvct24xlqsj76z42w2ttw2d043oy3> Discussed: Requesting Feedback on Airflow Survey < https://lists.apache.org/thread/jvsrw4hfyqg3k24b391glm9cgyq0zcxp> API Clients Major version < https://lists.apache.org/thread/c1n6gqrx3v00z1qr1nr10obvvn2v1rcz> Airflow UI DAG Composer < https://lists.apache.org/thread/p3png5oltm1ztm4rtlp5lnj3f0vpvbmx> Move FAB auth manager to a new provider < https://lists.apache.org/thread/70gyol2jwn6b39owgyg53c4k69zzv967> Suspend (Remove?) Daskexecutor provider < https://lists.apache.org/thread/ptwjf5g87lyl5476krt91bzfrm96pnb1> Suspend/Remove Apache Scoop provider < https://lists.apache.org/thread/tfj4147nopmn564jc0m49h65pd440wv8> Suspend/Remove Plexus provider < https://lists.apache.org/thread/offqjnrrj10tgr0lpxdbhvwtffs05dsf> Announced: Apache Airflow 2.7.3 Released < https://lists.apache.org/thread/s9vh3ydn25913132ftch6vmq5m72g4pr> How to reproducibly install Airflow < https://lists.apache.org/thread/khcnysjy8m479lpkthozg52onl1odk1f> The 2023 Airflow Survey is LIVE < https://lists.apache.org/thread/n9h1fb1vzyotc7b9pbt1mkfq8wqr3f0t> New committer: Jens Scheffler < https://lists.apache.org/thread/d8wy1yfcmgtp4cc4kc48ssnt1qpj4j31> Initiative on Strengthening security for Apache Airflow < https://lists.apache.org/thread/s3o68ypn9mbd8qk2zrg7td0bbmc3jf8q> Apache Airflow Providers prepared on November 08, 2023 are released < https://lists.apache.org/thread/zf8hhpbpfj03osdm15gzlp99gon2os0n> Apache Airflow Python Client 2.7.3 Released < https://lists.apache.org/thread/6yvf2gjvvjrf6nqq20cbdb7mfp2vb2y5> Apache Airflow Providers prepared on November 12, 2023 are released < https://lists.apache.org/thread/6t82oszwprj7hkjcbmjq9od09x8ghy5j> Apache Airflow 2.8.0b1 available for testing < https://lists.apache.org/thread/qjfwvdkfh97y8mob63fz5wvg1gj1ofol> Apache Airflow Providers prepared on November 24, 2023 are released < https://lists.apache.org/thread/7bczr0mkkb6dh7pysf8cb7yv7g53box6> Proposed: Change default docker image to point to "latest supported" < https://lists.apache.org/thread/xy192wmv6k1sy2wlx5vy64p678skrhko> Deprecate URI Connection representation in favor of JSON < https://lists.apache.org/thread/3n832wdy8xf2ochw0s7kmys23vxmwd8j> Airflow Monthly Town-Hall < https://lists.apache.org/thread/zw5zdsqxmy078qy8jfv6bfcl5551fk8y> IN-DEVELOPMENT AIP-51 Removing Executor Coupling from Core Airflow < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-51+Removing+Executor+Coupling+from+Core+Airflow > AIP-56: Extensible user management < https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > AIP-58 Airflow ObjectStore (AS) < https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263430565> See the Wiki for more of the project’s approved AIPs. < https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals > AIRFLOW FACT OF THE MONTH Did you know that the word technology was coined in 330 BC by Aristotle? In ancient times, technology was defined by Homer and Hesiod as the spoken word of manual craft or cunning skill. By 330 BC, Aristotle coined the Greek term technologia and split scientific knowledge into three parts: theoretical science, practical science, and productive science (technology). It’s fascinating to think how far we have come in the realm of technology, and how much we have yet to learn! STAY TUNED Please subscribe to this newsletter to ensure you always receive the latest issue: <http://eepurl.com/hXUA3r> Know of an upcoming event or publication the community should know about? Let me know at briana.oky...@astronomer.io.
