non-slim images has been pushed and are now available. Slim images are in progress.
https://hub.docker.com/repository/docker/apache/airflow/tags?page=&page_size=&ordering=&name=2.10.0rc1 On Mon, 12 Aug 2024 at 20:07, Utkarsh Sharma <utkarsh.sha...@astronomer.io.invalid> wrote: > Dear Airflow Community, > > We are currently experiencing some CI-related issues, which will > temporarily delay the availability of the Docker image for Airflow 2.10.0 > RC1. We anticipate that it will take a few hours to resolve these issues. > > Thanks, > > Utkarsh Sharma > > On Tue, Aug 13, 2024 at 12:33 AM Utkarsh Sharma < > utkarsh.sha...@astronomer.io> wrote: > > > Hey fellow Airflowers, > > > > I have cut Airflow 2.10.0rc1. This email is calling a vote on the > release, > > which will last at least 72 hours, from Monday, August 12, 2024 at > 6:00 pm > > UTC > > until Wednesday, August 15, 2024 at 6:00 pm UTC > > < > https://www.timeanddate.com/worldclock/fixedtime.html?msg=8&iso=20240815T1800&p1=1440 > >, > > and until 3 binding +1 votes have been received. > > > > > > > https://www.timeanddate.com/worldclock/fixedtime.html\?msg\=8\&iso\=20211011T1600\&p1\=1440 > > > > Consider this my (non-binding) +1. As I’m not a member of the PMC, > Ephraim > > signed the distribution. > > > > Airflow 2.10.0rc1 is available at: > > https://dist.apache.org/repos/dist/dev/airflow/2.10.0rc1/ > > > > *apache-airflow-2.10.0-source.tar.gz* is a source release that comes with > > INSTALL instructions. > > *apache-airflow-2.10.0.tar.gz* is the binary Python "sdist" release. > > *apache_airflow-2.10.0-py3-none-any.whl* is the binary Python wheel > > "binary" release. > > > > Public keys are available at: > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > Please vote accordingly: > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove with the reason > > > > Only votes from PMC members are binding, but all members of the community > > are encouraged to test the release and vote with "(non-binding)". > > > > The test procedure for PMC members is described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmc-members > > > > The test procedure for contributors and members of the community who > would > > like to test this RC is described in: > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-contributors > > > > > > Please note that the version number excludes the `rcX` string, so it's > now > > simply 2.10.0. This will allow us to rename the artifact without > modifying > > the artifact checksums when we actually release. > > > > Release Notes: > > https://github.com/apache/airflow/blob/2.10.0rc1/RELEASE_NOTES.rst > > > > For information on what goes into a release please see: > > > https://github.com/apache/airflow/blob/main/dev/WHAT_GOES_INTO_THE_NEXT_RELEASE.md > > > > *Changes since 2.9.3:* > > > > *Significant Changes* > > > > *Datasets no longer trigger inactive DAGs (#38891)* > > > > Previously, when a DAG is paused or removed, incoming dataset events > would > > still > > trigger it, and the DAG would run when it is unpaused or added back in a > > DAG > > file. This has been changed; a DAG's dataset schedule can now only be > > satisfied > > by events that occur when the DAG is active. While this is a breaking > > change, > > the previous behavior is considered a bug. > > > > The behavior of time-based scheduling is unchanged, including the > > timetable part > > of ``DatasetOrTimeSchedule``. > > > > *``try_number`` is no longer incremented during task execution (#39336)* > > > > Previously, the try number (``try_number``) was incremented at the > > beginning of task execution on the worker. This was problematic for many > > reasons. > > For one it meant that the try number was incremented when it was not > > supposed to, namely when resuming from reschedule or deferral. And it > also > > resulted in > > the try number being "wrong" when the task had not yet started. The > > workarounds for these two issues caused a lot of confusion. > > > > Now, instead, the try number for a task run is determined at the time the > > task is scheduled, and does not change in flight, and it is never > > decremented. > > So after the task runs, the observed try number remains the same as it > was > > when the task was running; only when there is a "new try" will the try > > number be incremented again. > > > > One consequence of this change is, if users were "manually" running tasks > > (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks > > run``), > > try number will no longer be incremented. Airflow assumes that tasks are > > always run after being scheduled by the scheduler, so we do not regard > this > > as a breaking change. > > > > *``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145)* > > > > The ``/logout`` endpoint's method in FAB Auth Manager has been changed > > from ``GET`` to ``POST`` in all existing > > AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, > > ``AuthOIDView``, ``AuthRemoteUserView``), and > > now includes CSRF protection to enhance security and prevent unauthorized > > logouts. > > > > *OpenTelemetry Traces for Apache Airflow (#37948).* > > > > This new feature adds capability for Apache Airflow to emit 1) airflow > > system traces of scheduler, > > triggerer, executor, processor 2) DAG run traces for deployed DAG runs in > > OpenTelemetry format. Previously, only metrics were supported which > emitted > > metrics in OpenTelemetry. > > This new feature will add richer data for users to use OpenTelemetry > > standard to emit and send their trace data to OTLP compatible endpoints. > > > > *Decorator for Task Flow ``(@skip_if, @run_if)`` to make it simple to > > apply whether or not to skip a Task. (#41116)* > > > > This feature adds a decorator to make it simple to skip a Task. > > > > *Using Multiple Executors Concurrently (#40701)* > > > > Previously known as hybrid executors, this new feature allows Airflow to > > use multiple executors concurrently. DAGs, or even individual tasks, can > be > > configured > > to use a specific executor that suits its needs best. A single DAG can > > contain tasks all using different executors. Please see the Airflow > > documentation for > > more details. Note: This feature is still experimental. See > `documentation > > on Executor < > > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#using-multiple-executors-concurrently > >`_ > > for a more detailed description. > > > > *Scarf based telemetry: Does Airflow collect any telemetry data? > (#39510)* > > > > Airflow integrates Scarf to collect basic usage data during operation. > > Deployments can opt-out of data collection by setting the > > ``[usage_data_collection]enabled`` option to False, or the > > SCARF_ANALYTICS=false environment variable. > > See `FAQ on this < > > > https://airflow.apache.org/docs/apache-airflow/stable/faq.html#does-airflow-collect-any-telemetry-data > >`_ > > for more information. > > > > > > *New Features* > > > > - AIP-61 Hybrid Execution (`AIP-61 < > > > https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22 > > >`_) > > - AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 < > > > https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22 > > >`_) > > - AIP-64 TaskInstance Try History (`AIP-64 < > > > https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22 > > >`_) > > - AIP-44 Internal API (`AIP-44 < > > > https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed > > >`_) > > - Enable ending the task directly from the triggerer without going into > > the worker. (#40084) > > - Extend dataset dependencies (#40868) > > - Feature/add token authentication to internal api (#40899) > > - Add DatasetAlias to support dynamic Dataset Event Emission and Dataset > > Creation (#40478) > > - Add example DAGs for inlet_events (#39893) > > - Implement ``accessors`` to read dataset events defined as inlet > (#39367) > > - Decorator for Task Flow, to make it simple to apply whether or not to > > skip a Task. (#41116) > > - Add start execution from triggerer support to dynamic task mapping > > (#39912) > > - Add try_number to log table (#40739) > > - Added ds_format_locale method in macros which allows localizing > datetime > > formatting using Babel (#40746) > > - Add DatasetAlias to support dynamic Dataset Event Emission and Dataset > > Creation (#40478, #40723, #40809, #41264, #40830, #40693, #41302) > > - Use sentinel to mark dag as removed on re-serialization (#39825) > > - Add parameter for the last number of queries to the DB in DAG file > > processing stats (#40323) > > - Add prototype version dark mode for Airflow UI (#39355) > > - Add ability to mark some tasks as successful in ``dag test`` (#40010) > > - Allow use of callable for template_fields (#37028) > > - Filter running/failed and active/paused dags on the home page(#39701) > > - Add metrics about task CPU and memory usage (#39650) > > - UI changes for DAG Re-parsing feature (#39636) > > - Add Scarf based telemetry (#39510, #41318) > > - Add dag re-parsing request endpoint (#39138) > > - Redirect to new DAGRun after trigger from Grid view (#39569) > > - Display ``endDate`` in task instance tooltip. (#39547) > > - Implement ``accessors`` to read dataset events defined as inlet > (#39367, > > #39893) > > - Add color to log lines in UI for error and warnings based on keywords > > (#39006) > > - Add Rendered k8s pod spec tab to ti details view (#39141) > > - Make audit log before/after filterable (#39120) > > - Consolidate grid collapse actions to a single full screen toggle > (#39070) > > - Implement Metadata to emit runtime extra (#38650) > > - Add executor field to the DB and parameter to the operators (#38474) > > - Implement context accessor for DatasetEvent extra (#38481) > > - Add dataset event info to dag graph (#41012) > > - Add button to toggle datasets on/off in dag graph (#41200) > > - Add ``run_if`` & ``skip_if`` decorators (#41116) > > - Add dag_stats rest api endpoint (#41017) > > - Add listeners for Dag import errors (#39739) > > - Allowing DateTimeSensorAsync, FileSensor and TimeSensorAsync to start > > execution from trigger during dynamic task mapping (#41182) > > > > > > *Improvements* > > > > - Allow set Dag Run resource into Dag Level permission: extends Dag's > > access_control feature to allow Dag Run resource permissions. (#40703) > > - Improve security and error handling for the internal API (#40999) > > - Datasets UI Improvements (#40871) > > - Change DAG Audit log tab to Event Log (#40967) > > - Make standalone dag file processor works in DB isolation mode (#40916) > > - Show only the source on the consumer DAG page and only triggered DAG > run > > in the producer DAG page (#41300) > > - Update metrics names to allow multiple executors to report metrics > > (#40778) > > - Format DAG run count (#39684) > > - Update styles for ``renderedjson`` component (#40964) > > - Improve ATTRIBUTE_REMOVED sentinel to use class and more context > (#40920) > > - Make XCom display as react json (#40640) > > - Replace usages of task context logger with the log table (#40867) > > - Rollback for all retry exceptions (#40882) (#40883) > > - Support rendering ObjectStoragePath value (#40638) > > - Add try_number and map_index as params for log event endpoint (#40845) > > - Rotate fernet key in batches to limit memory usage (#40786) > > - Add gauge metric for 'last_num_of_db_queries' parameter (#40833) > > - Set parallelism log messages to warning level for better visibility > > (#39298) > > - Add error handling for encoding the dag runs (#40222) > > - Use params instead of dag_run.conf in example DAG (#40759) > > - Load Example Plugins with Example DAGs (#39999) > > - Stop deferring TimeDeltaSensorAsync task when the target_dttm is in the > > past (#40719) > > - Send important executor logs to task logs (#40468) > > - Open external links in new tabs (#40635) > > - Attempt to add ReactJSON view to rendered templates (#40639) > > - Speeding up regex match time for custom warnings (#40513) > > - Refactor DAG.dataset_triggers into the timetable class (#39321) > > - add next_kwargs to StartTriggerArgs (#40376) > > - Improve UI error handling (#40350) > > - Remove double warning in CLI when config value is deprecated (#40319) > > - Implement XComArg concat() (#40172) > > - Added ``get_extra_dejson`` method with nested parameter which allows > you > > to specify if you want the nested json as string to be also deserialized > > (#39811) > > - Add executor field to the task instance API (#40034) > > - Support checking for db path absoluteness on Windows (#40069) > > - Introduce StartTriggerArgs and prevent start trigger initialization in > > scheduler (#39585) > > - Add task documentation to details tab in grid view (#39899) > > - Allow executors to be specified with only the class name of the > Executor > > (#40131) > > - Remove obsolete conditional logic related to try_number (#40104) > > - Allow Task Group Ids to be passed as branches in BranchMixIn (#38883) > > - Javascript connection form will apply CodeMirror to all textarea's > > dynamically (#39812) > > - Determine needs_expansion at time of serialization (#39604) > > - Add indexes on dag_id column in referencing tables to speed up deletion > > of dag records (#39638) > > - Add task failed dependencies to details page (#38449) > > - Remove webserver try_number adjustment (#39623) > > - Implement slicing in lazy sequence (#39483) > > - Unify lazy db sequence implementations (#39426) > > - Add ``__getattr__`` to task decorator stub (#39425) > > - Allow passing labels to FAB Views registered via Plugins (#39444) > > - Simpler error message when trying to offline migrate with sqlite > (#39441) > > - Add soft_fail to TriggerDagRunOperator (#39173) > > - Rename "dataset event" in context to use "outlet" (#39397) > > - Resolve ``RemovedIn20Warning`` in ``airflow task`` command (#39244) > > - Determine fail_stop on client side when db isolated (#39258) > > - Refactor cloudpickle support in Python operators/decorators (#39270) > > - Update trigger kwargs migration to specify existing_nullable (#39361) > > - Allowing tasks to start execution directly from triggerer without going > > to worker (#38674) > > - Better ``db migrate`` error messages (#39268) > > - Add stacklevel into the ``suppress_and_warn`` warning (#39263) > > - Support searching by dag_display_name (#39008) > > - Allow sort by on all fields in MappedInstances.tsx (#38090) > > - Expose count of scheduled tasks in metrics (#38899) > > - Use ``declarative_base`` from ``sqlalchemy.orm`` instead of > > ``sqlalchemy.ext.declarative`` (#39134) > > - Add example DAG to demonstrate emitting approaches (#38821) > > - Give ``on_task_instance_failed`` access to the error that caused the > > failure (#38155) > > - Simplify dataset serialization (#38694) > > - Add heartbeat recovery message to jobs (#34457) > > - Remove select_column option in TaskInstance.get_task_instance (#38571) > > - Don't create session in get_dag if not reading dags from database > > (#38553) > > - Add a migration script for encrypted trigger kwargs (#38358) > > - Implement render_templates on TaskInstancePydantic (#38559) > > - Handle optional session in _refresh_from_db (#38572) > > - Make type annotation less confusing in task_command.py (#38561) > > - Use fetch_dagrun directly to avoid session creation (#38557) > > - Added ``output_processor`` parameter to ``BashProcessor`` (#40843) > > - Improve serialization for Database Isolation Mode (#41239) > > - Only orphan non-orphaned Datasets (#40806) > > - Adjust gantt width based on task history dates (#41192) > > - Enable scrolling on legend with high number of elements. (#41187) > > > > *Bug Fixes* > > > > - Bugfix for get_parsing_context() when ran with LocalExecutor (#40738) > > - Validating provider documentation urls before displaying in views > > (#40933) > > - Move import to make PythonOperator working on Windows (#40424) > > - Fix dataset_with_extra_from_classic_operator example DAG (#40747) > > - Call listener on_task_instance_failed() after ti state is changed > > (#41053) > > - Add ``never_fail`` in BaseSensor (#40915) > > - Fix tasks API endpoint when DAG doesn't have ``start_date`` (#40878) > > - Fix and adjust URL generation for UI grid and older runs (#40764) > > - Rotate fernet key optimization (#40758) > > - Fix class instance vs. class type in > > validate_database_executor_compatibility() call (#40626) > > - Clean up dark mode (#40466) > > - Validate expected types for args for DAG, BaseOperator and TaskGroup > > (#40269) > > - Exponential Backoff Not Functioning in BaseSensorOperator Reschedule > > Mode (#39823) > > - local task job: add timeout, to not kill on_task_instance_success > > listener prematurely (#39890) > > - Move Post Execution Log Grouping behind Exception Print (#40146) > > - Fix triggerer race condition in HA setting (#38666) > > - Pass triggered or existing DAG Run logical date to DagStateTrigger > > (#39960) > > - Passing ``external_task_group_id`` to ``WorkflowTrigger`` (#39617) > > - ECS Executor: Set tasks to RUNNING state once active (#39212) > > - Only heartbeat if necessary in backfill loop (#39399) > > - Fix trigger kwarg encryption migration (#39246) > > - Fix decryption of trigger kwargs when downgrading. (#38743) > > - Fix wrong link in TriggeredDagRuns (#41166) > > - Pass MapIndex to LogLink component for external log systems (#41125) > > - Add NonCachingRotatingFileHandler for worker task (#41064) > > - Add argument include_xcom in method resolve an optional value (#41062) > > - Sanitizing file names in example_bash_decorator DAG (#40949) > > - Show dataset aliases in dependency graphs (#41128) > > - Render Dataset Conditions in DAG Graph view (#41137) > > - Add task duration plot across dagruns (#40755) > > - Add start execution from trigger support for existing core sensors > > (#41021) > > - add example dag for dataset_alias (#41037) > > - Add dataset alias unique constraint and remove wrong dataset alias > > removing logic (#41097) > > - Set "has_outlet_datasets" to true if "dataset alias" exists (#41091) > > - Make HookLineageCollector group datasets by (#41034) > > - Enhance start_trigger_args serialization (#40993) > > - Refactor ``BaseSensorOperator`` introduce ``skip_policy`` parameter > > (#40924) > > - Fix viewing logs from triggerer when task is deferred (#41272) > > - Refactor how triggered dag run url is replaced (#41259) > > - Added support for additional sql alchemy session args (#41048) > > - Allow empty list in TriggerDagRun failed_state (#41249) > > - Clean up the exception handler when run_as_user is the airflow user > > (#41241) > > - Collapse docs when click and folded (#41214) > > - Update updated_at when saving to db as session.merge does not trigger > > on-update (#40782) > > - Fix query count statistics when parsing DAF file (#41149) > > - Method Resolution Order in operators without ``__init__`` (#41086) > > - Ensure try_number incremented for empty operator (#40426) > > > > *Miscellaneous* > > > > - Remove the Experimental flag from ``OTel`` Traces (#40874) > > - Bump packaging version to 23.0 in order to fix issue with older otel > > (#40865) > > - Simplify _auth_manager_is_authorized_map function (#40803) > > - Use correct unknown executor exception in scheduler job (#40700) > > - Add D1 ``pydocstyle`` rules to pyproject.toml (#40569) > > - Enable enforcing ``pydocstyle`` rule D213 in ruff. (#40448, #40464) > > - Update ``Dag.test()`` to run with an executor if desired (#40205) > > - Update jest and babel minor versions (#40203) > > - Refactor BashOperator and Bash decorator for consistency and simplicity > > (#39871) > > - Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors > > (#38778) > > - ruff version bump 0.4.5 (#39849) > > - Bump ``pytest`` to 8.0+ (#39450) > > - Remove stale comment about TI index (#39470) > > - Configure ``back_populates`` between > ``DagScheduleDatasetReference.dag`` > > and ``DagModel.schedule_dataset_references`` (#39392) > > - Remove deprecation warnings in endpoints.py (#39389) > > - Fix SQLA deprecations in Airflow core (#39211) > > - Use class-bound attribute directly in SA (#39198, #39195) > > - Fix stacklevel for TaskContextLogger (#39142) > > - Capture warnings during collect DAGs (#39109) > > - Resolve ``B028`` (no-explicit-stacklevel) in core (#39123) > > - Rename model ``ImportError`` to ``ParseImportError`` for avoid > shadowing > > with builtin exception (#39116) > > - Add option to support cloudpickle in PythonVenv/External Operator > > (#38531) > > - Suppress ``SubDagOperator`` examples warnings (#39057) > > - Add log for running callback (#38892) > > - Use ``model_dump`` instead of ``dict`` for serialize Pydantic V2 model > > (#38933) > > - Widen cheat sheet column to avoid wrapping commands (#38888) > > - Update hatchling to latest version (1.22.5) (#38780) > > - bump uv to 0.1.29 (#38758) > > - Add missing serializations found during provider tests fixing (#41252) > > - Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) > > - Improve typing for allowed/failed_states in TriggerDagRunOperator > > (#39855) > > > > *Doc Only Changes* > > > > - Add ``filesystems`` and ``dataset-uris`` to "how to create your own > > provider" page (#40801) > > - Fix (TM) to (R) in Airflow repository (#40783) > > - Set ``otel_on`` to True in example airflow.cfg (#40712) > > - Add warning for _AIRFLOW_PATCH_GEVENT (#40677) > > - Update multi-team diagram proposal after Airflow 3 discussions (#40671) > > - Add stronger warning that MSSQL is not supported and no longer > > functional (#40565) > > - Fix misleading mac menu structure in howto (#40440) > > - Update k8s supported version in docs (#39878) > > - Add compatibility note for Listeners (#39544) > > - Update edge label image in documentation example with the new graph > view > > (#38802) > > - Update UI doc screenshots (#38680) > > - Add section "Manipulating queued dataset events through REST API" > > (#41022) > > - Add information about lack of security guarantees for docker compose > > (#41072) > > - Add links to example dags in use params section (#41031) > > - Change ``task_id`` from ``send_email`` to ``send_email_notification`` > in > > ``taskflow.rst`` (#41060) > > - Remove unnecessary nginx redirect rule from reverse proxy documentation > > (#38953) > > > > Cheers, > > Utkarsh Sharma > > >
