Fwd: Outstanding review requests

Ping! On 4 September 2019 at 14:47:41, Bolke de Bruin (bdbr...@gmail.com) wrote: Hello Team! I have some outstanding review requests and I would appreciate feedback or (duh) merging. Can someone take a look: * Add order by (sort) to basic search : https://reviews.apache.org/r/71431/ * Use

Re: Outstanding review requests

Wed, Sep 11, 2019 at 10:12 PM Tao Feng wrote: >> >> Hey Bolke, >> >> I think you sent the wrong alias. The review should go to the atlas alias >> :). >> >>> On Wed, Sep 11, 2019 at 12:57 PM Bolke de Bruin wrote: >>> >>>

Re: Airflow node different versions

I actually think that it is not that risky (although ymmv). Worker nodes are pretty independent from the scheduler/webserver. As long as the datamodel hasnt changed and nodes dont change their reporting (new statusses) to the db (that hasnt happened for a long time) you are probably okay. So th

Re: Announcing SIG-Knative/ The Monthly Knative Executor Meetup/Call-for-contributers

Really cool! Verstuurd vanaf mijn iPad > Op 16 okt. 2019 om 20:51 heeft Jarek Potiuk het > volgende geschreven: > > Fantastic! > >> On Wed, Oct 16, 2019 at 8:26 PM Driesprong, Fokko >> wrote: >> >> Awesome Daniel! >> >> Cheers, Fokko >> >> Op wo 16 okt. 2019 om 20:16 schreef Kevin Yang :

Re: Can an xcom push and pull to the same task instance on the same execution date?

I’m not convinced that this PR should actually go in. It affect task idempotency in my perspective and should have been given more thought. Now the Xcom values and task runtimes can be out of sync (e.g. imagine - simplified - if it pushed “success” to xcom, but after clearing the task failed). What

Drop Python 3.5 support?

Hi All, Can we drop python 3.5 support and switch to 3.6 as a minimum? Cheers Bolke

Re: Drop Python 3.5 support?

Json loads supports binary format - >https://docs.python.org/3/whatsnew/3.6.html#json - this has already >bitten us as well. there was code working fine in py2.7 and 3.6 but not >working with 3.5(!). > > Last but not least - it might free some resources on Travis (I hope

Re: Drop Python 3.5 support?

t;>>>> wrote: >>>>>> >>>>>>> +1 >>>>>>> >>>>>>> On Tue, Nov 12, 2019 at 10:48 PM Kaxil Naik >>>>>> wrote: >>>>>>>> >>>>>>>

Re: Closing JIRA Issue for Merged PRs

I wrote that script. It’s cli only unfortunately. On 18 November 2019 at 18:22:04, Dan Davydov (ddavy...@twitter.com.invalid) wrote: Wait this doesn't happen automatically!? I thought way-back-when someone wrote a script to automatically close the JIRA tickets (maybe that script is not run when

Re: Closing JIRA Issue for Merged PRs

example, the one that allows to "Automatically transition an issue >to >done when a pull request whose name contains the issue key is merged"? >Here is Atlassian repo: https://github.com/atlassian/gajira > >Bests, >Tomek > > >On Mon, Nov 18, 2019 at 7:22 PM Bolke

Re: [DISCUSS] Using shared memory for inter-task communication

My 2 cents: I don’t think this makes sense at all as it goes against to core of Airflow: Airflow does not do data processing by itself. So the only think you should share between tasks is meta data and that you do through XCom. We can redesign com if you want but it is also the only viable option

Re: [DISCUSS] Using shared memory for inter-task communication

Nov 27, 2019 at 11:40 AM Tomasz Urbaszek < tomasz.urbas...@polidea.com> wrote: > I agree with Bolke, Airflow is not a data processing tool. Also it should > not become > one as we already have some awesome solutions like Apache Storm, Flink or > Beam. > > Tomek > > &

Re: Improving the Airflow UI

+1 The react issue was solved some time ago and is fine to use. A challenge might be to keep track of all the licenses of the subcomponents react can pull in. Superset has some experience there. Superset is also based around the same components as airflow (FAB/React) B. Sent from my iPhone >

[DISCUSS] Improving security by decoupling the CLI from the database

Hi All, I’ve noticed that we are still implementing new features or are doing refactoring of CLI commands that directly interface with the database instead of using the abstractions that should be made available from the API specification. Why is this an issue? The CLI is used by arbitrary user to

Re: [DISCUSS] Improving security by decoupling the CLI from the database

omatically verified - we can easily add > pylint plugin that can check CLI package for > db usage). In my opinion we cannot expect people to use API until it goes > out of experimental/ we have a viable > stable long term alternative agreed. > > Once this is in place - no new CLI com

Re: [DISCUSS] Improving security by decoupling the CLI from the database

to work out a solution that will satisfy everyone. Thank you also for adding this mode to the project, because I know that it solves the problem of many users, but unfortunately my clients cannot use it and want to develop a solution that can be used by current and future users. On Sat, Jan 18

[DISCUSS] Lineage improvements and standardization of operator signatures

Dear All, Over last few weeks I made serious improvements to the lineage support that Airflow has. Whilst not complete it’s starting to shape up and I think it is good to share some thoughts and directions. Much has been discussed with several organisations like Polidea, Daily Motion and Lyft. Som

Re: [DISCUSS] Lineage improvements and standardization of operator signatures

t; would return the inlets and outlets of a given task, and then it's up > to > > > the operator how it wants to set these inlets/outlets like the > Papermill > > > operator currently does. This also keeps allows inlets/outlets more > dynamic > > > (e.g. i

Re: Shorter release vote or 1.10.9 to pin Werkzeug dep

+1 binding Sent from my iPhone > On 7 Feb 2020, at 14:08, Jarek Potiuk wrote: > > Agree. > >> On Fri, Feb 7, 2020 at 2:05 PM Ash Berlin-Taylor wrote: >> >> Hello all, >> >> So Werkzeug 1.0 was released this morning, and it finally broke a number >> of direct or indirect dependencies of ou

Re: [DISCUSS] Stop using Jira (since we aren't using it properly)

Honestly, I think both suck. So I can go either way On 16 March 2020 at 12:33:27, Ash Berlin-Taylor (a...@firemirror.com) wrote: The subject pretty much says it all. We aren't using Jira very well in most cases, and the requirement for a Jira ticket for a code change leads to people just creati

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

Well it became a long list eh and it was used for bootstrapping the community. So I think it's value has changed. You might want to consider categorizing it or make it into the "one million dollar" page (placing the logos on 1mio squared pixels) or a kind of overview page. Say as a potential

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

On topic: Airflow has a pretty loosely defined contract. E.g. operators do not have strict ordering of keywords, our naming is not very consistent (conn_id vs oss_conn_id and many more) and we do weird thing in tasks vs dag/scheduler/executor :-). Design wise we are very flexible (or ?). This is a

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

I beg to differ too and I do think that Alexander is right in what he wants to accomplish. Large installations do want to do rolling upgrades and not bring a cluster down for an upgrade. It should be possible to run Airflow Core 2.4 with for example 2.3 workers. It should however not happen through

Re: [VOTE] Release Airflow 2.5.0 from 2.5.0rc2

+1 binding.Starting to enjoy the dsl a bit again.Sent from my iPhoneOn 28 Nov 2022, at 13:58, Ash Berlin-Taylor wrote:+1 binding.Tested a few dags, poked around the new dataset view (nice one Brent and Drew) - small improvements that'll make a big difference to usability.-ashOn Nov 27 2022, at 6:

Re: [DISCUSSION] Assessing what is a breaking change for Airflow (SemVer context)

GridView. > > 4) Airflow also has also a few non-code interfaces that are considered > as part of the platform: statsd metrics is one of them. I can't think > of any more but maybe there are more. We could simply make an > inventory and discuss our approach on those ONCE and document it. This > will avoid discussions, discussions, discussions, and let our users > have some clear expectations and maintainers making quick decisions > when approving (or not) PRs. > > # Question > > Does it sound like a good plan? Is it worth making such an effort ? Or > maybe what we have as status-quo is "good enough" and that would be a > waste of effort? WDYT? > > J. > > > > > J > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [VOTE] Release Airflow 2.6.0 from 2.6.0rc1

next_run_calculation > > > > > > (#28172) > > > > > > > > > > > > *Misc/Internal* > > > > > > - Make eager upgrade additional dependencies optional (#30811) > > > > > > - Upgrade to pip 23.1.1 (#30808) > > > > > > - Remove protobuf limitation from eager upgrade (#30182) > > > > > > - Remove protobuf limitation from eager upgrade (#30182) > > > > > > - Deprecate ``skip_exit_code`` in ``BashOperator`` (#30734) > > > > > > - Remove gauge ``scheduler.tasks.running`` (#30374) > > > > > > - Bump json5 to 1.0.2 and eslint-plugin-import to 2.27.5 in > > > > > > ``/airflow/www`` (#30568) > > > > > > - Add tests to PythonOperator (#30362) > > > > > > - Add asgiref as a core dependency (#30527) > > > > > > - Discovery safe mode toggle comment clarification (#30459) > > > > > > - Upgrade moment-timezone package to fix Tehran tz (#30455) > > > > > > - Bump loader-utils from 2.0.0 to 2.0.4 in ``/airflow/www`` > > (#30319) > > > > > > - Bump babel-loader from 8.1.0 to 9.1.0 in ``/airflow/www`` > > (#30316) > > > > > > - DagBag: Use ``dag.fileloc`` instead of ``dag.full_filepath`` in > > > > exception > > > > > > message (#30610) > > > > > > - Change log level of serialization information (#30239) > > > > > > - Minor DagRun helper method cleanup (#30092) > > > > > > - Improve type hinting in stats.py (#30024) > > > > > > - Limit ``importlib-metadata`` backport to < 5.0.0 (#29924) > > > > > > - Align cncf provider file names with AIP-21 (#29905) > > > > > > - Upgrade FAB to 4.3.0 (#29766) > > > > > > - Clear ExecutorLoader cache in tests (#29849) > > > > > > - Lazy load Task Instance logs in UI (#29827) > > > > > > - added warning log for max page limit exceeding api calls > (#29788) > > > > > > - Aggressively cache entry points in process (#29625) > > > > > > - Don't use ``importlib.metadata`` to get Version for speed > > (#29723) > > > > > > - Upgrade Mypy to 1.0 (#29468) > > > > > > - Rename ``db export-cleaned`` to ``db export-archived`` (#29450) > > > > > > - listener: simplify API by replacing SQLAlchemy event-listening > by > > > > direct > > > > > > calls (#29289) > > > > > > - No multi-line log entry for bash env vars (#28881) > > > > > > - Switch to ruff for faster static checks (#28893) > > > > > > - Remove horizontal lines in TI logs (#28876) > > > > > > - Make allowed_deserialization_classes more intuitive (#28829) > > > > > > - Propagate logs to stdout when in k8s executor pod (#28440) > > > > > > - Fix code readability, add docstrings to json_client (#28619) > > > > > > - AIP-51 - Misc. Compatibility Checks (#28375) > > > > > > - Fix is_local for LocalKubernetesExecutor (#28288) > > > > > > - Move Hive macros to the provider (#28538) > > > > > > - Rerun flaky PinotDB integration test (#28562) > > > > > > - Add pre-commit hook to check session default value (#28007) > > > > > > - Refactor get_mapped_group_summaries for web UI (#28374) > > > > > > - Add support for k8s 1.26 (#28320) > > > > > > - Replace ``freezegun`` with time-machine (#28193) > > > > > > - Completed D400 for ``airflow/kubernetes/*`` (#28212) > > > > > > - Completed D400 for multiple folders (#27969) > > > > > > - Drop k8s 1.21 and 1.22 support (#28168) > > > > > > - Remove unused task_queue attr from k8s scheduler class (#28049) > > > > > > - Completed D400 for multiple folders (#27767, #27768) > > > > > > > > > > > > > > > > > > *Doc only changes* > > > > > > - Add instructions on how to avoid accidental airflow > > > upgrade/downgrade > > > > > > (#30813) > > > > > > - Add explicit information about how to write task logs (#30732) > > > > > > - Better explanation on how to log from tasks (#30746) > > > > > > - Use correct import path for Dataset (#30617) > > > > > > - Create ``audit_logs.rst`` (#30405) > > > > > > - Adding taskflow API example for sensors (#30344) > > > > > > - Add clarification about timezone aware dags (#30467) > > > > > > - Clarity params documentation (#30345) > > > > > > - Fix unit for task duration metric (#30273) > > > > > > - Update dag-run.rst for dead links of cli commands (#30254) > > > > > > - Add Write efficient Python code section to Reducing DAG > > complexity > > > > > > (#30158) > > > > > > - Allow to specify which connection, variable or config are being > > > > looked up > > > > > > in the backend using ``*_lookup_pattern`` parameters (#29580) > > > > > > - Add Documentation for notification feature extension (#29191) > > > > > > - Clarify that executor interface is public but instances are not > > > > (#29200) > > > > > > - Add Public Interface description to Airflow documentation > > (#28300) > > > > > > - Add documentation for task group mapping (#28001) > > > > > > - Some fixes to metrics doc (#30290) > > > > > > > > > > > > Cheers, > > > > > > Ephraim > > > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: AIP-56 Extensible user management

;> user management natively. As a result, if this AIP gets approved and go > >> through, the multi tenancy feature will be implemented as a second step > in > >> a new user manager and not in Airflow directly. > >> > >> > >> As always, feel very free to give your opinion on this email thread or > on > >> the AIP by adding comments. > >> > >> > >> References: > >> - AIP: > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > >> < > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-56+Extensible+user+management > >> > > >> - Initial email list discussion: > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61 < > >> https://lists.apache.org/thread/lf585xvvxpqtzhfyc6drzrf3rmg37w61> > >> > >> > >> Vincent > >> > >> > >> > >> > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [VOTE] Release Airflow 2.6.0 from 2.6.0rc5

I think I found a critical bug in rc5 Just kidding :-) +1 binding Bolke Sent from my iPhone > On 29 Apr 2023, at 20:14, Elad Kalif wrote: > > +1 binding > > בתאריך שבת, 29 באפר׳ 2023, 21:08, מאת Pierre Jeambrun ‏< > pierrejb...@gmail.com>: > >> +1 binding. (licences, signatures,

Re: AIP-56 Extensible user management

erm. Though, as a first iteration I >> think it is better to define a user manager using FAB as default user >> manager. The benefit of doing so is to have a backward compatible >> transition between before AIP and after AIP. This also simplifies the AIP >> since it is eas

Re: AIP-56 Extensible user management

t's it. > > Flask will then use this API to get the user name and display it in the > UI. > > My experience in Flask is pretty limited so I might miss something but > the > > way I see it is, you need a way to retrieve your user name from whatever > > user manager you

Re: [DISCUSS] Future of Pendulum in Airflow

replacement? > > Maybe someone from Airflow Community could propose their help with > maintenance of library: > - https://github.com/sdispater/pendulum/issues/590 > > Maybe we should get rid of the pendulum at all, as a last resort solution. > I can't imagine how we could do that, because a lot of stuff depends on the > pendulum and removing it would be a breaking change. > > > Best Wishes > *Andrey Anshin* > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Future of Pendulum in Airflow

er libraries might have a different kind of issue including > compatibility with databases. > More close replacement it is dateutil, but it also maintained by one person > last release was 2 years ago and contains quite a few issues with > timezones/DTS (no blame, that is just a fact)

Re: [DISCUSS] Future of Pendulum in Airflow

n Thu, 28 Sept 2023 at 15:12, Bolke de Bruin wrote: > for serialization I am not too worried about ZoneInfo. We do not use > pickling by default as we roll our own serialization format. We probably > just need the key (zoneinfo.key). > > I'm not sure what happened about thi

[DISCUSS] AIP-58 Add Airflow FS

ts. Bolke -- -- Bolke de Bruin bdbr...@gmail.com

[VOTE] AIP-58 Airflow ObjectStore

lease vote accordingly: [ ] + 1 approve [ ] + 0 no opinion [ ] - 1 disapprove with the reason Only votes from PMC members and committers are binding, but other members of the community are encouraged to check the AIP and vote with "(non-binding)". Cheers Bolke -- -- Bolke de Bruin bdbr...@gmail.com

Re: [VOTE] AIP-58 Airflow ObjectStore

s now. > > J. > > > > On Thu, Oct 19, 2023 at 4:34 PM Igor Kholopov > > wrote: > > > Thanks for incorporating the feedback! > > > > +1 (non-binding) > > > > On Thu, Oct 19, 2023 at 1:55 PM Dennis Akpenyi > > wrote: > > > >

Re: [VOTE] AIP-58 Airflow ObjectStore

31.17 -> released 1 Aug 2023 > > And it seems like everyone was waiting for it : > https://github.com/fsspec/s3fs/pull/809- the s3fs change for it was merged > yesterday. > > So yes +1! I hope the s3fs release will happen before we merge AIP-58. > > J. > > > > On Thu,

Re: [VOTE] AIP-58 Airflow ObjectStore

work that you were thinking or was it part of AIP >completion? >2. We would contribute the File abstraction as a follow-up to this AIP >too, which will help with the Dataset story too > > > Regards, > Kaxil > > On Thu, 19 Oct 2023 at 20:21, Bolke de Bruin w

Re: [VOTE] AIP-58 Airflow ObjectStore

of the scope of this AIP but will be possible). > > > > +1 (binding) > > > > My only concern is the stability of fsspec packages; I had a bad > experience > > with s3fs and gcsfs in the past due to patch/minor releases with breaking > > changes or conflict with

[RESULT] [VOTE] AIP-58 Airflow ObjectStore

Hello, AIP-58 is accepted. 4 +1 binding votes have been cast: Bolke Jarek Kaxil Hussein 4 +1 non binding Jens Avi Dennis Igor Vote thread: https://lists.apache.org/thread/wokt58k15g81cjnsytq9k1ofvspb4d5c The pr is almost done and I intend to finalize it by the end of this week. Please revi

Re: [DISCUSS] Removing Qubole provider (and adding removal process)

I suggest also removing it from pypi for security reasons. If there is a security issue with it then the issue will remain with us. B. Sent from my iPhone > On 26 Oct 2023, at 20:20, Jarek Potiuk wrote: > > Hello Airflow community, > > How do we feel about removing the Qubole provider compl

Re: [DISCUSS] Future of Pendulum in Airflow

> > Cons: > > - Current serialization model can't deal with backport packages. E.g. > > timezone which are serialized in backport_zoneinfo can't be deserialized > in > > zoneinfo > > > > Maybe we should replace parse datetime with another solution. Does anyone > > know a good replacement? > > > > Maybe someone from Airflow Community could propose their help with > > maintenance of library: > > - https://github.com/sdispater/pendulum/issues/590 > > > > Maybe we should get rid of the pendulum at all, as a last resort > solution. > > I can't imagine how we could do that, because a lot of stuff depends on > the > > pendulum and removing it would be a breaking change. > > > > > > Best Wishes > > *Andrey Anshin* > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [PROPOSAL] Deprecate URI Connection representation in favor of JSON

s://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#generating-a-connection-uri > > > > , there is no such of method for generate JSON, but I don't see the > > > problem > > > > to create such of method in Connection object which return string > > > > representation of JSON connection. > > > > > > > > So my suggestion would be pretty simple: > > > > 1. Deprecate accept URI as parameter to the Connection object > > > > 2. Deprecate uri property > > > > 3. Replace get_uri by get_json > > > > 4. Replace all URI examples in Providers Connections by JSON > > alternatives > > > > > > > > > > > > WDYT? Maybe I miss something and we should keep Airflow Connection as > > URI > > > > in the future Airflow's major versions and support both ways to > provide > > > > connections as JSON and URI. > > > > > > > > > > > > Best Wishes > > > > *Andrey Anshin* > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Suspend/Remove Apache Scoop provider

+1 Go Sent from my iPhone > On 23 Nov 2023, at 10:52, Andrey Anshin wrote: > > Greetings everyone! > > Since we began to actively use the mechanism to suspend/remove providers I > want to start the discussion about suspend and potential remove Apache > Scoop [1] provider. > > Apache Scoop

Re: [VOTE] Airflow Providers prepared on November 24, 2023

https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/7.10.0rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-common-io/1.1.0rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.8.1rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-databricks/5.0.1rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-dbt-cloud/3.4.1rc1/ > > > > https://pypi.org/project/apache-airflow-providers-docker/3.8.2rc1/ > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-elasticsearch/5.2.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-google/10.12.0rc1/ > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/8.3.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-odbc/4.2.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-openai/1.0.1rc1/ > > > > https://pypi.org/project/apache-airflow-providers-opsgenie/5.3.0rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-papermill/3.5.0rc1/ > > > > https://pypi.org/project/apache-airflow-providers-redis/3.4.1rc1/ > > > > > https://pypi.org/project/apache-airflow-providers-snowflake/5.1.2rc1/ > > > > https://pypi.org/project/apache-airflow-providers-trino/5.4.1rc1/ > > > > > > > > Cheers, > > > > Elad Kalif > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Future of Pendulum in Airflow

gt; mid-term >>>>>>>> if we could gain influence and even take part and speed up >>> releases >>>>> and >>>>>>>> merging of issues that are blocking us (or would be blocking us >> in >>>>> the >>>>&

[DISCUSS] Serialization in Apache Airflow: Clarifications and Future Directions

Hi All! I am reaching out to initiate a discussion on a topic that has been causing some confusion within the community: serialization in Apache Airflow. The purpose of this email is to shed light on the current state of serialization and, if necessary, open the floor for discussions on the way f

Re: [DISCUSS] Serialization in Apache Airflow: Clarifications and Future Directions

ore generic serialization starts and decrypt after deserialization > ends.So rather than > building in encryption in serialization, we should provide simple > primitives ("encrypt", "decrypt") that > serializer should use to encrypt it's data, but > serialization/deserialization should not care about > whether the field is encrypted or not. Maybe this is what you also thought > about - if so - I am good > with it. > > We do as you say at the moment also in the linked PR. The serializer does not deal with encrypted values directly. It is up to the object to determine what needs to be encrypted and what not. > > · What do we define as a best practice for interacting with the > > serializers? Are we okay with losing semantic information if using an > > intermediate format? Or do we find it the best practice to provide a > > serializer? > > > > I think it should be case-by-case. And we should clearly define where the > XCom serializer > should be used in and in what context. And where other serializers might > and should be > used. I see no particular value in applying the same rules and assumptions > for the > different serialization cases we might have. > > Case-by-case is fine to me as long as the choice is explicit and thus options are considered. > > > *Other links:* > > > > · Docs on serde / Xcom serializer: > > https://github.com/apache/airflow/pull/35885 > > > > · Encryption: https://github.com/apache/airflow/pull/35867 > > > > · Discussion on PyODBC: > https://github.com/apache/airflow/pull/32319 > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Allowlist in serializatin [was: Serialization in Apache Airflow]

somewhere else that is a serious issue. Basically you can rely on pickle when you are sure you control both sides and the user cannot interfere. This is what Spark does for example with Spark Connect. We cannot rely on that because the user is in between. Bolke -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Allowlist in serializatin [was: Serialization in Apache Airflow]

as the potential of giving > our users false sense of security - if we do not properly describe it in > the model. So we need to be very clear when we are describing the intended > level of isolation provided. > > We are not just talking about the different DAG authors here. The DAG of U1 might run in an environment different from the DAG of MA1. If MA1 wants to get extra information on that environment by doing something in the context of U1 it affects OPS which is, in good practices, not (managed by) the same person as U1. Bolke -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Using serde for PyODBC case/other cases [was Serialization in Apache Airflow]

on what we want to do with data awareness (separate > > discussion). If you think: > > > > ObjectStoragePath : Table : Dataframe > > > > It is useful to have dataframe as part of core. This could be pandas or > > maybe polars which is so much faster :-). > > >

Re: [PROPOSAL] NEED URGENT FEEDBACK! MySQL Badly signed repo breaks Airflow images

gt; Both myself and Andrey's recommendation is to go to MariaDB for main and > > 2.8.0. We are pretty fed-up with a number of past issues we had to fight > > with - and even if Oracle fixes it now, we have no faith Oracle can be > > trusted as MySQL steward for the repos (Andrey did not have faith for > quite > > some time now and I lost it today). > > > > In order to respond to potential edge cases, we can also keep the option > > that users who really want it, might be able to rebuild the image with > > MySQL drivers - similarly as we give the users the option to use bullseye > > in 2.8.0. And we can even run a CI build with it to see that it continues > > to work. > > > > I believe this is the best option and we can get it implemented quickly > > today and get RC4 of Airflow with mariadb clients. If we see a > > support/consensus for the community we will **just** proceed with it and > > start a formal LAZY_CONSENSUS thread. > > > > WDYT? > > > > J. > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [VOTE] Airflow Providers prepared on December 12, 2023

+1 binding Let's get this out. B Sent from my iPhone > On 17 Dec 2023, at 09:20, Rahul Vats wrote: > > +1 non-binding > > Regards, > Rahul Vats > 9953794332 > > >> On Sun, 17 Dec 2023 at 13:18, utkarsh sharma wrote: >> >> +1 non-binding >> >> On Sun, 17 Dec 2023 at 1:15 PM, Pankaj Sing

Re: [DISCUSS] Future of Pendulum in Airflow

> > Example from May 2022: > > > > https://github.com/apache/airflow/issues/23400 > > > > And Example from March 2020 ( which might or might not be fixed by the > open > > PR): > > > > https://github.com/apache/airflow/issues/7999 > > > &

Re: [ANNOUNCE] Apache Airflow 2.8.0 Released

🥳 B. Sent from my iPhone > On 18 Dec 2023, at 20:11, Kaxil Naik wrote: > > That's awesome 🎉 > >> On Tue, 19 Dec 2023 at 00:39, Jarek Potiuk wrote: >> >> Wooohooo! >> >>> On Mon, Dec 18, 2023 at 8:05 PM Ephraim Anierobi >>> wrote: >>> >>> Dear Airflow community, >>> >>> I'm happy to ann

Re: [DISCUSS] "Require conversation resolution" in our PRs before merge?

I'm less enthusiastic. What problem are we solving with this? If something has not been addressed it can be done in a follow up or of if it was just part of the conversation it won't have impact on the code. In addition, the ones that need to deal with it are the ones merging and those are not n

Re: [DISCUSS] "Require conversation resolution" in our PRs before merge?

This reflect my feelings as well. I'm not convinced we are solving something that needs to be solved. B. Sent from my iPhone > On 19 Dec 2023, at 21:05, Ash Berlin-Taylor wrote: > > Weak -1 from me, because I don't think this needs to be enforced/required > part of the workflow. > > I.e. C

Re: [Discussion] AIP-60 Standard URI representation for Airflow Datasets

gt; - > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > --------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

Just checking: I see that some improvements to Airflow.io (two weeks ago) were not included and some provider updates neither. Haven't checked anything else yet. Is that intentional? Ie. is that the purpose of this release. Other big(ger) and more recent changes have been included hence askin

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

github.com/apache/airflow?tab=readme-ov-file#what-goes-into-the-next-release > > Regards > - Ephraim > > On Wed, 17 Jan 2024 at 09:03, Bolke de Bruin wrote: > > > Just checking: > > > > I see that some improvements to Airflow.io (two weeks ago) were not >

Re: [VOTE] Release Airflow 2.8.1 from 2.8.1rc1

any regression. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > On Thu, Jan 18, 2024 at 12:16 AM Jed Cunningham < > > jedcunning...@apache.org> > > > wrote: > > > > > > > +1 (binding) > > > > > > > > Checked signatures, checksums, licences. Used it with the helm chart > > > with a > > > > few different configs > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: What goes into the next release (was [VOTE] Release Airflow 2.8.1 from 2.8.1rc1)

mail about it at our devlist, or the > documentation about it in README was too succinct? > > Or maybe we need a better way of communicating it - I am not sure. But I > hope this email will clarify a lot of it, and will prove valuable when > searching in devlist. > > Maybe even some

Re: [ANNOUNCE] Starting experimenting with "Require conversation resolution" setting

t; > > > >>>> > > > > >>>>> Wooho! Looking to see how this turns out for airflow 😃 > > > > >>>>> > > > > >>>>> On Sat, 30 Dec 2023 at 1:35 PM, Jarek Potiuk > > > > > >> wrote: > > > > >>>>> > > > > >>>>>> Hello everyone, > > > > >>>>>> > > > > >>>>>> As discussed in > > > > >>>>>> > > https://lists.apache.org/thread/cs6mcvpn2lk9w2p4oz43t20z3fg5nl7l > > > I > > > > >>> just > > > > >>>>>> enabled "require conversation resolution" for our main/stable > > > > >>> branches. > > > > >>>>> We > > > > >>>>>> have not used it in the past so it might not work as we think > or > > > we > > > > >>>>> might > > > > >>>>>> need to tweak something. > > > > >>>>>> > > > > >>>>>> Generally speaking (if all works) all conversations on PRs > > should > > > be > > > > >>>>>> resolved before we can merge the PR. This "resolving" is > > > encouraged > > > > >> to > > > > >>>>> be > > > > >>>>>> done by the author when they think the conversation is > resolved, > > > but > > > > >>> it > > > > >>>>> can > > > > >>>>>> also be done by reviewers or the maintainer who wants to merge > > the > > > > >> PR. > > > > >>>>>> > > > > >>>>>> We attempted to describe some basic rules and expectations > here: > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > >> > > > > > > > > > > https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#step-5-pass-pr-review > > > > >>>>>> but undoubtedly there will be questions and issues that we > might > > > > >> want > > > > >>> to > > > > >>>>>> solve - so feel free to discuss it here or raise > question/issues > > > in > > > > >>>>>> #development channel in slack (I am also happy to be pinged > > > directly > > > > >>>>> about > > > > >>>>>> it and help to resolve any issues/gather feedback). > > > > >>>>>> > > > > >>>>>> J. > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > - > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [ANNOUNCE] Starting experimenting with "Require conversation resolution" setting

esolving discussions. > > > > -a > > > > On 31 January 2024 11:00:11 GMT, Ash Berlin-Taylor > wrote: > > >I'm a -1 on keeping this as I don't see it gives us any real benefit > > other than a rubber-stamp. Let's treat people as intelligent grown

Re: [VOTE] January 2024 PR of the Month

I'll give 22253 my vote. That one was from outside and resurrected from the dead. Jarek's is also great. Bolke Sent from my iPhone > On 25 Jan 2024, at 01:52, Mehta, Shubham wrote: > > +1 for #36537. > > Shubham > > > On 2024-01-23, 3:03 PM, "Scheffler Jens (XC-AS/EAE-ADA-T)" >

Re: [RESULT][VOTE] January 2024 PR of the Month

Ohh haha. Mail was trickling on slowly. Well deserved Jarek. Sent from my iPhone > On 30 Jan 2024, at 20:07, Briana Okyere > wrote: > > Hey All, > > Congratulations to Jarek on winning PR of the Month for January 2044 with PR > #36537: Standardize airflow build process and switch to Hatchlin

Re: [VOTE] "Require conversation resolution" setting in PRs as permanent solution

-1, binding. Sent from my iPhone > On 5 Feb 2024, at 21:07, Scheffler Jens (XC-AS/EAE-ADA-T) > wrote: > > +1 binding > > Sent from Outlook for iOS > > From: Pankaj Koti > Sent: Monday, February 5, 2024 3:36:41 PM > To: dev@airflow.ap

Re: [DISCUSS] Rename channels on slack

his is > > > only > > > > my > > > > >> > > personal opinion. I am not against the names Jarek suggested. > > > > >> > > > > > > >> > > On 2024/02/08 11:54:34 Jarek Potiuk wrote: > > > > >> > > > Hey here, > > > > >> > > > > > > > >> > > > The number of "troubleshooting/best-practices" questions we > > have > > > > in > > > > >> > > > #development channel on Slack reached the level where we > have > > > more > > > > >> of > > > > >> > > those > > > > >> > > > than discussion about airflow development. > > > > >> > > > > > > > >> > > > There were few slack proposals, that we should change the > > names, > > > > >> but it's > > > > >> > > > quite a significant change for everyone, so we need to > discuss > > > and > > > > >> have > > > > >> > > > [LAZY CONSENSUS] here. > > > > >> > > > > > > > >> > > > My proposal is to rename them like that (but I am totally > open > > > to > > > > >> other > > > > >> > > > ideas): > > > > >> > > > > > > > >> > > > #development -> #contriuting-to-airflow > > > > >> > > > #troubleshooting -> #troubleshooting-questions > > > > >> > > > > > > > >> > > > And I propose to create a new channel: > > > > >> > > > > > > > >> > > > #best-practices > > > > >> > > > > > > > >> > > > There users will be able to discuss best-practices on how to > > use > > > > >> airflow > > > > >> > > > (it's not troubleshooting, it's more "please advise how I > > should > > > > do > > > > >> > > that). > > > > >> > > > We should make some announcements there, update our > community > > > > pages > > > > >> and > > > > >> > > > welcome messages on slack to mention this channel. > > > > >> > > > > > > > >> > > > WDYT? > > > > >> > > > > > > > >> > > > > > > >> > > > > > > - > > > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > - > > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > >> For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >> > > > > >> > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] Deprecate cli options in Airflow Configurations and airflow.api.client

rent_api_client` should able to return None, > if it return None, then it should use implementation from the > airflow/cli/commands, otherwise use deprecated client, with raising > RemovedInAirflow3Warning > > WDYT? > -- -- Bolke de Bruin bdbr...@gmail.com

Re: Bad mixing of decorated and classic operators (users shooting themselves in their foot)

hat well the @task decoration magic, but maybe we could > > somehow detect the case where someone instantiates the operator (and > runs > > execute) inside a decorated task and give a helpful error in this case? I > > am afraid people will start using it more and more and the sooner we add > > protection against it, the better chance we have to contain it. > > > > J. > > -- -- Bolke de Bruin bdbr...@gmail.com

Re: Apache Airflow 2.9.0b2 available for testing

Pendulum 3 serializes ISO8601 slightly different (missing T) AFAIK. I thought someone opened an issue for that (don't have it handy). Maybe it is something we should at least mention in the release notes? Sent from my iPhone > On 28 Mar 2024, at 00:54, Ephraim Anierobi wrote: > > Hey fellow

Re: Apache Airflow 2.9.0b2 available for testing

7 > > From: Bolke de Bruin > Date: Thursday, 28 March 2024 at 13:01 > To: dev@airflow.apache.org > Subject: Re: Apache Airflow 2.9.0b2 available for testing > Pendulum 3 serializes ISO8601 slightly different (missing T) AFAIK. I thought > someone opened an issue for that (don&

Re: Apache Airflow 2.9.0b2 available for testing

ersion < 3.12. > > While possibly we should warn people that Pendulum 3 serialization changes > for ISO8601, it's not really connected with the Airflow 2.9 release. > > J. > >> On Thu, Mar 28, 2024 at 4:47 PM Bolke de Bruin wrote: >> >> It is as w

Re: [VOTE] Release Airflow 2.9.0 from 2.9.0rc2

+1 binding Deployed here no issues found. Bolke Sent from my iPhone > On 5 Apr 2024, at 19:02, Jed Cunningham wrote: > > +1 (binding) > > Checked reproducibility, signatures, checksums, licences. Used it with the > helm chart with a few different configs. All looks good! -

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

I have left several comments :-). And on interactive dag runs even after the explanation of Vikram I still don't have a clue what we want to accomplish there :-P. I would like to see a mantra or team for Airflow 3. That helps nudging people in the same direction. Suggestions in the comments. B

Re: [HUGE DISCUSSION] Airflow3 and tactical (Airflow 2) vs strategic (Airflow 3) approach

that connection info be passed "as part of > task > > matadata". That would require some mechanism of declaring prior to task > > execution what connections would be used. This is a thought that has > come > > up when thinking about execution of non-python t

Re: [VOTE] AIP-69 Remote Executor

ZQilaxeu9E%3D&reserved=0 > > > > >> > > > > > >> > > > > > >> > > > > > >> > Since the first draft was raised and the discussion thread went > > > > along, I > > > > >> > integrated a bit of feedback and now also added some technical > > > > details as > > > > >> > proposed development. After successful vote I’d raise a draft > > > > >> > PR as > > > > PoC. > > > > >> > As a big wave of emails was posted by Jarek after I dropped > > > > >> > this AIP > > > > I’d > > > > >> > like to highlight that I propose to make this a tactical > > > > implementation > > > > >> > which might be a base for some discussion how to distribute > > > > >> > work in > > > a > > > > >> > future Airflow 3.0. I would assume if interfaces and structures > > > > change, > > > > >> > rework will be needed and it is fully accepted that breaking > > > > >> > changes > > > > need > > > > >> > to be applied if moving to Airflow 3. > > > > >> > > > > > >> > > > > > >> > > > > > >> > Looking forward to releasing this for Airflow 2.10 (but > > > > >> > depending > > > how > > > > >> fast > > > > >> > I can make it and also depending on if somebody wants to join > > > forces). > > > > >> > > > > > >> > > > > > >> > > > > > >> > Consider this my +1 binding vote. > > > > >> > > > > > >> > > > > > >> > > > > > >> > The vote will last until 6:00 PM GMT/UTC on May 23, 2024, and > > > > >> > until > > > at > > > > >> > least 3 binding votes have been cast. I have it a bit longer as > > > usual > > > > >> > because of a public holiday in some countries. > > > > >> > > > > > >> > > > > > >> > > > > > >> > Please vote accordingly: > > > > >> > > > > > >> > > > > > >> > > > > > >> > [ ] + 1 approve > > > > >> > > > > > >> > [ ] + 0 no opinion > > > > >> > > > > > >> > [ ] - 1 disapprove with the reason > > > > >> > > > > > >> > > > > > >> > > > > > >> > Only votes from PMC members and committers are binding, but > > > > >> > other > > > > members > > > > >> > of the community are encouraged to check the AIP and vote with > > > > >> > "(non-binding)". > > > > >> > > > > > >> > Mit freundlichen Grüßen / Best regards > > > > >> > > > > > >> > Jens Scheffler > > > > >> > > > > > >> > Alliance: Enabler - Tech Lead (XC-AS/EAE-ADA-T) Robert Bosch > > > > >> > GmbH | Hessbruehlstraße 21 | 70565 Stuttgart-Vaihingen > > > | > > > > >> > GERMANY | www.bosch.com > > > > >> > Tel. +49 711 811-91508 | Mobil +49 160 90417410 | > > > > >> > jens.scheff...@de.bosch.com<mailto:jens.scheff...@de.bosch.com> > > > > >> > > > > > >> > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB > > > > >> > 14000; > > > > >> > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer; > > > > >> > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. > > > > Markus > > > > >> > Forschner, > > > > >> > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja > > > > >> > Rückert ​ > > > > >> > > > > > >> > > > > > > > > > > -- -- Bolke de Bruin bdbr...@gmail.com

[DISCUSS] AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

-- -- Bolke de Bruin bdbr...@gmail.com

Re: [DISCUSS] AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

ge > and this AIP solves only the part that Airflow machines will be able to > fetch the files from storage. Is that correct? > > On Sun, May 26, 2024 at 10:55 AM Bolke de Bruin wrote: > > > Hi All, > > > > I would like to discuss a new AIP aimed at enhancing t

Re: [DISCUSS] AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

lly be available without changes to your code. > > What about SQL files a task uses, either as a template or via something else > such as dbt? How about YAML based dag generators? > > (This might be mentioned in the wiki page, but it's not loading for me right > now) >

Re: [DISCUSS] AIP-71 Generalizing DAG Loader and Processor for Ephemeral Storage

It goes a bit backwards from "let's think about what we should remove from > Airflow and delegate rather than add to it", but might be worth it if we > get other benefits for free (versioning and performance IMHO are worth > looking at). > > J. > > On Sun, May 2

Re: [DISCUSS] AIP-63, AIP-64, and AIP-65: DAG Versioning

In context of AIP-71 - slightly directing your attetion there for discussion purposes I think it would be nice to do a dag = dag_load(ObjectStoragePath("dagfs://mydag?version=1)) Having dag versioning as an fs implementation would open up additional interesting avenues for DAG manipulation. BT

Re: [VOTE] Release Apache Airflow 2.0.0 form 2.0.0rc3

+1, binding. AWESOME WORK We talked about this so much and now it's finally there. It almost.akes my cry ;-) Bolke Sent from my iPhone > On 14 Dec 2020, at 23:01, Jarek Potiuk wrote: > >  > +1 (binding): RAT, SHA, GPG are all OK. > > * I run a few combinations of Python/Backends. > *

Re: Airflow 1.10.2b1 available now on PyPI - PLEASE TEST

Thanks Kaxil. As always be really careful with calling things a release. This is not a release in anyway. Just a convenience package for developers. For the future I suggest not using the word ‘release’ in this context. It will confuse people. Can you please include: https://github.com/apache/

Re: Airflow 1.10.2b1 available now on PyPI - PLEASE TEST

Also my GPL removal is not done yet :-(. Working on it... Verstuurd vanaf mijn iPad > Op 13 jan. 2019 om 19:59 heeft Felix Uellendall het > volgende geschreven: > > Hey kaxil, > > thank your for running this release. :) > > I just want to let you know some duplicate lines in the CHANGELOG.tx

Re: [VOTE] Airflow 1.10.2rc3

+1, binding Checked: * KEYS * Signatures * Git Tag Bolke > On 19 Jan 2019, at 17:35, Kaxil Naik wrote: > > Hey all, > > I have cut Airflow 1.10.2 RC3. This email is calling a vote on the release, > which will last for 72 hours. Consider this my (binding) +1. > > Airflow 1.10.2 RC3 is availa

Re: [VOTE] Airflow 1.10.2rc3

Can I suggest that we get in the habit of time based releases, now that we don’t have to vote twice? In this context I suggest that not having documentation is not blocking for release, but that we should do a quick .3 release Tao can you change your vote to +1? Verstuurd vanaf mijn iPad > Op

Re: Airflow 1.10.2 is released

Great work Kaxil! Thanks. Verstuurd vanaf mijn iPad > Op 23 jan. 2019 om 02:18 heeft Kaxil Naik het volgende > geschreven: > > Dear Airflow community, > > I'm happy to announce that* Airflow 1.10.2* was just released. > > The source release, as well as the binary "sdist" release, are availab

Re: [PROPOSAL] Add a landing page for Apache Airflow

+1, not sure about an AIP. Why not just do it? Sent from my iPhone > On 25 Jan 2019, at 08:41, Driesprong, Fokko wrote: > > +1 would be awesome! > > Op vr 25 jan. 2019 om 07:52 schreef Kenneth Knowles : > >> Since Beam was mentioned, please do take any of our automation you find >> useful. We

Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?

I agree. Maybe we should make a choice on what we consider 1st class executors and move others to contrib (I'm looking at you celery) Sent from my iPhone > On 25 Jan 2019, at 12:46, Ash Berlin-Taylor wrote: > > Is anyone using the Mesos Executor? I think we should deprecate and remove it. > >

Re: Usage of airflow/contrib/auth/backends/password_auth.py

Kerberos auth also.works with the API. So let's properly remove it. Sent from my iPhone > On 31 Jan 2019, at 14:52, Deng Xiaodong wrote: > > Thanks Niels. > > In this case, one solution we’re thinking of is to refactor this module to > make it work with the `ab_user` table (currently it’s wo

Re: Proposal to remove json_client

Hi David, I assume that you then want to ensure that all calls from the CLI are being made through the Rest-API. In other words the json_client would be the only client remaining. The local_client is a security problem as it needs direct database access which makes it problematic to have normal

Re: Proposal to remove json_client

LI access the DB in the same manner the REST API does. > > In my diagram all DB be access would be in the API/common/experimental/*.py > files, with both REST and CLI APIs making calls to this internal API. > > Can you explain further the security concern with the CLI? > > -

Re: Contributing to the Kubernetes Exeuctor

Hi Darren These improvements sound super useful and will benefit many. Starting to contribute is as easy as opening a PR as you have done. There is no reason to ask for permission to the community. Sometimes it might be handy to provide an explanation of what you try to accomplish particularly

Re: [VOTE] Accept AIP-13: OpenAPI 3 based API definition

I’m not sure if we need to vote on this (what is the consequence of -1?), but I don’t mind +1, binding On 21 February 2019 at 19:58:50, Driesprong, Fokko (fo...@driesprong.frl) wrote: +1 (binding) Would be a great improvement to consolidate the API. Cheers, Fokko Op do 21 feb. 2019 om 19:36

Re: [DISCUSS] AIP-12 Persist DAG into DB

I agree with Fokko here. We have been discussing serialisation for a veerrryyy long time and nothing has come of it :-). Probably because we are making it too big. As Fokko states, there are several goals and each of them probably warrants an AIP: 1) Make the webserver stateless: needs the graph

Re: Multiple Schedulers - "scheduler_lock"

I have done quite some work on making it possible to run multiple schedulers at the same time. At the moment I don’t think there are real blockers actually to do so. We just don’t actively test it. Database locking is mostly in place (DagRuns and TaskInstances). And I think the worst that can

Re: Welcome Daniel Imberman as a new committer!

Welcome! :-) Verstuurd vanaf mijn iPad > Op 19 mrt. 2019 om 01:13 heeft jiajie zhong het > volgende geschreven: > > Congratulations, Daniel! > > -jiajie > > Sent from my iPhone > >> On Mar 19, 2019, at 07:45, Daniel Imberman wrote: >> >> Thank you everyone 🙂. Excited for this opportunity!

Re: [VOTE] Accept AIP-3: Drop support for Python 2

+1, binding Sent from my iPhone > On 24 Mar 2019, at 05:26, Deng Xiaodong wrote: > > +1 (non-bonding) > >> On Sun, Mar 24, 2019 at 11:49 Tao Feng wrote: >> >> +1 (binding) >> >> On Sat, Mar 23, 2019 at 4:19 PM Driesprong, Fokko >> wrote: >> >>> Dear Airflow community, >>> >>> This email

Re: Database referral integrity

as it slows down inserts in fact tables and straight out >>> prevents bulk loading. >>> >>> Another approach is to avoid deleting in general, especially referenced >>> fks, and setting up some activity/visibility flag to false instead. >>> >

  1   2   >