tacharya
> Sent: Thursday, 30 January 2025 15:09
> To: dev@airflow.apache.org
> Subject: Re: [ANNOUNCEMENT] Moving Providers to separate sub-projects
> soon-ish
>
> [You don't often get email from kunal.ju...@gmail.com. Learn why this is
> important at https://aka.ms/Le
uary 2025 15:09
To: dev@airflow.apache.org
Subject: Re: [ANNOUNCEMENT] Moving Providers to separate sub-projects soon-ish
[You don't often get email from kunal.ju...@gmail.com. Learn why this is
important at https://aka.ms/LearnAboutSenderIdentification ]
EXTERNAL MAIL: Indien je de afzender
Bummer, thank you for letting me know Kunal š
-Original Message-
From: Kunal Bhattacharya
Sent: Thursday, 30 January 2025 15:09
To: dev@airflow.apache.org
Subject: Re: [ANNOUNCEMENT] Moving Providers to separate sub-projects soon-ish
[You don't often get email from kun
-Original Message-
> From: Jarek Potiuk
> Sent: Sunday, 26 January 2025 12:47
> To: dev@airflow.apache.org
> Subject: Re: [ANNOUNCEMENT] Moving Providers to separate sub-projects
> soon-ish
>
> EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze
> niet vert
FYI: I've started a new PR to move the Microsoft related providers to the new
folder structure
-Original Message-
From: Jarek Potiuk
Sent: Sunday, 26 January 2025 12:47
To: dev@airflow.apache.org
Subject: Re: [ANNOUNCEMENT] Moving Providers to separate sub-projects soon-ish
EXT
Ah: one small, important thing I forgot: For the new providers, NO LONGER
dependencies are maintained in `provider.yaml` - each new provider has it's
own `pyproject.toml` and you should edit all kinds of dependencies
(required, optional, devel) in the respective `pyproject.toml` of the
provider - s
After the initial move of few providers - the move is in progress
https://github.com/apache/airflow/issues/46045 , also we have a dedicated
channel in our Slack: https://apache-airflow.slack.com/archives/C08AA9EJFB5.
-> #new-providers-strucuture
Already few people are moving providers (Thanks Kalya
Ah and one comment (Pierre already encountered it) - before running `breeze
start-airflow` you will need to rebuild the image.
Breeze will nicely ask you to do so, so pressing `y` when it does will do
the job.
This will happen more as we move the providers, so for a while if you see
missing import
Update:
We have now 3 providers moved: *airbyte, celery and edge - pre-release*
(thanks Jens for moving edge one!). I have already fixed a few initial
issues and improvements and added some consistency checks, there are two or
three more providers to move with some of the things I still need to ad
I will try to get celery next - we need a bugfix for it after yanking the
last release.
On Tue, Jan 14, 2025 at 9:46āÆAM Elad Kalif wrote:
> I plan to cut a wave soon but we can't test on airbyte as there were no
> changes since last release
>
> On Tue, Jan 14, 2025 at 8:52āÆAM Jarek Potiuk wrote
I plan to cut a wave soon but we can't test on airbyte as there were no
changes since last release
On Tue, Jan 14, 2025 at 8:52āÆAM Jarek Potiuk wrote:
> First provider (airbyte) merged. Managed to walk around Sphinx without
> answering too many riddles finally.
>
> In the next coming days I will
First provider (airbyte) merged. Managed to walk around Sphinx without
answering too many riddles finally.
In the next coming days I will migrate a few more providers and once we get
through a next provider release cycle with those few providers and fix/see
any kind of teething issues I will add s
Same, I am not strongly opinionated, just a preference :)
On 2025/01/10 15:36:05 Vikram Koka wrote:
> I agree this makes sense.
>
> I was originally concerned that this would make it more difficult to ensure
> compatibility across providers for capabilities such as common.sql,
> objectstore, and
I agree this makes sense.
I was originally concerned that this would make it more difficult to ensure
compatibility across providers for capabilities such as common.sql,
objectstore, and so on.
However, seeing that the "common" pattern would remain the same and it's
only the code layout that is ch
So I propose letting the "doer" make the decision if we are split.
On Fri, Jan 10, 2025 at 11:52āÆAM Ash Berlin-Taylor wrote:
> Not strong at all, preference Is all. It sounds like Vincent and I are in
> the hyphen camp and you and Maciej are in the slash camp.
>
> +1 on the āI donāt care what co
Not strong at all, preference Is all. It sounds like Vincent and I are in the
hyphen camp and you and Maciej are in the slash camp.
+1 on the āI donāt care what code style is used as long as it is
programmatically enforcedā.
-a
> On 10 Jan 2025, at 09:41, Jarek Potiuk wrote:
>
> Is there any
Is there anything else that "tastes" Ash ? A concrete reason that makes you
think the "-" prefix in this case is better than the "/" folder? How
strong is your "taste" preference and do you think it will have some
lasting effect if we choose to flatten the folder structure?
I might make a small v
My preference is for being āmore directā and not having deeply nested things
where possible ā I think Microsoft might be the one case where having extra
folders makes sense. And Iām fine with things not being consistent across
providers/groups of providers.
-ash
> On 8 Jan 2025, at 17:18, Ja
We still will have to keep common-{compat/io/sql/...?} providers with
nested structure, as
they aren't "compat" provider, but "common-compat" provider - without the
nested name it
does not make sense IMO.
År., 8 sty 2025 o 18:21 Jarek Potiuk napisaÅ(a):
> Can you give an example of what might br
Can you give an example of what might break why having
`providers/aapche-beam/src/airflow/providers/apache/beam`?
Nothing will break. It's just:
* the code will have to be a little more complex as it will have to do some
conditional writes of "-" "/"
* there will be inconsistency in the depth of
> And we already have a number of mappings and conventions to handle that.
> For example provider I'd mapping to dirs (apache.beam -> apache/beam), and
> 'apache-airflow-providers-apache-beam' as package na e and
> airflow/providers/apache/beam as packages inside the distribution. Those
> will rem
Also - one more thing.. Doing this migration to the new structure- we could
*potrentially* use that opportunity to discontinue some providers. There
are quite likely quite a few that make very little point to maintain any
more and we could use the Airflow 3 release opportunity to say "those
provide
Since each provider will self contained, it will be easier for isolation
and moving providers one-by-one to Airflow 3.
On Wed, 8 Jan 2025 at 00:04, Jarek Potiuk wrote:
> I think it will be better to keep it.
>
> The reason we have varying levels were to group things together - mainly
> Apache re
I think it will be better to keep it.
The reason we have varying levels were to group things together - mainly
Apache related providers, but also Microsoft.
And we already have a number of mappings and conventions to handle that.
For example provider I'd mapping to dirs (apache.beam -> apache/bea
Good question. I always found it confusing to have some providers at different
level. Examples:
- "airbyte" in "providers" directory (I would qualify it as "regular" provider)
- "hive" in "providers/apache"
- "amazon" in "providers" but which contains only one sub directory "aws"
I would be in fa
+1 one to this on general terms, it will hopefully reduce a lot of the
boilerplate we need.
As for the amazon/aws example specifically that does bring up a question,
should we have `/` or `-`.. to give some examples:
cncf kubernetes: ./providers/cncf/kubernetes or ./providers/cncf-kubernetes
Ap
Oh. . And one other benefit of it will be that we will be able to get rid
of about 40% of the "Providers Manager" code. Currently, in Providers
manager we have a lot of "ifs" that make it possible to use providers in
breeze and local environment from the sources. In "production" installation
we are
Those are very good questions :)
On Mon, Jan 6, 2025 at 10:54āÆPM Ferruzzi, Dennis
wrote:
> To clarify that I understand your diagram correctly, let's say you clone
> the Airflow repo to ~/workspace/airflow/. Does this mean that the AWS Glue
> Hook which used to live at
> ~/workspace/airflow/pro
low.apache.org
Subject: [EXT] [ANNOUNCEMENT] Moving Providers to separate sub-projects soon-ish
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier Ć©lectroni
Hello everyone,
I have almost ready PR (I need to fix doc building and do some more tests /
docs for local development environments), for the first step of provider
separation to separate sub-projects inside our mono-repo - which has been
discussed and agreed to a long time ago - and I had a POC f
30 matches
Mail list logo
