; >
> > > But regardless of the curiosity above, I'll return to the drawing board,
> > and see what else can be done for this particular problem. If there are
> > other Bundle types who need to solve the same problem perhaps we can find a
> > more acc
If there are
> other Bundle types who need to solve the same problem perhaps we can find a
> more acceptable implementation in Airflow core to support this. And if not,
> I'll proceed with externalizing the storage of the S3 Bundle version
> metadata outside of Airflow.
> >
>
ow core to support this. And if not, I'll
> proceed with externalizing the storage of the S3 Bundle version metadata
> outside of Airflow.
>
> Cheers,
> Niko
>
> ________________
> From: Jarek Potiuk
> Sent: Wednesday, July 9, 2025 11:59:06 PM
>
his. And if not, I'll
proceed with externalizing the storage of the S3 Bundle version metadata
outside of Airflow.
Cheers,
Niko
From: Jarek Potiuk
Sent: Wednesday, July 9, 2025 11:59:06 PM
To: dev@airflow.apache.org
Subject: RE: [EXT] S3 Dag Bundle Versions and DB Mana
be afraid of doing sometimes difficult work to
> make a good product for our users, it's for them in the end :)
>
> However, I see your perspectives as well, making our code and DB
> management more complex is more work and complication for us. And from the
> feedback so f
From: Jens Scheffler
Sent: Wednesday, July 9, 2025 12:07:08 PM
To: dev@airflow.apache.org
Subject: RE: [EXT] S3 Dag Bundle Versions and DB Manager
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confir
st for the data to be stored transparently to the user and
co-located with the data it strongly relates to (i.e. the dag runs that are
associated with those bundle versions).
Is using DB Manager completely unacceptable these days? What are folks'
thoughts on that?
Cheers,
Niko
_____
i.e. the dag runs that are
> associated with those bundle versions).
>
> Is using DB Manager completely unacceptable these days? What are folks'
> thoughts on that?
>
> Cheers,
> Niko
>
>
> From: Jarek Potiuk
> Sent: Wednesday,
bject: RE: [EXT] S3 Dag Bundle Versions and DB Manager
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur extern
> Another option also would be Using dynamodb table? that also supports
snapshots and i feel it works very well with state management.
Yep that would also work.
Anything "Amazon" to keep state would do. I think that it should be our
"default" approach that if we have to keep state and the state i
Agree another s3 bucket also works here
Another option also would be Using dynamodb table? that also supports
snapshots and i feel it works very well with state management.
Pavan
On Wed, Jul 9, 2025 at 2:06 PM Jarek Potiuk wrote:
> One of the options would be to use a similar approach as terr
One of the options would be to use a similar approach as terraform uses -
i.e. use dedicated "metadata" state storage in a DIFFERENT s3 bucket than
DAG files. Since we know there must be an S3 available (obviously) - it
seems not too excessive to assume that there might be another bucket,
independe
Hey folks,
tl;dr I’d like to get some thoughts on a proposal to use DB Manager for S3 Dag
Bundle versioning.
The initial commit for S3 Dag Bundles was recently merged [1] but it lacks
Bundle versioning (since this isn’t trivial with something like S3, like it is
with Git). The proposed solutio
13 matches
Mail list logo
