Is there any open PR in Elyra, where any of the mentioned approach (api, s3, git-sync etc) has been tried out? I think it's important for Airflow devs to know the underlying architecture of Elyra to suggest something.
By a quick look at the .pipeline JSON file, it seems like you would need to do a mapping of it with a serialized DAG's JSON. However, it would solve only Scheduler & webserver part, for workers some special mechanism would be needed. I can join the call as well, if it is scheduled between 8am-11pm India time. On Tue, Nov 3, 2020 at 6:21 AM Alan K Chin <akc...@us.ibm.com> wrote: > Hi All, > > Thanks for responding! We are pretty excited about using Airflow with > Elyra. Regarding a time to meet and chat this week, let's shoot for > sometime between Wednesday and Friday. We can set something more concrete > tomorrow once we figure out everyone's timezones and availability. > > Look forward to hearing from all of you! > > -- > Alan Chin > CODAIT, San Francisco > Email - akc...@us.ibm.com > > > ----- Original message ----- > From: Jarek Potiuk <jarek.pot...@polidea.com> > To: dev@airflow.apache.org > Cc: luckbr1...@gmail.com, akc...@us.ibm.com > Subject: [EXTERNAL] Re: A Visual Editor for Airflow pipelines > Date: Mon, Nov 2, 2020 1:06 PM > > Cool!. I also think it's an interesting one:). But it would be great to > have such integration possible from Elyra :). Let us know what comes out of > it :). > > J. > > > On Mon, Nov 2, 2020 at 10:02 PM Ry Walker <r...@rywalker.com> wrote: > > Hi Luciano - > > Elyra looks like an interesting project — we'd love to connect and talk > through the opportunity. > > You can compare your cal to mine and grab a slot here: > https://calendly.com/ryw/60min > <https://urldefense.proofpoint.com/v2/url?u=https-3A__calendly.com_ryw_60min&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=aOWVqGrjKRUgW0EWaWXLRLpndFjU1PzNSlI6oWopbsU&e=> > — and I'll be sure to get a few of the Airflow PMC members to join as well. > > > -Ry > > Ry Walker > Founder/CTO of Astronomer + Airflow Committer > > > On Mon, Nov 2, 2020 at 12:00 AM Luciano Resende <luckbr1...@gmail.com> > wrote: > > Hi All, > > As mentioned in the user list [1] we are working on a visual editor > for pipelines and adding Airflow as one of the supported backends. > > https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html > <https://urldefense.proofpoint.com/v2/url?u=https-3A__elyra.readthedocs.io_en_latest_user-5Fguide_pipelines.html&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=8f8kamB87CAFy88IyKQf-8ncdas2jx4HXTXIWlerVfo&e=> > > As you are the Airflow devs, we would invite you to help us implement > the best integration possible, in two steps: > > 1) Getting a solid integration for building and running pipelines with > python scripts and jupyter notebooks > > 2) Expand the available list component types and enable more generic > operators > > One of the questions raised in the original e-mail is related to how > to best submit the pipeline dag to be executed by the Airflow runtime, > we have tried a few different options, starting from the experimental > REST API, S3 bucket syncs and these seem to not be the ideal solution, > will be looking into git-sync next, but would really appreciate some > suggestions on the best options, particularly if someone has already > done some external integration similar to this. > > Feel free to create issues for discussion and or more details > > https://github.com/elyra-ai/elyra/issues > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_elyra-2Dai_elyra_issues&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=SGo8ml8UvVVZb6o0hTZmzmZO1PrtNwTnyS2BgY5JE4w&e=> > > Or use this thread for suggestions > > [1] > https://lists.apache.org/thread.html/r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9%40%3Cusers.airflow.apache.org%3E > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread.html_r19ca5e61a90910a6b5de6feea186d9138a4cd47c91ea34dd4cce6ff9-2540-253Cusers.airflow.apache.org-253E&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=FsDxHEyjQeUPKhX6SKQrp3sxoLm6O5JgmohQ1DTcDX8&e=> > > > -- > Luciano Resende > http://twitter.com/lresende1975 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_lresende1975&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=rrHOp_HyFiYz0ObxnOfzX6A_fudgvIJBcptsbNWlZPU&e=> > http://lresende.blogspot.com/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__lresende.blogspot.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=licSt0rU7lNnqEB7hRm-In85VLVg-i7MsjTzfj-feQs&e=> > > > > -- > > Jarek Potiuk > Polidea > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=jl0naplQ--zX_Ab1cCycs6qLUgEt6qjWRUHGSONmZ80&e=> > | > Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.polidea.com_&d=DwMFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=9sHnXRUkfGz7tZzoTnwwsCnQBT7JEWS9x3yvHWFZpSo&m=e4s3K-7Xq43WmxGqVFqNU5jQ-GPBbvbR7Shp6MWghSk&s=jl0naplQ--zX_Ab1cCycs6qLUgEt6qjWRUHGSONmZ80&e=> > > > >
