Yes, that makes sense. On Fri, Nov 8, 2019 at 3:22 PM Kamil Breguła <kamil.breg...@polidea.com> wrote:
> In the case of Hadoop, it is published by Apache, so it can be in the > apache directory. This will mimic the grouping presented in the > documentation. > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#software-operators-and-hooks > > On Fri, Nov 8, 2019 at 3:47 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > I think we should keep the vote open at least until mid next week to have > > more thought and inputs on this one. > > > > In general, I am happy with the approach but operators/hooks and sensors > > shouldn't be a provider. "hadoop" can be its provider and hdfs can be a > > part of it. > > > > providers/ > > google > > cloud > > operators > > hooks > > sensors > > gsuite > > operators > > ... > > amazon > > aws > > operators > > ... > > microsoft > > azure > > operators > > ... > > hadoop > > hdfs > > operators > > ... > > > > We can also define what is a "provider" so we know what to add in it in > the > > future. SSH/FTP/SFTP belongs to the same family group. Do we want to have > > separate providers for each one of them ??? > > > > Regards, > > Kaxil > > > > On Fri, Nov 8, 2019 at 9:08 AM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > I really like to make everything a provider. That's a great idea ! > This way > > > everything "backportable" will have to be in "providers" package. > Really > > > nice and clean separation (and less mess in "airflow"). And we will not > > > have to have any artificial grouping (we can still group them at the > > > documentation level). > > > > > > We do not need backport in name. And I think it's more of technical > detail > > > on naming the package which we can work out while reviewing PRs and we > can > > > agree final naming of the released packaged on PMC level (PMCs will > have to > > > vote on releasing those). > > > > > > The thinking is that it's intention is really to be only backported to > 1.10 > > > - we are not going (yet) to use the packages in Airflow 2.*. so I > thought > > > by naming them backport we can express that intent more clearly. > > > > > > So let me clarify the structure of folders we are going to have if we > > > follow it (i just added some examples) including the already agreed > changes > > > from AIP-21: > > > > > > providers/ > > > google > > > cloud > > > operators > > > hooks > > > sensors > > > gsuite > > > operators > > > ... > > > amazon > > > aws > > > operators > > > ... > > > microsoft > > > azure > > > operators > > > ... > > > operators > > > sqlite.py > > > oracle.py > > > docker.py > > > hooks > > > hdfs.py > > > sqlite.py > > > sensors > > > http.py > > > sql.py > > > > > > > > > J. > > > > > > On Fri, Nov 8, 2019 at 9:43 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > > > Do we need to include `-backport,`? What was the thinking behind > that? > > > > > > > > I think software and protocol should be merged. I would also say > > > > _everything_ is a provider, so airflow.providers.ssh.SSHOperator for > > > > instance is what I would prefer > > > > > > > > -a > > > > > > > > On 8 November 2019 08:32:42 GMT, Jarek Potiuk < > jarek.pot...@polidea.com> > > > > wrote: > > > > >One more day to go. I would love to see some opinions on this AIP-21 > > > > >update > > > > >:). > > > > > > > > > >Executive summary: > > > > > > > > > >* we will be moving a number of integrations to sub-packages of > > > > >airflow. > > > > >* they will be backportable to 1.10.*. There will be > > > > >'apache-airflow-[package]-backport' pypi installable with python 3 > that > > > > >will make Airflow 2.0 operators/hooks etc. available with 1.10* > > > > >operators. > > > > >* the current proposal for sub-packages is > > > > >"protocols/software/providers/" > > > > >(but if you think merging protocols and software makes sense - > please > > > > >express your opinion > > > > >* we are not moving "fundamental" operators/hooks etc.. > > > > >* Airflow 2.0 is still going to be installed as a single package > with > > > > >all > > > > >operators (so we are not yet implementing AIP-8) > > > > > > > > > >J. > > > > > > > > > >On Wed, Nov 6, 2019 at 10:07 AM Jarek Potiuk < > jarek.pot...@polidea.com> > > > > >wrote: > > > > > > > > > >> I think all this cases are valid but maybe I was not super-clear. > > > > >It's > > > > >> only the transfer operators that we need to decide where to put - > not > > > > >> hooks. > > > > >> Usually the complexity of communication with particular storages > is > > > > >(or at > > > > >> least should be) in the Hooks rather than Operators. > > > > >> > > > > >> Operators should be just thin wrappers over the logic in the > hooks. > > > > >> Hooks are going to stay where they belong - S3 Hooks in amazon, > GCS > > > > >Hooks > > > > >> in google.cloud, GoogleSheet Hooks in google.gsuite. > > > > >> > > > > >> Since we actually have mono-repo - this will be no problem (and no > > > > >cross > > > > >> dependencies problem) to have S3 -> GCS operator in google and > use > > > > >hooks > > > > >> from both google/amazon. > > > > >> > > > > >> I hope this alleviates your concern Daniel ? > > > > >> > > > > >> J. > > > > >> > > > > >> > > > > >>> What about GoogleSheetsToS3? GoogleSheetsToGCS? These you would > > > > >put in > > > > >>> the target, i.e. the storage? But GoogleSheetsToSftp would be in > > > > >google > > > > >>> sheets operators file? The complexity, and the shared code, are > in > > > > >the > > > > >>> gsheet component -- not into the storage destination. > > > > >>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >>> On Tue, Nov 5, 2019 at 5:46 PM Jarek Potiuk > > > > ><jarek.pot...@polidea.com> > > > > >>> wrote: > > > > >>> > > > > >>> > Hello Airflow Community, > > > > >>> > > > > > >>> > The email calls for a vote to update AIP-21 Changes in import > > > > >paths > > > > >>> > < > > > > >>> > > > > > >>> > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > > > > >>> > > > > > > >>> > with > > > > >>> > the changes described below. The vote will last till Saturday > 8th > > > > >2am > > > > >>> CEST > > > > >>> > (72 hours). Committers have a binding vote but everyone from > the > > > > >>> community > > > > >>> > is encouraged to cast an advisory vote. > > > > >>> > > > > > >>> > *Summary*: > > > > >>> > > > > > >>> > The proposal is to update AIP-21 to move all non-core > > > > >>> > operators/hooks/sensor (and related files) to sub-packages > within > > > > >>> airflow > > > > >>> > (protocols/software/providers) or (software/providers). > > > > >>> > I am also happy to merge protocols+software, so if you have a > > > > >strong > > > > >>> > opinion on it - please state it with your vote and we can > decide > > > > >based > > > > >>> on > > > > >>> > majority. > > > > >>> > > > > > >>> > Those packages will be separately released (schedule/process > TBD) > > > > >and > > > > >>> will > > > > >>> > be backportable to 1.10.* airflow series, so that users can > > > > >install it > > > > >>> and > > > > >>> > start using new Airflow2.0 operators in their Python 3 Airflow > > > > >1.10 > > > > >>> > environments (only Python 3.5+ is supported). > > > > >>> > > > > > >>> > We will proceed with migrating the providers package to already > > > > >agreed > > > > >>> > paths without waiting for the final vote (following current > > > > >version of > > > > >>> > AIP-21). Since we have working POC - we know the agreed paths > will > > > > >work > > > > >>> for > > > > >>> > us. > > > > >>> > > > > > >>> > *Previous discussions: * > > > > >>> > > > > > >>> > - > > > > >>> > > > > > >>> > > > > > >>> > > > > > > > > > > > > > https://lists.apache.org/thread.html/b07a93c9114e3d3c55d4ee514955bac79bc012c7a00db627c6b4c55f@%3Cdev.airflow.apache.org%3E > > > > >>> > - > > > > >>> > > > > > >>> > > > > > >>> > > > > > > > > > > > > > https://lists.apache.org/thread.html/e25ddc546e367a4af3e594fecbd4431959bd5a89045e748e4206e7ff@%3Cdev.airflow.apache.org%3E > > > > >>> > > > > > >>> > *More Details*: > > > > >>> > > > > > >>> > 1) Information that we are going in the direction of AIP-8 but > not > > > > >yet > > > > >>> > reaching it - focusing on separating out backportable packages > > > > >>> installable > > > > >>> > in Airflow releases 1.10.* . Airflow 2.0 will still be > installed > > > > >as a > > > > >>> whole > > > > >>> > and all the source will be kept in one repo, but we now have a > way > > > > >to > > > > >>> build > > > > >>> > backportable packages for groups of operators. POC available > here: > > > > >>> > https://github.com/apache/airflow/pull/6507 (based on Ash's > > > > >>> > https://github.com/ashb/airflow-submodule-test) > > > > >>> > > > > > >>> > 2) We move all integrations to new packages (keeping deprecated > > > > >import > > > > >>> > aliases in the old places). The following split (according to > > > > >>> "stewardship" > > > > >>> > over the integrations): > > > > >>> > > > > > >>> > - *fundamentals* - core of ariflow - they are really part of > > > > >Apache > > > > >>> > Airflow. Stewards - core Airflow team. Not > > > > >backportable/separated > > > > >>> out. > > > > >>> > - *protocols* - are not owned by anyone, they are public and > > > > >the > > > > >>> > implementation is fully "open". There are no particular > > > > >stewards (no > > > > >>> > need). > > > > >>> > Users of particular protocols should mainly maintain those > and > > > > >add > > > > >>> > support > > > > >>> > for different versions of the protocols. > > > > >>> > - *software* - both API and software are controlled by > someone > > > > >>> outside > > > > >>> > of Airflow (commercial or open-source project), but the > > > > >deployment of > > > > >>> > that > > > > >>> > software is "owned" by the user installing Airflow. The > > > > >"stewardship" > > > > >>> > might > > > > >>> > be also the users but the controlling party (Oracle for > > > > >example) > > > > >>> might > > > > >>> > be > > > > >>> > interested in maintaining those operators as well. > > > > >>> > - *providers* - API/software/deployments are fully > controlled > > > > >by a > > > > >>> 3rd > > > > >>> > party. Here most likely "provider" will be interested in > > > > >maintaining > > > > >>> the > > > > >>> > operators (and for example like Google - provide integration > > > > >>> guidelines > > > > >>> > < > > > > >>> > > > > > >>> > > > > > > > > > > > > > https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?usp=drive_web&ouid=112320280470690058978 > > > > >>> > > > > > > >>> > for > > > > >>> > their hooks/operators/sensors) > > > > >>> > > > > > >>> > > > > > >>> > 3) Between-providers transfer operators should be kept at the > > > > >"target" > > > > >>> > rather than "source" > > > > >>> > For example S3 -> GCS should be in "google" provider, but > GCS-> S3 > > > > >>> should > > > > >>> > be in "amazon". > > > > >>> > > > > > >>> > 4) One-side provider transfer operators should be kept at the > > > > >"provider" > > > > >>> > regardless if they are target or source. > > > > >>> > For example GCS-> SFTP or SFTP -> GCS should be in "google" > > > > >provider. > > > > >>> > > > > > >>> > 5) If in doubt we will discuss individual cases separately. > > > > >>> > > > > > >>> > J. > > > > >>> > > > > > >>> > -- > > > > >>> > > > > > >>> > Jarek Potiuk > > > > >>> > Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > >>> > > > > > >>> > M: +48 660 796 129 <+48660796129> > > > > >>> > [image: Polidea] <https://www.polidea.com/> > > > > >>> > > > > > >>> > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Jarek Potiuk > > > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > >> > > > > >> M: +48 660 796 129 <+48660796129> > > > > >> [image: Polidea] <https://www.polidea.com/> > > > > >> > > > > >> > > > > > > > > > >-- > > > > > > > > > >Jarek Potiuk > > > > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > >M: +48 660 796 129 <+48660796129> > > > > >[image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > >
