I would like to move transfer operators to the new package - transfer airflow.providers.*.transfer.* airflow.providers.*.operator.* airflow.providers.*.sensor.*
Transfer operators are very often separated in the documentation and it would be helpful if this division was included in the code. https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html On Fri, May 29, 2020, 21:27 Xinbin Huang <bin.huan...@gmail.com> wrote: > I also vote [1]: S3ToHiveOperator > > Since there is no such concept on Airflow about transfer operators, we > better not introduce extra complexity with a potential generic `Transfer` > entity. > > Here are some personal thoughts: > - there are only `import *.operators.*` and `*.sensors.*` in the codebase > - `Transfer` is just a subclass of `.operators` > - the implementation of a transfer is rather similar to an operator - do > something in the `.execute` as opposed to returning True/False in the > `.poke` > > Another thing worth mentioning is that I saw that from different places > there was a misconception that there are 3 types of operators: Operators, > Sensors, and Transfers. > So whatever the final decision is, we need to communicate this more clearly > in the doc https://airflow.apache.org/docs/stable/concepts.html#operators > > On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dpstand...@gmail.com> > wrote: > > > I also vote [1]: S3ToHiveOperator > > Transfer is redundant. > > And why not 3 --- as of now there is no such distinction and I am not > > convinced it is justified. As of now, this is just another operator. > > Probably most operators do some kind of "transferring" of data and trying > > to decide what is transfer and what is not at this time would be rather > > squishy. > > > > On Fri, May 29, 2020 at 9:11 AM Samuel Tu <samue...@gmail.com> wrote: > > > > > I am in favor of [2] as some operators can handle operations such as > > > branching. > > > > > > Thanks > > > Sam > > > > > > > On May 29, 2020, at 9:00 AM, Jarek Potiuk <jarek.pot...@polidea.com> > > > wrote: > > > > > > > > Hello everyone > > > > > > > > We had a discussion in Slack which turned out that we have yet > another > > > > opportunity to name the operators/hooks a bit more consistently. > Seems > > > that > > > > we did not have any rule on how to name Transfer operators and we > have > > > > different conventions already. > > > > > > > > Explanation those are the two examples of conventions we have for > > > transfer > > > > operators: > > > > > > > > [1] *S3ToHiveOperator* > > > > [2] *S3ToHiveTransferOperator* > > > > [3] *S3ToHiveTransfer* > > > > [4] We do not care about consistency > > > > > > > > Some initial comments that I gathered from the discussions:: > > > > > > > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant > > > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to > > see > > > > that it's an Operator, but you get the Transfer purpose as well. > > > sometimes > > > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from > > the > > > > acronyms. > > > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with > > > "Operator" > > > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor", > > "Operator", > > > > so maybe "Transfer" should be another "entity" and in the future, we > > > might > > > > implement a more generic Transfer approach > > > > > > > > I will let the discussion run till the end of today and cast a formal > > > vote > > > > afterwards > > > > > > > > I do not yet cancel Backport RC3, because I am not sure if this is > > > > something we might want to do - maybe after discussion we decide we > > leave > > > > the status quo. > > > > > > > > Discussion in slack here: > > > > > > > > > > https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943 > > > > > > > > > > > > J. > > > > > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > >
