But your underlying point remains that it is hard/impossible for everyone to notice all changes (even those of us fortunate enough to be working on Airflow fulltime!) so nothing is every going to perfect. But we can get better -- but I am wary of wanting to avoid "design by committee" (which I've never seen end well)
And a general point about UI changes: <https://xkcd.com/1172/> :D -ashOn Thu, May 5 2022 at 02:59:03 -0700, Kevin Yang <yrql...@gmail.com> wrote:
Thanks Jarek, it's very good learning for me hearing very different feelings about things in Airflow. Our divergence may come a lot from that. We don't feel that strongly against the tree view (or we just got used to it). This may be because of how we use Airflow--we still use Airflow primarily for the traditional relative static offline data pipelinesI personally am not 100% convinced that we can bring it to absolute par with the tree view like how we fix a bug given their foundational differences. (But I really hope so) I agree that sometimes we need to shape our user needs/habits (thanks to Apple for removing my AUX jack 🙂). And taking more ambitious moves with good intention should be warranted in appropriate cases. It just wasn't justified as one of the cases in my book, but if the majority of the community agrees differently then I shall comply.Similarly, I think we differ in how we view the trade off between gains from Dynamic Task Group and the tree view. For us, it would unlikely be a slight delay/effort to migrate and I didn't notice overwhelmingly positive responses through my lens. This can totally be me missing information from the community and I shall comply with our collective decision.About when/where to raise concerns, agree we can do better. I didn't keep a good track of what will be included in each release and missed the voting thread and window--I should've asked someone from our team to do it when I was on vacation. I did realize it will be an replacement, quoting my last comment in the PR:But with the current implementation (+ optional partial graph view), I'm not convinced that it should be a replacement of the existing tree view, but rather a nice additional table view in our collections to better serve some use cases, e.g. task group heavy DAGs.And then I had the impression that Brent will show us some updates addressing the concern when they are completed--then I lost track of the discussion. I agree with you, I should've raised my concern here in the dev mailing list earlier given how I consider it as an impactful change--I will do that going forward.Though to be honest, we are moving quite fast and I cannot guarantee I'll always be able to keep track of the changes or seize the 72 hrs windows and finish all evaluation/testing/benchmarking/etc. Some trial/pilot/baking/grace period where I get more time to react would be much appreciated, in this case maybe a co-existing period for the views and a deprecation warning. Staying on an earlier release is indeed a valid alternative solution, though that might mean a bigger upgrade later, missed opportunity to sell grid view to our users while they still have tree view to fall back to and later access to other fixes/nice features we added while catching up.And yes, let's continue our github discussion on more specific use cases.Cheers, Kevin YOn Wed, May 4, 2022 at 5:12 AM Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote:> Replacing one of the most used interfaces is a pretty big deal. I like the grid view but it is not a superset of tree view, thus user workflows can break with the change. Yes we can argue that people can update their workflow and get used to it. But I'm not quite sure "we changed it to this, get used to it" is always the best approach we take to make Airflow a success when it's not strictly an improvement.My opinion: Let's work on improving it to be better and respond to the needs.I think not everyone has to move to 2.3.0 immediately and we have an opportunity to bring it to absolute "par" with the tree-view (minus all the oddities of the TreeView which we all knew about). I consider the current 2.3.0 version a little "buggy" Tree view replacement. "Buggy" in the sense it does not respond well to all the needs. and we will "fix" those needs in the coming 2.3.* releases. Yes, that's a little over-interpretation of "bug", I know. But this is how I think about it because the TreeView had so little sense in general, that we had no idea what and how people used some of its properties and we could not make a viable replacement without repeating the nonsense behaviour. And yes. Removing Grid View will make our users more eager to actually report problems they see with the new Grid View, knowing they have no other option. And yes - this is a little "manipulation" with the users of ours, but one that has very good intentions. And I understand users will complain. Tough. We should bite the bullet for a good cause. You just cannot make everyone happy when you build a product, sometimes you have to "shape" your user needs as well, "forcing" the users to adapt if previous choices made no sense.There is not much we can do now for the TreeView, we can't bring it back without huge effort of making it compatible with Dynamic Task Mapping (which is another reason why it's good we dropped it). I'd say it was a great decision to drop it in this context, because if we did not, the highly demanded dynamic task mapping would have taken a lot longer to deliver (and it already took an enormous effort). So I personally absolutely support this decision. No slightest doubt about it. If I consider "slight" delays in migration of people/companies who need to adjust their workflows, compared to overwhelmingly positive response to the new Dynamic Task mapping and opportunities it opens, this is quite incomparable. And for me - clear winner is - yes let's take the "backlash" work on improving the UI based on the feedback, but we should not let the old view which had plenty of problems hold us back. Yep. Some "older users" will have to bite the bullet, but if we work with them and respond to their needs BEFORE they get to actually migrate to 2.3 line, we are good. And that should be our goal.I think those users can still move to 2.2.5 while we adjust grid view to better respond to the needs of those users. Anyhow - the main reason to migrate to 2.3 will be to use Dynamic Task Mapping. And if you don't want to use dynamic task mapping now, staying on 2.2.5 until we catch-up, is a good idea. I think trying to keep the Tree View working with all the new changes would be next to impossible.However, I think indeed we probably missed the discussions and PRs and I think what we actually missed was an earlier discussion on it here in the devlist. On the other hand the discussion mostly happened here and in <https://github.com/apache/airflow/pull/18675> with a lot of arguments and it was clear from the beginning that the solution is to "replace" Tree View with the "New Tree view" which was later renamed to "Grid View". Anyone who was discussing it, could have raised the doubts here before (I personally did not think there was a case for that as I consider UI as mostly an "add-on" to the actual scheduler "core". It's a little late to raise the concerns on the devlist now, I think.I see you commented on it Kevin so I would love to understand - did you not realise it was going to be a replacement (I took part and I knew perfectly well it was going to be replacement)? Was it not clear? It's interesting to see why - if there were concerns - no-one brought it before? I'd love to understand so that we can avoid any problems like that in the future. Can you share your view on it?Maybe it was not entirely clear that replacing is going to happen. Maybe the authors and people who drove the idea could have raised it here anticipating it needs a more detailed devlist discussion. Maybe people who had concerns about it should do that if the original authors did not realise it's "really important to discuss here". Maybe that's a sign that we should pay more attention to UI changes in the future and always make sure to post such changes here. Let's focus on what we can do better.I think we should focus on answering this question and making the new Tree View respond better to the needs of the users now. And I really hope those users will be (they already are) helping in explaining how they found the Tree View missing and what they miss in the Grid View.From the current discussion it's really clear that two things are missing: <https://github.com/apache/airflow/discussions/23413>:* expanding subtrees of groups * differentiating task groups from tasks* ability to quickly see/navigate immediate downstream/possibly upstream dependencies of a task you are focused onI'd really love to see more users' comments on those uses, and raise concrete "I want to be able to do this" rather than "I want my Tree View back".J.On Wed, May 4, 2022 at 6:12 AM Kevin Yang <yrql...@gmail.com <mailto:yrql...@gmail.com>> wrote:Thanks for the pointer. I'll move the comments from the PR to the github discussion and continue usage/technical discussion there. In the meantime, I'd like to discuss a bit on what is the best way to release such change.Replacing one of the most used interfaces is a pretty big deal. I like the grid view but it is not a superset of tree view, thus user workflows can break with the change. Yes we can argue that people can update their workflow and get used to it. But I'm not quite sure "we changed it to this, get used to it" is always the best approach we take to make Airflow a success when it's not strictly an improvement.I do understand that we want to move fast. But as an established project widely used by many companies in production to power their business, we can maybe at the same time give more buffer for such change and incorporate more feedback. Like how we do deprecation, give people some time to react. People would naturally move to it if they find themselves much more productive using it.DAG versioning is very nice and I want it badly too. However I see it depends on the grid view but not the deprecation of tree view--we can always claim that tree view does not support versioning, another soft push for people to use grid view 😁Hope I'm making sense, looking forward to hearing your thoughts. Cheers, Kevin YOn Tue, May 3, 2022 at 12:19 PM Brent Bovenzi <br...@astronomer.io.invalid> wrote:Seconding what Jarek said. Please comment in that github discussion. I would very much like to get feedback and then iterate the UI much faster than before.On Tue, May 3, 2022 at 3:14 PM Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote:I personally don't think so.People will have to get used to it. It's far superior. And backlash is expected.I would rather read and focus on what's there to make it respond to the needs of users than have it in parallel with Tree view. Discussion already started here <https://github.com/apache/airflow/discussions/23413> and I'd say we should do everything to figure out what people REALLY miss in the Grid View comparing to Tree view and iterate on it and add it (And from earlier discussions with Brent, I think this is the plan).The Tree view held us back like crazy.The biggest problem with Tree view was that DAG versioning effort was (I believe) held back by it because it was next to impossible to think of "versioning" when Tree view was there.This is a bold move - but very much needed and even overdue I think.J.On Tue, May 3, 2022 at 9:05 PM Kevin Yang <yrql...@gmail.com <mailto:yrql...@gmail.com>> wrote:Hey guys sorry I saw this thread late--was on vacation last week.I noticed that we made a bold move replacing the tree view with the new grid view. I like the new grid view, but given the open discussion in this PR <https://github.com/apache/airflow/pull/18675> and popularity of tree view (in our use case it's visited 13x times more than graph view), should we consider making it an *addition* for now rather than *replacement*?On Sat, Apr 30, 2022 at 9:48 AM Kaxil Naik <kaxiln...@gmail.com <mailto:kaxiln...@gmail.com>> wrote:+1 (binding) - Big milestone since 2.0On Sat, 30 Apr 2022 at 09:24, Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote:Fanatic -> fantastic work... Seems like some drag&drop issue with my Chrome :)JOn Sat, Apr 30, 2022 at 10:21 AM Jarek Potiuk <ja...@potiuk.com <mailto:ja...@potiuk.com>> wrote:>> +1 (binding). Checked signatures, licences, checksums. All looks good.>> There is a TON of cool stuff in this release. I am confident it's > ready to get to the hands of our users after all the fanatic plenty of > people put in it. Just two things that I think deserve highlighting:>> * There is the obvious Dynamic Task Mapping (which is the highlight of > it - Ash and TP particularly - but all the other people who tested it > have spent countless hours on it and it is life- and future- changing> for Airflow).> * I particularly like the new Grid View. It will take people a bit of > getting used to the new screen - so this is a bit of a bold move and > there will be backlash I am sure. But I am quite confident it is soooo > much better and makes Airflow 2.3 finally showing up the modern UI> approach as well (After all the internal modernization) .> Brent and the team - big Kudos for all the work there and the boldness> in rethinking this one from grounds-up :). > > J. > > > On Fri, Apr 29, 2022 at 10:28 PM Vikram Koka > <vik...@astronomer.io.invalid> wrote: > > > > +1 (non-binding) > > > > Dynamic Task Mapping is a huge improvement! > >> > On Fri, Apr 29, 2022 at 11:34 AM Josh Fell <josh.d.f...@astronomer.io <mailto:josh.d.f...@astronomer.io>.invalid> wrote:> >> > >> +1 (non-binding) > >> > >> Dynamic Task Mapping feels life-changing. > >>> >> On Fri, Apr 29, 2022 at 12:41 PM Abhishek Bhakat <abhishek.bha...@astronomer.io.invalid> wrote:> >>>> >>> Other than that issue, have tested the version and would like to change my vote to +1 (non-binding)> >>>> >>> On Fri, Apr 29, 2022 at 9:49 PM Elad Kalif <elad...@apache.org <mailto:elad...@apache.org>> wrote:> >>>> > >>>> +1 (binding) > >>>>> >>>> On Fri, Apr 29, 2022 at 6:34 PM Dennis Akpenyi <dennisakpe...@gmail.com <mailto:dennisakpe...@gmail.com>> wrote:> >>>>> > >>>>> +1 (non-binding) > >>>>>> >>>>> On Fri 29. Apr 2022 at 17:30, Brent Bovenzi <br...@astronomer.io.invalid> wrote:> >>>>>> > >>>>>> +1 (non-binding) > >>>>>>> >>>>>> On Fri, Apr 29, 2022 at 11:13 AM Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> wrote:> >>>>>>> > >>>>>>> No, the same issue is fine. > >>>>>>>> >>>>>>> My point is that since this is no worse in 2.3.0rc2 than it was in 2.2.5 it shouldn't stop the release of 2.3.0.> >>>>>>> > >>>>>>> Cheers, > >>>>>>> Ash > >>>>>>>> >>>>>>> On Fri, Apr 29 2022 at 20:28:49 +0530, Abhishek Bhakat <abhishek.bha...@astronomer.io.INVALID> wrote:> >>>>>>>> >>>>>>> I am facing this issue with 2.3.0rc2. I did make a comment on the same issue. Shall I create another issue for 2.3.0rc2 ?> >>>>>>>> >>>>>>> On Fri, Apr 29, 2022 at 8:20 PM Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> wrote:> >>>>>>>> > >>>>>>>> Hi Abhishek, > >>>>>>>>> >>>>>>>> The issue you have linked to says the version that happens is in 2.2.5, so since this isn't a regression in 2.3 I'd ask if you could remove your -1 vote?> >>>>>>>>> >>>>>>>> On Fri, Apr 29 2022 at 20:13:31 +0530, Abhishek Bhakat <abhishek.bha...@astronomer.io.INVALID> wrote:> >>>>>>>> > >>>>>>>> -1 (non-binding)> >>>>>>>> An issue with parsing dynamic tasks, causes the scheduler to crash.> >>>>>>>> <https://github.com/apache/airflow/issues/23361> > >>>>>>>>> >>>>>>>> On Fri, Apr 29, 2022 at 7:58 PM Collin McNulty <col...@astronomer.io.invalid> wrote:> >>>>>>>>> > >>>>>>>>> +1 (non-binding) > >>>>>>>>> > >>>>>>>>> Made and ran several test DAGs using mapped tasks. > >>>>>>>>>> >>>>>>>>> On Fri, Apr 29, 2022 at 3:54 AM Ash Berlin-Taylor <a...@apache.org <mailto:a...@apache.org>> wrote:> >>>>>>>>>> > >>>>>>>>>> +1 binding. > >>>>>>>>>>> >>>>>>>>>> On Wed, Apr 27 2022 at 21:50:48 +0100, Ephraim Anierobi <ephraimanier...@apache.org <mailto:ephraimanier...@apache.org>> wrote:> >>>>>>>>>> > >>>>>>>>>> Hey fellow Airflowers, > >>>>>>>>>>> >>>>>>>>>> I have cut Airflow 2.3.0rc2. This email is calling a vote on the release, > >>>>>>>>>> which will last for 72 hours, from Wednesday, April 27, 2022, at 08:48 pm UTC > >>>>>>>>>> until Saturday, April 30, 2022, at 08:48 pm UTC, and until 3 binding +1 votes have been received.> >>>>>>>>>> > >>>>>>>>>> Consider this my (binding) +1. > >>>>>>>>>> > >>>>>>>>>> Airflow 2.3.0rc2 is available at:> >>>>>>>>>> <https://dist.apache.org/repos/dist/dev/airflow/2.3.0rc2/>> >>>>>>>>>>> >>>>>>>>>> *apache-airflow-2.3.0-source.tar.gz* is a source release that comes with INSTALL instructions. > >>>>>>>>>> *apache-airflow-2.3.0.tar.gz* is the binary Python "sdist" release. > >>>>>>>>>> *apache_airflow-2.3.0-py3-none-any.whl* is the binary Python wheel "binary" release.> >>>>>>>>>> > >>>>>>>>>> Public keys are available at:> >>>>>>>>>> <https://dist.apache.org/repos/dist/release/airflow/KEYS>> >>>>>>>>>> > >>>>>>>>>> Please vote accordingly: > >>>>>>>>>> > >>>>>>>>>> [ ] +1 approve > >>>>>>>>>> [ ] +0 no opinion > >>>>>>>>>> [ ] -1 disapprove with the reason > >>>>>>>>>>> >>>>>>>>>> Only votes from PMC members are binding, but all members of the community > >>>>>>>>>> are encouraged to test the release and vote with "(non-binding)".> >>>>>>>>>>> >>>>>>>>>> The test procedure for PMCs and Contributors who would like to test this RC are described in > >>>>>>>>>> https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md\#verify-the-release-candidate-by-pmcs <https://github.com/apache/airflow/blob/main/dev/README_RELEASE_AIRFLOW.md%5C#verify-the-release-candidate-by-pmcs>> >>>>>>>>>>> >>>>>>>>>> Please note that the version number excludes the `rcX` string, so it's now > >>>>>>>>>> simply 2.3.0. This will allow us to rename the artifact without modifying> >>>>>>>>>> the artifact checksums when we actually release. > >>>>>>>>>>> >>>>>>>>>> Release Notes: <https://github.com/apache/airflow/blob/2.3.0rc2/RELEASE_NOTES.rst>> >>>>>>>>>> > >>>>>>>>>> New features since 2.2.5: > >>>>>>>>>>> >>>>>>>>>> - Add dynamic task mapping (<https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-42+milestone%3A%22Airflow+2.3.0%22>)> >>>>>>>>>> - New Grid View replaces Tree View (#18675)> >>>>>>>>>> - Templated ``requirements.txt`` in Python Operators (#17349)> >>>>>>>>>> - Allow reuse of decorated tasks (#22941)> >>>>>>>>>> - Move the database configuration to a new section (#22284)> >>>>>>>>>> - Add ``SmoothOperator`` (#22813)> >>>>>>>>>> - Make operator's ``execution_timeout`` configurable (#22389)> >>>>>>>>>> - Events Timetable (#22332)> >>>>>>>>>> - Support dag serialization with custom ``ti_deps`` rules (#22698)> >>>>>>>>>> - Support log download in task log view (#22804)> >>>>>>>>>> - support for continue backfill on failures (#22697)> >>>>>>>>>> - Add ``dag-processor`` cli command (#22305)> >>>>>>>>>> - Add possibility to create users in LDAP mode (#22619) > >>>>>>>>>> - Add ``ignore_first_depends_on_past`` for scheduled jobs (#22491) > >>>>>>>>>> - Update base sensor operator to support XCOM return value (#20656) > >>>>>>>>>> - Add an option for run id in the ui trigger screen (#21851) > >>>>>>>>>> - Enable JSON serialization for connections (#19857) > >>>>>>>>>> - Add REST API endpoint for bulk update of DAGs (#19758) > >>>>>>>>>> - Add queue button to click-on-DagRun interface. (#21555) > >>>>>>>>>> - Add ``list-import-errors`` to ``airflow dags`` command (#22084) > >>>>>>>>>> - Store callbacks in database if ``standalone_dag_processor`` config is True. (#21731)> >>>>>>>>>> - Add LocalKubernetesExecutor (#19729)> >>>>>>>>>> - Add ``celery.task_timeout_error`` metric (#21602)> >>>>>>>>>> - Airflow ``db downgrade`` cli command (#21596) > >>>>>>>>>> - Add ``ALL_SKIPPED`` trigger rule (#21662)> >>>>>>>>>> - Add ``db clean`` CLI command for purging old data (#20838)> >>>>>>>>>> - Add ``celery_logging_level`` (#21506)> >>>>>>>>>> - Support different timeout value for dag file parsing (#21501) > >>>>>>>>>> - Support generating SQL script for upgrades (#20962) > >>>>>>>>>> - Add option to compress Serialized dag data (#21332)> >>>>>>>>>> - Branch python operator decorator (#20860) > >>>>>>>>>> - Add Audit Log View to Dag View (#20733)> >>>>>>>>>> - Add missing StatsD metric for failing SLA Callback notification (#20924) > >>>>>>>>>> - Add ``ShortCircuitOperator`` configurability for respecting downstream trigger rules (#20044) > >>>>>>>>>> - Allow using Markup in page title in Webserver (#20888) > >>>>>>>>>> - Add Listener Plugin API that tracks TaskInstance state changes (#20443) > >>>>>>>>>> - Add context var hook to inject more env vars (#20361) > >>>>>>>>>> - Add a button to set all tasks to skipped (#20455)> >>>>>>>>>> - Cleanup pending pods (#20438)> >>>>>>>>>> - Add config to warn public deployment exposure in UI (#18557)> >>>>>>>>>> - Log filename template records (#20165) > >>>>>>>>>> - Added windows extensions (#16110)> >>>>>>>>>> - Showing approximate time until next dag_run in Airflow (#20273)> >>>>>>>>>> - Extend config window on UI (#20052)> >>>>>>>>>> - Add show dag dependencies feature to CLI (#19985) > >>>>>>>>>> - Add cli command for 'airflow dags reserialize` (#19471) > >>>>>>>>>> - Add missing description field to Pool schema(REST API) (#19841) > >>>>>>>>>> - Introduce DagRun action to change state to queued. (#19353)> >>>>>>>>>> - Add DAG run details page (#19705) > >>>>>>>>>> - Add role export/import to cli tools (#18916)> >>>>>>>>>> - Adding ``dag_id_pattern`` parameter to the ``/dags`` endpoint (#18924)> >>>>>>>>>> > >>>>>>>>>> Thanks > >>>>>>>>>> Ephraim > >>>>> > >>>>> --> >>>>> Dr. Dennis Akpenyi, Airflow Core Developer, Astronomer Inc.
