I’m not convinced that this PR should actually go in. It affect task idempotency in my perspective and should have been given more thought. Now the Xcom values and task runtimes can be out of sync (e.g. imagine - simplified - if it pushed “success” to xcom, but after clearing the task failed). What is your view on this @Fokko?
On 26 October 2019 at 00:21:03, Driesprong, Fokko (fo...@driesprong.frl) wrote: It hasn't been targeted for 1.10.6. If you want this in, you need to convince the release manager Ash :-) Cheers, Fokko Op vr 25 okt. 2019 om 22:46 schreef Bas Harenslak < basharens...@godatadriven.com>: > Hi Damian, > > Currently XComs cannot be shared between instances of the same task (i.e. > retries/sensors). This is because XComs of a task instance are cleared when > rerunning the same task instance. Don’t know why that choice was made, but > this PR: https://github.com/apache/airflow/pull/6370 was recently merged > to master which fixes that problem, I don’t know what version it will be > released in though. > > Cheers, > Bas > > On 25 Oct 2019, at 18:08, Shaw, Damian P. <damian.sha...@credit-suisse.com > <mailto:damian.sha...@credit-suisse.com>> wrote: > > Hi all, > > I am trying to pull the latest xcom value that could include the current > task from a previous retry. In the UI I can see today's xcom value and > yesterday's xcom value. I am retrying the task and unable to get today's > value: > > Attempt 1: context['ti'].xcom_pull(key=key, include_prior_dates=True) -> > Yesterday's xcom value > Attempt 2: context['ti'].xcom_pull(key=key) -> None > Attempt 3: XCom.get_many(key=key, > execution_date=context['execution_date'] + timedelta(days=1), > include_prior_dates=True) -> List containing yesterday's xcom object > > It occurred to me at this point that maybe what I'm trying isn't possible > with xcom, does anyone know? > > (FYI I'm on Airflow 1.10.3 but I've looked at the source code between this > and master and seen no significant difference.) > Thanks, > Damian > > > > =============================================================================== > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > =============================================================================== > >
