[
https://issues.apache.org/jira/browse/AIRFLOW-3415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuri Bendana updated AIRFLOW-3415:
----------------------------------
Description:
When triggering dagruns in a loop, the imported references become null on the
second iteration. Here is an example [ gist |
[https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ]. For the
purposes here, you can ignore the DagRunSensor task. On the first iteration the
'sleeper' dag gets triggered but on the second iteration I see a
{noformat}
TypeError: 'NoneType' object is not callable{noformat}
To workaround this, I have to copy the import (in this case trigger_dag) inside
the loop.
Here is the stacktrace:
{code:java}
[2018-11-28 18:23:42,492] {{models.py:1789}} INFO - All retries failed; marking
task as FAILED
[2018-11-28 18:23:42,517] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 Traceback (most recent call last):
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File "/opt/venv/sgmo/bin/airflow", line 32, in <module>
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 args.func(args)
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in
wrapper
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 return f(*args, **kwargs)
[2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 490, in
run
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 _run(args, dag, ti)
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 406, in
_run
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 pool=args.pool,
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in
wrapper
[2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 return func(*args, **kwargs)
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/models.py", line 1659, in
_run_raw_task
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 result = task_copy.execute(context=context)
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py",
line 95, in execute
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 return_value = self.execute_callable()
[2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File
"/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py",
line 100, in execute_callable
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 return self.python_callable(*self.op_args, **self.op_kwargs)
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 File "/opt/airflow/dags/sleepersensor.py", line 11, in
triggersleeper
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 tdr = trigger_dag('sleeper')
[2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
Subtask t1 TypeError: 'NoneType' object is not callable
{code}
was:
When triggering dagruns in a loop, the imported references become null on the
second iteration. Here is an example [ gist |
[https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ]. For the
purposes here, you can ignore the DagRunSensor task. On the first iteration the
'sleeper' dag gets triggered but on the second iteration I see a
{noformat}
TypeError: 'NoneType' object is not callable{noformat}
To workaround this, I have to copy the import (in this case trigger_dag) inside
the loop.
> Imports become null when triggering dagruns in a loop
> -----------------------------------------------------
>
> Key: AIRFLOW-3415
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3415
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
> Affects Versions: 1.10.1
> Environment: CentOS 7
> Reporter: Yuri Bendana
> Priority: Minor
> Attachments: triggerdag_stacktrace.txt
>
>
> When triggering dagruns in a loop, the imported references become null on the
> second iteration. Here is an example [ gist |
> [https://gist.github.com/ybendana/3bc0791fe00b099be04aca47a8d524c9] ]. For
> the purposes here, you can ignore the DagRunSensor task. On the first
> iteration the 'sleeper' dag gets triggered but on the second iteration I see a
> {noformat}
> TypeError: 'NoneType' object is not callable{noformat}
> To workaround this, I have to copy the import (in this case trigger_dag)
> inside the loop.
> Here is the stacktrace:
> {code:java}
> [2018-11-28 18:23:42,492] {{models.py:1789}} INFO - All retries failed;
> marking task as FAILED
> [2018-11-28 18:23:42,517] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 Traceback (most recent call last):
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File "/opt/venv/sgmo/bin/airflow", line 32, in <module>
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 args.func(args)
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/cli.py", line 74,
> in wrapper
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 return f(*args, **kwargs)
> [2018-11-28 18:23:42,518] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 490, in
> run
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 _run(args, dag, ti)
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/bin/cli.py", line 406, in
> _run
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 pool=args.pool,
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/utils/db.py", line 74, in
> wrapper
> [2018-11-28 18:23:42,519] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 return func(*args, **kwargs)
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/models.py", line 1659, in
> _run_raw_task
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 result = task_copy.execute(context=context)
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py",
> line 95, in execute
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 return_value = self.execute_callable()
> [2018-11-28 18:23:42,520] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File
> "/opt/venv/sgmo/lib/python2.7/site-packages/airflow/operators/python_operator.py",
> line 100, in execute_callable
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 return self.python_callable(*self.op_args, **self.op_kwargs)
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 File "/opt/airflow/dags/sleepersensor.py", line 11, in
> triggersleeper
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 tdr = trigger_dag('sleeper')
> [2018-11-28 18:23:42,521] {{base_task_runner.py:101}} INFO - Job 1354170:
> Subtask t1 TypeError: 'NoneType' object is not callable
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
