Hello,
See this is the code:
from airflow import DAG
from datetime import datetime,timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
So this code use simplehttpopeator(
https://airflow.incubator.apache.org/code.html) which is used to create the
cluster or post the information on this url(qa.qubole.net/api) this is same
as requests.post() in python.
As you can see i am using passwd here
> endpoint='/v2/clusters?auth_token=%s' % (passwd),
and i am making many dags like that which are connected and they all use
the password(auth_token) to send the data so i want to pass passwd as
command line argument.
Argument parser is not working here
i tried argument parser and also sys.argv but it doesn't work.
from airflow import DAG
from airflow.operators import SimpleHttpOperator
from datetime import datetime,timedelta
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument("passwd")
args = parser.parse_args()
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
print args.passwd
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
file=os.path.join(os.path.dirname(os.path.abspath(__file__)),"check.json")
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (args.passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': args.passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
if i use this its not able to trigger the dag.
for triggering the dag we use
airflow trigger_dag dagid -r runid
argumentparser work in python
python script.py --passwd passwd #this work
but this is not a python script
airflow trigger_dag dagid -r runid --passwd passwd
sorry for my english.
On Sun, May 28, 2017 at 9:17 PM, shubham goyal <skgoyal...@gmail.com> wrote:
> Hello all,
>
>
> You are not understanding my point I understand command line argument
> ,sys.argv[],argparser to pass arguments via command line but I want to do
> these things when I trigger the dag
> And how I trigger the dag
> airflow trigger_dag dagname -s runid
>
> This triggering of dag doesn't support to pass command line argument but I
> want to pass authentication token as command line argument when I trigger
> the dag(you can see in airflow docs) so I am asking is there a way to do it.
>
> I tried argument parser but when I use that it's not able to trigger the
> dag.
>
> I want something like this (in cli)
>
> airflow trigger_dag dagname -r runid --passwd auth_token
>
> And want to access this authtoken in Python script.
>
>
> On May 28, 2017 8:18 PM, "Francois Dion" <francois.d...@gmail.com> wrote:
>
>> My mailbox if full of similar stories: companies dumping airflow on their
>> ETL (or similar) group. Those who knew Python succeeded, those who didn't
>> failed, and some even moved to other companies because they couldn't cope
>> with all this complexity dumped on them all at once.
>>
>> Moral of the story, it is best to learn python first, become good at it,
>> then get into specific tools.
>>
>> And, of course, this is not specific to a relatively straightforward
>> module
>> / application like airflow. I've seen the same with scikit-learn. Same
>> also
>> in other languages. And to make matters worse, the curricula (and books)
>> always starts with a crash course in a specific language, then go on and
>> on
>> about the application, perpetuating the myth that learning the programming
>> language well is totally unimportant.
>>
>> Francois
>>
>> On Sun, May 28, 2017 at 9:46 AM, Alan Gauld via Tutor <tutor@python.org>
>> wrote:
>>
>> > On 28/05/17 04:37, shubham goyal wrote:
>> > > Does anybody have answer?
>> >
>> > You received two answers, both of which asked
>> > you to try something and get back to us for more
>> > information. Did you try printing sys.argv?
>> > What was the result?
>> >
>> > And did you try Peter's argparse code?
>> >
>> > You still haven't explained what your dag is?
>> > What library are you using? It may treat args
>> > in a non standard way, but we can't tell if
>> > you don't give us the information we need to
>> > help you.
>> >
>> > --
>> > Alan G
>> > Author of the Learn to Program web site
>> > http://www.alan-g.me.uk/
>> > http://www.amazon.com/author/alan_gauld
>> > Follow my photo-blog on Flickr at:
>> > http://www.flickr.com/photos/alangauldphotos
>> >
>> >
>> > _______________________________________________
>> > Tutor maillist - Tutor@python.org
>> > To unsubscribe or change subscription options:
>> > https://mail.python.org/mailman/listinfo/tutor
>> >
>>
>>
>>
>> --
>> raspberry-python.blogspot.com - www.pyptug.org - www.3DFutureTech.info -
>> @f_dion
>> _______________________________________________
>> Tutor maillist - Tutor@python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>
_______________________________________________
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
