shubham goyal wrote: > He guys, > > I want to ask that can we pass the parameters as commandline arguments in > airflow when we are triggering the dag and access them inside the dag's > python script/file. > script: > like this here i am trying to create a cluster but i need to pass password > as cli arguments when i trigger the dag. can we do that. please help.
Have a look at argparse: <https://docs.python.org/3.6/library/argparse.html> Provided (1) you are OK with the password being echoed when you type it (2) the password doesn't require escaping to be a valid part of the `endpoint` argument the code might look like this: > from airflow import DAG > from datetime import datetime,timedelta import argparse parser = argparse.ArgumentParser() parser.add_argument("passwd") args = parser.parse_args() > default_args = { > 'owner': 'airflow', > 'depends_on_past': False, > 'start_date': datetime.now(), > 'email': ['airf...@airflow.com'], > 'email_on_failure': False, > 'email_on_retry': False > } > MAIN_DAG='check_dag' > dag = DAG(dag_id=MAIN_DAG, default_args=default_args, > schedule_interval=None) > > with open(file, "r") as f: > payload = f.read() # Reading the json data from a file > SimpleHttpOperator( # creating cluster using SimpleHttpOperator > task_id='cluster_create', > method='POST', > http_conn_id='qubole_default', > # for directing to https://qa.qubole.net/api endpoint='/v2/clusters?auth_token=%s' % args.passwd, > data=payload, > headers={"Content-Type": "application/json"}, params={'auth_token': args.passwd}, > response_check=lambda response: True if response.status_code == > 200 > else False, > dag=dag > ) > To address (1) there is <https://docs.python.org/3.6/library/getpass.html> (2) can be solved with <https://docs.python.org/3.6/library/urllib.parse.html> _______________________________________________ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
