SHARE
TWEET

Untitled

a guest Jun 18th, 2019 49 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. from airflow import DAG
  2.  
  3. from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
  4. from datetime import datetime, timedelta
  5.  
  6.  
  7. args = {
  8.     'owner': 'airflow',
  9.     'start_date': datetime(2019, 5, 22)
  10. }
  11. dag = DAG('spark_example_new', default_args=args, schedule_interval="*/10 * * * *")
  12.  
  13. operator = SparkSubmitOperator(
  14.     task_id='spark_submit_job_from_airflow',
  15.     conn_id='spark_default',
  16.     java_class='org.apache.spark.examples.JavaWordCount',
  17.     application='local:///opt/spark/examples/jars/spark-examples_2.12-2.4.1.jar',
  18.     total_executor_cores='1',
  19.     executor_cores='2',
  20.     executor_memory='2g',
  21.     num_executors='1',
  22.     name='airflow-spark-example-coming-from-aws-k8s',
  23.     verbose=True,
  24.     driver_memory='1g',
  25.     application_args=["/opt/spark/data/graphx/users.txt"],
  26.     dag=dag,
  27. )
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
 
Top