Guest User

Untitled

a guest
Jan 2nd, 2020
475
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 45.39 KB | None | 0 0
  1. .. Licensed to the Apache Software Foundation (ASF) under one
  2. or more contributor license agreements. See the NOTICE file
  3. distributed with this work for additional information
  4. regarding copyright ownership. The ASF licenses this file
  5. to you under the Apache License, Version 2.0 (the
  6. "License"); you may not use this file except in compliance
  7. with the License. You may obtain a copy of the License at
  8.  
  9. .. http://www.apache.org/licenses/LICENSE-2.0
  10.  
  11. .. Unless required by applicable law or agreed to in writing,
  12. software distributed under the License is distributed on an
  13. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
  14. KIND, either express or implied. See the License for the
  15. specific language governing permissions and limitations
  16. under the License.
  17.  
  18. Configuration Reference
  19. =======================
  20.  
  21. .. _config-ref/core:
  22.  
  23. ``[core]``
  24. ^^^^^^^^^^
  25.  
  26. .. _config-ref/core/dags_folder:
  27.  
  28. ``dags_folder``
  29. ***************
  30.  
  31. The folder where your airflow pipelines live, most likely a subfolder in a code repository. This path must be absolute
  32.  
  33. .. _config-ref/core/hostname_callable:
  34.  
  35. ``hostname_callable``
  36. *********************
  37.  
  38. Hostname by providing a path to a callable, which will resolve the hostname. The format is "package:function". For example, default value "socket:getfqdn" means that result from getfqdn() of "socket" package will be used as hostname. No argument should be required in the function specified. gIf using IP address as hostname is preferred, use value "airflow.utils.net:get_host_ip_address"
  39.  
  40. .. _config-ref/core/default_timezone:
  41.  
  42. ``default_timezone``
  43. ********************
  44.  
  45. Default timezone in case supplied date times are naive. Can be utc (default), system, or any IANA timezone string (e.g. Europe/Amsterdam)
  46.  
  47. .. _config-ref/core/executor:
  48.  
  49. ``executor``
  50. *************
  51.  
  52. The executor class that airflow should use. Choices include SequentialExecutor, LocalExecutor, CeleryExecutor, DaskExecutor, KubernetesExecutor
  53.  
  54. .. _config-ref/core/sql_alchemy_conn:
  55.  
  56. ``sql_alchemy_conn``
  57. ********************
  58.  
  59. The SqlAlchemy connection string to the metadata database. SqlAlchemy supports many different database engine, more information their website
  60.  
  61. .. _config-ref/core/sql_engine_encoding:
  62.  
  63. ``sql_engine_encoding``
  64. ***********************
  65.  
  66. The encoding for the databases
  67.  
  68. .. _config-ref/core/sql_alchemy_pool_enabled:
  69.  
  70. ``sql_alchemy_pool_enabled``
  71. ****************************
  72.  
  73. If SqlAlchemy should pool database connections.
  74.  
  75. .. _config-ref/core/sql_alchemy_pool_size:
  76.  
  77. ``sql_alchemy_pool_size``
  78. *************************
  79.  
  80. The SqlAlchemy pool size is the maximum number of database connections in the pool. 0 indicates no limit.
  81.  
  82. .. _config-ref/core/sql_alchemy_max_overflow:
  83.  
  84. ``sql_alchemy_max_overflow``
  85. ****************************
  86.  
  87. The maximum overflow size of the pool. When the number of checked-out connections reaches the size set in pool_size, additional connections will be returned up to this limit. When those additional connections are returned to the pool, they are disconnected and discarded. It follows then that the total number of simultaneous connections the pool will allow is pool_size + max_overflow, and the total number of "sleeping" connections the pool will allow is pool_size. max_overflow can be set to -1 to indicate no overflow limit; no limit will be placed on the total number of concurrent connections. Defaults to 10.
  88.  
  89. .. _config-ref/core/sql_alchemy_pool_recycle:
  90.  
  91. ``sql_alchemy_pool_recycle``
  92. ****************************
  93.  
  94. The SqlAlchemy pool recycle is the number of seconds a connection can be idle in the pool before it is invalidated. This config does not apply to sqlite. If the number of DB connections is ever exceeded, a lower config value will allow the system to recover faster.
  95.  
  96. .. _config-ref/core/sql_alchemy_pool_pre_ping:
  97.  
  98. ``sql_alchemy_pool_pre_ping``
  99. *****************************
  100.  
  101. Check connection at the start of each connection pool checkout. Typically, this is a simple statement like "SELECT 1". More information here: https://docs.sqlalchemy.org/en/13/core/pooling.html#disconnect-handling-pessimistic
  102.  
  103. .. _config-ref/core/sql_alchemy_schema:
  104.  
  105. ``sql_alchemy_schema``
  106. **********************
  107.  
  108. The schema to use for the metadata database. SqlAlchemy supports databases with the concept of multiple schemas.
  109.  
  110. .. _config-ref/core/sql_alchemy_connect_args:
  111.  
  112. ``sql_alchemy_connect_args``
  113. ****************************
  114.  
  115. Import path for connect args in SqlAlchemy. Default to an empty dict. This is useful when you want to configure db engine args that SqlAlchemy won't parse in connection string. See https://docs.sqlalchemy.org/en/13/core/engines.html#sqlalchemy.create_engine.params.connect_args
  116.  
  117. .. _config-ref/core/parallelism:
  118.  
  119. ``parallelism``
  120. ***************
  121.  
  122. The amount of parallelism as a setting to the executor. This defines the max number of task instances that should run simultaneously on this airflow installation
  123.  
  124. .. _config-ref/core/dag_concurrency:
  125.  
  126. ``dag_concurrency``
  127. *******************
  128.  
  129. The number of task instances allowed to run concurrently by the scheduler
  130.  
  131. .. _config-ref/core/dags_are_paused_at_creation:
  132.  
  133. ``dags_are_paused_at_creation``
  134. *******************************
  135.  
  136. Are DAGs paused by default at creation
  137.  
  138. .. _config-ref/core/max_active_runs_per_dag:
  139.  
  140. ``max_active_runs_per_dag``
  141. ***************************
  142.  
  143. The maximum number of active DAG runs per DAG
  144.  
  145. .. _config-ref/core/load_examples:
  146.  
  147. ``load_examples``
  148. *****************
  149.  
  150. Whether to load the examples that ship with Airflow. It's good to get started, but you probably want to set this to False in a production environment
  151.  
  152. .. _config-ref/core/plugins_folder:
  153.  
  154. ``plugins_folder``
  155. **********************
  156.  
  157. Where your Airflow plugins are stored
  158.  
  159. .. _config-ref/core/fernet_key:
  160.  
  161. ``fernet_key``
  162. **************
  163.  
  164. Secret key to save connection passwords in the db
  165.  
  166. .. _config-ref/core/donot_pickle:
  167.  
  168. ``donot_pickle``
  169. ****************
  170.  
  171. Whether to disable pickling dags
  172.  
  173. .. _config-ref/core/dagbag_import_timeout:
  174.  
  175. ``dagbag_import_timeout``
  176. *************************
  177.  
  178. How long before timing out a python file import
  179.  
  180. .. _config-ref/core/dag_file_processor_timeout:
  181.  
  182. ``dag_file_processor_timeout``
  183. ******************************
  184.  
  185. How long before timing out a DagFileProcessor, which processes a dag file
  186.  
  187. .. _config-ref/core/task_runner:
  188.  
  189. ``task_runner``
  190. ***************
  191.  
  192. The class to use for running task instances in a subprocess
  193. Can be used to de-elevate a sudo user running Airflow when executing tasks
  194.  
  195. .. _config-ref/core/default_impersonation:
  196.  
  197. ``default_impersonation``
  198. *************************
  199.  
  200. If set, tasks without a ``run_as_user`` argument will be run with this user
  201.  
  202. .. _config-ref/core/security:
  203.  
  204. ``security``
  205. ************
  206.  
  207. What security module to use (for example kerberos):
  208.  
  209. .. _config-ref/core/secure_mode:
  210.  
  211. ``secure_mode``
  212. ***************
  213.  
  214. If set to False enables some unsecure features like Charts and Ad Hoc Queries. In 2.0 will default to True.
  215.  
  216. .. _config-ref/core/unit_test_mode:
  217.  
  218. ``unit_test_mode``
  219. ******************
  220.  
  221. Turn unit test mode on (overwrites many configuration options with test values at runtime)
  222.  
  223. .. _config-ref/core/enable_xcom_pickling:
  224.  
  225. ``enable_xcom_pickling``
  226. ************************
  227.  
  228. Whether to enable pickling for xcom (note that this is insecure and allows for RCE exploits). This will be deprecated in Airflow 2.0 (be forced to False).
  229.  
  230. .. _config-ref/core/killed_task_cleanup_time:
  231.  
  232. ``killed_task_cleanup_time``
  233. ****************************
  234.  
  235. When a task is killed forcefully, this is the amount of time in seconds that it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
  236.  
  237. .. _config-ref/core/dag_run_conf_overrides_params:
  238.  
  239. ``dag_run_conf_overrides_params``
  240. *********************************
  241.  
  242. Whether to override params with dag_run.conf. If you pass some key-value pairs through ``airflow dags backfill -c`` or ``airflow dags trigger -c``, the key-value pairs will override the existing ones in params.
  243.  
  244. .. _config-ref/core/worker_precheck:
  245.  
  246. ``worker_precheck``
  247. *******************
  248.  
  249. Worker initialisation check to validate Metadata Database connection
  250.  
  251. .. _config-ref/core/dag_discovery_safe_mode:
  252.  
  253. ``dag_discovery_safe_mode``
  254. ***************************
  255.  
  256. When discovering DAGs, ignore any files that don't contain the strings ``DAG`` and ``airflow``.
  257.  
  258. .. _config-ref/core/default_task_retries:
  259.  
  260. ``default_task_retries``
  261. ************************
  262.  
  263. The number of retries each task is going to have by default. Can be overridden at dag or task level.
  264.  
  265. .. _config-ref/core/store_serialized_dags:
  266.  
  267. ``store_serialized_dags``
  268. *************************
  269.  
  270. Whether to serialises DAGs and persist them in DB. If set to True, Webserver reads from DB instead of parsing DAG files More details: https://airflow.apache.org/docs/stable/dag-serialization.html
  271.  
  272. .. _config-ref/core/min_serialized_dag_update_interval:
  273.  
  274. ``min_serialized_dag_update_interval``
  275. **************************************
  276.  
  277. Updating serialized DAG can not be faster than a minimum interval to reduce database write rate.
  278.  
  279. .. _config-ref/core/check_slas:
  280.  
  281. ``check_slas``
  282. **************
  283.  
  284. On each dagrun check against defined SLAs
  285.  
  286. .. _config-ref/logging:
  287.  
  288. ``[logging]``
  289. ^^^^^^^^^^^^^
  290.  
  291. .. _config-ref/logging/base_log_folder:
  292.  
  293. ``base_log_folder``
  294. *******************
  295.  
  296. The folder where airflow should store its log files This path must be absolute
  297.  
  298. .. _config-ref/logging/remote_logging:
  299.  
  300. ``remote_logging``
  301. ******************
  302.  
  303. Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic Search. Users must supply an Airflow connection id that provides access to the storage location. If remote_logging is set to true, see UPDATING.md for additional configuration requirements.
  304.  
  305. .. _config-ref/logging/remote_log_conn_id:
  306.  
  307. ``remote_log_conn_id``
  308. **********************
  309.  
  310. .. _config-ref/logging/remote_base_log_folder:
  311.  
  312. ``remote_base_log_folder``
  313. **************************
  314.  
  315. .. _config-ref/logging/encrypt_s3_logs:
  316.  
  317. ``encrypt_s3_logs``
  318. *******************
  319.  
  320. .. _config-ref/logging/logging_level:
  321.  
  322. ``logging_level``
  323. *****************
  324.  
  325. .. _config-ref/logging/fab_logging_level:
  326.  
  327. ``fab_logging_level``
  328. *********************
  329.  
  330. .. _config-ref/logging/Logging class:
  331.  
  332. Logging ``class``
  333. *****************
  334.  
  335. Specify the class that will specify the logging configuration
  336. This class has to be on the python classpath
  337.  
  338. .. _config-ref/logging/logging_config_class:
  339.  
  340. ``logging_config_class``
  341. ************************
  342.  
  343. .. _config-ref/logging/Log format:
  344.  
  345. Log ``format``
  346. **************
  347.  
  348. Colour the logs when the controlling terminal is a TTY.
  349.  
  350. .. _config-ref/logging/colored_console_log:
  351.  
  352. ``colored_console_log``
  353. ***********************
  354.  
  355. .. _config-ref/logging/colored_log_format:
  356.  
  357. ``colored_log_format``
  358. **********************
  359.  
  360. .. _config-ref/logging/colored_formatter_class:
  361.  
  362. ``colored_formatter_class``
  363. ***************************
  364.  
  365. .. _config-ref/logging/log_format:
  366.  
  367. ``log_format``
  368. **************
  369.  
  370. .. _config-ref/logging/simple_log_format:
  371.  
  372. ``simple_log_format``
  373. *********************
  374.  
  375. .. _config-ref/logging/task_log_prefix_template:
  376.  
  377. ``task_log_prefix_template``
  378. ****************************
  379.  
  380. Specify prefix pattern like mentioned below with stream handler TaskHandlerWithCustomFormatter
  381.  
  382.  
  383. .. _config-ref/logging/log_filename_template:
  384.  
  385. ``log_filename_template``
  386. *************************
  387.  
  388. Log filename format
  389.  
  390. .. _config-ref/logging/log_processor_filename_template:
  391.  
  392. ``log_processor_filename_template``
  393. ***********************************
  394.  
  395. .. _config-ref/logging/dag_processor_manager_log_location:
  396.  
  397. ``dag_processor_manager_log_location``
  398. **************************************
  399.  
  400. Name of handler to read task instance logs. Default to use task handler.
  401.  
  402. .. _config-ref/logging/task_log_reader:
  403.  
  404. ``task_log_reader``
  405. *******************
  406.  
  407. .. _config-ref/logging/cli:
  408.  
  409. ``cli``
  410. *******
  411.  
  412. .. _config-ref/cli/api_client:
  413.  
  414. ``api_client``
  415. **************
  416.  
  417. .. _config-ref/api_client/endpoint_url:
  418.  
  419. ``endpoint_url``
  420. ****************
  421.  
  422. In what way should the cli access the API. The LocalClient will use the database directly, while the json_client will use the api running on the webserver
  423.  
  424. If you set web_server_url_prefix, do NOT forget to append it here, ex: endpoint_url. So api will look like: http://localhost:8080/myroot/api/experimental/...
  425.  
  426. .. _config-ref/debug:
  427.  
  428. ``[debug]``
  429. ^^^^^^^^^^^
  430.  
  431. .. _config-ref/debug/fail_fast:
  432.  
  433. ``fail_fast``
  434. *************
  435.  
  436. Used only with DebugExecutor. If set to True DAG will fail with first failed task. Helpful for debugging purposes.
  437.  
  438. .. _config-ref/api:
  439.  
  440. ``[api]``
  441. ^^^^^^^^^
  442. .. _config-ref/api/auth_backend:
  443.  
  444. ``auth_backend``
  445. ****************
  446.  
  447. How to authenticate users of the API
  448.  
  449. ``[lineage]``
  450. ^^^^^^^^^^^
  451.  
  452. .. _config-ref/lineage/backend:
  453.  
  454. ``backend``
  455. ***********
  456.  
  457. what lineage backend to use
  458.  
  459. .. _config-ref/atlas:
  460.  
  461. ``[atlas]``
  462. ^^^^^^^^^^^
  463.  
  464. .. _config-ref/atlas/sasl_enabled:
  465.  
  466. ``sasl_enabled``
  467. ****************
  468.  
  469. .. _config-ref/atlas/host:
  470.  
  471. ``host``
  472. ********
  473.  
  474. .. _config-ref/atlas/port:
  475.  
  476. ``port``
  477. ********
  478.  
  479. .. _config-ref/atlas/username:
  480.  
  481. ``username``
  482. ************
  483.  
  484. .. _config-ref/atlas/password:
  485.  
  486. ``password``
  487. ************
  488.  
  489. .. _config-ref/operators:
  490.  
  491. ``[operators]``
  492. ^^^^^^^^^^^^^^^
  493.  
  494. .. _config-ref/SECTION/default_owner:
  495.  
  496. ``default_owner``
  497. *****************
  498.  
  499. The default owner assigned to each new operator, unless provided explicitly or passed via ``default_args``
  500.  
  501. .. _config-ref/operators/default_cpus:
  502.  
  503. ``default_cpus``
  504. ****************
  505.  
  506. .. _config-ref/operators/default_ram:
  507.  
  508. ``default_ram``
  509. ***************
  510.  
  511. .. _config-ref/operators/default_disk:
  512.  
  513. ``default_disk``
  514. ****************
  515.  
  516. .. _config-ref/operators/default_gpus:
  517.  
  518. ``default_gpus``
  519. ****************
  520.  
  521. .. _config-ref/operators/allow_illegal_arguments:
  522.  
  523. ``allow_illegal_arguments``
  524. ***************************
  525.  
  526. Is allowed to pass additional/unused arguments (args, kwargs) to the BaseOperator operator. If set to False, an exception will be thrown, otherwise only the console message will be displayed.
  527.  
  528. .. _config-ref/hive:
  529.  
  530. ``[hive]``
  531. ^^^^^^^^^^
  532.  
  533. .. _config-ref/hive/default_hive_mapred_queue:
  534.  
  535. ``default_hive_mapred_queue``
  536. *****************************
  537.  
  538. Default mapreduce queue for HiveOperator tasks
  539. .. _config-ref/hive/mapred_job_name_template:
  540.  
  541. ``mapred_job_name_template``
  542. ****************************
  543.  
  544. Template for mapred_job_name in HiveOperator, supports the following named parameters: hostname, dag_id, task_id, execution_date
  545.  
  546. .. _config-ref/webserver:
  547.  
  548. ``[webserver]``
  549. ^^^^^^^^^^^^^^^
  550.  
  551. .. _config-ref/webserver/base_url:
  552.  
  553. ``base_url``
  554. ************
  555.  
  556. The base url of your website as airflow cannot guess what domain or cname you are using. This is used in automated emails that airflow sends to point links to the right web server
  557.  
  558. .. _config-ref/webserver/web_server_host:
  559.  
  560. ``web_server_host``
  561. ***************
  562.  
  563. The ip specified when starting the web server
  564.  
  565. .. _config-ref/webserver/web_server_port:
  566.  
  567. ``web_server_port``
  568. *******************
  569.  
  570. The port on which to run the web server
  571.  
  572. .. _config-ref/webserver/web_server_ssl_cert:
  573.  
  574. ``web_server_ssl_cert``
  575. ***********************
  576.  
  577. Paths to the SSL certificate and key for the web server. When both are provided SSL will be enabled. This does not change the web server port.
  578.  
  579. .. _config-ref/webserver/web_server_ssl_key:
  580.  
  581. ``web_server_ssl_key``
  582. **********************
  583.  
  584. .. _config-ref/webserver/web_server_master_timeout:
  585.  
  586. ``web_server_master_timeout``
  587. *****************************
  588.  
  589. Number of seconds the webserver waits before killing gunicorn master that doesn't respond
  590.  
  591. .. _config-ref/webserver/web_server_worker_timeout:
  592.  
  593. ``web_server_worker_timeout``
  594. *****************************
  595.  
  596. Number of seconds the gunicorn webserver waits before timing out on a worker
  597.  
  598. .. _config-ref/webserver/worker_refresh_batch_size:
  599.  
  600. ``worker_refresh_batch_size``
  601. *************************
  602.  
  603. Number of workers to refresh at a time. When set to 0, worker refresh is disabled. When nonzero, airflow periodically refreshes webserver workers by bringing up new ones and killing old ones.
  604.  
  605. .. _config-ref/webserver/worker_refresh_interval:
  606.  
  607. ``worker_refresh_interval``
  608. ***************************
  609.  
  610. Number of seconds to wait before refreshing a batch of workers.
  611.  
  612. .. _config-ref/webserver/secret_key:
  613.  
  614. ``secret_key``
  615. **************
  616.  
  617. Secret key used to run your flask app. It should be as random as possible
  618. .. _config-ref/webserver/workers:
  619.  
  620. ``workers``
  621. ***********
  622.  
  623. Number of workers to run the Gunicorn web server
  624.  
  625. .. _config-ref/webserver/worker_class:
  626.  
  627. ``worker_class``
  628. ****************
  629.  
  630. The worker class gunicorn should use. Choices include sync (default), eventlet, gevent
  631. .. _config-ref/webserver/access_logfile:
  632.  
  633. ``access_logfile``
  634. ******************
  635.  
  636. Log files for the gunicorn webserver. '-' means log to stderr.
  637.  
  638. .. _config-ref/webserver/error_logfile:
  639.  
  640. ``error_logfile``
  641. *****************
  642.  
  643. .. _config-ref/webserver/expose_config:
  644.  
  645. ``expose_config``
  646. *****************
  647.  
  648. Expose the configuration file in the web server
  649.  
  650. .. _config-ref/webserver/expose_hostname:
  651.  
  652. ``expose_hostname``
  653. *******************
  654.  
  655. Expose hostname in the web server
  656.  
  657. .. _config-ref/webserver/expose_stacktrace:
  658.  
  659. ``expose_stacktrace``
  660. *********************
  661.  
  662. Expose stacktrace in the web server
  663.  
  664. .. _config-ref/webserver/dag_default_view:
  665.  
  666. ``dag_default_view``
  667. ********************
  668.  
  669. Default DAG view. Valid values are: tree, graph, duration, gantt, landing_times
  670.  
  671. .. _config-ref/webserver/dag_orientation:
  672.  
  673. ``dag_orientation``
  674. *******************
  675.  
  676. Default DAG orientation. Valid values are: LR (Left->Right), TB (Top->Bottom), RL (Right->Left), BT (Bottom->Top)
  677.  
  678. .. _config-ref/webserver/demo_mode:
  679.  
  680. ``demo_mode``
  681. *************
  682.  
  683. Puts the webserver in demonstration mode; blurs the names of Operators for privacy.
  684.  
  685. .. _config-ref/webserver/log_fetch_timeout_sec:
  686.  
  687. ``log_fetch_timeout_sec``
  688. *************************
  689.  
  690. The amount of time (in secs) webserver will wait for initial handshake while fetching logs from other worker machine
  691.  
  692. .. _config-ref/webserver/hide_paused_dags_by_default:
  693.  
  694. ``hide_paused_dags_by_default``
  695. *******************************
  696.  
  697. By default, the webserver shows paused DAGs. Flip this to hide paused DAGs by default
  698.  
  699. .. _config-ref/webserver/page_size:
  700.  
  701. ``page_size``
  702. *************
  703.  
  704. Consistent page size across all listing views in the UI
  705.  
  706. .. _config-ref/webserver/navbar_color:
  707.  
  708. ``navbar_color``
  709. ****************
  710.  
  711. Define the color of navigation bar
  712.  
  713. .. _config-ref/webserver/default_dag_run_display_number:
  714.  
  715. ``default_dag_run_display_number``
  716. **********************************
  717.  
  718. Default dagrun to show in UI
  719.  
  720. .. _config-ref/webserver/enable_proxy_fix:
  721.  
  722. ``enable_proxy_fix``
  723. ********************
  724.  
  725. Enable werkzeug ``ProxyFix`` middleware
  726.  
  727. .. _config-ref/webserver/cookie_secure:
  728.  
  729. ``cookie_secure``
  730. *****************
  731.  
  732. Set secure flag on session cookie
  733.  
  734. .. _config-ref/webserver/cookie_samesite:
  735.  
  736. ``cookie_samesite``
  737. *******************
  738.  
  739. Set samesite policy on session cookie
  740.  
  741. .. _config-ref/webserver/default_wrap:
  742.  
  743. ``default_wrap``
  744. ****************
  745.  
  746. Default setting for wrap toggle on DAG code and TI log views.
  747.  
  748. .. _config-ref/webserver/analytics_tool:
  749.  
  750. ``analytics_tool``
  751. ******************
  752.  
  753. Send anonymous user activity to your analytics tool
  754.  
  755. .. _config-ref/webserver/analytics_id:
  756.  
  757. ``analytics_id``
  758. *****************
  759.  
  760. .. _config-ref/webserver/update_fab_perms:
  761.  
  762. ``update_fab_perms``
  763. ********************
  764.  
  765. Update FAB permissions and sync security manager roles on webserver startup
  766.  
  767. .. _config-ref/webserver/force_log_out_after:
  768.  
  769. ``force_log_out_after``
  770. ***********************
  771.  
  772. Minutes of non-activity before logged out from UI 0 means never get forcibly logged out
  773.  
  774.  
  775. .. _config-ref/email:
  776.  
  777. ``[email]``
  778. ^^^^^^^^^^^
  779.  
  780. .. _config-ref/SECTION/email_backend:
  781.  
  782. ``email_backend``
  783. *****************
  784.  
  785. .. _config-ref/smtp:
  786.  
  787. ``[smtp]``
  788. ^^^^^^^^^^
  789.  
  790. .. _config-ref/smtp/smtp_starttls:
  791.  
  792. ``smtp_starttls``
  793. *****************
  794.  
  795. If you want airflow to send emails on retries, failure, and you want to use the airflow.utils.email.send_email_smtp function, you have to configure an
  796.  
  797. .. _config-ref/smtp/smtp_ssl:
  798.  
  799. ``smtp_ssl``
  800. ************
  801.  
  802. .. _config-ref/smtp/smtp_user:
  803.  
  804. ``smtp_user``
  805. *************
  806.  
  807. Uncomment and set the user/pass settings if you want to use SMTP AUTH
  808.  
  809. .. _config-ref/smtp/smtp_password:
  810.  
  811. ``smtp_password``
  812. *****************
  813.  
  814. .. _config-ref/smtp/smtp_port:
  815.  
  816. ``smtp_port``
  817. *************
  818.  
  819. .. _config-ref/smtp/smtp_mail_from:
  820.  
  821. ``smtp_mail_from``
  822. ******************
  823.  
  824. .. _config-ref/sentry:
  825.  
  826. ``[sentry]``
  827. ^^^^^^^^^^^^
  828. .. _config-ref/SECTION/sentry_dsn:
  829.  
  830. Sentry (https://docs.sentry.io) integration
  831.  
  832. ``sentry_dsn``
  833. **************
  834.  
  835.  
  836. .. _config-ref/celery:
  837.  
  838. ``[celery]``
  839. ^^^^^^^^^^^^
  840.  
  841. This celery only applies if you are using the CeleryExecutor in
  842. ``[core]`` celery above
  843.  
  844. .. _config-ref/celery/celery_app_name:
  845.  
  846. ``celery_app_name``
  847. *******************
  848.  
  849. The app name that will be used by celery
  850.  
  851. .. _config-ref/celery/worker_concurrency:
  852.  
  853. ``worker_concurrency``
  854. **********************
  855.  
  856. The concurrency that will be used when starting workers with the "airflow celery worker" command. This defines the number of task instances that a worker will take, so size up your workers based on the resources on your worker box and the nature of your tasks
  857.  
  858. .. _config-ref/celery/worker_autoscale:
  859.  
  860. ``worker_autoscale``
  861. ********************
  862.  
  863. The maximum and minimum concurrency that will be used when starting workers with the "airflow celery worker" command (always keep minimum processes, but grow to maximum if necessary). Note the value should be "max_concurrency,min_concurrency" Pick these numbers based on resources on worker box and the nature of the task. If autoscale option is available, worker_concurrency will be ignored. http://docs.celeryproject.org/en/latest/reference/celery.bin.worker.html#cmdoption-celery-worker-autoscale
  864.  
  865. .. _config-ref/celery/worker_log_server_port:
  866.  
  867. ``worker_log_server_port``
  868. **************************
  869.  
  870. When you start an airflow worker, airflow starts a tiny web server subprocess to serve the workers local log files to the airflow main web server, who then builds pages and sends them to users. This defines the port on which the logs are served. It needs to be unused, and open visible from the main web server to connect into the workers.
  871.  
  872. .. _config-ref/celery/broker_url:
  873.  
  874. ``broker_url``
  875. **************
  876.  
  877. The Celery broker URL. Celery supports RabbitMQ, Redis and experimentally a sqlalchemy database. Refer to the Celery documentation for more information. http://docs.celeryproject.org/en/latest/userguide/configuration.html#broker-settings
  878.  
  879. .. _config-ref/celery/result_backend:
  880.  
  881. ``result_backend``
  882. ******************
  883.  
  884. The Celery result_backend. When a job finishes, it needs to update the metadata of the job. Therefore it will post a message on a message bus, or insert it into a database (depending of the backend) This status is used by the scheduler to update the state of the task The use of a database is highly recommended http://docs.celeryproject.org/en/latest/userguide/configuration.html#task-result-backend-settings
  885.  
  886. .. _config-ref/celery/flower_host:
  887.  
  888. ``flower_host``
  889. ***************
  890.  
  891. Celery Flower is a sweet UI for Celery. Airflow has a shortcut to start it ``airflow flower``. This defines the IP that Celery Flower runs on
  892.  
  893. .. _config-ref/celery/flower_url_prefix:
  894.  
  895. ``flower_url_prefix``
  896. *********************
  897.  
  898. The root URL for Flower Ex: flower_url_prefix
  899.  
  900. .. _config-ref/celery/flower_port:
  901.  
  902. ``flower_port``
  903. ***************
  904.  
  905. This defines the port that Celery Flower runs on
  906.  
  907. Securing Flower with Basic Authentication
  908. Accepts user:password pairs separated by a comma
  909. Example: flower_basic_auth
  910. flower_basic_auth
  911.  
  912. .. _config-ref/celery/default_queue:
  913.  
  914. ``default_queue``
  915. *****************
  916.  
  917. Default queue that tasks get assigned to and that worker listen on.
  918.  
  919. .. _config-ref/celery/sync_parallelism:
  920.  
  921. ``sync_parallelism``
  922. ********************
  923.  
  924. How many processes CeleryExecutor uses to sync task state. 0 means to use max(1, number of cores - 1) processes.
  925.  
  926. .. _config-ref/celery/celery_config_options:
  927.  
  928. ``celery_config_options``
  929. *************************
  930.  
  931. Import path for celery configuration options
  932.  
  933. .. _config-ref/in_case_of_ssl :
  934.  
  935. [In case of using SSL]
  936. ^^^^^^^^^^^^^^^^^^^^^^
  937. .. _config-ref/celery/ssl_active:
  938.  
  939. ``ssl_active``
  940. **************
  941.  
  942. .. _config-ref/celery/ssl_key:
  943.  
  944. ``ssl_key``
  945. ***********
  946.  
  947. .. _config-ref/celery/ssl_cacert:
  948.  
  949. ``ssl_cacert``
  950. **************
  951.  
  952. .. _config-ref/celery/ssl_cert:
  953.  
  954. ``ssl_cert``
  955. ************
  956.  
  957. .. _config-ref/celery_pool_imp :
  958.  
  959. [Celery Pool implementation.]
  960. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  961. Choices include: prefork (default), eventlet, gevent or solo.
  962. See:
  963. https://docs.celeryproject.org/en/latest/userguide/workers.html#concurrency
  964. https://docs.celeryproject.org/en/latest/userguide/concurrency/eventlet.html
  965. pool
  966.  
  967. .. _config-ref/celery_broker_transport_options:
  968.  
  969. ``[celery_broker_transport_options]``
  970. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  971. This section is for specifying options which can be passed to the
  972. underlying celery broker transport. See:
  973. http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-broker_transport_options
  974.  
  975. .. _config-ref/celery_broker_transport_options/visibility_timeout:
  976.  
  977. ``visibility_timeout``
  978. **********************
  979.  
  980. The visibility timeout defines the number of seconds to wait for the worker to acknowledge the task before the message is redelivered to another worker. Make sure to increase the visibility timeout to match the time of the longest ETA you're planning to use.
  981.  
  982. visibility_timeout is only supported for Redis and SQS celery brokers.
  983. See:
  984. http://docs.celeryproject.org/en/master/userguide/configuration.html#std:setting-broker_transport_options
  985.  
  986.  
  987. .. _config-ref/dask:
  988.  
  989. ``[dask]``
  990. ^^^^^^^^^^
  991.  
  992. This section only applies if you are using the DaskExecutor in
  993. [core] section above
  994.  
  995. .. _config-ref/dask/cluster_address:
  996.  
  997. ``cluster_address``
  998. *******************
  999.  
  1000. The IP address and port of the Dask cluster's scheduler.
  1001.  
  1002. .. _config-ref/dask/tls_ca:
  1003.  
  1004. ``tls_ca``
  1005. **********
  1006.  
  1007. TLS/ SSL settings to access a secured Dask scheduler.
  1008.  
  1009. .. _config-ref/dask/tls_key:
  1010.  
  1011. ``tls_key``
  1012. ***********
  1013.  
  1014. .. _config-ref/dask/tls_cert:
  1015.  
  1016. ``tls_cert``
  1017. ************
  1018.  
  1019. .. _config-ref/scheduler:
  1020.  
  1021. ``[scheduler]``
  1022. ^^^^^^^^^^^^^^
  1023.  
  1024. .. _config-ref/scheduler/job_heartbeat_sec:
  1025.  
  1026. ``job_heartbeat_sec``
  1027. *********************
  1028.  
  1029. Task instances listen for external kill signal (when you clear tasks
  1030. from the CLI or the UI), this defines the frequency at which they should
  1031. listen (in seconds).
  1032.  
  1033. .. _config-ref/scheduler/scheduler_heartbeat_sec:
  1034.  
  1035. ``scheduler_heartbeat_sec``
  1036. ***************************
  1037.  
  1038. The scheduler constantly tries to trigger new tasks (look at the scheduler scheduler in the docs for more information). This defines how often the scheduler should run (in seconds).
  1039.  
  1040. .. _config-ref/scheduler/num_runs:
  1041.  
  1042. ``num_runs``
  1043. ************
  1044.  
  1045. The number of times to try to schedule each DAG file
  1046. -1 indicates unlimited number
  1047.  
  1048.  
  1049. .. _config-ref/scheduler/processor_poll_interval:
  1050.  
  1051. ``processor_poll_interval``
  1052. ***************************
  1053.  
  1054. The number of seconds to wait between consecutive DAG file processing
  1055.  
  1056. .. _config-ref/scheduler/min_file_process_interval:
  1057.  
  1058. ``min_file_process_interval``
  1059. *****************************
  1060.  
  1061. after how much time (seconds) a new DAGs should be picked up from the filesystem
  1062.  
  1063. .. _config-ref/scheduler/dag_dir_list_interval:
  1064.  
  1065. ``dag_dir_list_interval``
  1066. *************************
  1067.  
  1068. How often (in seconds) to scan the DAGs directory for new files. Default to 5 minutes.
  1069.  
  1070. .. _config-ref/scheduler/print_stats_interval:
  1071.  
  1072. ``print_stats_interval``
  1073. ************************
  1074.  
  1075. How often should stats be printed to the logs
  1076.  
  1077. .. _config-ref/scheduler/scheduler_health_check_threshold:
  1078.  
  1079. ``scheduler_health_check_threshold``
  1080. ************************************
  1081.  
  1082. If the last scheduler heartbeat happened more than scheduler_health_check_threshold ago (in seconds), scheduler is considered unhealthy. This is used by the health check in the "/health" endpoint
  1083.  
  1084. .. _config-ref/scheduler/child_process_log_directory:
  1085.  
  1086. ``child_process_log_directory``
  1087. *******************************
  1088.  
  1089. .. _config-ref/scheduler/scheduler_zombie_task_threshold:
  1090.  
  1091. ``scheduler_zombie_task_threshold``
  1092. ***********************************
  1093.  
  1094. Local task jobs periodically heartbeat to the DB. If the job has not heartbeat in this many seconds, the scheduler will mark the associated task instance as failed and will re-schedule the task.
  1095.  
  1096. .. _config-ref/scheduler/catchup_by_default:
  1097.  
  1098. ``catchup_by_default``
  1099. **********************
  1100.  
  1101. .. _config-ref/scheduler/max_tis_per_query:
  1102.  
  1103. ``max_tis_per_query``
  1104. *********************
  1105.  
  1106. Turn off scheduler catchup by setting this to False. Default behavior is unchanged and Command Line Backfills still work, but the scheduler will not do scheduler catchup if this is False, however it can be set on a per DAG basis in the DAG definition (catchup)
  1107.  
  1108. This changes the batch size of queries in the scheduling main loop. If this is too high, SQL query performance may be impacted by one or more of the following:
  1109. - reversion to full table scan
  1110. - complexity of query predicate
  1111. - excessive locking
  1112.  
  1113. Additionally, you may hit the maximum allowable query length for your db.
  1114.  
  1115. Set this to 0 for no limit (not advised)
  1116.  
  1117.  
  1118. .. _config-ref/statsd:
  1119.  
  1120. [statsd] (https://github.com/etsy/statsd) integration settings
  1121. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  1122. .. _config-ref/scheduler/statsd_on:
  1123.  
  1124. ``statsd_on``
  1125. *************
  1126.  
  1127. .. _config-ref/scheduler/statsd_host:
  1128.  
  1129. ``statsd_host``
  1130. ***************
  1131.  
  1132. .. _config-ref/scheduler/statsd_port:
  1133.  
  1134. ``statsd_port``
  1135. ***************
  1136.  
  1137. .. _config-ref/scheduler/statsd_prefix:
  1138.  
  1139. ``statsd_prefix``
  1140. *****************
  1141.  
  1142. .. _config-ref/scheduler/max_threads:
  1143.  
  1144. ``max_threads``
  1145. ***************
  1146.  
  1147. If you want to avoid send all the available metrics to StatsD, you can configure an allow list of prefixes to send only the metrics that start with the elements of the list (e.g: scheduler,executor,dagrun) statsd_allow_list
  1148.  
  1149. The scheduler can run multiple threads in parallel to schedule dags. This defines how many threads will run.
  1150.  
  1151.  
  1152. .. _config-ref/scheduler/authenticate:
  1153.  
  1154. ``authenticate``
  1155. ^^^^^^^^^^^^^^^^
  1156.  
  1157. .. _config-ref/scheduler/use_job_schedule:
  1158.  
  1159. ``use_job_schedule``
  1160. ********************
  1161.  
  1162. Turn off scheduler use of cron intervals by setting this to False. DAGs submitted manually in the web UI or with trigger_dag will still run.
  1163.  
  1164. .. _config-ref/ldap:
  1165.  
  1166. ``[ldap]``
  1167. ^^^^^^^^^^
  1168.  
  1169. .. _config-ref/SECTION/uri:
  1170.  
  1171. ``uri``
  1172. *******
  1173.  
  1174. set this to ldaps://<your.ldap.server>:<port>
  1175.  
  1176. .. _config-ref/ldap/user_filter:
  1177.  
  1178. ``user_filter``
  1179. ***************
  1180.  
  1181. .. _config-ref/ldap/user_name_attr:
  1182.  
  1183. ``user_name_attr``
  1184. ******************
  1185.  
  1186. .. _config-ref/ldap/group_member_attr:
  1187.  
  1188. ``group_member_attr``
  1189. *********************
  1190.  
  1191. .. _config-ref/ldap/superuser_filter:
  1192.  
  1193. ``superuser_filter``
  1194. ********************
  1195.  
  1196. .. _config-ref/ldap/data_profiler_filter:
  1197.  
  1198. ``data_profiler_filter``
  1199. ************************
  1200.  
  1201. .. _config-ref/ldap/bind_user:
  1202.  
  1203. ``bind_user``
  1204. *************
  1205.  
  1206. .. _config-ref/ldap/bind_password:
  1207.  
  1208. ``bind_password``
  1209. *****************
  1210.  
  1211. .. _config-ref/ldap/basedn:
  1212.  
  1213. ``basedn``
  1214. **********
  1215.  
  1216. .. _config-ref/ldap/cacert:
  1217.  
  1218. ``cacert``
  1219. **********
  1220.  
  1221. .. _config-ref/ldap/search_scope:
  1222.  
  1223. ``search_scope``
  1224. ****************
  1225.  
  1226. .. _config-ref/ldap/ignore_malformed_schema:
  1227.  
  1228. ``ignore_malformed_schema``
  1229. ***************************
  1230.  
  1231. This setting allows the use of LDAP servers that either return a broken schema, or do not return a schema.
  1232.  
  1233. .. _config-ref/kerberos:
  1234.  
  1235. ``[kerberos]``
  1236. ^^^^^^^^^^^^^^
  1237.  
  1238. .. _config-ref/SECTION/ccache:
  1239.  
  1240. ``ccache``
  1241. **********
  1242.  
  1243. .. _config-ref/SECTION/principal:
  1244.  
  1245. ``principal``
  1246. *************
  1247.  
  1248. gets augmented with fqdn
  1249.  
  1250. .. _config-ref/SECTION/reinit_frequency:
  1251.  
  1252. ``reinit_frequency``
  1253. ********************
  1254.  
  1255. .. _config-ref/SECTION/kinit_path:
  1256.  
  1257. ``kinit_path``
  1258. **************
  1259.  
  1260. .. _config-ref/SECTION/keytab:
  1261.  
  1262. ``keytab``
  1263. **********
  1264.  
  1265. .. _config-ref/github_enterprise:
  1266.  
  1267. ``github_enterprise``
  1268. ^^^^^^^^^^^^^^^^^^^^^
  1269.  
  1270. .. _config-ref/SECTION/api_rev:
  1271.  
  1272. ``api_rev``
  1273. ***********
  1274.  
  1275. .. _config-ref/admin:
  1276.  
  1277. ``[admin]``
  1278. ^^^^^^^^^^^
  1279.  
  1280. .. _config-ref/admin/hide_sensitive_variable_fields:
  1281.  
  1282. ``hide_sensitive_variable_fields``
  1283. **********************************
  1284.  
  1285. UI to hide sensitive variable fields when set to True
  1286.  
  1287. .. _config-ref/elasticsearch:
  1288.  
  1289. ``[elasticsearch]``
  1290. ^^^^^^^^^^^^^^^^^^^
  1291.  
  1292. .. _config-ref/elasticsearch/host:
  1293.  
  1294. ``host``
  1295. ********
  1296.  
  1297. Elasticsearch host
  1298.  
  1299. .. _config-ref/elasticsearch/end_of_log_mark:
  1300.  
  1301. ``end_of_log_mark``
  1302. *******************
  1303.  
  1304. .. _config-ref/elasticsearch/frontend:
  1305.  
  1306. ``frontend``
  1307. ************
  1308.  
  1309. Format of the log_id, which is used to query for a given tasks logs Used to mark the end of a log stream for a task Qualified URL for an elasticsearch frontend (like Kibana) with a template argument for log_id Code will construct log_id using the log_id template from the argument above. NOTE: The code will prefix the https:// automatically, don't include that here.
  1310.  
  1311. .. _config-ref/elasticsearch/write_stdout:
  1312.  
  1313. ``write_stdout``
  1314. ****************
  1315.  
  1316. Write the task logs to the stdout of the worker, rather than the default files
  1317.  
  1318. .. _config-ref/elasticsearch/json_format:
  1319.  
  1320. ``json_format``
  1321. ***************
  1322.  
  1323. Instead of the default log formatter, write the log lines as JSON
  1324.  
  1325. .. _config-ref/elasticsearch/json_fields:
  1326.  
  1327. ``json_fields``
  1328. ***************
  1329.  
  1330. Log fields to also attach to the json output, if enabled
  1331.  
  1332. .. _config-ref/elasticsearch_configs:
  1333.  
  1334. ``[elasticsearch_configs]``
  1335. ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  1336. .. _config-ref/elasticsearch_configs/use_ssl:
  1337.  
  1338. ``use_ssl``
  1339. ***********
  1340.  
  1341. .. _config-ref/elasticsearch_configs/verify_certs:
  1342.  
  1343. ``verify_certs``
  1344. ****************
  1345.  
  1346. .. _config-ref/kubernetes:
  1347.  
  1348. ``[kubernetes]``
  1349. ^^^^^^^^^^^^^^^^^
  1350.  
  1351. .. _config-ref/kubernetes/worker_container_repository:
  1352.  
  1353. ``worker_container_repository``
  1354. *******************************
  1355.  
  1356. The repository, tag and imagePullPolicy of the Kubernetes Image for the Worker to Run
  1357.  
  1358. .. _config-ref/kubernetes/worker_container_tag:
  1359.  
  1360. ``worker_container_tag``
  1361. ************************
  1362.  
  1363. .. _config-ref/kubernetes/worker_container_image_pull_policy:
  1364.  
  1365. ``worker_container_image_pull_policy``
  1366. **************************************
  1367.  
  1368. .. _config-ref/kubernetes/delete_worker_pods:
  1369.  
  1370. ``delete_worker_pods``
  1371. **********************
  1372.  
  1373. If True (default), worker pods will be deleted upon termination
  1374.  
  1375. .. _config-ref/kubernetes/worker_pods_creation_batch_size:
  1376.  
  1377. ``worker_pods_creation_batch_size``
  1378. ***********************************
  1379.  
  1380. Number of Kubernetes Worker Pod creation calls per scheduler loop
  1381.  
  1382. .. _config-ref/kubernetes/namespace:
  1383.  
  1384. ``namespace``
  1385. *************
  1386.  
  1387. The Kubernetes namespace where airflow workers should be created. Defaults to ``default``
  1388.  
  1389. .. _config-ref/kubernetes/airflow_configmap:
  1390.  
  1391. ``airflow_configmap``
  1392. *********************
  1393.  
  1394. The name of the Kubernetes ConfigMap Containing the Airflow Configuration (this file)
  1395.  
  1396. .. _config-ref/kubernetes/dags_in_image:
  1397.  
  1398. ``dags_in_image``
  1399. *****************
  1400.  
  1401. For docker image already contains DAGs, this is set to ``True``, and the worker will search for dags in dags_folder, otherwise use git sync or dags volume claim to mount DAGs
  1402.  
  1403. .. _config-ref/kubernetes/dags_volume_subpath:
  1404.  
  1405. ``dags_volume_subpath``
  1406. ***********************
  1407.  
  1408. For either git sync or volume mounted DAGs, the worker will look in this subpath for DAGs
  1409.  
  1410. .. _config-ref/kubernetes/dags_volume_claim:
  1411.  
  1412. ``dags_volume_claim``
  1413. *********************
  1414.  
  1415. For DAGs mounted via a volume claim (mutually exclusive with git-sync and host path)
  1416.  
  1417. .. _config-ref/kubernetes/logs_volume_subpath:
  1418.  
  1419. ``logs_volume_subpath``
  1420. ***********************
  1421.  
  1422. For volume mounted logs, the worker will look in this subpath for logs
  1423.  
  1424. .. _config-ref/kubernetes/logs_volume_claim:
  1425.  
  1426. ``logs_volume_claim``
  1427. *********************
  1428.  
  1429. A shared volume claim for the logs
  1430.  
  1431. .. _config-ref/kubernetes/dags_volume_host:
  1432.  
  1433. ``dags_volume_host``
  1434. ********************
  1435.  
  1436. For DAGs mounted via a hostPath volume (mutually exclusive with volume claim and git-sync) Useful in local environment, discouraged in production
  1437.  
  1438. .. _config-ref/kubernetes/logs_volume_host:
  1439.  
  1440. ``logs_volume_host``
  1441. ********************
  1442.  
  1443. A hostPath volume for the logs Useful in local environment, discouraged in production
  1444.  
  1445. .. _config-ref/kubernetes/env_from_configmap_ref:
  1446.  
  1447. ``env_from_configmap_ref``
  1448. **************************
  1449.  
  1450. A list of configMapsRefs to envFrom. If more than one configMap is specified, provide a comma separated list: configmap_a,configmap_b
  1451.  
  1452. .. _config-ref/kubernetes/env_from_secret_ref:
  1453.  
  1454. ``env_from_secret_ref``
  1455. ***********************
  1456.  
  1457. A list of secretRefs to envFrom. If more than one secret is specified, provide a comma separated list: secret_a,secret_b
  1458.  
  1459. .. _config-ref/kubernetes/git_repo:
  1460.  
  1461. ``git_repo``
  1462. ************
  1463.  
  1464. Git credentials and repository for DAGs mounted via Git (mutually exclusive with volume claim)
  1465.  
  1466. .. _config-ref/kubernetes/git_branch:
  1467.  
  1468. ``git_branch``
  1469. **************
  1470.  
  1471. .. _config-ref/kubernetes/git_subpath:
  1472.  
  1473. ``git_subpath``
  1474. ***************
  1475.  
  1476. .. _config-ref/kubernetes/git_sync_rev:
  1477.  
  1478. ``git_sync_rev``
  1479. ****************
  1480.  
  1481. The specific rev or hash the git_sync init container will checkout This becomes GIT_SYNC_REV environment variable in the git_sync init container for worker pods
  1482.  
  1483. .. _config-ref/kubernetes/git_user:
  1484.  
  1485. ``git_user``
  1486. ************
  1487.  
  1488. Use git_user and git_password for user authentication or git_ssh_key_secret_name and git_ssh_key_secret_key for SSH authentication
  1489.  
  1490. .. _config-ref/kubernetes/git_password:
  1491.  
  1492. ``git_password``
  1493. ****************
  1494.  
  1495. .. _config-ref/kubernetes/git_sync_root:
  1496.  
  1497. ``git_sync_root``
  1498. *****************
  1499.  
  1500. .. _config-ref/kubernetes/git_sync_dest:
  1501.  
  1502. ``git_sync_dest``
  1503. *****************
  1504.  
  1505. .. _config-ref/kubernetes/git_dags_folder_mount_point:
  1506.  
  1507. ``git_dags_folder_mount_point``
  1508. *******************************
  1509.  
  1510. Mount point of the volume if git-sync is being used. i.e. {AIRFLOW_HOME}/dags
  1511.  
  1512. To get Git-sync SSH authentication set up follow this format
  1513.  
  1514. .. code-block:: yaml
  1515.  
  1516. airflow-secrets.yaml:
  1517. ---
  1518. apiVersion: v1
  1519. kind: Secret
  1520. metadata:
  1521. name: airflow-secrets
  1522. data:
  1523. # key needs to be gitSshKey
  1524. gitSshKey: <base64_encoded_data>
  1525. ---
  1526. airflow-configmap.yaml:
  1527. apiVersion: v1
  1528. kind: ConfigMap
  1529. metadata:
  1530. name: airflow-configmap
  1531. data:
  1532. known_hosts: |
  1533. github.com ssh-rsa <...>
  1534. airflow.cfg: |
  1535. ...
  1536.  
  1537. .. _config-ref/kubernetes/git_ssh_key_secret_name:
  1538.  
  1539. ``git_ssh_key_secret_name``
  1540. ***************************
  1541.  
  1542. .. _config-ref/kubernetes/git_ssh_known_hosts_configmap_name:
  1543.  
  1544. ``git_ssh_known_hosts_configmap_name``
  1545. **************************************
  1546.  
  1547. .. _config-ref/kubernetes/git_ssh_key_secret_name:
  1548.  
  1549. ``git_ssh_key_secret_name``
  1550. ***************************
  1551.  
  1552. .. _config-ref/kubernetes/git_ssh_known_hosts_configmap_name:
  1553.  
  1554. ``git_ssh_known_hosts_configmap_name``
  1555. **************************************
  1556.  
  1557. #TODO FIX CODE SAMPLE
  1558.  
  1559. To give the git_sync init container credentials via a secret, create a secret with two fields: GIT_SYNC_USERNAME and GIT_SYNC_PASSWORD (example below) and add `git_sync_credentials_secret
  1560.  
  1561. Secret Example:
  1562.  
  1563. .. code-block:: yaml
  1564.  
  1565. apiVersion: v1
  1566. kind: Secret
  1567. metadata:
  1568. name: git-credentials
  1569. data:
  1570. GIT_SYNC_USERNAME: <base64_encoded_git_username>
  1571. GIT_SYNC_PASSWORD: <base64_encoded_git_password>
  1572. git_sync_credentials_secret
  1573.  
  1574. For cloning DAGs from git repositories into volumes: https://github.com/kubernetes/git-sync
  1575.  
  1576. .. _config-ref/kubernetes/git_sync_container_repository:
  1577.  
  1578. ``git_sync_container_repository``
  1579. *********************************
  1580.  
  1581. .. _config-ref/kubernetes/git_sync_container_tag:
  1582.  
  1583. ``git_sync_container_tag``
  1584. **************************
  1585.  
  1586. .. _config-ref/kubernetes/git_sync_init_container_name:
  1587.  
  1588. ``git_sync_init_container_name``
  1589. ********************************
  1590.  
  1591. .. _config-ref/kubernetes/git_sync_run_as_user:
  1592.  
  1593. ``git_sync_run_as_user``
  1594. ************************
  1595.  
  1596. .. _config-ref/kubernetes/worker_service_account_name:
  1597.  
  1598. ``worker_service_account_name``
  1599. *******************************
  1600.  
  1601. The name of the Kubernetes service account to be associated with airflow workers, if any. Service accounts are required for workers that require access to secrets or cluster resources. See the Kubernetes RBAC documentation for more: https://kubernetes.io/docs/admin/authorization/rbac/
  1602.  
  1603. .. _config-ref/kubernetes/image_pull_secrets:
  1604.  
  1605. ``image_pull_secrets``
  1606. **********************
  1607.  
  1608. Any image pull secrets to be given to worker pods, If more than one secret is required, provide a comma separated list: secret_a,secret_b
  1609.  
  1610. .. _config-ref/kubernetes/in_cluster:
  1611.  
  1612. ``in_cluster``
  1613. **************
  1614.  
  1615. Use the service account kubernetes gives to pods to connect to kubernetes cluster. It's intended for clients that expect to be running inside a pod running on kubernetes. It will raise an exception if called from a process not running in a kubernetes environment.
  1616.  
  1617. .. _config-ref/kubernetes/cluster_context:
  1618.  
  1619. ``cluster_context``
  1620. *******************
  1621.  
  1622. When running with in_cluster options to Kubernetes client. Leave blank these to use default behaviour like ````kubectl```` has.
  1623.  
  1624. .. _config-ref/kubernetes/config_file:
  1625.  
  1626. ``config_file``
  1627. ***************
  1628.  
  1629. .. _config-ref/kubernetes/affinity:
  1630.  
  1631. ``affinity``
  1632. ************
  1633.  
  1634. Affinity configuration as a single line formatted JSON object. See the affinity model for top-level key names (e.g. ``nodeAffinity``, etc.): https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#affinity-v1-core
  1635.  
  1636. .. _config-ref/kubernetes/tolerations:
  1637.  
  1638. ``tolerations``
  1639. ***************
  1640.  
  1641. A list of toleration objects as a single line formatted JSON array See: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/#toleration-v1-core
  1642.  
  1643. .. _config-ref/kubernetes/kube_client_request_args:
  1644.  
  1645. ``kube_client_request_args``
  1646. ****************************
  1647.  
  1648. keywords parameters to pass while calling a kubernetes client core_v1_api methods from Kubernetes Executor provided as a single line formatted JSON dictionary string. List of supported keyworddsparams are similar for all core_v1_apis, hence a single config variable for all apis See: https://raw.githubusercontent.com/kubernetes-client/python/master/kubernetes/client/apis/core_v1_api.py
  1649.  
  1650. Worker pods security context options
  1651. See:
  1652. https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  1653.  
  1654. .. _config-ref/kubernetes/run_as_user:
  1655.  
  1656. ``run_as_user``
  1657. ***************
  1658.  
  1659. Specifies the uid to run the first process of the worker pods containers as
  1660.  
  1661. .. _config-ref/kubernetes/fs_group:
  1662.  
  1663. ``fs_group``
  1664. ************
  1665.  
  1666. Specifies a gid to associate with all containers in the worker pods if using a git_ssh_key_secret_name use an fs_group that allows for the key to be read, e.g. 65533
  1667.  
  1668. .. _config-ref/kubernetes/worker_annotations:
  1669.  
  1670. ``worker_annotations``
  1671. **********************
  1672.  
  1673. Annotations configuration as a single line formatted JSON object. See the naming convention in: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/
  1674.  
  1675.  
  1676. .. _config-ref/kubernetes_node_selectors:
  1677.  
  1678. ``[kubernetes_node_selectors]``
  1679. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  1680.  
  1681. The Key-value pairs to be given to worker pods.
  1682. The worker pods will be scheduled to the nodes of the specified key-value pairs.
  1683. Should be supplied in the format: key
  1684.  
  1685. .. _config-ref/kubernetes_environment_variables:
  1686.  
  1687. ``[kubernetes_environment_variables]``
  1688. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  1689.  
  1690. The scheduler sets the following environment variables into your workers. You may define as
  1691. many environment variables as needed and the kubernetes launcher will set them in the launched workers.
  1692. Environment variables in this section are defined as follows
  1693. <environment_variable_key>
  1694.  
  1695. For example if you wanted to set an environment variable with value ``prod`` and key
  1696. ``ENVIRONMENT`` you would follow the following format:
  1697. ENVIRONMENT
  1698.  
  1699. Additionally you may override worker airflow settings with the AIRFLOW__<SECTION>__<KEY>
  1700. formatting as supported by airflow normally.
  1701.  
  1702. .. _config-ref/kubernetes_secrets:
  1703.  
  1704. ``[kubernetes_secrets]``
  1705. ^^^^^^^^^^^^^^^^^^^^^^^^
  1706.  
  1707. The scheduler mounts the following secrets into your workers as they are launched by the
  1708. scheduler. You may define as many secrets as needed and the kubernetes launcher will parse the
  1709. defined secrets and mount them as secret environment variables in the launched workers.
  1710. Secrets in this section are defined as follows
  1711. <environment_variable_mount>
  1712.  
  1713. For example if you wanted to mount a kubernetes secret key named ``postgres_password`` from the
  1714. kubernetes secret object ``airflow-secret`` as the environment variable ``POSTGRES_PASSWORD`` into
  1715. your workers you would follow the following format:
  1716. POSTGRES_PASSWORD
  1717.  
  1718. Additionally you may override worker airflow settings with the ``AIRFLOW__<SECTION>__<KEY>``
  1719. formatting as supported by airflow normally.
  1720.  
  1721. .. _config-ref/kubernetes_labels:
  1722.  
  1723. ``[kubernetes_labels]``
  1724. ^^^^^^^^^^^^^^^^^^^^^^^
  1725.  
  1726. The Key-value pairs to be given to worker pods.
  1727. The worker pods will be given these static labels, as well as some additional dynamic labels
  1728. to identify the task.
  1729. Should be supplied in the format: key
Add Comment
Please, Sign In to add comment