Installation guide¶
Kedro setup¶
First, you need to install base Kedro package in <17.0
version
Kedro 17.0 is supported by kedro-airflow-k8s, but not by kedro-mlflow yet, so the latest version from 0.16 family is recommended.
$ pip install 'kedro<0.17'
Plugin installation¶
Install from PyPI¶
You can install kedro-airflow-k8s
plugin from PyPi
with pip
:
pip install --upgrade kedro-airflow-k8s
Install from sources¶
You may want to install the develop branch which has unreleased features:
pip install git+https://github.com/getindata/kedro-airflow-k8s.git@develop
Available commands¶
You can check available commands by going into project directory and runnning:
$ kedro airflow-k8s
Usage: kedro airflow-k8s [OPTIONS] COMMAND [ARGS]...
Options:
-e, --env TEXT Environment to use.
-h, --help Show this message and exit.
Commands:
compile Create an Airflow DAG for a project
init Initializes configuration for the plugin
list-pipelines List pipelines generated by this plugin
run-once Uploads pipeline to Airflow and runs once
schedule Uploads pipeline to Airflow with given schedule
ui Open Apache Airflow UI in new browser tab
upload-pipeline Uploads pipeline to Airflow DAG location
compile
¶
compile
command takes one argument, which is the directory name containing configuration (relative to conf
folder).
As an outcome, dag
directory contains python file with generated DAG.
init
¶
init
command adds default plugin configuration to the project, based on Apache Airflow CLI input. It also allows
optionally adding github actions, to streamline project build and upload.
list-pipelines
¶
list-pipelines
lists all pipelines generated by this plugin which exist in Airflow server. All generated DAGs are
tagged with tag generated_with_kedro_airflow_k8s:$PLUGIN_VERSION
and the prefix of this tag is used to distinguish
among the other tags.
run-once
¶
run-once
command generates DAG from the pipeline, uploads it Airflow DAG location and triggers the DAG run as soon as
the new DAG instance is available. It optionally allows waiting for DAG run completion, checking if success
status is
returned.
schedule
¶
schedule
command takes three arguments, one is the directory name containing configuration (relative to conf
folder), the second one is the output location of generated dag, the third is cron like expression that relates to
Airflow DAG schedule_interval
.
ui
¶
ui
simplifies access to Apache Airflow console. It also allows open UI for the specific DAG.
upload-pipeline
¶
upload-pipeline
command takes two arguments, one is the directory name containing configuration (relative to conf
folder), the second one is the output location of generated dag.