文档章节

AirFlow-Install

openthings
 openthings
发布于 2018/07/10 13:41
字数 413
阅读 52
收藏 0

AirFlow-Installation

Getting Airflow

The easiest way to install the latest stable version of Airflow is with pip:

pip install apache-airflow

You can also install Airflow with support for extra features like s3 or postgres:

pip install "apache-airflow[s3, postgres]"

Extra Packages

The apache-airflow PyPI basic package only installs what’s needed to get started. Subpackages can be installed depending on what will be useful in your environment. For instance, if you don’t need connectivity with Postgres, you won’t have to go through the trouble of installing the postgres-devel yum package, or whatever equivalent applies on the distribution you are using.

Behind the scenes, Airflow does conditional imports of operators that require these extra dependencies.

Here’s the list of the subpackages and what they enable:

subpackage install command enables
all pip install apache-airflow[all] All Airflow features known to man
all_dbs pip install apache-airflow[all_dbs] All databases integrations
async pip install apache-airflow[async] Async worker classes for gunicorn
devel pip install apache-airflow[devel] Minimum dev tools requirements
devel_hadoop pip install apache-airflow[devel_hadoop] Airflow + dependencies on the Hadoop stack
celery pip install apache-airflow[celery] CeleryExecutor
crypto pip install apache-airflow[crypto] Encrypt connection passwords in metadata db
druid pip install apache-airflow[druid] Druid.io related operators & hooks
gcp_api pip install apache-airflow[gcp_api] Google Cloud Platform hooks and operators (using google-api-python-client)
jdbc pip install apache-airflow[jdbc] JDBC hooks and operators
hdfs pip install apache-airflow[hdfs] HDFS hooks and operators
hive pip install apache-airflow[hive] All Hive related operators
kerberos pip install apache-airflow[kerberos] kerberos integration for kerberized hadoop
ldap pip install apache-airflow[ldap] ldap authentication for users
mssql pip install apache-airflow[mssql] Microsoft SQL operators and hook, support as an Airflow backend
mysql pip install apache-airflow[mysql] MySQL operators and hook, support as an Airflow backend
password pip install apache-airflow[password] Password Authentication for users
postgres pip install apache-airflow[postgres] Postgres operators and hook, support as an Airflow backend
qds pip install apache-airflow[qds] Enable QDS (qubole data services) support
rabbitmq pip install apache-airflow[rabbitmq] Rabbitmq support as a Celery backend
s3 pip install apache-airflow[s3] S3KeySensor, S3PrefixSensor
samba pip install apache-airflow[samba] Hive2SambaOperator
slack pip install apache-airflow[slack] SlackAPIPostOperator
vertica pip install apache-airflow[vertica] Vertica hook support as an Airflow backend
cloudant pip install apache-airflow[cloudant] Cloudant hook
redis pip install apache-airflow[redis] Redis hooks and sensors

© 著作权归作者所有

openthings
粉丝 322
博文 1137
码字总数 687066
作品 1
东城
架构师
私信 提问
airflow单机版搭建记录——不使用mysql,redis

[toc] airflow单机版搭建记录 环境准备 Python(pip)——airflow由python编写 安装airflow pip install apache-airflow 环境变量配置 本人是在root用户下执行,可自行选择 export AIRFLOW_HOM...

helplove
07/03
81
0
airflow使用mysql数据库,LocalExecutor并发调度

mysql-airflow 在mysql上执行 create database airflow; —— 创建数据库 GRANT all privileges on airflow.* TO 'airflow'@'%' IDENTIFIED BY '123456'; —— 将数据库airflow的所有权限授权......

helplove
07/03
225
0
Airflow在Kubernetes上的操作器

Airflow在Kubernetes (第一部分): 一种不同类型的Operator 作者: Daniel Imberman (Bloomberg LP) 译者:openthings,2019.02.26. 原文:https://kubernetes.io/blog/2018/06/28/airflow-on-......

openthings
2018/07/07
745
0
基于Kubernetes的瓜子云的任务调度系统

很大的挑战。 接下来我讲详细介绍一下瓜子云的任务调度系统搭建所遇到的问题和解决方案。 需求 瓜子最早的时候,任务调度用的是Crontab,后来由于数据仓库的复杂调度需求,我们引入了Airflow...

店家小二
2018/12/14
0
0
使用Airflow来调度Data Lake Analytics的任务

今天我们来介绍一下使用Airflow来调度 Data Lake Analytics(后面简称DLA)的任务执行。DLA作为一个数据湖的解决方案, 客户有每天周期性的调度一些任务从DLA查询数据回流到业务系统的需求。因...

迷你芊宝宝
03/22
74
0

没有更多内容

加载失败,请刷新页面

加载更多

IT兄弟连 HTML5教程 介绍HTML5给你认识 习题

1.关于HTML5说法正确的是:(C) A.HTML5只是对HTML4的一个简单升级 B.所有主流浏览器都支持HTML5 C.HTML5新增了离线缓存机制 D.HTML5主要是针对移动端进行了优化 2.为了标识一个HTML文...

老码农的一亩三分地
7分钟前
2
0
关于1加手机rom的分析过程

1、关于清理app缓存的信息 framework.jar和services.jar文件,都位于/system/framework目录下面 framework.jar的android.os.Intent类里面定义了action "android.intent.action.CLEAR_PKG",这......

shatian
8分钟前
1
0
[docker]使用root进入docker

docker exec -it --user root ed0 /bin/bash https://blog.csdn.net/kaifeng86/article/details/73237056...

Danni3
21分钟前
2
0
通过重编译icu替换icuxxxx.dll减少QT5发布体积

最近需要使用QT写一个程序,选择了比较新的QT5.3以获得更多特性进行快速开发。但是QTCreator编译出来的程序是动态链接版本,接着就头痛了,自己程序很小,但是所需要携带的动态链接库却很大,...

shzwork
23分钟前
3
0
关于二叉树的前序、中序、后序三种遍历

二叉树遍历分为三种:前序、中序、后序,其中序遍历最为重要。为啥叫这个名字?是根据根节点的顺序命名的。 比如上图正常的一个满节点,A:根节点、B:左节点、C:右节点,前序顺序是ABC(根...

城市之雾
28分钟前
5
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部