airflow安装教程(local模式)

环境介绍

OS:centos 7.6

python:3.6.8

pip:21.2.4

具体安装

pip配置阿里云清华源

mkdir -p $HOME/.config/pip/

vim $HOME/.config/pip/pip.conf

新增如下内容:

[global]

index-url = https://pypi.tuna.tsinghua.edu.cn/simple

执行安装:

export AIRFLOW_HOME=/data/airflow

AIRFLOW_VERSION=2.2.2

export PYTHON_VERSION=3.6

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

pip install virtualenv

启动airflow

airflow standalone

报错

[root@hadoop1 ~]# airflow standalone
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 1129, in <module>
    conf.validate()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 224, in validate
    self._validate_config_dependencies()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 278, in _validate_config_dependencies
    f"error: sqlite C library version too old (< {min_sqlite_version}). "
airflow.exceptions.AirflowConfigException: error: sqlite C library version too old (< 3.15.0). See https://airflow.apache.org/docs/apache-airflow/2.2.2/howto/set-up-database.html#setting-up-a-sqlite-database

解决方法:

参考:Set up a Database Backend — Airflow Documentation (apache.org),重新配置sqlite

wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
cd sqlite/
export CFLAGS="-DSQLITE_ENABLE_FTS3 \
    -DSQLITE_ENABLE_FTS3_PARENTHESIS \
    -DSQLITE_ENABLE_FTS4 \
    -DSQLITE_ENABLE_FTS5 \
    -DSQLITE_ENABLE_JSON1 \
    -DSQLITE_ENABLE_LOAD_EXTENSION \
    -DSQLITE_ENABLE_RTREE \
    -DSQLITE_ENABLE_STAT4 \
    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
    -DSQLITE_SOUNDEX \
    -DSQLITE_TEMP_STORE=3 \
    -DSQLITE_USE_URI \
    -O2 \
    -fPIC"
export PREFIX="/usr/local"
LIBS="-lm" ./configure --disable-tcl --enable-shared --enable-tempstore=always --prefix="$PREFIX"
make
make install

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

新增到/etc/profile中,并执行source /etc/profile生效

再次启动airflow

airflow standalone

可以看到:

 webserver | ____________       _____________
 webserver | ____    |__( )_________  __/__  /________      __
 webserver | ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
 webserver | ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 webserver | _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
 webserver | [2021-12-03 03:36:04,583] {dagbag.py:500} INFO - Filling up the DagBag from /dev/null
 webserver | [2021-12-03 03:36:04,851] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create Admin
 webserver | [2021-12-03 03:36:07 -0500] [5428] [INFO] Starting gunicorn 20.1.0
 webserver | [2021-12-03 03:36:07 -0500] [5428] [INFO] Listening at: http://0.0.0.0:8080 (5428)
 webserver | [2021-12-03 03:36:07 -0500] [5428] [INFO] Using worker: sync
 webserver | [2021-12-03 03:36:07 -0500] [5433] [INFO] Booting worker with pid: 5433
 webserver | [2021-12-03 03:36:07 -0500] [5434] [INFO] Booting worker with pid: 5434
 webserver | [2021-12-03 03:36:07 -0500] [5435] [INFO] Booting worker with pid: 5435
 webserver | [2021-12-03 03:36:07 -0500] [5436] [INFO] Booting worker with pid: 5436

进入$AIRFLOW_HOME查看admin密码

cat $AIRFLOW_HOME/standalone_admin_password.txt

airflow安装教程(local模式)

airflow安装教程(local模式)

但是请注意,SQLite只适合测试场景,不能使用其作为生产环境部署使用,生产环境部署建议使用Postgres或者MySQL。

0 0 投票数
文章评分

本文为从大数据到人工智能博主「xiaozhch5」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。

原文链接:https://lrting.top/backend/8410/

(0)
上一篇 2022-07-26 19:42
下一篇 2022-07-27 11:21

相关推荐

订阅评论
提醒
guest
0 评论
内联反馈
查看所有评论
0
希望看到您的想法,请您发表评论x