文档章节

python分布式调度celery无法启动或自动退出排查步骤

testwork
 testwork
发布于 2017/01/16 13:22
字数 1369
阅读 1059
收藏 0

Celery是个好东西,但是不具体去分析,等出现错误会显得手忙脚乱

1) 启动调试方法

    celery worker -c 2 -A project -B -l debug -f /var/log/project.log

    日志基本没有,控制台就是一些debug和warning日志

[2017-01-16 13:06:35,449: INFO/Beat] beat: Starting...
[2017-01-16 13:06:35,461: INFO/MainProcess] Connected to amqp://username:**@127.0.0.1:5672/
[2017-01-16 13:06:35,482: DEBUG/Beat] Current schedule:
<Entry: 任务名称 <crontab: */3 * * * * (m/h/d/dM/MY)>
[2017-01-16 13:06:35,482: DEBUG/Beat] beat: Ticking with max interval->5.00 minutes
[2017-01-16 13:06:35,483: INFO/MainProcess] mingle: searching for neighbors
[2017-01-16 13:06:35,503: INFO/Beat] Scheduler: Sending due task 任务名称 (任务名称)
[2017-01-16 13:06:36,500: INFO/MainProcess] mingle: all alone
[2017-01-16 13:06:36,528: WARNING/MainProcess] celery@localhost ready.
[2017-01-16 13:06:36,559: DEBUG/MainProcess] Canceling task consumer...
[2017-01-16 13:06:36,561: INFO/MainProcess] beat: Shutting down...
[2017-01-16 13:06:37,587: DEBUG/MainProcess] Canceling task consumer...
[2017-01-16 13:06:37,588: DEBUG/MainProcess] Closing consumer channel...

然后怎么办,懵逼了...

这个时候,不能随便猜了,不信你可要猜一晚上试试

去git上下celery的源码,搜索关键字

Canceling task consumer

估计原因是在xx地方报错了,然后打印了2行日志出来.

找到代码在worker/consumer.py里面

   def shutdown(self, c):
        if c.task_consumer:
            import traceback
            debug("start shutdown traceback ====")
            traceback.print_stack()
            self.stop(c)
            debug('Closing consumer channel...')
            ignore_errors(c, c.task_consumer.close)
            c.task_consumer = None

添加一个traceback,打印出从哪里调用进来的.

再启动一下celery,你就会发现不一样了

[2017-01-16 13:06:37,579: WARNING/MainProcess] File "/usr/local/bin/celery", line 9, in <module>
[2017-01-16 13:06:37,580: WARNING/MainProcess] load_entry_point('celery==3.1.23', 'console_scripts', 'celery')()
[2017-01-16 13:06:37,580: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
[2017-01-16 13:06:37,580: WARNING/MainProcess] main()
[2017-01-16 13:06:37,580: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
[2017-01-16 13:06:37,581: WARNING/MainProcess] cmd.execute_from_commandline(argv)
[2017-01-16 13:06:37,581: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
[2017-01-16 13:06:37,581: WARNING/MainProcess] super(CeleryCommand, self).execute_from_commandline(argv)))
[2017-01-16 13:06:37,581: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 311, in execute_from_commandline
[2017-01-16 13:06:37,581: WARNING/MainProcess] return self.handle_argv(self.prog_name, argv[1:])
[2017-01-16 13:06:37,582: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 785, in handle_argv
[2017-01-16 13:06:37,582: WARNING/MainProcess] return self.execute(command, argv)
[2017-01-16 13:06:37,582: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 717, in execute
[2017-01-16 13:06:37,582: WARNING/MainProcess] ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 179, in run_from_argv
[2017-01-16 13:06:37,583: WARNING/MainProcess] return self(*args, **options)
[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 274, in __call__
[2017-01-16 13:06:37,583: WARNING/MainProcess] ret = self.run(*args, **kwargs)
[2017-01-16 13:06:37,583: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bin/worker.py", line 212, in run
[2017-01-16 13:06:37,584: WARNING/MainProcess] state_db=self.node_format(state_db, hostname), **kwargs
[2017-01-16 13:06:37,584: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 211, in start
[2017-01-16 13:06:37,584: WARNING/MainProcess] self.stop()
[2017-01-16 13:06:37,584: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 252, in stop

[2017-01-16 13:06:37,584: WARNING/MainProcess] self._shutdown(warm=True)
[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 266, in _shutdown
[2017-01-16 13:06:37,585: WARNING/MainProcess] self.blueprint.stop(self, terminate=not warm)
[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 183, in stop
[2017-01-16 13:06:37,585: WARNING/MainProcess] self.on_stopped()
[2017-01-16 13:06:37,585: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 164, in on_stopped
[2017-01-16 13:06:37,586: WARNING/MainProcess] self.consumer.shutdown()
[2017-01-16 13:06:37,586: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 306, in shutdown
[2017-01-16 13:06:37,586: WARNING/MainProcess] self.blueprint.shutdown(self)
[2017-01-16 13:06:37,586: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 171, in shutdown
[2017-01-16 13:06:37,586: WARNING/MainProcess] self.send_all(parent, 'shutdown')
[2017-01-16 13:06:37,587: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 155, in send_all
[2017-01-16 13:06:37,587: WARNING/MainProcess] fun(parent, *args)
[2017-01-16 13:06:37,587: WARNING/MainProcess] File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 628, in shutdown
[2017-01-16 13:06:37,587: WARNING/MainProcess] traceback.print_stack()

怀疑: start之后为啥马上就stop了.奇怪.

vim /usr/local/lib/python2.7/site-packages/celery/worker/__init__.py

到211行

    def start(self):
        """Starts the workers main loop."""
        try:
            self.blueprint.start(self)
        except WorkerTerminate:
            self.terminate()
        except Exception as exc:
            logger.error('Unrecoverable error: %r', exc, exc_info=True)
            self.stop()
        except (KeyboardInterrupt, SystemExit):
            self.stop()

为啥这地方使用了logger.error,我们没有看到日志呢,艹

剩下的就是想办法吧日志全打印出来,

通过代码跟踪需要配置2个日志名称celery.worker和celery.task

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'direct': {
            'format': '%(asctime)s - %(message)s'
        },
        'json': {
            'format': '%(message)s'
        },
    },
    'handlers': {
        'celery.worker': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/celery.worker.log',
            'formatter': 'direct'
        },
        'celery.task': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': '/var/log/celery.task.log',
            'formatter': 'direct'
        },
    },
    'loggers': {
        'celery.worker': {
            'handlers': ['celery.worker'],
            'level': 'DEBUG'
        },
        'celery.task': {
            'handlers': ['celery.task'],
            'level': 'DEBUG'
        },
    }
}

配置上面,

然后启动celery,

[2017-01-16 13:06:36,550: ERROR/MainProcess] Unrecoverable error: WorkerLostError('Could not start worker processes',)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 279, in start
    blueprint.start(self)
  File "/usr/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/usr/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 841, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python2.7/site-packages/celery/worker/loops.py", line 48, in asynloop
    raise WorkerLostError('Could not start worker processes')
WorkerLostError: Could not start worker processes

然后你会发现不同环境的这个错误是不一样的.

celery worker -c 2 -A project -l debug -P gevent

添加一个参数-P gevent能够正常启动了

查看一下-B参数的说明
  -B, --beat            Also run the celery beat periodic task scheduler.
                        Please note that there must only be one instance of
                        this service

© 著作权归作者所有

testwork
粉丝 4
博文 75
码字总数 21164
作品 0
崇明
程序员
私信 提问
python任务调度模块celery

python任务调度模块celery celery简介 Celery是一个python开发的异步分布式任务调度模块。 Celery本身并不提供消息服务,使用第三方服务,也就是borker来传递任务,一般使用rabbitMQ或者Red...

laoba
2018/05/28
626
0
Django框架17: Celery的使用

Celery介绍 1.什么是Celery Celery是一个python模块,它在官网的定义:Celery is asynchronous task queue/job based on distributed message passing. It is focused on real-time operatio......

代码打碟手
2018/09/26
150
0
Python celery简介

Celery异步分布式 什么是celery? 他是一个python开发的异步分布式任务调度模块 celery本身不提供消息服务,使用第三方服务,也就是broker来传递任务,目前支持rabbitmq,redis,数据库等等。...

lemonwater
2018/08/15
60
0
python-celery使用教程

Celery Celery是Python开发的分布式任务调度模块。分为任务分发,任务队列,worker3个部分。celery的出现,解决了python运行后台任务的需求。 这篇文章介绍的celery版本是3.1.18 celery架构 ...

go-skyblue
2015/07/23
1K
0
Python 并行分布式框架之 Celery

Celery (芹菜)是基于Python开发的分布式任务队列。它支持使用任务队列的方式在分布的机器/进程/线程上执行任务调度。 架构设计 Celery的架构由三部分组成,消息中间件(message broker)...

naughty
2015/03/12
4.2K
6

没有更多内容

加载失败,请刷新页面

加载更多

log4j起不来: No appenders could be found for logger

在mvn test时,log4j一起起不来,log message打不出来 原因: log4j.properties 文件没不存在,或存在,但放错路径 解:把log4j.properties放入mvn 默认的resources跑路径下面:./src/main/...

Rebecca_Hu
30分钟前
5
0
ETH 开发工作记录

测试是否连接成功 String web3ClientVersion = web3j.web3ClientVersion().send().getWeb3ClientVersion(); log.info("version=" + web3ClientVersion);...

xiaodong16
32分钟前
3
0
ntpserver配置

# For more information about this file, see the man pages # ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5). driftfile /var/lib/ntp/drift # Permit ti......

Archer99
38分钟前
3
0
技术分享 | delete 语句引发大量 sql 被 kill 问题分析

作者:王航威 有赞 MySQL DBA,擅长分析和解决数据库的性能问题,利用自动化工具解决日常需求。 现象 某个数据库经常在某个时间点比如凌晨 2 点或者白天某些时间段发出如下报警 [Critical][p...

爱可生
43分钟前
5
0
Spring Boot 2.X(十六):应用监控之 Spring Boot Actuator 使用及配置

Actuator 简介 Actuator 是 Spring Boot 提供的对应用系统的自省和监控功能。通过 Actuator,可以使用数据化的指标去度量应用的运行情况,比如查看服务器的磁盘、内存、CPU等信息,系统的线程...

朝雾轻寒
47分钟前
4
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部