文档章节

python greenlet

MtrS
 MtrS
发布于 2015/04/26 18:18
字数 2372
阅读 120
收藏 2
点赞 0
评论 0

greenlet: Lightweight concurrent programming

Motivation

The “greenlet” package is a spin-off of Stackless, a version of CPython that supports micro-threads called “tasklets”. Tasklets run pseudo-concurrently (typically in a single or a few OS-level threads) and are synchronized with data exchanges on “channels”.

A “greenlet”, on the other hand, is a still more primitive notion of micro-thread with no implicit scheduling; coroutines, in other words. This is useful when you want to control exactly when your code runs. You can build custom scheduled micro-threads on top of greenlet; however, it seems that greenlets are useful on their own as a way to make advanced control flow structures. For example, we can recreate generators; the difference with Python’s own generators is that our generators can call nested functions and the nested functions can yield values too. (Additionally, you don’t need a “yield” keyword. See the example in test/test_generator.py).

Greenlets are provided as a C extension module for the regular unmodified interpreter. Example

Let’s consider a system controlled by a terminal-like console, where the user types commands. Assume that the input comes character by character. In such a system, there will typically be a loop like the following one:

def process_commands(*args): while True: line = '' while not line.endswith('\n'): line += read_next_char() if line == 'quit\n': print "are you sure?" if read_next_char() != 'y': continue # ignore the command process_command(line)

Now assume that you want to plug this program into a GUI. Most GUI toolkits are event-based. They will invoke a call-back for each character the user presses. [Replace “GUI” with “XML expat parser” if that rings more bells to you :-)] In this setting, it is difficult to implement the read_next_char() function needed by the code above. We have two incompatible functions:

def event_keydown(key): ??

def read_next_char(): ?? should wait for the next event_keydown() call

You might consider doing that with threads. Greenlets are an alternate solution that don’t have the related locking and shutdown problems. You start the process_commands() function in its own, separate greenlet, and then you exchange the keypresses with it as follows:

def event_keydown(key): # jump into g_processor, sending it the key g_processor.switch(key)

def read_next_char(): # g_self is g_processor in this simple example g_self = greenlet.getcurrent() # jump to the parent (main) greenlet, waiting for the next key next_char = g_self.parent.switch() return next_char

g_processor = greenlet(process_commands) g_processor.switch(*args) # input arguments to process_commands()

gui.mainloop()

In this example, the execution flow is: when read_next_char() is called, it is part of the g_processor greenlet, so when it switches to its parent greenlet, it resumes execution in the top-level main loop (the GUI). When the GUI calls event_keydown(), it switches to g_processor, which means that the execution jumps back wherever it was suspended in that greenlet – in this case, to the switch() instruction in read_next_char() – and the key argument in event_keydown() is passed as the return value of the switch() in read_next_char().

Note that read_next_char() will be suspended and resumed with its call stack preserved, so that it will itself return to different positions in process_commands() depending on where it was originally called from. This allows the logic of the program to be kept in a nice control-flow way; we don’t have to completely rewrite process_commands() to turn it into a state machine. Usage Introduction

A “greenlet” is a small independent pseudo-thread. Think about it as a small stack of frames; the outermost (bottom) frame is the initial function you called, and the innermost frame is the one in which the greenlet is currently paused. You work with greenlets by creating a number of such stacks and jumping execution between them. Jumps are never implicit: a greenlet must choose to jump to another greenlet, which will cause the former to suspend and the latter to resume where it was suspended. Jumping between greenlets is called “switching”.

When you create a greenlet, it gets an initially empty stack; when you first switch to it, it starts the run a specified function, which may call other functions, switch out of the greenlet, etc. When eventually the outermost function finishes its execution, the greenlet’s stack becomes empty again and the greenlet is “dead”. Greenlets can also die of an uncaught exception.

For example:

from greenlet import greenlet

def test1(): print 12 gr2.switch() print 34

def test2(): print 56 gr1.switch() print 78

gr1 = greenlet(test1) gr2 = greenlet(test2) gr1.switch()

The last line jumps to test1, which prints 12, jumps to test2, prints 56, jumps back into test1, prints 34; and then test1 finishes and gr1 dies. At this point, the execution comes back to the original gr1.switch() call. Note that 78 is never printed. Parents

Let’s see where execution goes when a greenlet dies. Every greenlet has a “parent” greenlet. The parent greenlet is initially the one in which the greenlet was created (this can be changed at any time). The parent is where execution continues when a greenlet dies. This way, greenlets are organized in a tree. Top-level code that doesn’t run in a user-created greenlet runs in the implicit “main” greenlet, which is the root of the tree.

In the above example, both gr1 and gr2 have the main greenlet as a parent. Whenever one of them dies, the execution comes back to “main”.

Uncaught exceptions are propagated into the parent, too. For example, if the above test2() contained a typo, it would generate a NameError that would kill gr2, and the exception would go back directly into “main”. The traceback would show test2, but not test1. Remember, switches are not calls, but transfer of execution between parallel “stack containers”, and the “parent” defines which stack logically comes “below” the current one. Instantiation

greenlet.greenlet is the greenlet type, which supports the following operations:

greenlet(run=None, parent=None) Create a new greenlet object (without running it). run is the callable to invoke, and parent is the parent greenlet, which defaults to the current greenlet. greenlet.getcurrent() Returns the current greenlet (i.e. the one which called this function). greenlet.GreenletExit This special exception does not propagate to the parent greenlet; it can be used to kill a single greenlet.

The greenlet type can be subclassed, too. A greenlet runs by calling its run attribute, which is normally set when the greenlet is created; but for subclasses it also makes sense to define a run method instead of giving a run argument to the constructor. Switching

Switches between greenlets occur when the method switch() of a greenlet is called, in which case execution jumps to the greenlet whose switch() is called, or when a greenlet dies, in which case execution jumps to the parent greenlet. During a switch, an object or an exception is “sent” to the target greenlet; this can be used as a convenient way to pass information between greenlets. For example:

def test1(x, y): z = gr2.switch(x+y) print z

def test2(u): print u gr1.switch(42)

gr1 = greenlet(test1) gr2 = greenlet(test2) gr1.switch("hello", " world")

This prints “hello world” and 42, with the same order of execution as the previous example. Note that the arguments of test1() and test2() are not provided when the greenlet is created, but only the first time someone switches to it.

Here are the precise rules for sending objects around:

g.switch(*args, **kwargs) Switches execution to the greenlet g, sending it the given arguments. As a special case, if g did not start yet, then it will start to run now. Dying greenlet If a greenlet’s run() finishes, its return value is the object sent to its parent. If run() terminates with an exception, the exception is propagated to its parent (unless it is a greenlet.GreenletExit exception, in which case the exception object is caught and returned to the parent).

Apart from the cases described above, the target greenlet normally receives the object as the return value of the call to switch() in which it was previously suspended. Indeed, although a call to switch() does not return immediately, it will still return at some point in the future, when some other greenlet switches back. When this occurs, then execution resumes just after the switch() where it was suspended, and the switch() itself appears to return the object that was just sent. This means that x = g.switch(y) will send the object y to g, and will later put the (unrelated) object that some (unrelated) greenlet passes back to us into x.

Note that any attempt to switch to a dead greenlet actually goes to the dead greenlet’s parent, or its parent’s parent, and so on. (The final parent is the “main” greenlet, which is never dead.) Methods and attributes of greenlets

g.switch(*args, **kwargs) Switches execution to the greenlet g. See above. g.run The callable that g will run when it starts. After g started, this attribute no longer exists. g.parent The parent greenlet. This is writeable, but it is not allowed to create cycles of parents. g.gr_frame The current top frame, or None. g.dead True if g is dead (i.e. it finished its execution). bool(g) True if g is active, False if it is dead or not yet started. g.throw([typ, [val, [tb]]])

Switches execution to the greenlet g, but immediately raises the given exception in g. If no argument is provided, the exception defaults to greenlet.GreenletExit. The normal exception propagation rules apply, as described above. Note that calling this method is almost equivalent to the following:

def raiser():
    raise typ, val, tb
g_raiser = greenlet(raiser, parent=g)
g_raiser.switch()

except that this trick does not work for the greenlet.GreenletExit exception, which would not propagate from g_raiser to g.

Greenlets and Python threads

Greenlets can be combined with Python threads; in this case, each thread contains an independent “main” greenlet with a tree of sub-greenlets. It is not possible to mix or switch between greenlets belonging to different threads. Garbage-collecting live greenlets

If all the references to a greenlet object go away (including the references from the parent attribute of other greenlets), then there is no way to ever switch back to this greenlet. In this case, a GreenletExit exception is generated into the greenlet. This is the only case where a greenlet receives the execution asynchronously. This gives try:finally: blocks a chance to clean up resources held by the greenlet. This feature also enables a programming style in which greenlets are infinite loops waiting for data and processing it. Such loops are automatically interrupted when the last reference to the greenlet goes away.

The greenlet is expected to either die or be resurrected by having a new reference to it stored somewhere; just catching and ignoring the GreenletExit is likely to lead to an infinite loop.

Greenlets do not participate in garbage collection; cycles involving data that is present in a greenlet’s frames will not be detected. Storing references to other greenlets cyclically may lead to leaks. Tracing support

Standard Python tracing and profiling doesn’t work as expected when used with greenlet since stack and frame switching happens on the same Python thread. It is difficult to detect greenlet switching reliably with conventional methods, so to improve support for debugging, tracing and profiling greenlet based code there are new functions in the greenlet module:

greenlet.gettrace() Returns a previously set tracing function, or None. greenlet.settrace(callback)

Sets a new tracing function and returns a previous tracing function, or None. The callback is called on various events and is expected to have the following signature:

def callback(event, args):
    if event == 'switch':
        origin, target = args
        # Handle a switch from origin to target.
        # Note that callback is running in the context of target
        # greenlet and any exceptions will be passed as if
        # target.throw() was used instead of a switch.
        return
    if event == 'throw':
        origin, target = args
        # Handle a throw from origin to target.
        # Note that callback is running in the context of target
        # greenlet and any exceptions will replace the original, as
        # if target.throw() was used with the replacing exception.
        return

For compatibility it is very important to unpack args tuple only when event is either 'switch' or 'throw' and not when event is potentially something else. This way API can be extended to new events similar to sys.settrace().

C API Reference

Greenlets can be created and manipulated from extension modules written in C or C++, or from applications that embed Python. The greenlet.h header is provided, and exposes the entire API available to pure Python modules. Types Type name Python name PyGreenlet greenlet.greenlet Exceptions Type name Python name PyExc_GreenletError greenlet.error PyExc_GreenletExit greenlet.GreenletExit Reference

PyGreenlet_Import() A macro that imports the greenlet module and initializes the C API. This must be called once for each extension module that uses the greenlet C API. int PyGreenlet_Check(PyObject *p) Macro that returns true if the argument is a PyGreenlet. int PyGreenlet_STARTED(PyGreenlet *g) Macro that returns true if the greenlet g has started. int PyGreenlet_ACTIVE(PyGreenlet *g) Macro that returns true if the greenlet g has started and has not died. PyGreenlet *PyGreenlet_GET_PARENT(PyGreenlet *g) Macro that returns the parent greenlet of g. int PyGreenlet_SetParent(PyGreenlet *g, PyGreenlet *nparent) Set the parent greenlet of g. Returns 0 for success. If -1 is returned, then g is not a pointer to a PyGreenlet, and an AttributeError will be raised. PyGreenlet *PyGreenlet_GetCurrent(void) Returns the currently active greenlet object. PyGreenlet *PyGreenlet_New(PyObject *run, PyObject *parent) Creates a new greenlet object with the callable run and parent parent. Both parameters are optional. If run is NULL, then the greenlet will be created, but will fail if switched in. If parent is NULL, the parent is automatically set to the current greenlet. PyObject *PyGreenlet_Switch(PyGreenlet *g, PyObject *args, PyObject *kwargs) Switches to the greenlet g. args and kwargs are optional and can be NULL. If args is NULL, an empty tuple is passed to the target greenlet. If kwargs is NULL, no keyword arguments are passed to the target greenlet. If arguments are specified, args should be a tuple and kwargs should be a dict. PyObject *PyGreenlet_Throw(PyGreenlet *g, PyObject *typ, PyObject *val, PyObject *tb) Switches to greenlet g, but immediately raise an exception of type typ with the value val, and optionally, the traceback object tb. tb can be NULL.

© 著作权归作者所有

共有 人打赏支持
MtrS
粉丝 31
博文 549
码字总数 352132
作品 0
榆林
Ubuntu 安装gevent

Gevent是一个基于greenlet的Python的并发框架,以赖于greenlet和libevent库,因此安装Gevent前,首先需要安装greenlet和libevent。 libevent的安装,这里就不介绍了,网上的安装教程一大箩,...

jackliu8722
2012/07/16
0
0
ubutun 下安装wsshd

wssh 可以让我们通过 HTTP 来调用远程的一个 shell,也就是说我们可以用浏览器来访问某个 Linux 服务器/虚拟机的终端(只要这个服务器上运行了 wsshd 服务器端) 1:需要软件: $ sudo apt...

七月杭州
2014/05/21
0
0
Cent os 下 libevent,greenlet,gevent 的安装 和介绍 记录

libevent libevent是一个事件触发的网络库,适用于windows、linux、bsd等多种平台,内部使用 select、epoll、kqueue等系统调用管理事件机制。著名分布式缓存软件memcached也是libevent base...

G0561
2014/04/05
0
0
openstack文件存储swift 2.15.1在CentOS7.4下的安装

下载swift : 注意 swift-2.15.1.tar.gz在 https://github.com/openstack/swift/releases下载的话,安装有问题。请在: https://tarballs.openstack.org/swift/swift-2.15.1.tar.gz下载 2.安装......

MichaelShu
2017/10/30
0
0
【整理】Python之JIT、Django、Greenlet和Stackless

【JIT】 即时编译(Just-in-time compilation),又称为动态编译,是一种提高程序运行效率的方法。 通常程序有两种编译方式:静态编译与动态编译(直译)。在静态编译中,程序在执行前全部被翻...

摩云飞
2012/11/06
0
0
python模块介绍-gevent介绍:基于协程的网络库

python模块介绍-gevent介绍:基于协程的网络库 介绍 gevent是基于协程的Python网络库。特点: 基于libev的快速事件循环(Linux上epoll,FreeBSD上kqueue)。 基于greenlet的轻量级执行单元。 ...

磁针石
2014/01/13
0
2
python,限制任意函数,线程的执行时间或根据条件终止.

想写一个通用的decorator限制一个函数的执行时间. 首先想到的就是把函数执行封在一个线程了,执行结果写到线程的属性里,join指定时间后读线程的属性并返回.问题在于若函数没有执行完,读不到属...

斑ban
2013/04/15
0
0
Python的并发编程工具包 Pykka

Pykka 项目的目的是为 Python 提供了易用的并发操作抽象层,使用 Actor 模式。 Pykka 提供了 Actor API ,包含两种不同的实现: ThreadingActor is built on the Python Standard Library’s...

openthings
2015/01/26
0
0
多进程(multiprocessing module)

一、多进程 1.1 多进程的概念   由于GIL的存在,python中的多线程其实并不是真正的多线程,如果想要充分地使用多核CPU的资源,在python中大部分情况需要使用多进程。Python提供了非常好用的...

Hme
昨天
0
0
Python gevent学习笔记 2

在上一篇里了解了gevent应用的IO模型概念之后,接下来开始真正了解gevent的使用。 Greenlet 在gevent里面最多应用到的就是greenlet,一个轻量级的协程实现。在任何时间点,只有一个greenlet处...

贱圣
2013/06/30
0
0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

20位活跃在Github上的国内技术大牛 leij 何小鹏 亚信

本文列举了20位在Github上非常活跃的国内大牛,看看其中是不是很多熟悉的面孔? 1. lifesinger(玉伯) Github主页: https://github.com/lifesinger 微博:@ 玉伯也叫射雕 玉伯(王保平),...

海博1600
11分钟前
0
0
高性能服务器本质论

一 服务器分类 从软件性能角度,高性能服务器分:cpu密集型服务器/IO密集型服务器 (1)CPU密集型:该类服务器没有对io的访问/没有同步点,性能瓶颈在于对cpu的充分利用。 典型的如转发服务器/...

码代码的小司机
12分钟前
0
0
Mybatis收集配置

一、Mybatis取Clob数据 1、Mapper.xml配置 <resultMap type="com.test.User" id="user"> <result column="id" property="id"/> <result column="json_data" property="jsonData" ......

星痕2018
37分钟前
0
0
centos7设置以多用户模式启动

1、旧版本linux系统修改inittab文件,在新版本执行vi /etc/inittab 会有以下提示 # inittab is no longer used when using systemd. # # ADDING CONFIGURATION HERE WILL HAVE NO EFFECT ON......

haha360
今天
0
0
OSChina 周日乱弹 —— 局长:怕你不爱我

Osc乱弹歌单(2018)请戳(这里) 【今日歌曲】 @ andonny :分享周二珂的单曲《孤独她呀》 《孤独她呀》- 周二珂 手机党少年们想听歌,请使劲儿戳(这里) @孤星闵月 :没事干,看一遍红楼梦...

小小编辑
今天
192
9
Java架构师知识体认识

源码分析 常用设计模式 Proxy代理模式 Factory工厂模式 Singleton单例模式 Delegate委派模式 Strategy策略模式 Prototype原型模式 Template模板模式 Spring5 beans 接口实例化 代理Bean操作 ...

小致dad
今天
0
0
SpringBoot | 第十章:Swagger2的集成和使用

前言 前一章节介绍了mybatisPlus的集成和简单使用,本章节开始接着上一章节的用户表,进行Swagger2的集成。现在都奉行前后端分离开发和微服务大行其道,分微服务及前后端分离后,前后端开发的...

oKong
今天
11
0
Python 最小二乘法 拟合 二次曲线

Python 二次拟合 随机生成数据,并且加上噪声干扰 构造需要拟合的函数形式,使用最小二乘法进行拟合 输出拟合后的参数 将拟合后的函数与原始数据绘图后进行对比 import numpy as npimport...

阿豪boy
今天
17
0
云拿 无人便利店

附近(上海市-航南路)开了家无人便利店.特意进去体验了一下.下面把自己看到的跟大家分享下. 经得现场工作人员同意后拍了几张照片.从外面看是这样.店门口的指导里强调:不要一次扫码多个人进入....

周翔
昨天
1
0
Java设计模式学习之工厂模式

在Java(或者叫做面向对象语言)的世界中,工厂模式被广泛应用于项目中,也许你并没有听说过,不过也许你已经在使用了。 简单来说,工厂模式的出现源于增加程序序的可扩展性,降低耦合度。之...

路小磊
昨天
254
1

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部