文档章节

Redis performance --- delete 100 records at maximum

Ocean_K
 Ocean_K
发布于 2019/04/03 10:38
字数 479
阅读 80
收藏 0

It deletes login token and corresponding data if there is more than 10 million records

QUIT = False
LIMIT = 10000000

def clean_session:
  while not QUIT:
    size = conn.zcard('recent:')
    if size <= LIMIT:
      time.sleep(1)
      continue

    # find out the range in `recent:` ZSET
    end_index = min(size-LIMIT, 100)
    tokens = conn.zrange('recent:', 0, end_index-1)

    # delete corresponding data
    session_keys = []
    for token in tokens:
      session_keys.append('viewed:' + token)

    conn.delete(*session_keys)
    conn.hdel('login:', *tokens)
    conn.zrem('recent:', *tokens)

 the question is: 

  • why delete 100 records at most per time?

  • why not just delete size - LIMIT records at once?

  • is there some performance consideration?

 

A:

I guess there are multiple reasons for that choice.

Redis is a single-threaded event loop. It means a large command (for instance a large zrange, or a large del, hdel or zrem) will be processed faster than several small commands, but with an impact on the latency for the other sessions. If a large command takes one second to execute, all the clients accessing Redis will be blocked for one second as well.

A first reason is therefore to minimize the impact of these cleaning operations on the other client processes. By segmenting the activity in several small commands, it gives a chance to other clients to execute their commands as well.

A second reason is the size of the communication buffers in Redis server. A large command (or a large reply) may take a lot of memory. If millions of items are to be cleaned out, the reply of the lrange command or the input of the del, hdel, zrem commands can represent megabytes of data. Past a certain limit, Redis will close the connection to protect itself. So it is better to avoid dealing with very large commands or very large replies.

A third reason is the memory of the Python client. If millions of items have to be cleaned out, Python will have to maintain very large list objects (tokens and session_keys). They may or may not fit in memory.

The proposed solution is incremental: whatever the number of items to delete, it will avoid consuming a lot of memory on both client and Redis sides. It will also avoid to hit the communication buffer limit (resulting in the connection to be closed), and will limit the impact on the performance of the other processes accessing Redis.

Note that the 100 value is arbitrary. A smaller value will allow for better latencies at the price of a lower session cleaning throughput. A larger value will increase the throughput of the cleaning algorithm at the price of higher latencies.

It is actually a classical trade-off between the throughput of the cleaning algorithm, and the latency of other operations.

Ocean_K
粉丝 28
博文 107
码字总数 40133
作品 0
昌平
程序员
私信 提问
加载中
请先登录后再评论。

暂无文章

如何处理ASP.NET MVC Framework中的多个提交按钮?

问题: Is there some easy way to handle multiple submit buttons from the same form? 是否有一些简单的方法可以处理来自同一表单的多个提交按钮? Example: 例: <% Html.BeginForm("MyA......

技术盛宴
28分钟前
6
0
Android NDK 工具链的使用方法(Standalone Toolchain)

首先需要确定目标机器的指令集。 如果是 x86 的机器,用 x86-4.4.3 版本的工具链;如果是 arm 指令的,用 arm-linux-androideabi-4.4.3 版本 (x86-4.4.3 和 arm-linux-androideabi-4.4.3 位于...

shzwork
今天
29
0
给定两个目录树,如何找出哪些文件有所不同? - Given two directory trees, how can I find out which files differ?

问题: If I want find the differences between two directory trees, I usually just execute: 如果我想找到两个目录树之间的差异,我通常只执行: diff -r dir1/ dir2/ This outputs ex......

法国红酒甜
今天
17
0
互联网领域唯一一家!百度牵头“工业智能实验室”入选工业互联网产业联盟实验室首批名单

本文作者:y****n 6月30日,工业互联网产业联盟首批实验室名单公布,百度牵头申报的“工业智能实验室”成功入选,百度在工业互联网领域领先的技术实力获得业界认可。 工业互联网产业联盟(简...

百度开发者中心
昨天
13
0
jQuery Ajax文件上传 - jQuery Ajax File Upload

问题: Can I use the following jQuery code to perform file upload using POST method of an ajax request ? 我可以使用以下jQuery代码使用ajax请求的POST方法执行文件上传吗? $.ajax({......

javail
今天
27
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部