文档章节

Redis performance --- delete 100 records at maximum

Ocean_K
 Ocean_K
发布于 04/03 10:38
字数 479
阅读 6
收藏 0

It deletes login token and corresponding data if there is more than 10 million records

QUIT = False
LIMIT = 10000000

def clean_session:
  while not QUIT:
    size = conn.zcard('recent:')
    if size <= LIMIT:
      time.sleep(1)
      continue

    # find out the range in `recent:` ZSET
    end_index = min(size-LIMIT, 100)
    tokens = conn.zrange('recent:', 0, end_index-1)

    # delete corresponding data
    session_keys = []
    for token in tokens:
      session_keys.append('viewed:' + token)

    conn.delete(*session_keys)
    conn.hdel('login:', *tokens)
    conn.zrem('recent:', *tokens)

 the question is: 

  • why delete 100 records at most per time?

  • why not just delete size - LIMIT records at once?

  • is there some performance consideration?

 

A:

I guess there are multiple reasons for that choice.

Redis is a single-threaded event loop. It means a large command (for instance a large zrange, or a large del, hdel or zrem) will be processed faster than several small commands, but with an impact on the latency for the other sessions. If a large command takes one second to execute, all the clients accessing Redis will be blocked for one second as well.

A first reason is therefore to minimize the impact of these cleaning operations on the other client processes. By segmenting the activity in several small commands, it gives a chance to other clients to execute their commands as well.

A second reason is the size of the communication buffers in Redis server. A large command (or a large reply) may take a lot of memory. If millions of items are to be cleaned out, the reply of the lrange command or the input of the del, hdel, zrem commands can represent megabytes of data. Past a certain limit, Redis will close the connection to protect itself. So it is better to avoid dealing with very large commands or very large replies.

A third reason is the memory of the Python client. If millions of items have to be cleaned out, Python will have to maintain very large list objects (tokens and session_keys). They may or may not fit in memory.

The proposed solution is incremental: whatever the number of items to delete, it will avoid consuming a lot of memory on both client and Redis sides. It will also avoid to hit the communication buffer limit (resulting in the connection to be closed), and will limit the impact on the performance of the other processes accessing Redis.

Note that the 100 value is arbitrary. A smaller value will allow for better latencies at the price of a lower session cleaning throughput. A larger value will increase the throughput of the cleaning algorithm at the price of higher latencies.

It is actually a classical trade-off between the throughput of the cleaning algorithm, and the latency of other operations.

本文转载自:https://stackoverflow.com/questions/24015330/redis-performance-delete-100-records-at-maximum

Ocean_K
粉丝 19
博文 104
码字总数 40133
作品 0
昌平
程序员
私信 提问
ORA-28056: Writing audit records to Windows Event Log failed

C:\>sqlplus / as sysdba SQL*Plus: Release 11.2.0.1.0 Production on 星期四 1月 5 13:55:44 2012 Copyright (c) 1982, 2010, Oracle. All rights reserved. ERROR: ORA-28056: Writing a......

晨曦之光
2012/04/25
539
0
How-to: Tune Your Apache Spark Jobs (Part 2)

In the conclusion to this series, learn how resource tuning, parallelism, and data representation affect Spark job performance. In this post, we’ll finish what we started in “......

Si2
2016/01/17
29
0
redis性能测试,测试并发性能

redis数据库windows客户端下载地址:https://github.com/dmajkic/redis/downloads 原文链接:http://redis.io/topics/benchmarks How fast is Redis? Redis includes the redis-benchmark u......

格鲁特
2013/01/16
0
2
standby 磁盘IO性能较差,影响Primary性能

1. 近日处理一个由于standby 磁盘IO性能较差,导致Primary的性能受到影响。 主库主要是等待"log file switch completion",通过ASH dump分析,最终发现实际等待事件是"LGWR-LNS wait on cha...

hsbxxl
2016/09/05
0
0
Redis 2.8.0 正式版发布

NoSQL Redis 2.8.0正式版发布!新的产品线。2013-11-22 高性能KV数据库。经过6个RC.上个版本是2013-08-29的2.6.16. 对比2.6所有新特性如下: * [NEW] Slaves are now able to partially res...

fei
2013/11/22
6.7K
25

没有更多内容

加载失败,请刷新页面

加载更多

IT兄弟连 Java语法教程 编写Java源代码

现在我们来一步一步的编写第一个Java程序,鼎鼎大名的“HelloWorld”。 编写Java源代码 编写Java源代码可以使用任何无格式的文本编辑器,在Windows操作系统上可以使用记事本、Edit Plus等程序...

老码农的一亩三分地
10分钟前
2
0
JavaScript箭头函数中的this详解

前言 箭头函数极大地简化了this的取值规则。 普通函数与箭头函数 普通函数指的是用function定义的函数: var hello = function () {console.log("Hello, Fundebug!");} 箭头函数指的是用...

开元中国2015
21分钟前
2
0
ETL测试工具简介

ETL测试 ETL测试过程与其他测试过程类似,包括一些阶段。 确定业务需求 测试计划 设计测试用例和测试数据 测试执行和错误报告 总结报告 测试结束 ETL测试的类型 生产验证测试: 也称为表平衡...

python测试开发人工智能安全
34分钟前
1
0
OSChina 周四乱弹 —— 不能空腹吃早餐

Osc乱弹歌单(2019)请戳(这里) 【今日歌曲】 @for_ :#今日歌曲推荐# 很好听的钢琴曲,节奏简单,有一点伤感。分享Fabrizio Paterlini的单曲《Veloma》: 《Veloma》- Fabrizio Paterlini ...

小小编辑
今天
1K
13
探讨android更新UI的几种方法

作为IT新手,总以为只要有时间,有精力,什么东西都能做出来。这种念头我也有过,但很快就熄灭了,因为现实是残酷的,就算一开始的时间和精力非常充足,也会随着项目的推进而逐步消磨殆尽。我...

天王盖地虎626
今天
8
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部