Redis data structure design for sorting time-based values

原创
2016/12/06 10:42
阅读数 123

Question:

I'm performing some analysis on a data stream and publishing the results on a Redis channel. Consumers subscribe to these channels and get real-time data feeds. All historical data analysis results are lost.

Now I want to add the ability to store historical data in Redis so that consumers can query this historical data (mainly by time). Since the analysis results are partitioned by time what would be the a good design to store the results in Redis?

Answer:

Use redis sorted sets.

Sorted sets store data based on "scores", so in your case, just use a time stamp in millis; the data will be sorted automatically, allowing you to retrieve historical items using start/end date ranges, here's an example...

Add items to a sorted set...

zadd historical <timestamp> <dataValue>

..add some sample data..

 zadd historical 1 data1
 zadd historical 2 data2
 zadd historical 3 data3
 zadd historical 4 data4
 zadd historical 5 data5
 zadd historical 6 data6
 zadd historical 7 data7

..retrieve a subset of items using start/end range...

 zrangebyscore historical 2 5

..returns...

1) "data2"
2) "data3"
3) "data4"
4) "data5"

So, in your case, if you want to retrieve all historical items for the last day, just do this...

zrangebyscore historical <currentTimeInMillis> <currentTimeInMillis - 86400000>

Additional

(1) What if two data inserts in the same milisecond?

While members are unique in a sorted set, scores (time stamps in this case) may be repeated.

(2) What if each data value is not unique?

you can add some random string before or after your data to keep it not repeated.

I think in your last code example you'd want to reverse the range, i.e. zrangebyscore historical <currentTimeInMillis - 86400000> <currentTimeInMillis> since it looks like it expects the lower value first.

Redis ZRANGEBYSCORE command returns all the elements in the sorted set at key with a score between min and max (including elements with score equal to min or max). The elements are considered to be ordered from low to high scores. The elements having the same score are returned in lexicographical order (this follows from a property of the sorted set implementation in Redis and does not involve further computation).

Reference:http://stackoverflow.com/questions/17153154/redis-data-structure-design-for-sorting-time-based-values

展开阅读全文
加载中
点击引领话题📣 发布并加入讨论🔥
0 评论
1 收藏
0
分享
返回顶部
顶部