文档章节

定期删除elasticsearch的过期index

Mark93
 Mark93
发布于 2016/12/12 16:36
字数 1718
阅读 540
收藏 2

环境要求:

1. elasticsearch curator(与zookeeper curator不一样)

2. crontab

安装curator

安装pip

[root@es1 ~]# wget https://bootstrap.pypa.io/get-pip.py
--2016-12-12 11:18:34--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io... 151.101.16.175
Connecting to bootstrap.pypa.io|151.101.16.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1595408 (1.5M) [text/x-python]
Saving to: et-pip.py

100%[====================================================================================================================>] 1,595,408   9.40K/s   in 2m 3s   


[root@es1 ~]# python get-pip.py 
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting pip
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% || 1.3MB 58kB/s 
Collecting setuptools
  Downloading setuptools-31.0.0-py2.py3-none-any.whl (477kB)
    100% || 481kB 55kB/s 
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% || 71kB 35kB/s 
Collecting argparse; python_version == "2.6" (from wheel)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, argparse, wheel
Successfully installed argparse-1.4.0 pip-9.0.1 setuptools-31.0.0 wheel-0.29.0
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.

安装curator

[root@es1 ~]# pip install elasticsearch-curator
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting elasticsearch-curator
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading elasticsearch-curator-4.2.4.tar.gz (134kB)
    100% || 143kB 54kB/s 
Collecting elasticsearch<3.0.0,>=2.4.0 (from elasticsearch-curator)
  Downloading elasticsearch-2.4.0-py2.py3-none-any.whl (54kB)
    100% || 61kB 64kB/s 
Collecting click>=6.0 (from elasticsearch-curator)
  Downloading click-6.6-py2.py3-none-any.whl (71kB)
    100% || 71kB 43kB/s 
Collecting pyyaml>=3.10 (from elasticsearch-curator)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% || 256kB 55kB/s 
Collecting voluptuous>=0.9.3 (from elasticsearch-curator)
  Downloading voluptuous-0.9.3.tar.gz
Collecting certifi>=2016.09.26 (from elasticsearch-curator)
  Downloading certifi-2016.9.26-py2.py3-none-any.whl (377kB)
    100% || 378kB 58kB/s 
Collecting urllib3<2.0,>=1.8 (from elasticsearch<3.0.0,>=2.4.0->elasticsearch-curator)
  Downloading urllib3-1.19.1-py2.py3-none-any.whl (104kB)
    100% || 112kB 48kB/s 
Requirement already satisfied: setuptools>=0.6b1 in /usr/lib/python2.6/site-packages (from voluptuous>=0.9.3->elasticsearch-curator)
Building wheels for collected packages: elasticsearch-curator, pyyaml, voluptuous
  Running setup.py bdist_wheel for elasticsearch-curator ... done
  Stored in directory: /root/.cache/pip/wheels/33/46/ac/5e17bb2d61b2139011742f37ca0a2d7fde039cca3d51b574f8
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for voluptuous ... done
  Stored in directory: /root/.cache/pip/wheels/4b/cc/a6/c1683326047075d71cdd74c966300efbfeda143fdf6c5f7587
Successfully built elasticsearch-curator pyyaml voluptuous
Installing collected packages: urllib3, elasticsearch, click, pyyaml, voluptuous, certifi, elasticsearch-curator
Successfully installed certifi-2016.9.26 click-6.6 elasticsearch-2.4.0 elasticsearch-curator-4.2.4 pyyaml-3.12 urllib3-1.19.1 voluptuous-0.9.3

检查curator

[root@es1 ~]# curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.


[root@es1 ~]# curator_cli --help
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

Options:
  --config PATH       Path to configuration file. Default:
                      ~/.curator/curator.yml
  --host TEXT         Elasticsearch host.
  --url_prefix TEXT   Elasticsearch http url prefix.
  --port TEXT         Elasticsearch port.
  --use_ssl           Connect to Elasticsearch through SSL.
  --certificate TEXT  Path to certificate to use for SSL validation.
  --client-cert TEXT  Path to file containing SSL certificate for client auth.
  --client-key TEXT   Path to file containing SSL key for client auth.
  --ssl-no-validate   Do not validate SSL certificate
  --http_auth TEXT    Use Basic Authentication ex: user:pass
  --timeout INTEGER   Connection timeout in seconds.
  --master-only       Only operate on elected master node.
  --dry-run           Do not perform any changes.
  --loglevel TEXT     Log level
  --logfile TEXT      log file
  --logformat TEXT    Log output format [default|logstash|json].
  --version           Show the version and exit.
  --help              Show this message and exit.

Commands:
  allocation        Shard Routing Allocation
  close             Close indices
  delete_indices    Delete indices
  delete_snapshots  Delete snapshots
  forcemerge        forceMerge index/shard segments
  open              Open indices
  replicas          Change replica count
  show_indices      Show indices
  show_snapshots    Show snapshots
  snapshot          Snapshot indices

配置curator

配置客户端连接

# curator_conf.yml

client:
    hosts: ["172.16.1.42:9200"]
    url_prefix: 
    use_ssl: False
    certificate: 
    client_cert: 
    client_key: 
    ssl_no_validate: False
    http_auth: 
    timeout: 30
    master_only: False

logging:
    loglevel: INFO
    logfile: /data/apps/curator/curator.log 
    logformat: default
    blacklist: ['elasticsearch','urllib3']

配置执行动作

# curator_actions.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 3 days (based on index name), for baihe_ha-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False 
    filters:
    - filtertype: pattern
      kind: prefix
      value: bh_ha-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 3
      exclude:
  2:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

测试curator脚本

curator --config ./curator_conf.yml ./curator_actions.yml

配置crontab

编写执行脚本

#!/bin/sh
/usr/bin/curator --config /data/apps/curator/curator_conf.yml /data/apps/curator/curator_actions.yml

#curator_start.sh

配置执行规则

crontab -e:打开了vi,输入:

30 5 * * * /data/apps/curator/curator-start.sh,之后保存退出vi

crontab -l:查看所有的root用户的定时任务
 

附:crontab的文件格式

分 时 日 月 星期 要运行的命令

第1列分钟1~59
第2列小时1~23(0表示子夜)
第3列日1~31
第4列月1~12
第5列星期0~7(0和7表示星期天)
第6列要运行的命令

troubleshooting

[root@S3-202 ~]# curator --help
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 631, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2489, in requires
    dm = self._dep_map
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2700, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2717, in _compute_dependencies
    from _markerlib import compile as compile_marker
ImportError: No module named _markerlib

这并不是需要安装markerlib,只要升级一下setuptools即可,方法如下:

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-19.2.zip
unzip setuptools-19.2.zip
cd setuptools-19.2
python setup.py install

[root@es1 setuptools-19.2]# pip install elasticsearch-curator
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Requirement already satisfied: elasticsearch-curator in /usr/lib/python2.6/site-packages
Requirement already satisfied: certifi>=2016.09.26 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: pyyaml>=3.10 in /usr/lib64/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: voluptuous>=0.9.3 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: click>=6.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: elasticsearch<3.0.0,>=2.4.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: setuptools>=0.6b1 in /usr/lib/python2.6/site-packages/setuptools-19.2-py2.6.egg (from voluptuous>=0.9.3->elasticsearch-curator)
Requirement already satisfied: urllib3<2.0,>=1.8 in /usr/lib/python2.6/site-packages (from elasticsearch<3.0.0,>=2.4.0->elasticsearch-curator)

参考:

http://www.cnblogs.com/java-zhao/p/5900590.html

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about.html

本文转载自:http://www.cnblogs.com/java-zhao/p/5900590.html

共有 人打赏支持
Mark93
粉丝 6
博文 140
码字总数 35695
作品 0
朝阳
私信 提问
加载中

评论(3)

Mark93
Mark93

引用来自“茶茶布丝”的评论

删除时以时间为维度删除时需要指定index名称吗?
是需要指定名称的,请留意curator_actions.yml文件中的filters部分
茶茶布丝
茶茶布丝
删除时以时间为维度删除时需要指定index名称吗?
茶茶布丝
茶茶布丝
(。・∀・)ノ゙嗨 你好 请问这个对所有index的过期删除吗?
Elasticsearch 5.5.1的安装和入门教程(转)

说明:转自老阮的文章,业界最简单的入门教程。一切的安装的运行建议不要用root权限,最好是当前用户下的权限。 作者: 阮一峰 日期: 2017年8月17日 全文搜索属于最常见的需求,开源的 Elas...

easonjim
01/21
0
0
springboot整合elasticsearch全文检索入门

只是简单的整合介绍 # 安装 下载elasticsearch与kibana https://www.elastic.co/start # 依赖 springBoot 2.0.5.RELEASE 起步依赖的elasticsearch的版本是 5.6.11 # 配置 可在application.y...

FutaoSmile丶
10/23
0
0
Elasticsearch 基本介绍及其与 Python 的对接实现

什么是 Elasticsearch 想查数据就免不了搜索,搜索就离不开搜索引擎,百度、谷歌都是一个非常庞大复杂的搜索引擎,他们几乎索引了互联网上开放的所有网页和数据。然而对于我们自己的业务数据...

崔庆才
08/01
0
0
SpringBoot集成ElasticSearch

1. ElasticSearch 1.1 ElasticSearch的简单介绍 ElasticSearch 是开源搜索平台领域的一个新成员。 ElasticSearch(简称 ES) 是一个基于 Lucene 构建的开源,分布式,RESTful 搜索引擎。 具有...

我叫钱多多_
09/12
0
0
ElasticSearch使用

安装之前,请参考https://github.com/richardwilly98/elasticsearch-river-mongodb根据你的MongoDB版本号决定需要的elasticsearch版本号和插件号。 1)安装ES 下载ElasticSearch_版本号.tar....

强子哥哥
2014/04/09
0
0

没有更多内容

加载失败,请刷新页面

加载更多

EOS官方钱包keosd

EOS官方钱包的名称是keosd,它负责管理你的私钥,并且帮你进行交易的签名。 不过不幸的是,keosd钱包对普通用户并不友好,它是一个命令行程序,目前还没有像以太坊的mist那样的图形化界面,而...

汇智网教程
36分钟前
1
0
ArrayList的实现原理以及实现线程安全

一、ArrayList概述 ArrayList是基于数组实现的,是一个动态的数字,可以自动扩容。 ArrayList不是线程安全的,效率比较高,只能用于单线程的环境中,在多线程环境中可以使用Collections.syn...

一看就喷亏的小猿
53分钟前
2
0
Netty 备录 (一)

入职新公司不久,修修补补1个月的bug,来了点实战性的技术---基于netty即时通信 还好之前对socket有所使用及了解,入手netty应该不是很难吧,好吧,的确有点难,刚看这玩意的时候,可能都不知道哪里...

_大侠__
昨天
5
0
Django简单介绍和用户访问流程

Python下有许多款不同的 Web 框架。Django是重量级选手中最有代表性的一位。许多成功的网站和APP都基于Django。 Django是一个开放源代码的Web应用框架,由Python写成。 Django遵守BSD版权,初...

枫叶云
昨天
8
0
Spring Cloud Stream消费失败后的处理策略(四):重新入队(RabbitMQ)

应用场景 之前我们已经通过《Spring Cloud Stream消费失败后的处理策略(一):自动重试》一文介绍了Spring Cloud Stream默认的消息重试功能。本文将介绍RabbitMQ的binder提供的另外一种重试...

程序猿DD
昨天
7
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部