定期删除elasticsearch的过期index
博客专区 > Mark93 的博客 > 博客详情
定期删除elasticsearch的过期index
Mark93 发表于1年前
定期删除elasticsearch的过期index
  • 发表于 1年前
  • 阅读 270
  • 收藏 2
  • 点赞 0
  • 评论 0

腾讯云 新注册用户 域名抢购1元起>>>   

摘要: 使用ELK的同学都知道,logstash会每天生成一个index,随着index数量的增加,文件磁盘越来越小,必须删除不需要的index才能释放空间.这里介绍一种可以定时清理过期的index方法.

环境要求:

1. elasticsearch curator(与zookeeper curator不一样)

2. crontab

安装curator

安装pip

[root@es1 ~]# wget https://bootstrap.pypa.io/get-pip.py
--2016-12-12 11:18:34--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io... 151.101.16.175
Connecting to bootstrap.pypa.io|151.101.16.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1595408 (1.5M) [text/x-python]
Saving to: et-pip.py

100%[====================================================================================================================>] 1,595,408   9.40K/s   in 2m 3s   


[root@es1 ~]# python get-pip.py 
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting pip
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% || 1.3MB 58kB/s 
Collecting setuptools
  Downloading setuptools-31.0.0-py2.py3-none-any.whl (477kB)
    100% || 481kB 55kB/s 
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% || 71kB 35kB/s 
Collecting argparse; python_version == "2.6" (from wheel)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, argparse, wheel
Successfully installed argparse-1.4.0 pip-9.0.1 setuptools-31.0.0 wheel-0.29.0
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.

安装curator

[root@es1 ~]# pip install elasticsearch-curator
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting elasticsearch-curator
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading elasticsearch-curator-4.2.4.tar.gz (134kB)
    100% || 143kB 54kB/s 
Collecting elasticsearch<3.0.0,>=2.4.0 (from elasticsearch-curator)
  Downloading elasticsearch-2.4.0-py2.py3-none-any.whl (54kB)
    100% || 61kB 64kB/s 
Collecting click>=6.0 (from elasticsearch-curator)
  Downloading click-6.6-py2.py3-none-any.whl (71kB)
    100% || 71kB 43kB/s 
Collecting pyyaml>=3.10 (from elasticsearch-curator)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% || 256kB 55kB/s 
Collecting voluptuous>=0.9.3 (from elasticsearch-curator)
  Downloading voluptuous-0.9.3.tar.gz
Collecting certifi>=2016.09.26 (from elasticsearch-curator)
  Downloading certifi-2016.9.26-py2.py3-none-any.whl (377kB)
    100% || 378kB 58kB/s 
Collecting urllib3<2.0,>=1.8 (from elasticsearch<3.0.0,>=2.4.0->elasticsearch-curator)
  Downloading urllib3-1.19.1-py2.py3-none-any.whl (104kB)
    100% || 112kB 48kB/s 
Requirement already satisfied: setuptools>=0.6b1 in /usr/lib/python2.6/site-packages (from voluptuous>=0.9.3->elasticsearch-curator)
Building wheels for collected packages: elasticsearch-curator, pyyaml, voluptuous
  Running setup.py bdist_wheel for elasticsearch-curator ... done
  Stored in directory: /root/.cache/pip/wheels/33/46/ac/5e17bb2d61b2139011742f37ca0a2d7fde039cca3d51b574f8
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for voluptuous ... done
  Stored in directory: /root/.cache/pip/wheels/4b/cc/a6/c1683326047075d71cdd74c966300efbfeda143fdf6c5f7587
Successfully built elasticsearch-curator pyyaml voluptuous
Installing collected packages: urllib3, elasticsearch, click, pyyaml, voluptuous, certifi, elasticsearch-curator
Successfully installed certifi-2016.9.26 click-6.6 elasticsearch-2.4.0 elasticsearch-curator-4.2.4 pyyaml-3.12 urllib3-1.19.1 voluptuous-0.9.3

检查curator

[root@es1 ~]# curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.


[root@es1 ~]# curator_cli --help
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

Options:
  --config PATH       Path to configuration file. Default:
                      ~/.curator/curator.yml
  --host TEXT         Elasticsearch host.
  --url_prefix TEXT   Elasticsearch http url prefix.
  --port TEXT         Elasticsearch port.
  --use_ssl           Connect to Elasticsearch through SSL.
  --certificate TEXT  Path to certificate to use for SSL validation.
  --client-cert TEXT  Path to file containing SSL certificate for client auth.
  --client-key TEXT   Path to file containing SSL key for client auth.
  --ssl-no-validate   Do not validate SSL certificate
  --http_auth TEXT    Use Basic Authentication ex: user:pass
  --timeout INTEGER   Connection timeout in seconds.
  --master-only       Only operate on elected master node.
  --dry-run           Do not perform any changes.
  --loglevel TEXT     Log level
  --logfile TEXT      log file
  --logformat TEXT    Log output format [default|logstash|json].
  --version           Show the version and exit.
  --help              Show this message and exit.

Commands:
  allocation        Shard Routing Allocation
  close             Close indices
  delete_indices    Delete indices
  delete_snapshots  Delete snapshots
  forcemerge        forceMerge index/shard segments
  open              Open indices
  replicas          Change replica count
  show_indices      Show indices
  show_snapshots    Show snapshots
  snapshot          Snapshot indices

配置curator

配置客户端连接

# curator_conf.yml

client:
    hosts: ["172.16.1.42:9200"]
    url_prefix: 
    use_ssl: False
    certificate: 
    client_cert: 
    client_key: 
    ssl_no_validate: False
    http_auth: 
    timeout: 30
    master_only: False

logging:
    loglevel: INFO
    logfile: /data/apps/curator/curator.log 
    logformat: default
    blacklist: ['elasticsearch','urllib3']

配置执行动作

# curator_actions.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 3 days (based on index name), for baihe_ha-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False 
    filters:
    - filtertype: pattern
      kind: prefix
      value: bh_ha-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 3
      exclude:
  2:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

测试curator脚本

curator --config ./curator_conf.yml ./curator_actions.yml

配置crontab

编写执行脚本

#!/bin/sh
/usr/bin/curator --config /data/apps/curator/curator_conf.yml /data/apps/curator/curator_actions.yml

#curator_start.sh

配置执行规则

crontab -e:打开了vi,输入:

30 5 * * * /data/apps/curator/curator-start.sh,之后保存退出vi

crontab -l:查看所有的root用户的定时任务
 

附:crontab的文件格式

分 时 日 月 星期 要运行的命令

第1列分钟1~59
第2列小时1~23(0表示子夜)
第3列日1~31
第4列月1~12
第5列星期0~7(0和7表示星期天)
第6列要运行的命令

troubleshooting

[root@S3-202 ~]# curator --help
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 631, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2489, in requires
    dm = self._dep_map
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2700, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2717, in _compute_dependencies
    from _markerlib import compile as compile_marker
ImportError: No module named _markerlib

这并不是需要安装markerlib,只要升级一下setuptools即可,方法如下:

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-19.2.zip
unzip setuptools-19.2.zip
cd setuptools-19.2
python setup.py install

[root@es1 setuptools-19.2]# pip install elasticsearch-curator
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Requirement already satisfied: elasticsearch-curator in /usr/lib/python2.6/site-packages
Requirement already satisfied: certifi>=2016.09.26 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: pyyaml>=3.10 in /usr/lib64/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: voluptuous>=0.9.3 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: click>=6.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: elasticsearch<3.0.0,>=2.4.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: setuptools>=0.6b1 in /usr/lib/python2.6/site-packages/setuptools-19.2-py2.6.egg (from voluptuous>=0.9.3->elasticsearch-curator)
Requirement already satisfied: urllib3<2.0,>=1.8 in /usr/lib/python2.6/site-packages (from elasticsearch<3.0.0,>=2.4.0->elasticsearch-curator)

参考:

http://www.cnblogs.com/java-zhao/p/5900590.html

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about.html

共有 人打赏支持
粉丝 6
博文 94
码字总数 33921
×
Mark93
如果觉得我的文章对您有用,请随意打赏。您的支持将鼓励我继续创作!
* 金额(元)
¥1 ¥5 ¥10 ¥20 其他金额
打赏人
留言
* 支付类型
微信扫码支付
打赏金额:
已支付成功
打赏金额: