文档章节

定期删除elasticsearch的过期index

Mark93
 Mark93
发布于 2016/12/12 16:36
字数 1718
阅读 330
收藏 2
点赞 0
评论 0

环境要求:

1. elasticsearch curator(与zookeeper curator不一样)

2. crontab

安装curator

安装pip

[root@es1 ~]# wget https://bootstrap.pypa.io/get-pip.py
--2016-12-12 11:18:34--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io... 151.101.16.175
Connecting to bootstrap.pypa.io|151.101.16.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1595408 (1.5M) [text/x-python]
Saving to: et-pip.py

100%[====================================================================================================================>] 1,595,408   9.40K/s   in 2m 3s   


[root@es1 ~]# python get-pip.py 
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting pip
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
    100% || 1.3MB 58kB/s 
Collecting setuptools
  Downloading setuptools-31.0.0-py2.py3-none-any.whl (477kB)
    100% || 481kB 55kB/s 
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% || 71kB 35kB/s 
Collecting argparse; python_version == "2.6" (from wheel)
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: pip, setuptools, argparse, wheel
Successfully installed argparse-1.4.0 pip-9.0.1 setuptools-31.0.0 wheel-0.29.0
/tmp/tmpONfJtN/pip.zip/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.

安装curator

[root@es1 ~]# pip install elasticsearch-curator
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Collecting elasticsearch-curator
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/lib/python2.6/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading elasticsearch-curator-4.2.4.tar.gz (134kB)
    100% || 143kB 54kB/s 
Collecting elasticsearch<3.0.0,>=2.4.0 (from elasticsearch-curator)
  Downloading elasticsearch-2.4.0-py2.py3-none-any.whl (54kB)
    100% || 61kB 64kB/s 
Collecting click>=6.0 (from elasticsearch-curator)
  Downloading click-6.6-py2.py3-none-any.whl (71kB)
    100% || 71kB 43kB/s 
Collecting pyyaml>=3.10 (from elasticsearch-curator)
  Downloading PyYAML-3.12.tar.gz (253kB)
    100% || 256kB 55kB/s 
Collecting voluptuous>=0.9.3 (from elasticsearch-curator)
  Downloading voluptuous-0.9.3.tar.gz
Collecting certifi>=2016.09.26 (from elasticsearch-curator)
  Downloading certifi-2016.9.26-py2.py3-none-any.whl (377kB)
    100% || 378kB 58kB/s 
Collecting urllib3<2.0,>=1.8 (from elasticsearch<3.0.0,>=2.4.0->elasticsearch-curator)
  Downloading urllib3-1.19.1-py2.py3-none-any.whl (104kB)
    100% || 112kB 48kB/s 
Requirement already satisfied: setuptools>=0.6b1 in /usr/lib/python2.6/site-packages (from voluptuous>=0.9.3->elasticsearch-curator)
Building wheels for collected packages: elasticsearch-curator, pyyaml, voluptuous
  Running setup.py bdist_wheel for elasticsearch-curator ... done
  Stored in directory: /root/.cache/pip/wheels/33/46/ac/5e17bb2d61b2139011742f37ca0a2d7fde039cca3d51b574f8
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for voluptuous ... done
  Stored in directory: /root/.cache/pip/wheels/4b/cc/a6/c1683326047075d71cdd74c966300efbfeda143fdf6c5f7587
Successfully built elasticsearch-curator pyyaml voluptuous
Installing collected packages: urllib3, elasticsearch, click, pyyaml, voluptuous, certifi, elasticsearch-curator
Successfully installed certifi-2016.9.26 click-6.6 elasticsearch-2.4.0 elasticsearch-curator-4.2.4 pyyaml-3.12 urllib3-1.19.1 voluptuous-0.9.3

检查curator

[root@es1 ~]# curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.


[root@es1 ~]# curator_cli --help
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

Options:
  --config PATH       Path to configuration file. Default:
                      ~/.curator/curator.yml
  --host TEXT         Elasticsearch host.
  --url_prefix TEXT   Elasticsearch http url prefix.
  --port TEXT         Elasticsearch port.
  --use_ssl           Connect to Elasticsearch through SSL.
  --certificate TEXT  Path to certificate to use for SSL validation.
  --client-cert TEXT  Path to file containing SSL certificate for client auth.
  --client-key TEXT   Path to file containing SSL key for client auth.
  --ssl-no-validate   Do not validate SSL certificate
  --http_auth TEXT    Use Basic Authentication ex: user:pass
  --timeout INTEGER   Connection timeout in seconds.
  --master-only       Only operate on elected master node.
  --dry-run           Do not perform any changes.
  --loglevel TEXT     Log level
  --logfile TEXT      log file
  --logformat TEXT    Log output format [default|logstash|json].
  --version           Show the version and exit.
  --help              Show this message and exit.

Commands:
  allocation        Shard Routing Allocation
  close             Close indices
  delete_indices    Delete indices
  delete_snapshots  Delete snapshots
  forcemerge        forceMerge index/shard segments
  open              Open indices
  replicas          Change replica count
  show_indices      Show indices
  show_snapshots    Show snapshots
  snapshot          Snapshot indices

配置curator

配置客户端连接

# curator_conf.yml

client:
    hosts: ["172.16.1.42:9200"]
    url_prefix: 
    use_ssl: False
    certificate: 
    client_cert: 
    client_key: 
    ssl_no_validate: False
    http_auth: 
    timeout: 30
    master_only: False

logging:
    loglevel: INFO
    logfile: /data/apps/curator/curator.log 
    logformat: default
    blacklist: ['elasticsearch','urllib3']

配置执行动作

# curator_actions.yml
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 3 days (based on index name), for baihe_ha-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False 
    filters:
    - filtertype: pattern
      kind: prefix
      value: bh_ha-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 3
      exclude:
  2:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

测试curator脚本

curator --config ./curator_conf.yml ./curator_actions.yml

配置crontab

编写执行脚本

#!/bin/sh
/usr/bin/curator --config /data/apps/curator/curator_conf.yml /data/apps/curator/curator_actions.yml

#curator_start.sh

配置执行规则

crontab -e:打开了vi,输入:

30 5 * * * /data/apps/curator/curator-start.sh,之后保存退出vi

crontab -l:查看所有的root用户的定时任务
 

附:crontab的文件格式

分 时 日 月 星期 要运行的命令

第1列分钟1~59
第2列小时1~23(0表示子夜)
第3列日1~31
第4列月1~12
第5列星期0~7(0和7表示星期天)
第6列要运行的命令

troubleshooting

[root@S3-202 ~]# curator --help
Traceback (most recent call last):
  File "/usr/bin/curator", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 3007, in <module>
    working_set.require(__requires__)
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 728, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 631, in resolve
    requirements.extend(dist.requires(req.extras)[::-1])
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2489, in requires
    dm = self._dep_map
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2700, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2717, in _compute_dependencies
    from _markerlib import compile as compile_marker
ImportError: No module named _markerlib

这并不是需要安装markerlib,只要升级一下setuptools即可,方法如下:

wget https://pypi.python.org/packages/source/s/setuptools/setuptools-19.2.zip
unzip setuptools-19.2.zip
cd setuptools-19.2
python setup.py install

[root@es1 setuptools-19.2]# pip install elasticsearch-curator
DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Requirement already satisfied: elasticsearch-curator in /usr/lib/python2.6/site-packages
Requirement already satisfied: certifi>=2016.09.26 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: pyyaml>=3.10 in /usr/lib64/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: voluptuous>=0.9.3 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: click>=6.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: elasticsearch<3.0.0,>=2.4.0 in /usr/lib/python2.6/site-packages (from elasticsearch-curator)
Requirement already satisfied: setuptools>=0.6b1 in /usr/lib/python2.6/site-packages/setuptools-19.2-py2.6.egg (from voluptuous>=0.9.3->elasticsearch-curator)
Requirement already satisfied: urllib3<2.0,>=1.8 in /usr/lib/python2.6/site-packages (from elasticsearch<3.0.0,>=2.4.0->elasticsearch-curator)

参考:

http://www.cnblogs.com/java-zhao/p/5900590.html

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about.html

© 著作权归作者所有

共有 人打赏支持
Mark93
粉丝 6
博文 96
码字总数 35695
作品 0
朝阳
Elasticsearch 5.5.1的安装和入门教程(转)

说明:转自老阮的文章,业界最简单的入门教程。一切的安装的运行建议不要用root权限,最好是当前用户下的权限。 作者: 阮一峰 日期: 2017年8月17日 全文搜索属于最常见的需求,开源的 Elas...

easonjim ⋅ 01/21 ⋅ 0

Centos6搭建elk系统,监控IIS日志

**所需程序: 服务器端:java、elasticsearch、kikbana 客 户 端:IIS、logstash** 一、服务器端(192.168.10.46)操作: 先建立一个ELK专门的目录: [root@Cent65 ~]mkdir /elk/ 上传到elk...

D杀手D ⋅ 04/24 ⋅ 0

CTO详细讲解海量日志处理ELK

ELK实时日志分析平台之Elasticsearch简介 Elasticsearch是一个高度灵活的开源全文检索和分析引擎。它能够迅速(几乎是实时地)地存储、查找和分析大规模数据。通常被用在有复杂的搜索要求的系...

Java架构分享 ⋅ 05/23 ⋅ 0

全文搜索引擎 Elasticsearch 教程

文章目录 一、安装 二、基本概念 三、新建和删除 Index 四、中文分词设置 五、数据操作 六、数据查询 七、参考链接 全文搜索属于最常见的需求,开源的 Elasticsearch (以下简称 Elastic)是...

John ⋅ 今天 ⋅ 0

Docker 部署ELK 日志分析

Docker 部署ELK 日志分析 elk集成镜像包 名字是 sebp/elk 安装 docke、启动 yum install docke service docker start Docker至少得分配3GB的内存;不然得加参数 -e ESMINMEM=128m -e ESMAXM...

yikayi ⋅ 06/08 ⋅ 0

Elasticsearch内核解析 - 写入流程

目前的Elasticsearch有两个明显的身份,一个是分布式搜索系统,另一个是分布式NoSQL数据库,对于这两种不同的身份,读写语义基本类似,但也有一点差异。 写操作 实时性: 搜索系统的Index一般...

少强 ⋅ 04/16 ⋅ 0

logstash、elasticsearch、kibana搭建日志平台

1、下载logstash a、官方下载地址:https://www.elastic.co/downloads/logstash b、解压:tar -xzvf logstash-5.1.1.tar.gz c、设置用户测试的配置文件:vim logstatsh_test.conf,编辑内容如...

binhu ⋅ 05/23 ⋅ 0

Docker 部署ELK 日志分析

Docker 部署ELK 日志分析 elk集成镜像包 名字是 sebp/elk 安装 docke、启动 yum install docke service docker start Docker至少得分配3GB的内存;不然得加参数 -e ESMINMEM=128m -e ESMAXM...

yikayi ⋅ 06/08 ⋅ 0

当ES赶超Redis,这份ES进修攻略不容错过!

从4月DB-Engines最新发布的全球数据库排名中,我们赫然发现ElasticSearch逆袭超越了Redis,从原先的第9名上升至第8名,而Redis则落后一名,排在了其后。 事实上,这场逆袭并不算太让人意外。...

DBAplus社群 ⋅ 04/15 ⋅ 0

海量事件数据存储与计算——高可用建设

作者简介 运小军 百度云资深研发工程师 负责百度智能运维方向大规模日志处理、海量事件数据存储相关设计研发工作,在分布式系统架构、大数据存储计算、高性能网络服务和即时通讯服务有广泛实...

g2v13ah ⋅ 05/25 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

js模拟栈和队列

栈和队列 栈:LIFO(先进后出)一种数据结构 队列:LILO(先进先出)一种数据结构 使用的js方法 1.push();可以接收任意数量的参数,把它们逐个推进队尾(数组末尾),并返回修改后的数组长度。 2....

LIAOJIN1 ⋅ 7分钟前 ⋅ 0

180619-Yaml文件语法及读写小结

Yaml文件小结 Yaml文件有自己独立的语法,常用作配置文件使用,相比较于xml和json而言,减少很多不必要的标签或者括号,阅读也更加清晰简单;本篇主要介绍下YAML文件的基本语法,以及如何在J...

小灰灰Blog ⋅ 15分钟前 ⋅ 0

IEC60870-5-104规约传送原因

1:周期循环2:背景扫描3:自发4:初始化5:请求6:激活7:激活确认8:停止激活9:停止激活确认10:激活结束11:远程命令引起的返送信息12:当地命令引起的返送信息13:文件传送20:响应总召...

始终初心 ⋅ 28分钟前 ⋅ 0

【图文经典版】冒泡排序

1、可视化排序过程 对{ 6, 5, 3, 1, 8, 7, 2, 4 }进行冒泡排序的可视化动态过程如下 2、代码实现    public void contextLoads() {// 冒泡排序int[] a = { 6, 5, 3, 1, 8, 7, 2, ...

pocher ⋅ 38分钟前 ⋅ 0

ORA-12537 TNS-12560 TNS-00530 ora-609解决

oracle 11g不能连接,卡住,ORA-12537 TNS-12560 TNS-00530 TNS-12502 tns-12505 ora-609 Windows Error: 54: Unknown error 解决方案。 今天折腾了一下午,为了查这个问题。。找了N多方案,...

lanybass ⋅ 53分钟前 ⋅ 0

IDEA反向映射Mybatis

1.首先在pom文件的plugins中添加maven对mybatis-generator插件的支持 ` <!-- mybatis逆向工程 --><plugin><groupId>org.mybatis.generator</groupId><artifactId>mybatis-generator-ma......

lichengyou20 ⋅ 57分钟前 ⋅ 0

4.10/4.11/4.12 lvm讲解 4.13 磁盘故障小案例

准备磁盘分区 fdisk /dev/sdb n 创建三个新分区,分别1G t 改变分区类型为8e 准备物理卷 pvcreate /dev/sdb1 pvcreate /dev/sdb2 pvcreate /dev/sdb3 pvdisplay/pvs 列出当前的物理卷 pvremo...

Linux_老吴 ⋅ 59分钟前 ⋅ 0

zabbix 3.4安装

#已装好lamp环境 1.安装相关yum仓库 rpm -i http://repo.zabbix.com/zabbix/3.4/rhel/7/x86_64/zabbix-release-3.4-2.el7.noarch.rpm #tip:rpm -ql zabbix-release 看上面这个软件装了哪些东......

山月关 ⋅ 今天 ⋅ 0

Java的Excel导出工具类

首先在POM中引入需要的Jar <dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.47</version></dependency><dependency><groupId>o......

Kxvz ⋅ 今天 ⋅ 0

springboot 使用jsp

目录结构: 启动文件的Application必须在contorller文件的父级 文件路径在src/main/webapp下面 我的配置:前缀是/WEB-INF/jsp/ pom.xml需要加入tomcat-embed-jasper, 对jsp的支持的依赖 <de...

夜醒者 ⋅ 今天 ⋅ 0

没有更多内容

加载失败,请刷新页面

加载更多

下一页

返回顶部
顶部