文档章节

Elasticsearch 维护(实验后整理)

Faye_Cai
 Faye_Cai
发布于 2016/06/21 13:37
字数 855
阅读 78
收藏 0

实验完了之后整理下来的,FYI

Scenarios :

1.  unassigned+ initializing shards are not zero , cluster is yellow/red status  :
When one node  left the cluster, the cluster status is yellow/red , and there will be unassigned/initializing shards , when the node comes back and join into the cluster , the unassigned shards will decrease , there will be initializing shards , it will take some time ,
Solution :
Generally, the cluster will become green ultimately , unassigned+ initializing shards will be 0 ;
But in some cases , some shards fail at starting
,  then we will take some actions
1) How to clean the unassigned shards ?
# Check health
[][root@test xing]# curl --max-time 180 -s -E {mysslpemfile}:*****--insecure https://myhost:9200/_cluster/health?pretty
{
  "cluster_name" : "***_es_cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 14,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0
}
# Check shards status, columns :  index shard shard_type shard_status ..... node_name (in my testing scenario , my index name is "xing" )
[][root@test˜]# curl --max-time 180 -s -E **** --insecure https://****:9200/_cat/shards?pretty
xing 3  p UNASSIGNED
xing 3  r UNASSIGNED
xing 10 p STARTED    1 2.4kb ******* esdev2
xing 10 r STARTED    1 2.4kb ******* esdev1
xing 6  p UNASSIGNED
xing 6  r UNASSIGNED
xing 2  p UNASSIGNED
xing 2  r UNASSIGNED
xing 1  p UNASSIGNED
xing 1  r UNASSIGNED
xing 0  p UNASSIGNED
xing 0  r UNASSIGNED
xing 7  p STARTED    0  144b ******* esdev2
xing 7  r STARTED    0  144b ******* esdev1
xing 11 p STARTED    1 2.4kb ******* esdev2
xing 11 r STARTED    1 2.4kb ******* esdev1
xing 9  p STARTED    0  144b ******* esdev2
xing 9  r STARTED    0  144b ******* esdev1
xing 5  p UNASSIGNED
xing 5  r UNASSIGNED
xing 4  p UNASSIGNED
xing 4  r UNASSIGNED
xing 8  p STARTED    0  144b ******* esdev2
xing 8  r STARTED    0  144b ******* esdev1

# Allocate the shard "3" to esdev1 (that is to put primary shard to esdev1)
[][root@test˜]# curl -XPOST --max-time 180 -s -E**** --insecure https://******:9200/_cluster/reroute?pretty -d '{
    "commands" : [ {
          "allocate" : {
              "index" : "xing",
              "shard" : 3,
              "node" : "test_node",
              "allow_primary" : true
          }
        }
    ]
}'
# check shards again
[][root@@esdev1 ˜]# curl --max-time 180 -s -E **** --insecure https://****:9200/_cat/shards?pretty
xing 3  r STARTED    0   79b ******* esdev2
xing 3  p STARTED    0  115b ******* esdev1
xing 10 p STARTED    1 2.4kb ******* esdev2
xing 10 r STARTED    1 2.4kb ******* esdev1
xing 6  p UNASSIGNED
xing 6  r UNASSIGNED
......

# If there are two many unassigned shards to allocate .
Notice : change the index name to your index ; change node name , do not put all primary shards into one node, balance the primary shards into all nodes
[][root@@esdev1 ˜]# for shard in $(curl -XGET --max-time 180 -s -E **** --insecure https://****:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do
    curl -XPOST --max-time 180 -s -E ***** --insecure https://****:9200/_cluster/reroute -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "xing",
                  "shard" : '$shard',
                  "node" : "esdev1",
                  "allow_primary" : true
              }
            }
        ]
    }'
    sleep 5
done
# check shard again , all are restored
[][root@esdev1 faye]# curl --max-time 180 -s -E ***** --insecure https://****:9200/_cat/shards?pretty
xing 3  r STARTED 0  144b ******* esdev2
.........
xing 2  p STARTED 0  144b *******  esdev1


How can I artificially produce the red status and unassigned shards? FYI .
stop ES service on all nodes , go to my index "xing" to delete all shards in node1 , delete part shards in node2 , then start ES on both nodes

2) how to clean the initializing shards ?
[][root@test ˜]# curl --max-time 180 -s -E ***** --insecure https://*****:9200/_cat/shards?pretty
t37       434 p STARTED 12221982  13.8gb 10.0.0.22   datanode02
t37       434 r STARTED 12221982  13.8gb 10.0.0.23   datanode03
t37       403 p INITIALIZING 21620252  28.3gb 10.0.0.22   datanode02
t37       404 p INITIALIZING 5720596    4.9gb 10.0.0.22   datanode02
t37       20  p UNASSIGNED
t37       20  r UNASSIGNED
t37       468 p INITIALIZING  8313898  12.3gb 10.0.0.22   datanode02
t37       470 p INITIALIZING  38868416 56.8gb 10.0.0.22   datanode02
t37       37  p UNASSIGNED
t37       37  r UNASSIGNED
t37       430 r STARTED 11806144  15.8gb 10.0.0.24   datanode04
t37       430 p STARTED 11806144  15.8gb 10.0.0.25   datanode05
t37       368 p STARTED 34530372    43gb 10.0.0.25   datanode05
t37       368 r STARTED 34530372    43gb 10.0.0.23   datanode03

Then, what can we do?
Answer :  stop ES on two nodes , then start ES on two nodes (Not restart ES one by one node)

2. when execute restore failed ,such as there are some corruption in the restore snapshots, there will be unassigned shard and initializing shards , cluster will be red status
eg :
[][root@esdev1 backup]# curl --max-time 180 -s -E ***** --insecure https://****:9200/_cluster/health?pretty
{
  "cluster_name" : "*****_es_cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 9,
  "active_shards" : 18,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 2,
  "number_of_in_flight_fetch" : 0
}
Solution :  stop ES on two nodes , then start ES on two nodes (Not restart ES one by one node) , ES will recover the restore issue itself , the color will be green and come to the original point before restoring action


3. Some primary shards data in one ES nodes are deleted/updated by mistake
Solution : Don't worry , stop ES on two nodes , then start ES on two nodes , ES will recover the shards iteself using its replication on another node when starting up again

© 著作权归作者所有

Faye_Cai
粉丝 0
博文 28
码字总数 5590
作品 0
海淀
高级程序员
私信 提问
ELK+Filebeat+Nginx集中式日志解决方案(二)——添加ElasticSearch集群

一、使用说明: Elasticsearch插件: Elasticsearch常用的几个插件为: # head 地址 https://github.com/mobz/elasticsearch-head mobz/elasticsearch-head kopf 地址 https://github.com/l......

在下郑小明
2018/06/26
0
0
Elasticsearch介绍和安装

版权声明:https://blog.csdn.net/weixin43814195?t=1 https://blog.csdn.net/weixin43814195/article/details/85275156 Elasticsearch 1.简介 1.1基本概念 Elasticsearch是基于Lucene的全文......

MIss.Fan
2018/12/27
0
0
[Success]Elasticsearch介绍和安装+ik

一.Elasticsearch介绍和安装 SprignBoot整合Spring Data Elasticsearch 以及使用 1.1.简介 1.1.1.Elastic Elastic官网:https://www.elastic.co/cn/ Elastic有一条完整的产品线:Elasticsear......

Armymans
2018/11/12
0
0
ELK 实验(二)安装Elastic Search 单节点

Linux 安装 java -version echo $JAVA_HOME 木有返回,环境没有设置好。。。 vi /etc/profile export JAVAHOME=/usr/java/jdk1.8.0162 export JREHOME=$JAVAHOME/jre export CLASSPATH=$JAV......

pcdog
2018/04/15
0
0
Elasticsearch中文分词研究

一、ES分析器简介 ES是一个实时搜索与数据分析引擎,为了完成搜索功能,必须对原始数据进行分析、拆解,以建立索引,从而实现搜索功能; ES对数据分析、拆解过程如下: 首先,将一块文本分成...

zhaipengfei1231
2018/04/18
0
0

没有更多内容

加载失败,请刷新页面

加载更多

CSS3 : transition 属性

本文转载于:专业的前端网站➧CSS3 : transition 属性 CSS3的 transition 属性用于状态过度效果! 1、语法: 1 transition: property duration timing-function delay;2 -moz-transition: ...

前端老手
22分钟前
5
0
一个简单的加密工具,性能貌似不行,待优化

一个简单的加密工具,性能貌似不行,待优化 package com.kxvz.common.crypt;import javax.crypto.Cipher;import javax.crypto.KeyGenerator;import javax.crypto.SecretKey;import java.i......

Kxvz
24分钟前
4
0
vue实现路由懒加载

一、为什么要实现懒加载 减少首屏加载时间,避免白屏 二、常用的懒加载方式有两种:即使用vue异步组件 和 ES中的import a、vue异步加载: import Vue from 'vue' import Router from 'vue-ro...

Bing309
27分钟前
5
0
axios拦截器

axios.interceptors.response.use(response => { if (response.data.code == 0) { return response.data } else if (response.data.code == 600) { Cookies.remove('Admin-Token') router.pu......

Cyoya
29分钟前
5
0
给大家分享下部署云桌面的几个小技巧

从去年4月份开始我们公司就开始使用云桌面来进行上网办公的,在这一年多的使用过程中并没有出现像网上和有些用户说的那样,说云桌面各种坑老是出现这样和那样的问题,而我们之所以用的还不错...

GZASD
29分钟前
4
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部