文档章节

Elasticsearch存储空间不够导致索引只读的解决方法

傲娇字符
 傲娇字符
发布于 01/17 10:51
字数 694
阅读 1976
收藏 0

问题描述

今天发现当天的索引在ES中并没有创建,logstash中不停的报错:

[INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "reason"=>"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"})

索引变成了只读:

index read-only / allow delete (api)];"}

es报错,es报错也是索引只读错误

[2018-08-23T17:30:35,546][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:134) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_144]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_144]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_144]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:135) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:118) ~[?:?]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:571) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:380) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:375) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:908) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockException(TransportReplicationAction.java:826) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:814) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:712) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:169) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:97) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:125) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:350) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:462) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:175) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:514) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:98) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.0.0.jar:6.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
	at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:182) ~[elasticsearch-6.0.0.jar:6.0.0]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:812) ~[elasticsearch-6.0.0.jar:6.0.0]
	... 20 more

解决办法

经过分析,发现是因为ES所在服务器磁盘空间太低引起,具体解决办法如下:

在kibana开发控制台执行下面语句即可:

PUT _settings
{
  "index": {
    "blocks": {
      "read_only_allow_delete": "false"
    }
  }
}

执行语句

原因

一旦在存储超过95%的磁盘中的节点上分配了一个或多个分片的任何索引,该索引将被强制进入只读模式

参考链接:https://www.aityp.com/%E8%A7%A3%E5%86%B3elasticsearch%E7%B4%A2%E5%BC%95%E5%8F%AA%E8%AF%BB/

© 著作权归作者所有

傲娇字符
粉丝 5
博文 45
码字总数 17482
作品 0
武汉
架构师
私信 提问
elasticsearch FORBIDDEN/12/index read-only / allow

Eleastisearch6.0.0由单节点升级到多节点集群cluster时候出现的分片同步错误问题解决 原创 2018年01月18日 16:33:21 5 启动多个节点的ES后,ES开始推举master节点并同步分片shard数据到新ES节...

蓝叶子Sheep
2018/08/14
0
0
Elasticsearch7.2中文文档翻译(一):Elasticsearch简介

Elasticsearch7.2系列文章:https://www.xugj520.cn/category/ES/![2019-07-17T09:38:26.png][1] 为了搜索和分析,你懂得!!! 简介 Elasticsearch是Elastic Stack核心的分布式搜索和分析引...

高效码农
07/24
115
0
小白都会超详细--ELK日志管理平台搭建教程

目录 一、介绍 二、安装JDK 三、安装Elasticsearch 四、安装Logstash 五、安装Kibana 六、Kibana简单使用 系统环境:CentOS Linux release 7.4.1708 (Core) 当前问题状况 开发人员不能登录线...

渣渣辉
2018/07/15
0
0
elasticsearch入门到放弃之spring boot elasticsearch x-pack

spring-boot-data-elasticsearch 代码地址:https://github.com/zhaoyunxing92/spring-boot-learn-box/tree/master/spring-boot-elasticsearch/spring-boot-data-elasticsearch 这个就跟我在......

zhaoyunxing
07/07
0
0
ElasticSearch学习笔记4--核心概念解读

4.1 索引 索引(index)是ElasticSearch存放具体数据的地方,是一类具有相似特征的文档的集合。ElasticSearch中索引的概念具有不同意思,这里的索引相当于关系数据库中的一个数据库实例(databa...

荆辰曦
2018/09/05
0
0

没有更多内容

加载失败,请刷新页面

加载更多

android抓取app启动类

adb shell,然后logcat | grep "ActivityManager: START"

安卓工程师王恒
28分钟前
13
0
并发编程之Java锁

一、重入锁 锁作为并发共享数据,保证一致性的工具,在JAVA平台有多种实现(如 synchronized(重量级) 和 ReentrantLock(轻量级)等等 ) 。这些已经写好提供的锁为我们开发提供了便利。 重入锁...

codeobj
32分钟前
13
0
云栖深度干货 | 打造“云边一体化”,时序时空数据库TSDB技术原理深度解密

本文选自云栖大会下一代云数据库分析专场讲师自修的演讲——《TSDB云边一体化时序时空数据库技术揭秘》 自修 —— 阿里云智能数据库产品事业部高级专家 认识TSDB 第一代时序时空数据处理工具...

开源中国小二
33分钟前
11
0
mobile-nfv运行步骤

1. 编译过程 在 [workspaces] 的 <workdir> 目录运行以下命令: $ nfv-build make clean all... $ nfv-build make mobile-package...*** PACKAGE BUILT AT: /vob/BUG_117504/casa/nfv/b......

Robotcl_Blog
42分钟前
13
0
如何对邮箱做域名验证

做客户开发的时候,邮件群发仍然是个有效的手段。如果发海外邮件的话,选个服务器在国外的邮箱比较好,因为邮件抵达率会比较高一些。比如gmail, zoho mail等等,都可以。本文以zoho mail为例...

Zoho云服务
43分钟前
17
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部