文档章节

elasticsearch threadpool setting

MrYx3en
 MrYx3en
发布于 2016/07/27 16:16
字数 383
阅读 116
收藏 1

#threadpool setting

2016-07-27 16:10:22,840 ERROR org.ctrip.ops.sysdev.outputs.Elasticsearch$1 pool-2-thread-2 RemoteTransportException[[a08.elastic.loganalyse.monitor.b28.youku][inet[/10.103.11.27:9300]][indices:data/write/
bulk[s]]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 50) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$PrimaryPhase$1@2b16cb10]; 
2016-07-27 16:10:22,840 INFO org.ctrip.ops.sysdev.outputs.Elasticsearch$1 pool-2-thread-2 2104 doc failed, 2104 need to retry
2016-07-27 16:10:22,840 INFO org.ctrip.ops.sysdev.outputs.Elasticsearch$1 pool-2-thread-2 sleep 1052millseconds after bulk failure

Threadpool Properties Prevent Data Loss[5] ElasticSearch node has several thread pools in order to improve how threads are managed within a node. At Loggly, we use bulk request extensively, and we have found that setting the right value for bulk thread pool using threadpool.bulk.queue_size property is crucial in order to avoid data loss or _bulk retries

threadpool.bulk.queue_size: 3000

This property value is for the bulk request. This tells ES the number of requests that can be queued for execution in the node when there is no thread available to execute a bulk request. This value should be set according to your bulk request load. If your bulk request number goes higher than queue size, you will get a RemoteTransportException as shown below.

Note that in ES the bulk requests queue contains one item per shard, so this number needs to be higher than the number of concurrent bulk requests you want to send if those request contain data for many shards. For example, a single bulk request may contain data for 10 shards, so even if you only send one bulk request, you must have a queue size of at least 10. Setting this value “too high” will chew up heap in your JVM, but does let you hand off queuing to ES, which simplifies your clients.

You either need to keep the property value higher than your accepted load or gracefully handle RemoteTransportException in your client code. If you don’t handle the exception, you will end up losing data. We simulated the exception shown below by sending more than 10 bulk requests with a queue size of 10.

RemoteTransportException[[<Bantam>][inet[/192.168.76.1:9300]][bulk/shard]]; nested: EsRejectedExecutionException[rejected execution (queue capacity 10) on org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1@13fe9be];

http://blog.csdn.net/huwei2003/article/details/41214391

http://stackoverflow.com/questions/20683440/elasticsearch-gives-error-about-queue-size

© 著作权归作者所有

共有 人打赏支持
上一篇: golang - log/syslog
下一篇: golang - interface
MrYx3en
粉丝 10
博文 131
码字总数 30542
作品 0
宝鸡
系统管理员
私信 提问
Elastic Search学习笔记1——安装elasticsearch2.4.6

Elastic Search 简介 1.基于Apache Lucene的开源搜索引擎 2.采用Java编写 RESTful API风格 3.较容易的横向扩展 应用场景 1.海量数据分析引擎 2.数据搜索引擎 3.数据仓库 官网 https://www.el...

晨猫
2018/03/09
0
0
Centos7.4部署配置Elasticsearch5.6集群

参考文档 https://www.elastic.co/guide/en/elasticsearch/reference/5.6/index.html https://www.elastic.co/guide/cn/elasticsearch/guide/current/important-configuration-changes.html ......

minminmsn
2018/07/09
0
0
CentOS7 部署 ElasticSearch 集群

环境 主机名 IP 操作系统 ES 版本 es227 192.168.1.227 CentOS7.5 6.5.4 es228 192.168.1.228 CentOS7.5 6.5.4 es229 192.168.1.229 CentOS7.5 6.5.4 下载 elasticsearch-6.5.4.tar.gz 各节点......

Colben
2018/12/27
0
0
Spark中hive的使用(hive操作es示例)

配置hive-site.xml <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <description>JDBC connect ......

守望者之父
2018/06/15
0
0
分布式搜索引擎 ElasticSearch 6.1.3 和 5.6.7 发布

ElasticSearch 6.1.3 和 5.6.7 发布了。 主要更新内容如下: 6.1.3 Engine Replica recovery could go into an endless flushing loop #28350 Internal Never return null from Strings.toke......

达尔文
2018/01/31
1K
3

没有更多内容

加载失败,请刷新页面

加载更多

编码规范

4.、编码时的一些建议 1、尽量指定类、方法的final修饰符——虚拟机会想办法内联所有的final方法来减少方法执行时创建栈帧的数量,从而降低栈溢出的风险 2、尽量重用对象——重复new对象会增...

呵呵哒灬
21分钟前
2
0
第一个docker化的java应用

Docker 思想 集装箱 标准化:运输方式/存储方式/API 接口 隔离

BeanHo
21分钟前
1
0
数据库技术-Mysql主从复制与数据备份

数据库技术-Mysql 主从复制的原理: MySQL中数据复制的基础是二进制日志文件(binary log file)。一台MySQL数据库一旦启用二进制日志后,其作为master,它的数据库中所有操作都会以“事件”...

须臾之余
昨天
13
0
Git远程仓库——GitHub的使用(一)

Git远程仓库——GitHub的使用(一) 一 、 Git远程仓库 由于你的本地仓库和GitHub仓库之间的传输是通过SSH加密的,所以需要一下设置: 步骤一、 创建SSH key 在用户主目录下,看看有没有.ss...

lwenhao
昨天
4
0
SpringBoot 整合

springBoot 整合模板引擎 SpringBoot 整合Mybatis SpringBoot 整合redis SpringBoot 整合定时任务 SpringBoot 整合拦截器...

细节探索者
昨天
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部