文档章节

ElasticSearch02索引一个文档视角看集群

AllenOR灵感
 AllenOR灵感
发布于 2017/09/10 01:18
字数 3131
阅读 1
收藏 0

本文是想透过一个空的elasticsearch实例应用了解创建一个索引在实体文件中占用多少磁盘空间, 所以在落实创建之前:

  • 操作之前做一些记录;
  • 利用python来创建一个索引;
  • 观察索引创建后每个环节发生的变化;
  • 激活另外一个空节点, 并观察数据同步情况;
  • 总结。

操作之前做一些记录

文件分布

[root@es-master data]# tree /usr/local/elasticsearch-5.5.1/data/
/usr/local/elasticsearch-5.5.1/data/
└── nodes                       # 目录
    └── 0                       # 目录
        ├── node.lock           # 文件
        └── _state              # 目录
            ├── global-16.st    # 文件
            └── node-17.st      # 文件

3 directories, 3 files

文件大小(bytes)

[root@es-master data]# find /usr/local/elasticsearch-5.5.1/data/ -type f | xargs du -lsb
0   /usr/local/elasticsearch-5.5.1/data/nodes/0/node.lock
71  /usr/local/elasticsearch-5.5.1/data/nodes/0/_state/node-17.st
140 /usr/local/elasticsearch-5.5.1/data/nodes/0/_state/global-16.st

节点信息

[root@es-master data]# curl -X GET http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "HawkEye",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

分片和索引的统计信息

[root@es-master data]# curl -X GET http://localhost:9200/_stats?pretty
{
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  },
  "_all" : {
    "primaries" : { },
    "total" : { }
  },
  "indices" : { }
}

 
 

利用python来创建一个索引

_01_create_index.py

# -.- coding:utf-8 -.-
from elasticsearch import Elasticsearch

es = Elasticsearch(hosts=["192.168.1.132"])
es.index(
    index="website",
    doc_type="blog",
    id=1,
    body={
        "title": "My first blog entry",
        "text":  "Just trying this out...",
        "date":  "2014/01/01"
    }
)

# 执行python文件, 落实索引创建
[root@es-master ~]# python _01_create_index.py

# 检查索引是否创建成功

[root@es-master data]# curl -XGET http://localhost:9200/website/blog/1?pretty
{
  "_index" : "website",
  "_type" : "blog",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "date" : "2014/01/01",
    "text" : "Just trying this out...",
    "title" : "My first blog entry"
  }
}

 
 

观察索引创建后每个环节发生的变化

elasticsearch应用日志

[2017-08-18T22:41:03,598][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [website] creating index, cause [auto(bulk api)], templates [ mappings []
[2017-08-18T22:41:04,490][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [website/YhxMl-cOQUq6WCYH1sKjRg] create_mapping [blog]

文件分布

[root@es-master data]# tree /usr/local/elasticsearch-5.5.1/data/
/usr/local/elasticsearch-5.5.1/data/
└── nodes
    └── 0
        ├── indices
        │   └── YhxMl-cOQUq6WCYH1sKjRg
        │       ├── 0
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       ├── 1
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       ├── 2
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       ├── 3
        │       │   ├── index
        │       │   │   ├── _0.cfe
        │       │   │   ├── _0.cfs
        │       │   │   ├── _0.si
        │       │   │   ├── segments_3
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-2.tlog
        │       │       └── translog.ckp
        │       ├── 4
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       └── _state
        │           └── state-4.st
        ├── node.lock
        └── _state
            ├── global-16.st
            └── node-17.st

26 directories, 32 files

文件大小(bytes)

[root@es-master data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-17.st
140 ./nodes/0/_state/global-16.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-2.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
632 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-4.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

[root@es-master data]# echo `find ./ -type f | xargs du -lsb | awk '{print $1}'` | tr " " "+" | bc
7272        # 存储总量(Bytes)  ==  7K大小

节点信息

[root@es-master data]# curl -X GET http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "HawkEye",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

分片和索引的统计信息

[root@es-master data]# curl -X GET http://localhost:9200/_stats/indexing,store?pretty
{
  "_shards" : {
    "total" : 10,
    "successful" : 5,
    "failed" : 0
  },
  "_all" : {
    "primaries" : {
      "store" : {
        "size_in_bytes" : 5304,
        "throttle_time_in_millis" : 0
      },
      "indexing" : {
        "index_total" : 1,
        "index_time_in_millis" : 90,
        "index_current" : 0,
        "index_failed" : 0,
        "delete_total" : 0,
        "delete_time_in_millis" : 0,
        "delete_current" : 0,
        "noop_update_total" : 0,
        "is_throttled" : false,
        "throttle_time_in_millis" : 0
      }
    },
    "total" : {
      "store" : {
        "size_in_bytes" : 5304,
        "throttle_time_in_millis" : 0
      },
      "indexing" : {
        "index_total" : 1,
        "index_time_in_millis" : 90,
        "index_current" : 0,
        "index_failed" : 0,
        "delete_total" : 0,
        "delete_time_in_millis" : 0,
        "delete_current" : 0,
        "noop_update_total" : 0,
        "is_throttled" : false,
        "throttle_time_in_millis" : 0
      }
    }
  },
  "indices" : {
    "website" : {
      "primaries" : {
        "store" : {
          "size_in_bytes" : 5304,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 1,
          "index_time_in_millis" : 90,
          "index_current" : 0,
          "index_failed" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0,
          "noop_update_total" : 0,
          "is_throttled" : false,
          "throttle_time_in_millis" : 0
        }
      },
      "total" : {
        "store" : {
          "size_in_bytes" : 5304,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 1,
          "index_time_in_millis" : 90,
          "index_current" : 0,
          "index_failed" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0,
          "noop_update_total" : 0,
          "is_throttled" : false,
          "throttle_time_in_millis" : 0
        }
      }
    }
  }
}

 
 

激活另外一个空节点,并观察数据同步情况

主节点应用日志

[2017-08-18T23:38:13,202][INFO ][o.e.c.s.ClusterService   ] [node-1] added {{y1FxQfR}{y1FxQfRnTpyKdtid8ISeLQ}{lJd8BNRhQ5ywFkjLhZ2bPQ}{192.168.1.104}{192.168.1.104:9300},}, reason: zen-disco-node-join[{y1FxQfR}{y1FxQfRnTpyKdtid8ISeLQ}{lJd8BNRhQ5ywFkjLhZ2bPQ}{192.168.1.104}{192.168.1.104:9300}]
[2017-08-18T23:38:14,038][WARN ][o.e.d.z.ElectMasterService] [node-1] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [-1], total number of master-eligible nodes used for publishing in this round: [2])
[2017-08-18T23:38:16,294][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[website][4]] ...]).

从节点应用日志

[2017-08-18T23:38:38,303][INFO ][o.e.d.DiscoveryModule    ] [y1FxQfR] using discovery type [zen]
[2017-08-18T23:38:40,023][INFO ][o.e.n.Node               ] initialized
[2017-08-18T23:38:40,023][INFO ][o.e.n.Node               ] [y1FxQfR] starting ...
[2017-08-18T23:38:40,420][INFO ][o.e.t.TransportService   ] [y1FxQfR] publish_address {192.168.1.104:9300}, bound_addresses {[::]:9300}
[2017-08-18T23:38:40,428][INFO ][o.e.b.BootstrapChecks    ] [y1FxQfR] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-08-18T23:38:43,810][INFO ][o.e.c.s.ClusterService   ] [y1FxQfR] detected_master {node-1}{VlKvChmBR7aQhObWv_e4Kw}{cVyIHfX9RI6glUw3YsLEVg}{192.168.1.132}{192.168.1.132:9300}, added {{node-1}{VlKvChmBR7aQhObWv_e4Kw}{cVyIHfX9RI6glUw3YsLEVg}{192.168.1.132}{192.168.1.132:9300},}, reason: zen-disco-receive(from master [master {node-1}{VlKvChmBR7aQhObWv_e4Kw}{cVyIHfX9RI6glUw3YsLEVg}{192.168.1.132}{192.168.1.132:9300} committed version [26]])
[2017-08-18T23:38:44,653][INFO ][o.e.h.n.Netty4HttpServerTransport] [y1FxQfR] publish_address {192.168.1.104:9200}, bound_addresses {[::]:9200}
[2017-08-18T23:38:44,653][INFO ][o.e.n.Node               ] [y1FxQfR] started

从节点数据目录结构

[root@es-slave01 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-6.st
140 ./nodes/0/_state/global-6.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-5.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_4
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

[root@es-slave01 data]# echo `find ./ -type f | xargs du -lsb | awk '{print $1}'` | tr " " "+" | bc
7387

大小发生了变化,所以要再看一下主节点的数据目录结构

主节点数据目录结构

[root@es-master data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-17.st
140 ./nodes/0/_state/global-16.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-2.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-9.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

[root@es-master data]# echo `find ./ -type f | xargs du -lsb | awk '{print $1}'` | tr " " "+" | bc
7387

集群生效后, 两个节点的数据完全一致, 这其实不是我想要看到的结果, 因为数据完全是镜像同步, 如果增加更多机器也是这种效果,那么就相当于是我所有的数据只存放在一台机器上一样,无非是多了灾备的情况和更高效的读,存储成本太高。
所以我需要多起两台空节点加入到集群中,然后再观察,发现数据存储方式发生了变化,elasticsearch内部将数据平摊到每个节点上了.

es-master

[root@es-master data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-17.st
140 ./nodes/0/_state/global-16.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-2.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-13.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

es-slave01

[root@es-slave01 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-6.st
140 ./nodes/0/_state/global-6.st
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-9.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_4
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

es-slave02

[root@es-slave02 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-10.st
140 ./nodes/0/_state/global-9.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-4.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog

es-slave03

[root@es-slave03 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-3.st
141 ./nodes/0/_state/global-3.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-1.st

总结

四个节点, 仔细看下来shards分片们分别分布在:
es-mater: 1 3 4
es-slave01: 3 4
es-slave02: 0 1 2
es-slave03: 2 0

五个节点:
es-master: 0 1
es-slave01: 3 4
es-slave02: 2 4
es-slave03: 2 3
es-slave04: 0 1

六个节点:
es-master: 0 1
es-slave01: 3 4
es-slave02: 2 4
es-slave03: 2 3
es-slave04: 1
es-slave05: 0

综合下来看,数据要求至少是一份主分片和一份副本分片,并且散落在不同的机器上。

补充(提高性能)

正常情况下, 一个索引(index)会拥有五个分片(shards),当机器数量达到5时,每个机器上都会均匀的分配到一个分片和一个副本分片。但是当拥有十台机器时,每个机器上就只会有一个分片(可能是主分片也可能是副本分片但是最终只有一份),性能理论上就会得到大大的提升。
如果这还不够,那么可以在加5台机器,并在/webiste/_settings中设置number_of_replicas为2(默认是一份副本),那么15台机器中将会有10台机器是副本分片5台机器是主分片,理论上来讲性能又会得到更多提升,同时灾备更健壮。
以此类推,20台机器可以将副本分片设置为3,25台机器可以将副本分片设置为4等等。

参考

-[x] 集群内的原理 中文英文

本文转载自:http://www.jianshu.com/p/2cac077e05cf

共有 人打赏支持
AllenOR灵感
粉丝 11
博文 2635
码字总数 83001
作品 0
程序员
私信 提问
详细介绍如何搭建ELK集群

Logstash 1.1 安装 注:安装在需要收集日志的机器上。 1.2 创建配置 1.3 启动 ElasticSearch集群(三台) 2.1 安装 2.2 配置 2.2.1 elasticsearch01 2.2.2 elasticsearch02 2.2.3 elasticsea...

禁区铁铍人
2017/12/07
0
0
Elasticsearch是什么以及核心概念

Elasticsearch是什么 Elasticsearch是一个高度可扩展的开源全文搜索和分析引擎。它可以在很短的时间内存储,搜索和分析大量的数据。它通常作为具有复杂搜索场景情况下的核心发动机。我们举几...

赛克蓝德
2016/02/03
1K
0
ElasticSearch入门--第一讲(理论篇)

集群 集群是一个或多个节点的集合,用来保存应用的全部数据并提供基于全部节点的集成式索引和搜索功能。每个集群都需要有一个惟一的名称。 节点 节点是一个集群中的单台服务器,用来保存数据...

有事没事
2016/10/11
6
0
elasticsearch基础操作01

ElasticSearch ES是一个基于Lucene实现的开源、分布式、Restful的全文本搜索引擎;此外,它还是一个分布式实时文档存储,其中每个文档的每个 field均是被索引的数据,且可被搜索;也是一个带...

limingyu0312
05/15
0
0
ElasticSearch5.1 基本概念和配置

一、说明 使用的是新版本5.1,直接从官网下载rpm包进行安装,https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.rpm 二、几个基本概念 接近实时(NRT) Elasticsea...

PeanutLike
2016/12/14
1K
1

没有更多内容

加载失败,请刷新页面

加载更多

区块链安全 - 以太坊短地址攻击

1 基础知识 EVM虚拟机在解析合约的字节码时,依赖的是ABI的定义,从而去识别各个字段位于字节码的什么地方。关于ABI,可以阅读这个文档: https://github.com/ethereum/wiki/wiki/Ethereum-C...

HiBlock
15分钟前
0
0
自定义函数及内部函数

变量的作用域 局部变量 global $Global及其他超全局数组 静态变量 仅初始化赋值 保留于内存直到response才销毁 global和static变量的区别 global:局部变量全局话 static:定义静态局部变量 函...

关元
16分钟前
0
0

中国龙-扬科
28分钟前
1
0
python包

https://www.lfd.uci.edu/~gohlke/pythonlibs/

陆朋
38分钟前
1
0
一文弄懂“分布式锁”,一直以来你的选择依据正确吗?

本文主要会关注的问题是“分布式锁”的问题。 多线程情况下对共享资源的操作需要加锁,避免数据被写乱,在分布式系统中,这个问题也是存在的,此时就需要一个分布式锁服务。 常见的分布式锁实...

Java干货分享
46分钟前
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部