文档章节

LevelDB的算分逻辑

强子1985
 强子1985
发布于 2017/07/23 11:07
字数 882
阅读 56
收藏 0

每次1个新文件,先算放到哪层上,这是根据是否有key的overlap来决定的,这个逻辑见

stop in org.iq80.leveldb.impl.Version.pickLevelForMemTableOutput

int pickLevelForMemTableOutput(Slice smallestUserKey, Slice largestUserKey) {
		System.out.println("" + new String(smallestUserKey.getBytes()));
		System.out.println("" + new String(largestUserKey.getBytes()));
		// ================岁月静好,现世安稳===================
		int level = 0;
		// ================岁月静好,现世安稳===================
		// ================岁月静好,现世安稳===================
		// ================岁月静好,现世安稳===================
		if (!overlapInLevel(0, smallestUserKey, largestUserKey)) {
			// ================岁月静好,现世安稳===================
			// Push to next level if there is no overlap in next level,
			// and the #bytes overlapping in the level after that are limited.
			// ================岁月静好,现世安稳===================
			InternalKey start = new InternalKey(smallestUserKey, MAX_SEQUENCE_NUMBER, ValueType.VALUE);
			InternalKey limit = new InternalKey(largestUserKey, 0, ValueType.VALUE);
			// ================岁月静好,现世安稳===================
			while (level < MAX_MEM_COMPACT_LEVEL) {
				// ================岁月静好,现世安稳===================
				if (overlapInLevel(level + 1, smallestUserKey, largestUserKey)) {
					break;
				}
				// ================岁月静好,现世安稳===================
				long sum = Compaction.totalFileSize(versionSet.getOverlappingInputs(level + 2, start, limit));
				if (sum > MAX_GRAND_PARENT_OVERLAP_BYTES) {
					break;
				}
				// ================岁月静好,现世安稳===================
				level++;
			}
		}
		return level;
		// ================岁月静好,现世安稳===================
	}

 

选好了层之后,需要apply一下,这个见

stop in  org.iq80.leveldb.impl.VersionSet$Builder.apply --- 选好level,放到对应的层级上

/**
		 * Apply the specified edit to the current state.
		 */
		public void apply(VersionEdit edit) {
			// 看到这里了
			// Update compaction pointers
			for (Entry<Integer, InternalKey> entry : edit.getCompactPointers().entrySet()) {
				Integer level = entry.getKey();
				InternalKey internalKey = entry.getValue();
				versionSet.compactPointers.put(level, internalKey);
			}
			// 看到这里了=======================
			// Delete files
			for (Entry<Integer, Long> entry : edit.getDeletedFiles().entries()) {
				Integer level = entry.getKey();
				Long fileNumber = entry.getValue();
				levels.get(level).deletedFiles.add(fileNumber);
				// todo missing update to addedFiles?
			}
			// 看到这里了=======================
			// Add new files
			for (Entry<Integer, FileMetaData> entry : edit.getNewFiles().entries()) {
				Integer level = entry.getKey();
				FileMetaData fileMetaData = entry.getValue();

				// We arrange to automatically compact this file after
				// a certain number of seeks. Let's assume:
				// (1) One seek costs 10ms
				// (2) Writing or reading 1MB costs 10ms (100MB/s)
				// (3) A compaction of 1MB does 25MB of IO:
				// 1MB read from this level
				// 10-12MB read from next level (boundaries may be misaligned)
				// 10-12MB written to next level
				// This implies that 25 seeks cost the same as the compaction
				// of 1MB of data. I.e., one seek costs approximately the
				// same as the compaction of 40KB of data. We are a little
				// conservative and allow approximately one seek for every 16KB
				// of data before triggering a compaction.
				int allowedSeeks = (int) (fileMetaData.getFileSize() / 16384);
				if (allowedSeeks < 100) {
					allowedSeeks = 100;
				}
				fileMetaData.setAllowedSeeks(allowedSeeks);

				levels.get(level).deletedFiles.remove(fileMetaData.getNumber());
				levels.get(level).addedFiles.add(fileMetaData);
			}
			// 看到这里了=======================
		}

 

 

接下来开始算分

stop in  org.iq80.leveldb.impl.VersionSet.finalizeVersion  --- 算分

private void finalizeVersion(Version version) {
		{
			printFile(version);
		}
		// Precomputed best level for next compaction
		// ==================================================================
		int bestLevel = -1;
		double bestScore = -1;
		// ==================================================================
		for (int level = 0; level < version.numberOfLevels() - 1; level++) {
			double score;
			if (level == 0) {
				// We treat level-0 specially by bounding the number of files
				// instead of number of bytes for two reasons:
				//
				// (1) With larger write-buffer sizes, it is nice not to do too
				// many level-0 compactions.
				//
				// (2) The files in level-0 are merged on every read and
				// therefore we wish to avoid too many files when the individual
				// file size is small (perhaps because of a small write-buffer
				// setting, or very high compression ratios, or lots of
				// overwrites/deletions).
				// ==================================================================
				score = 1.0 * version.numberOfFilesInLevel(level) / L0_COMPACTION_TRIGGER;
			} else {
				// Compute the ratio of current size to size limit.
				long levelBytes = 0;
				for (FileMetaData fileMetaData : version.getFiles(level)) {
					levelBytes += fileMetaData.getFileSize();
				}
				score = 1.0 * levelBytes / maxBytesForLevel(level);
			}

			if (score > bestScore) {
				bestLevel = level;
				bestScore = score;
			}
		}
		// 看到这里了
		version.setCompactionLevel(bestLevel);
		version.setCompactionScore(bestScore);
		// ==================================================================
	}

 

算完分后,开始选择是否需要compaction

stop in  org.iq80.leveldb.impl.VersionSet.pickCompaction ---根据分数来决定是否要compaction

public Compaction pickCompaction() {
		// ================岁月静好,现世安稳===================
		// We prefer compactions triggered by too much data in a level over
		// the compactions triggered by seeks.
		// ================岁月静好,现世安稳===================
		boolean sizeCompaction = (current.getCompactionScore() >= 1);
		boolean seekCompaction = (current.getFileToCompact() != null);

		int level;
		List<FileMetaData> levelInputs;
		// ================岁月静好,现世安稳===================
		if (sizeCompaction) {
			level = current.getCompactionLevel();
			// ================岁月静好,现世安稳===================
			Preconditions.checkState(level >= 0);
			Preconditions.checkState(level + 1 < NUM_LEVELS);
			// ================岁月静好,现世安稳===================
			// Pick the first file that comes after compact_pointer_[level]
			levelInputs = newArrayList();
			// ================岁月静好,现世安稳===================
			for (FileMetaData fileMetaData : current.getFiles(level)) {
				if (!compactPointers.containsKey(level)
						|| internalKeyComparator.compare(fileMetaData.getLargest(), compactPointers.get(level)) > 0) {
					levelInputs.add(fileMetaData);
					break;
				}
			}
			if (levelInputs.isEmpty()) {
				// Wrap-around to the beginning of the key space
				levelInputs.add(current.getFiles(level).get(0));
			}
		}
		// ================岁月静好,现世安稳===================
		else if (seekCompaction) {
			level = current.getFileToCompactLevel();
			levelInputs = ImmutableList.of(current.getFileToCompact());
		} else {
			return null;
		}

		// Files in level 0 may overlap each other, so pick up all overlapping
		// ones
		if (level == 0) {
			Entry<InternalKey, InternalKey> range = getRange(levelInputs);
			// Note that the next call will discard the file we placed in
			// c->inputs_[0] earlier and replace it with an overlapping set
			// which will include the picked file.
			levelInputs = getOverlappingInputs(0, range.getKey(), range.getValue());

			Preconditions.checkState(!levelInputs.isEmpty());
		}

		Compaction compaction = setupOtherInputs(level, levelInputs);
		return compaction;
	}

 

© 著作权归作者所有

共有 人打赏支持
强子1985

强子1985

粉丝 882
博文 1179
码字总数 870993
作品 8
南京
架构师
私信 提问
用Kyoto Tycoon挂载LevelDB存储

Kyoto Tycoon(以下简称KT)是TokyoTyrant的作者Mikio Hirabayashi 的系列作品之一,KT 是一个数据库网络层服务,它提供一个插件机制,可以挂载几乎所有的数据库存储设备。之前已经有过KT嫁接...

红薯
2011/07/31
2.3K
2
兄弟连区块链教程Fabric1.0源代码分析LevelDB KV数据库

Fabric 1.0源代码笔记 之 LevelDB(KV数据库) 1、LevelDB概述 LevelDB是Google开源的持久化KV单机数据库,具有很高的随机写,顺序读/写性能,但是随机读的性能很一般,也就是说,LevelDB很适...

兄弟连区块链入门教程
2018/11/01
0
0
LevelDB 入门 —— 全面了解 LevelDB 的功能特性

本节我们将全面了解一下 LevelDB 的各种特性。LevelDB 的开发语言是 C++,考虑到会使用 C++ 语言的同学不是很多,在本节我们将使用 Java 语言来描述 LevelDB 的特性。其它语言栈的同学也不必...

技术小能手
01/11
0
0
activemq集群假死

现象是:进程都还在,但是端口已经不监听,查看日志发现以下错误,集群重启后就好了,这个情况大家碰到过吗,有规避的办法吗,在这里先谢谢了 2017-01-04 04:43:12,184 | WARN | listeners a...

youarepp
2017/01/05
666
3
rl_set key val能否保证一致性,leveldb写成功,redis写失败如何处理?

redis-storage能否做到强一致性 比如:rl_set key val 往redis和leveldb写值, 优先顺序:leveldb > redis, leveldb如果失败,将中断往redis写,返回错误 那如果,写leveldb成功,写redis时失...

笑天居士
2014/10/27
154
0

没有更多内容

加载失败,请刷新页面

加载更多

谈谈lucene的DocValues特性之SortedDocValuesField

lucene的DocValues不同于document文档级别的存储,它是一个面向列的存储结构,提供从文档编号到值的映射功能。根据不同的数据类型和应用场景支持多个DocValuesField类型,SortedDocValuesFi...

FAT_mt
19分钟前
0
0
漏洞防御与修复工作

导读 漏洞管理工作是企业安全建设必不可少的一环,在风险管理工作中,漏洞管理能够防患于未然,企业对漏洞管理有着广泛的基础建设和实践经验。但随着攻防技术的发展,传统漏洞管理的安全技术...

问题终结者
22分钟前
0
0
做菜

https://www.meishij.net/zuofa/douchirou_3.html https://www.meishij.net/zuofa/doufupaohongshaorou_1.html...

阿锋zxf
22分钟前
0
0
TortoiseSVN使用简介

TortoiseSVN使用简介 1 安装及下载client 端 2 什么是SVN(Subversion)? 3 为甚么要用SVN? 4 怎么样在Windows下面建立SVN Repository? 5 建立一个Working目录 6 新增档案及目录到Repositor...

linjin200
36分钟前
0
0
ZStack--网络模型1:L2和L3网络

ZStack将网络模型抽象为L2和L3网络。L2网络提供一种二层网络隔离的方式,而L3网络主要和OSI七层模型中第4层~第7层网络服务相对应。我们的想法是使用管理员熟悉的术语和概念,来形容ZStack的网...

ZStack社区版
38分钟前
0
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部