文档章节

A Digest of Evernote’s Architecture (*) 转

黄俊que
 黄俊que
发布于 2014/05/31 22:57
字数 963
阅读 17
收藏 0

Let’s get things started with a coarse-grained overview of the physical makeup of the Evernote service. I won’t go into a lot of detail on each component here; we’ll aim to talk about the interesting bits in separate posts later.

Starting at the top-left corner of the diagram, all stats as of May 17th, 2011 …

Networking: Virtually all traffic to and from Evernote comes to www.evernote.com via HTTPS port 443. This includes all “web” activity, but also all client synchronization via our Thrift-based service API.  Altogether, that produces up to 150 million HTTPS requests per day, with peak traffic around 250 Mbps. (Unfortunately for our semi-nocturnal Operations team, this daily peak tends to arrive around 6:30 am, Pacific time.)

We use BGP to direct traffic through fully independent network feeds from our primary (NTT) and secondary (Level 3) providers. This is filtered through Vyatta on the way to the new A10 load balancers that we deployed in January when we hit the limit of SSL performance for our old balancers. We’re comfortably handling existing traffic using one AX 2500 plus a failover box, but we’re preparing to test their N+1 configuration in our staging cluster to prepare for future growth.

Shards: The core of the Evernote service is a farm of servers that we call “shards.”  Each shard handles all data and all traffic (web and API) for a cohort of 100,000 registered Evernote users.  Since we have more than 9 million users, this translates into around 90 shards.

Physically, shards are deployed as a pair a SuperMicro boxes with two quad-core Intel processors, a ton of RAM and full chassis of Seagate enterprise drives in mirrored RAID configurations. On top of each box, we run a base Debian host that manages two Xen virtual machines. The primary VM on a box runs our core application stack:  Debian + Java 6 + Tomcat + Hibernate + Ehcache +  Stripes +GWT MySQL (for metadata) + hierarchical local file systems (for file data).

All user data on the primary VM on one box is kept synchronously replicated to a secondary VM on a different box using DRBD. This means that each byte of user data is on at least four different enterprise drives across two different physical servers, plus nightly backups. If we have any problems with a server, we can fail the services from its primary VM over to the secondary on another box with minimal downtime via Heartbeat.

Since each users’ data is completely localized to one (virtual) shard host, we can run each shard as an independent island with virtually no crosstalk or dependencies. This means that issues on one shard don’t snowball to other shards.

To connect users with their shard, we push most of the work into the load balancers, which have a cascade of rules to find the shard in the URL and/or cookies.

UserStore: While the vast majority of all data is stored within the single-tier NoteStore shards, they all share a single master “UserStore” account database (also MySQL) with a small amount of information about each account, such as: username, MD5 password, and user shard ID. This database is small enough to fit in RAM, but we maintain high redundancy with the same combination of RAID mirroring, DRBD replication to a secondary, and nightly backups.

AIR processors: In order to allow you to search for words found within images in your notes, we maintain a pool of 28 servers that spend each day using their 8 cores to process new images. On a busy day, this translates into 1.3 or 1.4 million separate images. Currently, these use a mix of Linux and Windows, but we plan to convert them all to Debian by the end of the month now that we’ve removed some pesky legacy dependencies.

These servers run a pipeline of “Advanced Imaging and Recognition” (AIR) software developed by our R&D team. This software cleans up each image, identifies word-shaped regions, and then attempts to compile a weighted list of possible matches for each word using a set of “recognition engines” that each contribute a set of guesses.  This includes engines developed by our own team which specialize in, for example, handwriting recognition, as well as licensed technologies from best-of-breed commercial partners.

Other services: All of these servers are racked in a pair of dedicated cages at our data center in Santa Clara, California. In addition to the hardware that provides our core service, we also have smaller groups of servers for lighter-weight tasks that only require one or two boxes or Xen virtual machines. For example, our “incoming email” SMTP gateway is a pair of Debian servers with Postfixand a custom Java mail processor built on top of Dwarf. Our @myen Twitter gateway is a simple in-house daemon using twitter4j.

Our corporate web site is Apache, our blogs are WordPress, most of our fully redundant internal switching topology is from HP, we use Puppet for configuration management, and we monitor withZabbix, Opsview, and AlertSite. We run nightly backups with a combination of different software that migrates data over a dedicated 1Gbps link to a secondary data center.

Wait, but why? I realize this post leaves lots of obvious questions about why we’ve chosen to do X instead of Y in a number of different places. Why run our own servers instead of using a cloud provider? Why such stuffy old software (Java, SQL, local storage, etc.) instead of hot new magic bullets? …

We’ll try to get into more details to answer these questions in the next few months.

(*) UPDATE, June 29, 2011: The title of this post was changed from “Architectural Digest” at the request of Conde Nast.

本文转载自:

共有 人打赏支持
黄俊que
粉丝 1
博文 13
码字总数 5514
作品 0
长沙
架构师
私信 提问
17Python标准库系列之hashlib模块

Python标准库系列之hashlib模块 This module implements a common interface to many different secure hash and message digest algorithms. Included are the FIPS secure hash algorithm......

余二五
2017/11/07
0
0
[转] Leaving patterns & practices

[J.D. Meier's Blog]“Life is like skiing. Just like skiing, the goal is not to get to the bottom of the hill. It’s to have a bunch of good runs before the sun sets.” – Seth G......

长平狐
2012/09/04
192
0
跨平台就是脑残和平庸的代名词

 跟很多公司不同,在产品层面,我们从不信奉‘跨平台的一致性’(cross-platform consistency)。在我们看来,这种‘一致性’意味着‘平庸’而非‘卓越’;我们的目标是后者,代价再大也在所...

宏哥
2013/06/24
98
0
Camel概念【Architecture ①】

1.4 Camel’s architecture Let’s now turn our attention to Camel’s architecture. We’ll first take a look at the high-level architecture and then drill down into the specific c......

k_k_anna
2015/01/28
0
0
MessageDigest数字签名,加密

Java Cryptography Architecture,Java加密架构,java平台中用于访问和开发加密功能的框架。 MessageDigest 类 MessageDigest 类是一个引擎类,它是为了提供诸如 SHA1 或 MD5 等密码上安全的...

zh151832
2016/04/15
49
0

没有更多内容

加载失败,请刷新页面

加载更多

大数据处理也要安全--关于MaxCompute的安全科普

摘要: 企业从未像今天这样可以轻易地存储和使用大数据。然而,当您在使用大数据产品时,是否考虑过其中的安全问题呢?庆幸的是,阿里云产品专家和安全专家早就想你所想急你所急,先行一步将...

阿里云云栖社区
32分钟前
1
0
vue如何编写组件可以通过Vue.use()使用

一般平时用别人的组件时都是通过import引入然后Vue.use()来使用,那么如何让我们写的组件也可以用这种方式使用呢? 1.首先新建一个文件夹例如:Home,然后在该文件中新建两个文件Home.vue和i...

北辰丨丶
32分钟前
2
0
SpringBoot自动配置原理

前言 只有光头才能变强。 文本已收录至我的GitHub仓库,欢迎Star:https://github.com/ZhongFuCheng3y/3y 回顾前面Spring的文章(以学习的顺序排好): Spring入门这一篇就够了 Spring【依赖注...

Java3y
37分钟前
2
0
如何伪装成一个服务端开发(十) -- Spring MVC 源码

前言 在第七篇我们已经聊过了一些Spring MVC的运行原理,当然大多数人应该还是和我一样迷迷糊糊,只知道一个大概的运行过程,这一篇,我想要从源码的角度更加进一步去了解Spring MVC的整个运...

街角的小丑
41分钟前
1
0
应用前台耗电怎么破?功耗避雷指南已“佩奇”

使用应用时被用户吐槽手机掉电快、卡顿、过度发热,导致用户体验差,以上情况的产生,应用的功耗设计不足是直接症结。 当前,人们对性能体验的追求前所未有,应用设计功能越来越强大,界面也...

安卓绿色联盟
41分钟前
1
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部