无穷扩展性的秘密：NetFlix开源武器库清单

近日MaxCDN的首席执行官Chris Ueland转发了一篇关于NetFlix高扩展性背后的技术架构揭秘文章。

作者收集整理了互联网上的大量公开资料，为我们描画出了NetFlix的核心技术架构，并列出了开源“武器库”清单。

首先我们来看下NetFlix的总体技术架构，分为应用与数据、业务工具、基础服务、DevOps和Open Connect CDN五大块（下图）。

NetFlix技术架构高可扩展性的秘密在于六点：

1.可扩展性的企业文化：信任员工、不搞企业政治

2.开发自己的CDN和多重CDN能力

3.倚重Amazon的多种云服务：所有的存储都放在亚马逊S3云，以及大量的计算载荷…（参考阅读：全球最大的云计算实例：Netflix如何架构IT）

4.大量开发应用开源项目，从安全到测试、大数据，到亚马逊工具等

5.支持大量设备类型，超过50种

6.保持可扩展性算法的先进性，通过奖金高昂的NetFlix Prize竞赛不断推出更好的可扩展性算法

NetFlix高可扩展性背后的企业文化

NetFlix的开源“武器库”清单

NetFlix的工程师最近发布了一篇名为Netflix的开源进化的文章，读者还可以在这里（https://netflix.github.io/）找到NetFlix的开源工具清单，下面我们分门别类简单介绍一下这些开源工具：

大数据

• Genie – 一个NetFlix开发的，强大的基于REST的任务处理引擎，可运行包括Hadoop、Pig、Hive在内的各种数据处理任务。
• Inviso – Hadoop任务或集群性能的监控工具
• Lipstick – 简洁直观地展示Pig任务工作流
• Aegisthus – 从Cassandra中大规模抽取数据提供给下游的分析处理工作。

开发与交付工具

• Nebula – Effort at Netflix to share its internal build infrastructure.
• Aminator – A tool for creating EBS AMIs.
• Asgard – Web interface for application deployments and cloud management in Amazon Web Services (AWS).

通用Runtime服务与代码库

• Eureka – Service discovery for the Netflix cloud platform.
• Archaius – Distributed configuration.
• Ribbon – Resilent and intelligent inter-process and service communication.
• Hystrix – Provides reliability beyond single service calls. Isolates latency and fault tolerance at runtime.
• Karyon and Governator – JVM container services.
• Prana sidecar – Prana provides proxy capabilities within an instance.
• Zuul – Provides dyamically scriptable proxying at the edge of the cloud deployment.
• Fenzo – Provides advanced scheduling and resource management for cloud native frameworks.

数据持久性

• EVCache and Dynomite – For using Memcached and Redis at scale.
• Astyanax and Dyno – Client libraries to better consume datastores in the Cloud.

分析、可靠性与性能

• Atlas – Time-series telemetry platform
• Edda – Service to track changes in your cloud
• Spectator – Easy integration of Java application code with Atlas
• Vector – Exposes high-resolution host-level metrics with minimal overhead.
• Ice – Exposes ongoing cost and and cloud utilization trends.
• Simian Army – Tests Netflix instances for random failures.

安全工具

• Security Monkey – Helps monitor and secure large AWS-based environments.
• Scumblr – Leverages Internet-wide targeted searches to surface specific security issues for investigation.
• MSL – An extensible and flexible secure messaging protocol that addresses a number of secure communications use cases and requirements.
• Falcor – Represent remote data sources as a single domain model via a virtual JSON graph.
• Restify – node.js REST framework specifically meant for web service APIs
• RxJS – A reactive programming library for JavaScript