无穷扩展性的秘密:NetFlix开源武器库清单

近日MaxCDN的首席执行官Chris Ueland转发了一篇关于NetFlix高扩展性背后的技术架构揭秘文章。

作者收集整理了互联网上的大量公开资料,为我们描画出了NetFlix的核心技术架构,并列出了开源“武器库”清单。

首先我们来看下NetFlix的总体技术架构,分为应用与数据、业务工具、基础服务、DevOps和Open Connect CDN五大块(下图)。

Netflix技术架构

NetFlix技术架构高可扩展性的秘密在于六点:

1.可扩展性的企业文化:信任员工、不搞企业政治

2.开发自己的CDN和多重CDN能力

3.倚重Amazon的多种云服务:所有的存储都放在亚马逊S3云,以及大量的计算载荷…(参考阅读:全球最大的云计算实例:Netflix如何架构IT

4.大量开发应用开源项目,从安全到测试、大数据,到亚马逊工具等

5.支持大量设备类型,超过50种

6.保持可扩展性算法的先进性,通过奖金高昂的NetFlix Prize竞赛不断推出更好的可扩展性算法

 

NetFlix高可扩展性背后的企业文化

Netflix企业文化

 

NetFlix的开源“武器库”清单

NetFlix的工程师最近发布了一篇名为Netflix的开源进化的文章,读者还可以在这里(https://netflix.github.io/)找到NetFlix的开源工具清单,下面我们分门别类简单介绍一下这些开源工具:

大数据

  • Genie – 一个NetFlix开发的,强大的基于REST的任务处理引擎,可运行包括Hadoop、Pig、Hive在内的各种数据处理任务。
  • Inviso – Hadoop任务或集群性能的监控工具
  • Lipstick – 简洁直观地展示Pig任务工作流
  • Aegisthus – 从Cassandra中大规模抽取数据提供给下游的分析处理工作。

开发与交付工具

  • Nebula – Effort at Netflix to share its internal build infrastructure.
  • Aminator – A tool for creating EBS AMIs.
  • Asgard – Web interface for application deployments and cloud management in Amazon Web Services (AWS).

通用Runtime服务与代码库

  • Eureka – Service discovery for the Netflix cloud platform.
  • Archaius – Distributed configuration.
  • Ribbon – Resilent and intelligent inter-process and service communication.
  • Hystrix – Provides reliability beyond single service calls. Isolates latency and fault tolerance at runtime.
  • Karyon and GovernatorJVM container services.
  • Prana sidecar – Prana provides proxy capabilities within an instance.
  • Zuul – Provides dyamically scriptable proxying at the edge of the cloud deployment.
  • Fenzo – Provides advanced scheduling and resource management for cloud native frameworks.

数据持久性

  • EVCache and Dynomite For using Memcached and Redis at scale.
  • Astyanax and Dyno Client libraries to better consume datastores in the Cloud.

分析、可靠性与性能

  • Atlas – Time-series telemetry platform
  • Edda – Service to track changes in your cloud
  • Spectator – Easy integration of Java application code with Atlas
  • Vector – Exposes high-resolution host-level metrics with minimal overhead.
  • Ice – Exposes ongoing cost and and cloud utilization trends.
  • Simian Army – Tests Netflix instances for random failures.

安全工具

  • Security Monkey – Helps monitor and secure large AWS-based environments.
  • Scumblr – Leverages Internet-wide targeted searches to surface specific security issues for investigation.
  • MSL – An extensible and flexible secure messaging protocol that addresses a number of secure communications use cases and requirements.
  • Falcor – Represent remote data sources as a single domain model via a virtual JSON graph.
  • Restify – node.js REST framework specifically meant for web service APIs
  • RxJS – A reactive programming library for JavaScript

第一时间获取面向IT决策者的独家深度资讯,敬请关注IT经理网微信号:ctociocom

   

除非注明,本站文章均为原创或编译,未经许可严禁转载。

相关文章:


关于作者

传媒领域资深人士,IT行业净坛使者。 联系邮箱:zhanglin@ctocio.com