在线时间:8:00-16:00
迪恩网络APP
随时随地掌握行业动态
扫描二维码
关注迪恩网络微信公众号
开源软件名称:Kraken开源软件地址:https://gitee.com/mirrors/Kraken开源软件介绍:Kraken is a P2P-powered Docker registry that focuses on scalability and availability. It isdesigned for Docker image management, replication, and distribution in a hybrid cloud environment.With pluggable backend support, Kraken can easily integrate into existing Docker registry setupsas the distribution layer. Kraken has been in production at Uber since early 2018. In our busiest cluster, Kraken distributesmore than 1 million blobs per day, including 100k 1G+ blobs. At its peak production load, Krakendistributes 20K 100MB-1G blobs in under 30 sec. Below is the visualization of a small Kraken cluster at work:
Table of Contents
FeaturesFollowing are some highlights of Kraken:
DesignThe high-level idea of Kraken is to have a small number of dedicated hosts seeding content to anetwork of agents running on each host in the cluster. A central component, the tracker, will orchestrate all participants in the network to form apseudo-random regular graph. Such a graph has high connectivity and a small diameter. As a result, even with only one seeder andhaving thousands of peers joining in the same second, all participants can reach a minimum of 80%max upload/download speed in theory (60% with current implementation), and performance doesn'tdegrade much as the blob size and cluster size increase. For more details, see the team's techtalk at KubeCon + CloudNativeCon. Architecture
BenchmarkThe following data is from a test where a 3G Docker image with 2 layers is downloaded by 2600 hostsconcurrently (5200 blob downloads), with 300MB/s speed limit on all agents (using 5 trackers and5 origins):
UsageAll Kraken components can be deployed as Docker containers. To build the Docker images: $ make images For information about how to configure and use Kraken, please refer to the documentation. Kraken on KubernetesYou can use our example Helm chart to deploy Kraken (with an example HTTP fileserver backend) onyour k8s cluster: $ helm install --name=kraken-demo ./helm Once deployed, every node will have a docker registry API exposed on For more information on k8s setup, see README. DevclusterTo start a herd container (which contains origin, tracker, build-index and proxy) and two agentcontainers with development configuration: $ make devcluster Docker-for-Mac is required for making dev-cluster work on your laptop.For more information on devcluster, please check out devcluster README. Comparison With Other ProjectsDragonfly from AlibabaDragonfly cluster has one or a few "supernodes" that coordinates the transfer of every 4MB chunk of datain the cluster. While the supernode would be able to make optimal decisions, the throughput of the whole cluster islimited by the processing power of one or a few hosts, and the performance would degrade linearly aseither blob size or cluster size increases. Kraken's tracker only helps orchestrate the connection graph and leaves the negotiation of actual datatransfer to individual peers, so Kraken scales better with large blobs.On top of that, Kraken is HA and supports cross-cluster replication, both are required for areliable hybrid cloud setup. BitTorrentKraken was initially built with a BitTorrent driver, however, we ended up implementing our P2Pdriver based on BitTorrent protocol to allow for tighter integration with storage solutions and morecontrol over performance optimizations. Kraken's problem space is slightly different than what BitTorrent was designed for. Kraken's goal isto reduce global max download time and communication overhead in a stable environment, whileBitTorrent was designed for an unpredictable and adversarial environment, so it needs to preserve morecopies of scarce data and defend against malicious or bad behaving peers. Despite the differences, we re-examine Kraken's protocol from time to time, and if it's feasible, wehope to make it compatible with BitTorrent again. Limitations
ContributingPlease check out our guide. ContactTo contact us, please join our Slack channel. |
请发表评论