Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 1720. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCFhosted projects.

Lesson Learned on Running Hadoop on Kubernetes Chen Qiang, LinkedIn

LinkedIn operates one of the world’s largest Hadoop environments, with ~450PB used data, 2 billion files/blocks, and over 400K jobs/day. However, testing cluster features in an isolated fashion has been traditionally fairly difficult. Infra teams such as HDFS, YARN, and Azkaban often step on top of one another for testing new features in our existing test Hadoop clusters. Setting up a new test cluster requires coordination between hardware, infra, and security teams, usually taking weeks to months. We have recently extended Kubernetes’ usage to test Hadoop(HDFS/YARN) clusters, by deploying productionlike Hadoop cluster on Kubernetes. This has reduced infra setup time from weeks down to minutes with no network, hardware dependencies, and enables critical infra/workflow teams to test new features on the fly.

https://sched.co/ZeoG