Apache Beam Cuts Processing Time 94% For LinkedIn
datanami, April, 5th 2023
Like many large companies, LinkedIn relied on the Lamba architecture to run separate batch and streaming workloads, with a form of reconciliation at the end.
After implementing Apache Beam, it was able to combine batch and streaming workloads, thereby slashing its processing time by 94%, the company says.
LinkedIn is a big user of Apache Samza, a distributed stream processing engine that the company developed in-house in Scala and Java, alongside the Apache Kafka message bus. The company uses Samza to process 2 trillion messages per day, writes LinkedIn Senior Software Engineer Yuhong Cheng in a March 23 post to the company's engineering blog.