Minions, a powerful framework to handle complex operational tasks in Pinot (Jan 24th)
Wednesday, January 24th, 2024: 9:00 PM to 10:30 PM
Join us for this exclusive meetup to learn about Minions- a powerful framework to handle complex operational tasks in Pinot
Apache Pinot is a real-time distributed OLAP datastore that is designed to support high-throughput queries with low latency for various analytics use cases. To maintain data integrity, result accuracy, and system efficiency, Pinot performs background operational tasks such as data compaction, GDPR data purging, table repartitioning, and schema evolution reindexing. These tasks can be resource-intensive and potentially impact query performance if executed on the same component responsible for query execution.
To address this challenge, Pinot utilizes Minion, which is built upon Apache Helix's task framework. Minion handles computationally intensive operational tasks, effectively offloading these workloads from the query execution component. This separation ensures that operational tasks do not compromise query performance. Minion is designed to be easily extensible and pluggable, serving not only to address performance issues but also to create data ingestion and backfilling pipelines, saving time for operators who would otherwise need to build custom solutions.
Hosted by StarTree