tl;dr:"Within Spotify, we run 20,000 batch data pipelines defined in 1,000+ repositories, owned by 300+ teams — daily. The majority of our pipelines rely on two tools: Luigi (Python) and Flo (Java). The data orchestration team decided to move away from these tools, and in this post, the team details why the decision was made, and the journey they took to make the transition."