Overclocking DBT: Discord's Custom Solution In Processing Petabytes Of Data
- Chris Dong tl;dr: “At Discord, we faced a challenge that would make most data teams flinch: scaling dbt to process petabytes of data while supporting 100+ developers simultaneously working across 2,500+ models. What started as a simple implementation quickly hit critical limitations to accommodate millions of concurrent users generating petabytes of data.”featured in #606
Making Uber’s ExperimentEvaluation Engine 100x Faster
tl;dr: This blog post describes how we made efficiency improvements to Uber’s Experimentation platform to reduce the latencies of experiment evaluations by a factor of 100x, milliseconds to microseconds. We accomplished this by going from a remote evaluation architecture to a local evaluation architecture.featured in #603
In Defense Of Simple Architectures
- Dan Luu tl;dr: Dan discusses the effectiveness of simple architectures in software development, using Wave, a $1.7B company, as an example. Wave's architecture is a Python monolith on top of Postgres, allowing engineers to focus on delivering value to users. The article emphasizes that simple architectures can be created more cheaply and easily than complex ones, even for high-traffic apps. Despite the trend towards complex, microservice-based architectures, Dan argues for the "unreasonable effectiveness" of monoliths, detailing Wave's choices, mistakes, and areas of unavoidable complexity. Simplicity in architecture can lead to success, allowing companies to allocate complexity where it benefits the business.featured in #599
In S3 Simplicity Is Table Stakes
- Werner Vogels tl;dr: “A few months ago at re:Invent, I spoke about Simplexity – how systems that start simple often become complex over time as they address customer feedback, fix bugs, and add features. At Amazon, we’ve spent decades working to abstract away engineering complexities so our builders can focus on what matters most: their unique business logic. There’s perhaps no better example of this journey than S3.”featured in #599
Systems Ideas That Sound Good But Almost Never Work
- Steven Sinofsky tl;dr: “I started my list with “let’s just” because 9 out of 10 times when someone says “let’s just” what follows is going to be ultimately way more complicated than anyone in the room thought it would be. I’m going to say “9 out of 10 times” a lot below on purpose because…experience. I offer an example of two below but for each there are probably a half dozen I lived through.”featured in #599
Systems Ideas That Sound Good But Almost Never Work
- Steven Sinofsky tl;dr: “I started my list with “let’s just” because 9 out of 10 times when someone says “let’s just” what follows is going to be ultimately way more complicated than anyone in the room thought it would be. I’m going to say “9 out of 10 times” a lot below on purpose because…experience. I offer an example of two below but for each there are probably a half dozen I lived through.”featured in #598
A Software Architecture Reading List
- Dr Andrew Leigh tl;dr: “Dr Andrew Leigh reflects on 25 years of professional experience as a software architect and offers his insights and a reading list of recommendations.”featured in #597
Supercharging Discord Mobile: Our Journey To A Faster App
- Ruby Feinstein tl;dr: Ruby discusses key improvements include virtualizing the server list to reduce memory usage, optimizing chat performance for smoother scrolling, and rebuilding the emoji picker natively to prevent blank frames. They also enhanced animated emoji rendering with WebP and eliminated “blanking” issues in lists.featured in #597
9 Software Architecture Patterns For Distributed Systems
tl;dr: “Architectural patterns provide proven solutions to common problems encountered in distributed systems, ensuring reliability, scalability, and maintainability. Among these patterns, some patterns stand out as fundamental for managing data and communication flow effectively, which we will see in this article.”featured in #597
9 Software Architecture Patterns For Distributed Systems
tl;dr: “Architectural patterns provide proven solutions to common problems encountered in distributed systems, ensuring reliability, scalability, and maintainability. Among these patterns, some patterns stand out as fundamental for managing data and communication flow effectively, which we will see in this article.”featured in #596