/Data

Why Software Only Moves Forward

tl;dr: “Data is the biggest reason software only moves forward. Once you save state, your code will need to understand that state forever. This is double true for state that leaves your system and becomes distributed. Billing state, emails, and async jobs are a common early introduction to these issues.”

featured in #565

Use Data That Looks Like Data

- Thorsten Ball tl;dr: “Time for me to pass on something I've been practicing for years but haven't found written down somewhere. It's a simple thing. A practical thing. Forged in the trenches. It won't win any contests in which the audience gasps and says "oh, now that is clever." But it’s easy. Simple even. And it can save a lot of time and tears and, at the end of the day, isn't that some of the best stuff? When debugging or testing your program, do not use data that looks like a variable or type name.”

featured in #557

How We Built Ngrok's Data Platform

- Christian Hollinger tl;dr: “How we built it, what we learned, as well as some selective deep dives I found interesting enough to be worth sharing in more detail, since they’ll bridge the gap between what people usually understand by the term “data engineering” and how we run data here at ngrok. Some of this might even be useful for your own data platform endeavors, whether your team is big or small.”

featured in #556

Introducing Netflix’s Key-Value Data Abstraction Layer

tl;dr: “In this post, we dive deep into how Netflix’s KV abstraction works, the architectural principles guiding its design, the challenges we faced in scaling diverse use cases, and the technical innovations that have allowed us to achieve the performance and reliability required by Netflix’s global operations.”

featured in #552

Building And Scaling Notion’s Data Lake

tl;dr: “In the past three years Notion’s data has expanded 10x due to user and content growth, with a doubling rate of 6-12 months. Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake. Here’s how we did it.”

featured in #533

Data Loaders For The Win

- Allison Horst tl;dr: Slow data apps hinder data exploration by viewers and developers, leaving insights on the table. See how data loaders can help you speed up data apps by pushing bulky data access, wrangling and analysis “behind the scenes” on build instead of on page load.

featured in #516

Data Fetching Patterns In Single-Page Applications

- Juntao Qiu tl;dr: “When a single-page application needs to fetch data from a remote source, it needs to do so while remaining responsive and providing feedback to the user during an often slow query. Five patterns help with this. Asynchronous State Handler wraps these queries with meta-queries for the state of the query. Parallel Data Fetching minimizes wait time. Fallback Markup specifies fallback displays in markup. Code Splitting loads only code that's needed. Prefetching gathers data before it may needed to reduce latency when it is.”

featured in #515

Building A Weather Data Warehouse Part I: Loading A Trillion Rows Of Weather Data Into TimescaleDB

- Ali Ramadhan tl;dr: “I think it would be cool to have historical weather data from around the world to analyze for signals of climate change we’ve already had rather than think about potential future change.” Ali discusses the implementation of this analysis tool. 

featured in #510

Struggling with Snowflake Costs? Try our Cost Optimization Calculator

tl;dr: Snowflake costs skyrocket for SaaS providers because the need to deliver real-time, interactive analytics is always on. If your Snowflake bill is spiraling, try our cost optimization calculator to discover your potential savings when using a Snowflake warehouse for ad-hoc queries. (No form required)

featured in #501

Top 5 Challenges of Designing Your Data Warehouse for Multi-Tenant Analytics

tl;dr: Data warehouses are built to store large volumes of data from numerous sources, not for SaaS platforms working with multi-tenant analytics where data security is vital. This guide helps you avoid the headaches that come with that architecture mismatch featuring solutions from our analytics experts.

featured in #499