Unlock access to the data your product needs (Sponsored)
Most tools are still locked to their own database, blind to everything users already have in Slack, GitHub, Salesforce, Google Drive, and dozens of other apps. That's the ceiling on what you can build. WorkOS Pipes removes it. One API call connects your product to the apps your users live in. Pull context from their tools, cross-reference data across silos, power AI agents that act across services. All with fresh, managed credentials you never have to think about. Every Friday morning, a team at Spotify takes hundreds of code changes written by dozens of engineering teams and begins packaging them into a single app update. That update will eventually reach more than 675 million users on Android, iOS, and Desktop. They do this every single week. And somehow, more than 95% of those releases ship to every user without a hitch. The natural assumption is that they’re either incredibly careful and therefore slow, or incredibly fast and therefore reckless. The truth is neither. How do you ship to 675 million users every week, with hundreds of changes from dozens of teams running on thousands of device configurations, without breaking things? The answer is not to test really hard. Spotify built a release architecture where speed and safety reinforce each other. In this article, we will take a look at this process in detail and attempt to derive learnings. Disclaimer: This post is based on publicly shared details from the Spotify Engineering Team. Please comment if you notice any inaccuracies. The Two-Week Journey of a Spotify ReleaseTo see how this works, let us follow a single release from code merge to production. Spotify practices trunk-based development, which means that all developers merge their code into a single main branch as soon as it’s tested and reviewed. There are no long-lived feature branches where code sits in isolation for weeks. Everyone pushes to the same branch continuously, which keeps integration problems small but requires discipline and solid automated testing. See the diagram below that shows the concept of trunk-based development:
Each release cycle starts on a Friday morning. The version number gets bumped on the main branch. From that point, nightly builds start going out to Spotify employees and a group of external alpha testers. During this first week, teams develop and merge new code freely. Bug reports flow in from internal and alpha users. Crash rates and other quality metrics are tracked for each build, both automatically and by human review. When a crash or issue crosses a predefined severity threshold, a bug ticket gets created automatically. When something looks suspicious but falls below that threshold, the Release Manager can create one manually.
On the Friday of the second week, the release branch gets cut, meaning a separate copy of the codebase is created specifically for this release. This is the key moment in the release cycle. From this point, only critical bug fixes are allowed on the release branch. Meanwhile, the main branch keeps moving. New features and non-critical fixes continue to merge there, destined for next week’s release. This separation is the mechanism that lets Spotify develop at full speed while simultaneously stabilizing what’s about to ship. Teams then perform regression testing, checking that existing features still work correctly after the new changes, and report their results. Teams with high confidence in their automated tests and pre-merge routines can opt out of manual testing entirely. Beta testers receive builds from the more stable release branch, providing additional real-world runtime over the weekend. By Monday, the goal is to submit the app to the stores. By Tuesday, if the app store review passes and quality metrics look good, Spotify rolls it out to 1% of users. By Wednesday, if nothing alarming surfaces, they roll out to 100%. The flow below shows all the steps in a typical release process:
As an example, for version 8.9.2, which carried the Audiobooks feature launch in new markets, this timeline played out almost exactly as planned. What made that possible was everything happening behind the timeline. Rings of Exposure: Catching Bugs Where They’re Cheapest to FixThe code doesn’t go from a developer’s laptop to 675 million users in one jump. It passes through concentric rings of users, and each ring exists to catch a specific category of failure.
For reference, the Audiobooks launch in version 8.9.2 shows how this system works at an even more granular level. The Audiobooks feature didn’t just pass through these five rings of app release. It had its own layered rollout on top of that. The feature code had been sitting in the app for multiple releases already, hidden behind a backend feature flag. It was turned on for most employees first. The team watched for any crash, no matter how small, that might indicate trouble. Only after the app release itself reached a sufficient user base did the Audiobooks team begin gradually enabling the feature for real users in specific markets, using the same backend flag to control the percentage. See the diagram below that shows the concept of a feature flag:
This separation between deploying code and activating a feature is a powerful pattern in the Spotify release process. It allows code to sit in the app, baking in production conditions invisibly, and get turned on later. If something goes wrong after activation, the feature can be turned off without shipping a new release. At Spotify’s scale, feature flags are a core safety mechanism, though managing hundreds of them across a large organization, each with per-market and per-user-percentage controls, is its own engineering challenge. The Release Manager also made a deliberate coordination decision for 8.9.2. Since the Audiobooks feature was a high-stakes launch with marketing events already scheduled, another major feature that had been planned for the same release was rescheduled to the following week. Fewer variables in a single release means easier diagnosis if something goes wrong. That kind of judgment call is one of the things that separates release management from pure automation. From Jira to a Release Command CenterThe multi-ring system generates a lot of data, such as Crash rates, bug tickets, sign-off statuses, build verification results, and app store review progress. Someone has to make sense of all of it, and this wasn’t an easy task. Before the Release Manager Dashboard existed, everything lived in Jira. The Release Manager had to jump between tickets, check statuses across multiple views, and verify conditions manually, all while answering questions from teams on Slack. It was easy to miss a small detail, and a missed detail could mean extra work or a bug slipping through. So the Release team built a dedicated command center with clear goals:
The result was the Release Manager Dashboard, built as a plugin on Backstage, Spotify’s internal developer portal.
It pulls and aggregates data from around 10 different backend systems into a single view. For each platform (Android, iOS, Desktop), the dashboard shows blocking bugs, the latest build status, automated test results, crash rates normalized against actual usage (so a crash rate is meaningful whether 1,000 or 1,000,000 people are using the build), team sign-off progress, and rollout state. Everything is color-coded
Here’s an example of how the dashboard appears:
The dashboard also surfaces release criteria as a visible checklist:
When everything goes green, the release is ready to advance. The dashboard got off to a rocky start, however. The first version was slow and expensive. Every page reload triggered queries to all 10 of the source systems it depended on, causing long load times and high costs. The Spotify engineering team noted that each reload cost about as much as a decent lunch in Stockholm. After switching to caching and pre-aggregating data every five minutes, load time dropped to eight seconds, and the cost became negligible. The Robot: Automating the Predictable, Keeping Humans for the AmbiguousThe dashboard gave the Release Manager the information to make fast decisions. However, by analyzing the time-series data the dashboard generated, the team noticed that a lot of the time in the release cycle wasn’t spent on hard decisions, but waiting. The biggest time sinks were testing and fixing bugs (unavoidable), waiting for app store approval (outside Spotify’s control), and delays from manually advancing a release when a step was completed outside working hours. That last one alone could cost up to 12 hours. If the app store approved a build at 11 PM, the release just sat there until someone woke up and clicked “next.” Therefore, the team built what they called “the Robot.” It’s a backend service that models the release process as a state machine, a set of defined stages with specific conditions that must be met before moving to the next one. The Robot tracks seven states. The five states on the normal path forward are release branched, final release candidate (the build that will actually ship), submitted for app store review, rolled out to 1%, and rolled out to 100%. Two additional states handle problems, which means either the rollout gets paused or the release gets cancelled entirely. See the diagram below:
The Robot continuously checks whether the conditions for advancing to the next state are met. If manual testing is signed off, no blocking bugs are open, and automated tests pass on the latest commit on the release branch, the Robot automatically submits the build for app store review without human intervention. If the app store approves the build at 3 AM, the Robot initiates the 1% rollout immediately instead of waiting for someone to show up at the office. The result was an average reduction of about eight hours per release cycle. However, the Robot doesn’t make the hard calls. It doesn’t decide whether a crash affecting users in a specific region is severe enough to block a release. It doesn’t decide whether a bug in a new feature like Audiobooks, with marketing events already scheduled, should delay the entire release or just the feature rollout. It doesn’t negotiate with feature teams about timing. Those decisions require judgment, context, and sometimes difficult conversations. The Release Manager handles all of them. This split is deliberate. Predictable transitions that depend on rule-checks get automated. Ambiguous decisions that require coordination and judgment stay with humans. ConclusionSpotify ships weekly to 675 million users through a strong release architecture. Layered exposure catches bugs where they’re cheapest to fix and centralized tooling turns scattered data into fast decisions. Automation handles the predictable so humans can focus on the ambiguous. The key lesson here is that speed and safety aren’t opposites. At Spotify, each one enables the other. A weekly cadence means each release carries fewer changes. Fewer changes mean less risk per release. Less risk means shipping with confidence. Since a cancelled release only costs one week, not a month or a quarter, teams are more willing to kill a bad release rather than push it through and hope for the best. References:
|
Comments
Post a Comment