Release Management For Scale Mobile Platforms

The modern version of loading the Noah's Ark is essentially releasing features on a scaled mobile platform. In this blog, we cover strategies that help us coordinate feature rollouts to millions of our customers on JioCinema. Can't break anything and won't rollout anything to a 100% without data, and, this is just on the final step! Read on to learn how we conduct this dance weekly and manage to stay with time to the music.

What's a release?

The JioCinema mobile application (app) has hundreds of millions of users. Releasing features, especially if that ties into marketing announcements or large public events watched by millions like Bigg Boss, TATA IPL or Election News is non trivial.

When we want to mention something in a press-release (for eg; when we launched paid subscriptions), or when we want to put out marketing campaigns which you might have seen during IPL like Jeeto Dhan Dhana Dhan or IPL stickers, the definition of a feature being “released” would mean 100% of all users should have access to it, at the moment they read the press-release or watch the advert.

On the other hand, at a usual standup of the engineering team, when asked if something is “released”, an engineer might say YES, if their code had been merged into the most recent build published to App Store / Play Store.

A product manager manging the ‘rollout’ of a feature has to carefully bridge this gap between different people’s definitions of released.

In this post we'll try to understand various challenges associated with continuous releases on mobile apps with a large userbase taking a contrived example of a fictitious feature called BlueBoxes™ and charting its journey from being developed to be available to end users, through the software delivery pipeline at JioCinema, to see how we tackle these challenges.

Challenges of Releasing at Scale

Taking “end user / customer being able to use a certain feature” as a definition of release, here are the challenges with releasing features on our mobile app that we face.

1. Does everyone have it? If there is code to be shipped inside the ‘client’ (mobile apps) - then we need to account for time to rollout an upgrade and users to adopt that upgrade. Depending on platform and metric stability, the "rollout" can take upto 4-5 weeks.
2. Gradual Ramp Release: If the feature will bring traffic in orders of 10s/100s of millions to a new service, we might want to do it in phased manner. Rollout is 100%, but experience is phased.
3. 3P Dependency : If the feature brings traffic or involves usage of a third-party vendor / partner, we have to account for their ability to scale up.
4. Kill Switches : During the rollout, if something goes wrong, we want to have the ability to halt without having to ask users to “please updated to the fixed app”, because rolling out a fixed app also takes a lot of time.
5. Cohort Rollout : Often, we want to release the feature to a specific group of users only (some regions, or some languages, or a randomly selected percentage) and observe its impact on product metrics, before we decide it to be a good idea to roll it out to 100% of our users. A/B testing new user flows are important tools for the product team.

The next time someone says "rollout", pause and articulate the above. We've covered what happens AFTER some feature has been sent out in the world.

Breaking down a Feature Release

Lets cover the landmines that exist while we're building a feature that will impact "rollout" and create dependencies.

The new feature: BlueBoxes™️

Let us imagine today in our app we fetch some data from the server, and based on the data returned we show some red boxes in the app for every x1, x2, x3... etc items.

We want to move to a future where if the server sends from y1,y2,y3.... items those are rendered as blue boxes in the app instead of red boxes.

Rollout Differences: Backend vs Frontend

Clients (Mobile apps)

• Assume app version 3.1.1 has the code to render only red boxes and parse only x_ objects, and no ability to read y_ objects or render blue boxes
• These features will be added to app version 3.2.0 at some future date
• When 3.2.0 will be released to market, not immediately 100% users will update their apps to 3.2.0
• For a long enough period (weeks), a big % of people will be on 3.1.1 as well as 3.2.0 simultaneously

Clients (Website)

• Today, the website you can see on jiocinema.com is build no. 2.3.1 - it does not have the ability to read y_ objects and render blue boxes
• The website gets the functionality in build version 2.4.0
• The website build 2.4.0 can be deployed to canary.jiocinema.com which is only visible inside the company VPN for internal people to test it out
• When website build 2.4.0 is deployed to production i.e. jiocinema.com then 100% of users who come to the website get to see the new website. Users do not need to “install an update” of their website

Servers

• Today the server has no ability to generate y_ objects, only can generate x_ objects in response.
• Server has no versions. Every time code is merged to backend repository, new deployment of server is done (This is called “Continuous Deployment”)
• Tomorrow when the code to generate y_ objects is merged, the server can immediately start sending y_ objects in response.

In a ‘Continuous Delivery’ ecosystem, each ‘release’ of a server is not versioned, and in fact releases happen several times a day. “API endpoints” can (and should) be versioned but adding new types of attributes to some endpoints typically does not warrant a new API endpoint versioning scheme. /api/v1 to /api/v2 changes are typically much larger ones.

Backwards Compatibility

The one sacrosanct rule of doing phased releases in consumer mobile products - changes need to be backwards compatible. What does that mean?

The Old World had only x1,x2,x3... and red boxes. The New World has both x1,x2... and y1,y2... and both red and blue boxes.

But the way mobile apps are released (as we saw above), there is a time period when some people are in the Old World, and some are in the New World. What happens when an app from the Old World interacts with a server from the New World, and vice versa?

Case 1: Known Unknowns

Case 2: Unknown Knowns

Case 3: Unknown Unknowns

Those who forget backward compatbility will be condemned to firefight and hotfix.

Release Strategies and Tactics

Versioned Responses

One of the tools to handle backwards compatibility is the server having the ability to serve different responses to different app versions.

While this can handle the worst of backwards compatibility cases, it is a costly solution because:

• backend code will be full of if (appVer >= x) then ...
• Cost of caching API responses on CDNs increases are more ‘variants’ of response need caching

Client Feature Flags

Feature flags are configurations we can turn on/off via a config server. The app downloads configs every time it starts. The app would have shipped with ability to handle both Case A and B of a config, and we can simply toggle between Case A and B without making further new app releases.

A few benefits of this over versioned response approach are -

• Apart from backwards compatibility, this can also be used for A/B testing new feature. We can see if users love BlueBoxes by giving enabled_blue: false to some users and enabled_blue: true to some users and comparing their data
• At any later date, a released feature can still be turned off - if something goes wrong in how blue boxes are displayed or some bugs in blue boxes are identified
• This keeps server uncomplicated as server sends same response regardless of app version

Obviously this is not a possible option if app crashes when receiving “y” items in response. In that case a versioned response might be the only solve we have, or force-upgrade users to new versions to accelerate adoption.

Server Configs

The server can also be reading some configs, and behaving differently based on the config value. The benefit of configs is that they can be changed without having to change code and redeploy the entire server. Redeploys are costly, and infeasible in the middle of high-scale events.

• If apps didn’t have a way to disable the BlueBoxes feature, the server can still be configured to not send y_ objects till enough people are on an app version that can handle blue boxes properly.
• It can simply be used to globally turn off BlueBoxes at a future date too - as a product requirement, if the experiment didn't work out or if there are bugs in the feature

The Feature Release Journey

With all the challenges in mind and knowing the tools we have to manage a feature release, let’s see what does the lifecycle of a typical feature release look like in a date-wise table.

Product decision is to enable this feature only if it can be served to >50% users, because a big announcement for this will be made

Summary

If you have worked somewhere with a mobile app that is used by a more than few thousand users, this might sound familiar to some of the systems you are already using. If not, and this sounds overwhelmingly complex, don't worry - it actually isn't.

Once you have the proper tools (a feature flag framework, a server that supports versioned responses, a phased-rollout plan) and processes (release note generators, JIRA<>Github syncing, adherence to project management systems) in place, this becomes second nature.

At JioCinema we release close to a dozen such small and big new features every week. At any point of time there are hundreds of feature flags active that regulate the rollouts of new features and/or run A/B test experiments for new user experiences. If that sounds like something you'd like to experience from the inside, we are hiring in multiple engineering teams! Drop us a line.