Backward Compatibility
Podcast: Play in new window | Download (56.4MB) | Embed
Subscribe: Apple Podcasts | Spotify | Email | RSS | More
By its nature, software is subject to frequent change. In general, software isn’t particularly static and can be rapidly reconfigured to adapt quickly to changing circumstances in the market, security landscape, and whatever industry it serves. This is one of the main reasons that software is eating the world right now – it’s because of its rapid ability to change.
However, this adaptability can also be a weakness if you aren’t careful. Most software these days includes integration points of various sorts so that other software can work with it. If the people integrating with your software happen to be paying you, then it’s crucial that you control the way your software interfaces change so that you disrupt your users as little as possible. To make things more complicated, not everyone always interacts with your software in the way you expect. You may find, for instance, that people directly interact with an underlying database, use web automation to push data into your system, or even automate your user interface as a sort of “back door” integration method.
If you want to try and ensure backward compatibility to the degree possible, there are a few practices you can use that will make this easier. While we are planning to have more episodes discussing backward compatibility in various contexts, there are some general rules that you should follow regardless of how other software is interacting with your software. In particular, these rules are focused around the idea of limiting disruption to your users while still allowing as much flexibility as possible for your development team. Disrupting your users a little bit will irritate them, while disrupting them a lot makes it more likely that they will flee your platform for that of your competitors.
Keeping backward compatibility is a royal pain at best. As your software gets more mature and has more people integrating with it, don’t be surprised when you find that you have to spend significant effort keeping your code from breaking other people’s integrations with your product. Once your software is stable and in use, you no longer have the ability to change things at the edge as quickly as you could when you were starting out. However, if you are careful and plan ahead, you can often significantly reduce the amount of time and effort that are required to be able to make changes to your software that impact other people. Best of all, these best practices can often result in your integration points being more useful to your own team as well.
Episode Breakdown
Use Semver religiously.
If you can’t communicate version changes in a clear and widely understood manner, then everything else we are about to suggest will be impossible. Semantic versioning is a good way to do this. With Semver, versions are expressed in three segments (Major, Minor, Patch). Major version numbers are incremented when you make breaking changes. Minor versions are incremented when you add functionality in a backwards compatible manner. Patch versions are incremented when you add bug fixes in a backwards compatible manner. Semver is a great way to communicate changes to your users in a way that makes it easy for them to see how much trouble an upgrade will be. This doesn’t mean that you don’t need to include release notes, however.
Keep on top of YOUR dependencies.
Nothing can force you to implement breaking changes on your users faster than having one of your dependencies force you to quickly make sweeping changes in your code. While some vendors really seem to love to make sudden sweeping changes that cause major problems among their users, most vendors at least attempt to handle breaking changes in a sensible manner. If they don’t, you should reconsider your integration with them, because they are hurting your users in a way that causes you to be blamed. Try to find out where upcoming changes are announced for your dependencies and get on their mailing list, sign up for their RSS feeds, etc. It’s also a good idea to take an inventory of the dependencies of your application. Try to eliminate as many as you can, while trying to determine how likely breaking changes are for the rest.
Keep on top of relevant legal requirements that impact your software, its dependencies, and that of your clients.
Regulatory changes are usually fairly slow, but they are inevitable. They can force the dependencies of your application to making breaking changes quickly, with deadlines that can’t be negotiated. Additionally, keep an eye on the news if your application is subject to any regulatory bodies (if not, then start watching for people who want to regulate your sector, because it’s coming). Once a new law is passed, make sure that your company’s legal team (not development, and not management) tell you what changes you have to make. Prioritize getting those changes in place before the deadline so that you have plenty of time to deal with the fallout before you start accumulating fines. Understand that many of your clients and the systems you are dependent upon may also be under changing regulatory pressure. It may not be the same as yours. This can mean that you still have to deal with (for example) California or Denmark’s laws when you are a software company based out of Nashville.
Communicate regularly with users who are integrating with your software.
You don’t want to be in the situation where you can’t contact the users of your software when a breaking change is coming. Make sure that your company can get ahold of people WHEN something changes (don’t assume this is an IF). Changes should be communicated as early as you possibly can, so that your clients can plan ahead. Remember that integration with your software is probably not their highest priority, but simply a single feature out of many. Also make sure and have a focus group of select clients on whom you can test upcoming integration point changes. While this is typicaly considered to be the responsibility of product owners or management, if you don’t have this, you need to push to make it happen. This can keep you from making changes to your endpoints that drastically break someone else’s workflow. You also need a good way to correlate a particular set of contact information with an API key, so that as the deadline for converting comes closer, you can actually warn the impacted companies. If you don’t, your users are liable to get a very unpleasant surprise when you turn off the deprecated version of the system.
Have a means of collecting and reporting on usage statistics for any integration points you make available.
At the very least, you need to know how many clients use an endpoint (broken down to the lowest level you can get) and how frequently they use it. You may find that some integration points in your software are rarely used at all. If this is the case, there is less concern about tampering with that endpoint. You might also find that one or two clients account for all the usage of an endpoint. If this is the case, you may need to get more information about why they are using it. It could be that you can easily convince them to move to your new implementation if it offers advantages that the old one didn’t. Usage statistics are also handy when you get close to turning off the old implementation, so that you can estimate how many problems will be created by turning something off. It also helps you to track adoption of your newer integration points.
Push users away from the pit of failure.
Try to keep users from using approaches that are fragile, unless you want to support those approaches intentionally. For instance, if you don’t want users doing janky integrations directly with your table structure in your database, find another way to make similar functionality available to them and make it discoverable. Similarly, provide SDKs for working with your system that cover expected use cases so that users don’t try (and probably halfway fail) to roll their own. Make your libraries available through standard package managers for the various languages. This way, you can push updates that actually mark items as deprecated and surface new ones well before the cut-off. This makes it easier for consumers to gradually wean themselves off your old API. Provide useful error messages and other feedback if someone is using your API in a way that will soon be unsupported.
Consider documentation a first class citizen and ship it along with the version of any integration points you have.
If you are waiting to start writing documentation until your new version is complete, expect significant delays in getting people to move to the new version of your software. Developer documentation should generally be written by people with development experience and should not lag behind the released version of the code. Incorrect documentation is massively frustrating and will slow adoption of the newer system. Documentation should also be subject to pull request style revisions (we call that an editor) as well as testing (literally follow the darned things and see if the approach described works). A site where users can test out endpoints to see how they work is also an effective (and increasingly necessary) form of living documentation.
Give integrators as much time as possible before removing old versions.
If there isn’t a regulatory timeline that forces an update in short order, then you should give clients as long as you can to update before pulling the plug on the old version. Bear in mind that integrating with your software is not only thing that their software likely does – if you break things for them, they will consider that integration point to be risky, as they should. While this may make it more difficult to make changes in your system, this can also force you to more thoroughly decouple your architecture in general, making this less of a problem in the future. You may find that particularly large clients want to stay on the old version for a long time and you can’t migrate them because management won’t let you. You shouldn’t assume as a developer that you can force a client to do anything.
Allow mixed-use of versions where possible.
As clients start to use the newer parts of your system, make sure that they can seamlessly use both the old and new parts of the system in parallel. Their transition should be allowed to be gradual. Be especially careful about things like authentication, access to various resources, etc., as you will probably need to introduce code to the newer parts of the system to deal with things created in the older parts. You may also be forced to do the reverse. Pay close attention to errors, because if you do this, you will probably get more of them. However, those errors are often very informative about use cases that you didn’t realize you had.
Leverage custom SDKs
If you built a custom SDK for interacting with your software, then you should be leveraging that during this process. You should begin to expose new endpoints as soon as they are available and should marking things for deprecation as soon as possible (as previously mentioned). You should also make sure and message your userbase to let them know that changes are coming and that keeping their SDK up to date is the best way to catch breaking changes before they become a problem. Be sure to not only add any new endpoints and mark obsolescence, but to make sure that any new error messages coming out of the system are represented cleanly in the code you provide. You probably can’t get away with just generating an OpenAPI wrapper on this, tempting as it may be.
Tricks of the Trade
Moving into a leadership role can be daunting. The new role has new and sometimes opposing requirements to what you had been doing before. As front line developers we have the luxury of getting into the weeds or hyper focusing on our one area. Whereas moving up the ladder into a lead or managerial role one has to look at the bigger picture. It’s easy to still think in terms of how to get things done at a worker level. It takes learning a new skill but in order to lead the developers you need to make those skill backward compatible to the mindset you had as a developer. You need to be able to get into that mindset to work with and understand those you are leading, then leave it to think at an organizational level. This is quite a challenge, especially for those new to the role. So be lenient with your managers and lead developers when they may not be thinking in the same vein as you about something.