Making Apps Plugable
Podcast: Play in new window | Download (43.5MB) | Embed
Subscribe: Apple Podcasts | Spotify | Email | RSS | More
If an application is around for long enough, eventually the users will want to modify the behavior of the application for themselves. If your application is of sufficient size, it quickly becomes impossible to cover every use case that your users might have. Whether it is the import and export of data, integrations with other system, or additional functionality, it’s hard to predict how your end users will want to use your application in production. This is especially true if you are writing code for the enterprise, as larger companies tend to have their own teams of developers.
While you can just throw together a plugin system and call it a day, that’s an approach that is likely to end badly. Not only does such an approach lead to potential security nightmares, but it can also create stability problems in your application. Further, if you design your plugin model poorly, it can cripple your ability to make changes to your application for the rest of your users. Your clients might also find it difficult to develop and maintain plugins for your system, causing it to be a waste of time for everyone. Finally, a bad plugin model can also result in your clients wasting their time and coming to resent your app. Really it’s better to avoid having a plugin model for your application if you aren’t going to do it correctly.
If you still want to build a plugin system into your application, there are some things you need to plan out. The items that are of greatest concern will vary widely depending on what your application does and what sort of environment it will be running in. However, there are certain considerations that will always come up when building a plugin model, regardless of the application in question.
Writing a plugin system that will do anything other than create nightmares is hard. There’s a lot to consider and most of it needs to be fully baked before you show it to anyone who might try to use it. Further, when your app is hooking into other people’s code (or vice versa), it exponentially increases the number of problems that you may experience in production, while making it more complex to deal with them. Be very careful if you are ever told to make your application pluggable – it could be worth it, but it’s not as easy to do it well as you might think.
Episode Breakdown
The business value of a plugin system is important to understand.
If you don’t understand why your employer wants a plugin system, you don’t understand enough to start writing one. Don’t negotiate on this – you’ll be blamed for any design decisions you made if you made bad assumptions. There are widely different security and runtime things you’ll have to consider based on business goals.
Some possible reasons for wanting plugins. Enterprise clients want to extend the application and don’t want to pay for custom development from your company. Your company wishes to enable third parties to integrate with your app in general. This might be an effort to build an ecosystem around your product or turn it into a “platform”. Your company wishes to build their own plugins and sell them as optional upgrades to your product. Your company wishes to build a marketplace around plugins for the product and get a commission every time a plugin is sold. Your company wants to enable vetted strategic partners to build plugins for your application.
How you build your plugin system will vary greatly based on what business value you are attempting to create. If the general public will be writing plugins that are used by the general public, you are going to have to have a lot of infrastructure in place for discovery, plus you’re going to have to spend a ton of time on security. If your application is a software as a service application and is multitenant, you’ll have to spend a lot of time on security, plus you’ll have to make sure that plugins don’t degrade system performance. If enterprises will be building plugins, you’ll need to make them easy to debug and troubleshoot, or your helpdesk will be swamped with calls. If you are building a plugin marketplace, you’ll have to worry about fraud and you are going to have to spend a lot of time on security. If strategic partners will be building plugins, you’re going to have to have a good way of vetting the plugins for security and other issues and you will probably need development staff involved to help them.
Security
Plugins generally need to live in a sandboxed context. Unless you can completely trust the author of the plugin and prove that the plugin came from that author, it should live in a security context that limits its access to the rest of your application. This means that plugins will need to be signed and verified.
You will likely need a permissions system around plugins. This means that once the plugin is verified, that you check against some data store in the system to determine what actions a plugin can take. This means that the plugin will, out of necessity, need to be kept from having open access to the entire system. This also means writing APIs that plugins will use, and those APIs will have to validate permissions. Plugins with too many failed calls should also probably be disabled to limit abuse.
If your plugins are created by the general public, you also need to watch their upgrade paths. WordPress in particular has had issues with this, where a plugin is upgraded and the new version has vulnerabilities. You are probably going to need some method of validating plugins, and warning people who are using ones that are dangerous. Users will blame your product if a plugin causes problems, even if you didn’t write the plugin.
System Performance
Plugins may cause issues with system performance. Poorly written plugins can often consume far more system resources than is acceptable. They can also be slow due to the developers not really understanding your system. You’ll need a way to prove that the plugin is the problem rather than your code.
If you are running a SAAS, you also have to make sure that plugins (even if not installed on your servers) can’t hurt performance for other users. Generally, this means rate-limiting access to APIs. This also requires significant investment in instrumentation to make sure that you can catch performance issues proactively. You need to be able to quickly kill access for plugins that are causing performance issues.
You probably should never host user-built plugins on your server in a SAAS environment. This takes more investment than most companies are willing to make in order to keep from having problems. In general, the risk of a breach in a SAAS product hosting plugins from the general public is too high to consider.
User Permissions
If your plugins are user-facing or touch end user data, then you will probably need to build a permission model for the user’s data as well. This means that plugins will have to assert permissions before taking an action. It also means that any calls the plugin is making will need to check permissions. Be very careful about what happens after a permissions check failure. Make sure this doesn’t make your application unstable.
User permissions will need to be fairly granular, so that users aren’t forced to simply select all permissions. Plugins will need to also include a manifest indicating the permissions that are required. Along with this, you’ll need to make sure that you clearly explain what the individual permissions mean.
It’s also possible if your app allows users to share data, that the plugins will need asset-level permissions as well. For instance, lets say that you were making a plugin for instagram. You might need permissions at the user level, but there might also need to be restrictions for what user can do with their friend’s data. The lack of this, by the way, was part of the whole Cambridge Analytica mess with Facebook.
Plugin Upgrade Cycles
Plugin upgrades will also be an annoying fact of life. Your users may be reluctant to update your software if it will break any plugins they are using. This also means that if you wish to avoid this problem, you are going to have to have an area for plugin authors to test their code against newer application versions. You’ll also need to be able to disable old plugin versions upon upgrading your application, or force an upgrade. You may also want to consider pausing application updates until the user either removes incompatible plugins, or all plugins have compatible versions available.
When plugins upgrade, you’ll also need to deal with changes in their required permission sets. This means checking the aforementioned manifests, and prompting users to enable the needed permissions. If the users don’t assent to the new permissions, you need to figure out how to handle that.
Your own upgrade cycle is in the mix here. For instance, you’ll like make your permissions more granular over time and will need to migrate existing permissions. This can be a problem, depending on how you derive the new permissions. You should prompt users at any change in permissions in order to make sure that the plugin still really has those permissions.
Diagnostics
Plugins complicate diagnostics in general. When you are logging events in your application, you may find that you need to extend your logging to get better diagnostics, especially when plugins are sending data to or receiving data from your application. This is even more critical if plugins are allowed to modify data at some point during a workflow, as badly written plugins may change the data in ways you don’t expect.
Plugins themselves need diagnostics. Any plugins in the app also need a way to write log entries, such that you can determine what is going on. While you can catch a lot of activity in the logs for API if you built one, you may also want to expose APIs that allow plugins to write out to your logs as well. Be careful about how much data you allow plugins to emit, however, as particularly chatty plugins are apt to chew up disk space faster than you would think, especially if in heavy use.
You probably need to expose logs from plugins to the developers of the plugins as well. You have to be willing to work well with plugin developers if you are going to have a plugin system that is easy to use. This means giving them meaningful feedback about any errors that their plugins encounter in the wild. Their own logs may not have all the data they need (if they can even log anything at all).
Audit trails
If plugins are in the mix and can read or write data, you definitely need audit trails. When data is lost or destroyed in your system by a plugin, you need to be able to prove that. Reliable audit trails that cannot be tampered with by plugins, are necessary, especially if your app includes any risk of a lawsuit or legal compliance is in the mix.
Audit trails are also important from a forensics perspective. If a plugin is compromised and does get access to data, you want that to be logged. This may help you discover vulnerabilities in plugins while those vulnerabilities are in use the wild.
You should also put audit trails around any actions involving plugins. This includes enabling/disabling plugins, as well as updates. If any changes to permissions occur, these need to also be logged as well.
Compatibility Concerns
You will eventually run into compatibility issues that originate in a plugin. These could be anything from communication protocols changing (https) to plugins being unstable on newer OS environments. Basically, remember that plugins have all the problems of any other software, plus whatever problems they have from living in your runtime environment.
You need to have very robust error handling around any calls that you make to a plugin. Errors in a plugin must never take down your system. You also need to make sure that you aren’t swallowing errors that occur. These should be logged. This generally means having some sort of instrumentation in place so that you can see if a sudden spike in errors occurs. You probably can’t safely disable a plugin if errors occur, as it may cause data issues down the line, so notifying the author of the plugin quickly is a must.
Loading and Unloading Plugins
You may also need to be able to load and unload plugins quickly on the fly. Depending on the application architecture, this could involve various degrees of complexity. For instance, in .NET, if you dynamically load an assembly into the current app domain, you can’t unload it. You’ll run into similar issues with other platforms as well.
If you don’t allow hot reloading of plugins at runtime, this means you’ll probably need to shut down the app and restart it to load the new plugin configuration. Depending on the app (like games), this may be acceptable. If it is an enterprise-level service, that may not be acceptable. You have to be careful to pause any calls go into a plugin during a hot reload, lest you get weird errors that you can’t replicate.
Plugins also need hooks to know when they are being asked to shut down. The plugin may need to deallocate resources, write log entries, finish up transactions, etc., before shutdown. Once you have sent the message that you are requesting shutdown, you shouldn’t send any more data at the plugin. After a window of time, then you completely shut the plugin down. If you allow plugins to shutdown cleanly and clearly document how to do so, it’s harder to blame you for the kind of errors that would occur otherwise.
Managing Configuration
Plugins are likely to have configuration of their own. This could be anything from small items that need heavy security like connection strings, to larger items such as bitmaps and the like. You probably don’t want to store this stuff alongside your own application configuration, as that facility may be abused (for instance, they could store enough stuff in there to crash the app or degrade performance). If the storage is going to happen on your servers, you need to be really careful that you limit the amount of stuff that can be stored. If the plugins are being written by the general public, you probably don’t want to store any data from them on your servers. That could lead to your servers being seized if it turns out that they are storing anything illegal.
Plugins will probably need a few varieties of storage. For instance, there probably will need to be instance-level storage for data that that plugin needs across the board while running a particular environment. There will also probably be user-level storage, where the plugin stores data on a per-user basis. Depending on complexity, the plugins may need more than this.
Book Club
Art Matters: Because Your Imagination Can Change The World
Neil Gaiman
“The world always seems brighter when you’ve made something that wasn’t there before.” ~ Neil Gaiman
This month we’re going to do something a little different. Instead of a tech or business book I’ve picked one of my favorite fiction authors, Neil Gaiman. This book is a collection of his speeches, poems, and creative manifestos combined into a small book about the importance of creativity and art. It’s illustrated by Chris Riddell. The material in this book is very thought provoking. You may or may not agree with the author, I find myself doing both, but he will get you thinking about the topics. The first section is titled Credo and is about the longevity of an idea. In it he talks about his beliefs on beliefs. He even says it’s Ok to disagree with him.
Tricks of the Trade
Never express a problem like the ones in this episode to management in terms of difficulty or time. They don’t care about the former and they’ll take the latter as an excuse most of the time. Instead, express such problems in a written fashion in terms of risk to the organization. Keep the emails that you write about this sort of stuff in storage that you control. This way, if they try to force you to do something dumb, they can’t throw you under the bus. A plugin system is a great example of this – companies will try to make you “just” do something that doesn’t make sense, and then act like it was your fault when it went badly. Keep evidence that makes it clear who is at fault.