White Label Apps Part 2: Maintenance

06:50 Troubleshooting and Instrumentation

Many clients are not going to like random outbound connections coming out of their data center. This means that you generally aren’t going to get an advance notice that an error is cropping up on a client system until they call you. It also means that you are going to have to come up with some way to get logging and diagnostic data from a client to you manually when an error occurs.

You need to make sure your logging solution doesn’t clog their systems up. You can’t write logs forever. You need to rotate them and get rid of old ones so that you don’t consume all their disk space. Similarly, you need to be extra careful about using sensible logging levels. Debug-level messages shouldn’t be written as errors. Clients need to be able to configure their logging level in a focused manner and need to be able to hot swap the level while the system is running. Such logging should be focused. It is worse than useless to set the logging level for an entire system to Debug level, as it can hurt performance and problems will be hidden in the noise. Logging should also not significantly damage system performance or the clients will not want to use it. Errors in logging shouldn’t crash the thing that is trying to log. Logs should not contain sensitive information. If your environment is sensitive enough, you may want to rig tests to catch this. Log entries should contain sufficient data to trace something through the system. That means some sort of correlation id if multiple things are happening at once.

“Just because they paid for it, doesn’t mean that they understand it.”

You need to report to event logs and other sensible systems that admins already use. It’s not smart to roll your own logger and diagnostic tools unless your organization is very large. Your clients are probably going to want your stuff to integrate with whatever they are already using. This can be things like the windows event log, application insights, or other third parties. Your clients are unlikely to agree on the best solution, so expect to support several, possibly multiple in the same organization. Be prepared to offer guidance on how to use whatever tool they are using to troubleshoot your app.

19:55 Deployment

You may not have admin rights on the machines. This means coordinating with someone on their side. This can also mean that you can’t deploy at all, but have to provide a decent installer and instructions for the client to install. For larger clients, the coordination can be quite difficult, and this can mean that you have to support two or more versions of the application in production at your client sites.

You have to test your deployment code as well. Remember, this code is running on the client’s production server. If you break something else on there, they aren’t going to be happy. You need to carefully log every change you make a client machine, because when something breaks, they need to be able to fix it quickly. You need to make sure that rollback, repair, and uninstall leave the system in a consistent state as well. Be extremely careful about changing global configuration on the server. Log it if you do. Understand that your updates will get blamed for a broken system until you can prove that it wasn’t you.

“It’s really dangerous to send things over email.”

Signing is important. You need to sign your dll’s, executables and other assets, and provide checksums. The last thing you want is to fail to do this and have a client run something that looks like it is from you and isn’t, on a production environment. It goes without saying that you and the client need to control the entire chain of custody of your deployed code between yourself and them.

Backups are critical. You can’t anticipate everything, so you need to back up everything. This includes the old version of the code and the database(s). This also includes configuration information. You also need a plan beforehand on how you are going to roll back. If you (or support) has to do this on the phone with a client on the other end, the word “uh” shouldn’t be getting uttered.

31:35 Integration Points

Be careful and intentional about how you version any public interfaces that your client uses. As mentioned before, this includes the database. This also means that you have to carefully design how endpoints are going to act. You can add functionality, but you shouldn’t remove or alter it. This means a big design up front. Which means that your agile zealots aren’t going to be happy, but you have to modify for this. This also tends to mean a slower release cycle.

“You really need to apply the O of SOLID principles, Open/Closed.”

You need to communicate deprecation of interfaces in a reasonable manner, with appropriate warning. You can’t just remove stuff that someone might be using, especially in this scenario. This means accurate communication with the client about interfaces that are going away. You should also mark deprecated interfaces in code.

This also means regression testing of older, still supported interfaces. You can’t focus all tests and bug fixing on the new code. The old is still being used. This can mean that application changes are expensive, so they have to be considered carefully. You cannot allow resume-driven development on your team.

When you say that something is going away in six months, expect that to extend to a year or more. Give yourself a margin of error. Paying clients will push deadlines. This means that if something is going away for security reasons, you can’t wait until the last minute.

42:50 Help and Documentation

Your help system probably needs to ship with the app. While it’s great to have an online knowledgebase, depending on the client they may not be able to get there. This is also important for clients running an old version. The help system needs to match their system.

“There’s nothing more frustrating than a user that can’t get an answer.”

Help systems and documentation have to be first class citizens and need to be tested and kept up to date as well. Help that is inaccurate, out of date, or confusing can cause bigger problems. Help systems are not an afterthought, but should be considered a project feature and get prioritization in maintenance.

Your help system should allow lookup of errors. The error codes we mentioned before should be capable of being searched in both your knowledgebase and in help. This allows clients to resolve some of their own problems.

46:40 Support

A lot of support is going to be by phone. This means expensive employee time, and you need to optimize for being able to fix things quickly. In this scenario, unclear error messages can cost your company a lot of money, especially if support is included. If the client has to pay for support time, you’ll make them mad if the support time is wasted.

You need a means on your end to examine their logs. If they ship you a log, you probably need something more than a simple text editor to work with it. You need a clean way to follow some sort of correlation ID through the system. You probably also want to be able to load it into a database and query it.

You have to be prepared for situations where you support your clients’ clients. This is part of why clear error messages are important. Bear in mind that their clients may be programmatically integrating with the system as well.

“You may be writing a white label application that is being used by a white label application…it’s like white labelception.”

You may also have to support integration partners of your clients. Your clients may also have third parties who are integrating with their system and need help integrating with yours. They are probably going to expect development help.

You may need to support your clients’ developers, support staff, and DBAs. If your client is white-labeling your software, they will certainly have people doing branding who will need advising. You don’t want to skimp here, because if they do it the wrong way, it can constrain development for years to come. You need to be especially careful of DBAs and developers, as they will make assumptions and then build on them.