Documentation

Documentation seems to be a problem in many companies and is usually the component in a project that is most often overlooked. Every time that either a new employee starts or an external contractor is hired for an AD related project, instead of getting a binder with proper documentation, he or she is assigned a mentor or a buddy who explains the systems and infrastructure. Then, the first new task is to write the documentation that has been missing for the last X years. However, after the first week he or she realizes there is not enough information and when they ask for it, they get some vague pointers on where to look.

Unfortunately, the usual circle is that documentation is left for later stages in the project, and over time gets forgotten or information is passed on by word of mouth, or as a collection of links to websites, instead. Over time, the missing or incomplete documentation becomes a costly burden to the organization knowledge is lost and, because of its non-existence, is impossible to back up. The eventual creation of this documentation, which wouldn't have taken that much time to begin with, is a lengthy and expensive process.

Documentation is not really that hard to do, but it can be hard to convince your project or program manager allocate the extra time in order to complete it. Usually, the questions will arise as to why this needs to be done now and cannot be done later. A good argument for this kind of questioning would be to explain to him or her that at a later time, information is no longer fresh and remembered, or that it is necessary for backtracking problems. I have found that both of these work very well and generally managers will give you time to document properly. If, however, you don't get the time, please make sure that you obtain written confirmation regarding the project or program managers acceptance that there is no way of knowing what has been done, and no time to write proper documentation.

Getting documentation done is actually quite easy. It comprises two steps, and once you have done this a couple of times, it will flow easily and you will produce documentation that your manager will actually be proud to show around.

First open notepad or any text editor and write, in short points, what you do, every step. In some cases, I just copy and paste the command, or the output, or both, into a line and keep going. Once you have completed the task, take a standard company template and format it into four sections. The outline is shown in the following table:

If you stick to your template all the time when writing documentation, you will be the pride of your unit. Basically, should the need arise, you would be able to produce documentation where other people can't. It might look like a lot of extra work but as I mentioned already, it becomes very routine after a couple of times, and when you see a properly formatted document, it does make you proud. It also make your life easier when someone asks you how to do certain things,: you just open the document, print it, and point the person at the printer. A lot of time saved!

Of course, writing it is only one part of the equation. The other part is to keep it up-to-date. If you write a document about what group policies you are currently applying, then any change needs to be reflected in that document for it to be up-to-date.

Documentation plays a big part in disaster recovery, and sitting having a freshly-recovered domain, not knowing some of the settings that were applied earlier that now prevent things from working, dearly-it may even cost you even your job!

When writing your DR, please make sure that you have a printed copy in each location and at least one offsite copy per location. In some companies, it is standard practice for the domain or Enterprise admin at least to have a printed copy at home or on a USB key with him or her at all times. It is also good practice to have a printed copy or an electronic copy in the location's safe so that it can be retrieved very quickly.

Write your documents regarding your infrastructure as clearly as possible, and do not make any assumptions about who will be reading the documents. It could very well be a summer worker or a trainee, although very often companies rely on professional DR-specialized companies. Some of these companies not only do regular, twice a year, complete DR in an isolated environment, but also sometimes provide you with warm sites to get your infrastructure back up and running more quickly. However, you never know what the disaster situation will be and if it is bad, you will want to ensure that everything possible is provided in the instructions.

Backups

I worked in a company once with a seasoned sysadmin who always said: Backups are like rumors, everyone talks about them but no one verifies that they are true. I found this to be correct a lot of times. Another good saying is: There are no backups, only restores, successful or unsuccessful ones. Everything else is just an activity leading to a restore.

Companies have backup strategies in place and the occasional repair or deleted file isn't a problem. But what happens if you have a catastrophic failure? What happens when the tapes that are in the server room neatly on a stacked, and labeled Monday through Friday, get water damaged or, worse, fall prey to an electrical fire?

Backups are one of the most crucial parts of your infrastructure and, next to documentation, probably the one taken for granted the most. It is not enough that you have a big nice tape drive or tape robot and that it backs up 100GB every night. What matters is how fast can you restore the data to a usable state. How fast can you recover a system state for a domain controller if your communications link is down and your network is being rebuilt after a natural disaster?

Testing is the key word. In some of the companies that I have worked for, testing of backups is a routine, weekly, event. This might sound like overkill but when the data contained on those tapes is highly sensitive and costs millions of dollars to replace, you will gladly invest half an hour to make sure that you can restore valid and complete files.

Backups, just like documentation, can be a pain but, in the end, you have to weigh the risk versus benefit, and unless you are backing up completely useless data, I guarantee you that your business people will always show you that it is worth keeping an eye on the backups.