Active Directory database basics: Performing an offline defrag

http://searchwindowsserver.techtarget.com/tip/0,289483,sid68_gci1325578_mem1,00.html

For the most part, the Active Directory database just works. It performs automatic online defrag, rarely becomes corrupt and never requires manual database recovery procedures like Exchange Server does. Still, it's important for administrators to understand the basics of an Active Directory database – how it works and certain important maintenance procedures.
Figure 1 shows where the database (NTDS.DIT file) sits in the Active Directory architecture. You can see how protocols like LDAP interface with the Directory Service Agent (DSA). The DSA is responsible for functions such as schema enforcement of updates, access control enforcement, object identification, referrals and functional level definition. It is associated with a GUID -- specifically the objectGUID attribute of NTDSsettings object -- that is used in replication to identify replication partners. This is exposed in the Repadmin/showrepl command and is the GUID that is mapped to the server's FQDN in the DNS Alias record.
Figure 1 (click to enlarge)

There is also a database GUID that is the invocationID attribute of the NTDSsettings object. The database layer is responsible for the creation, deletion and modification of objects, as well as the retrieval of objects, attributes and the schema cache. The schema defines rules for the organization of the database in terms of classes and attributes.
The NTDS.dit -- located in the %systemroot%\ntds directory -- exists on every Windows server installation. It is a basic transactional Jet database just like Exchange Server, and it is recommended to store the database and logs on separate physical disks. The location for these files is determined during Dcpromo, but you can change the location using the NTDSUtil program described later in this article. Figure 2 shows a typical NTDS directory.
Figure 2 (click to enlarge)

NTDS.DIT is the database. During Dcpromo, it is enhanced with data from other domain controllers if it's joined to an existing domain or starts a new domain. As objects and attributes are added, deleted or modified, the database gains "whitespace" (unused space).
Adding users, computers, printers and other objects along with defining various attributes will cause the database to grow, and it can be anywhere from a few MB in size to several GB. Being able to load the database entirely in addressable memory will significantly improve operations such as authentication. Therefore it is normally recommended for DCs with 4 GB or more of physical RAM to use the \3 GB switch in Boot.ini to expand the user mode section of memory, which permits more of the NTDS.DIT to fit in there.
Windows Server 2003 made an important change that significantly reduced the database size. While a security descriptor value is stored for every object in Windows 2000 Server, Windows Server 2003 uses single instance descriptors, allowing multiple objects to use a single descriptor.
Defragging the Active Directory database
It's important to defragment the Active Directory database for best performance. Normally there is an online defragmentation that occurs about twice a day on the database, but this is more of a backup than defrag. It can give admins a false sense of security, thinking that the database has been defragged without any downtime.
The only way to truly defragment the Active Directory database, remove whitespace and decrease its size is with an offline defrag. This requires you to take Active Directory offline by booting a DC into Directory Service Restore Mode (DSRM), which boots up the DC in safe mode without mounting the AD database. Once booted, enter the NTDSUtil program and you will be able to perform a number of actions that are not possible with Active Directory online, including those in the File menu as shown in Figure 3.

Figure 3 (click to enlarge)

Significant functions include:

• Header -- Dumps the Jet database header with information, such as database location, the state (i.e., clean or dirty shutdown, last backup, database size, etc.). This is shown in Figure 4.
  
• Checksum -- Performs a Jet physical integrity check, a good thing to do before defragmenting. See Figure 2.
  
• Integrity -- Performs a logical integrity check of the Jet database. After running this command you will see a message recommending that you run the Semantic Database Analysis function. This option is found in the main NTDSUtil menu. This is a good one to run if you get events in the system event log indicating database inconsistency, corruption or other errors. The Semantic Database Analysis function will be covered in more detail in a future article.
  
• Compact to %S -- This command is the defrag operation that squeezes the whitespace out of the database.

Figure 4 (click to enlarge)

Note that there are other commands available that allow you to move the database and log files, set the online backup directory path and even perform a soft database recovery.
To perform the actual defrag of the Active Directory database, the following steps are required:

1. Create a directory that will temporarily store the compacted (defragmented) database. In our example here, we have created C:\ESE-Backup.
   
2. From NTDSUtil, go to the File menu option. In the File Maintenance prompt, enter: Compact c:\ESE-Backup
   When the compacting operation completes, you will see the following text:

   If compaction was successful, you need to:
         copy "c:\ese-backup\ntds.dit"
         "C:\WINDOWS\NTDS\ntds.dit"
   and delete the old log files:
         del C:\WINDOWS\NTDS\*.log

   
   
3. It follows that the next step is to copy the new compacted ntds.dit in c:\ese-backup to c:\windows\ntds\ntds.dit (overwrite the old file), and then delete the 4 logs. You will only have the edb.chk, temp.edb and ntds.dit files left.
   
4. Restart the DC in normal mode.

If you want to play with this in a test domain, you can record the size of the NTDS.DIT and then create a large number of users -- say 10,000 -- in the domain. Note the increased size of the NTDS.DIT. Then delete the users and follow the procedure just described to compact the database and replace the old one with the whitespace for the 10,000 deleted users. Compare the size of the new, compacted database with the size after you created the users. You could also wait for an online defrag of the database to occur (recorded with an event in the system event log) and see if that changes the NTDS.DIT size. Note that in a production situation, you want to perform database integrity checks to ensure stability.
It is not necessary to perform this offline defrag on a regular basis, but it's good to do it after significant changes have been made -- such as the removal of a large number of users or groups -- to keep the database at an efficient size.