How To Recover Forefront TMG From a Corrupt Configuration Database

We all know it is good practice to keep regular Forefront TMG configuration backups as they help you recover your deployment quickly and accurately in case of a failure or miss configuration.  There is however a scenario where these backups cannot be restored to bail you out.  When Forefront TMG has a corrupt configuration database, the backup and restore mechanism itself is broken and as such you need to fix this first before you can recover from backup.

Symptoms of a Corrupt Configuration

In the cases of corrupt configuration that I have seen, Forefront TMG generally keeps working as per normal, but you do not have any ability to change anything. Another symptom you may notice is an empty Firewall policy screen.

In Monitoring | Configuration, you may also see an error about not having synchronized since 1999/11/30.

Under Monitoring | Alerts and in the Windows event logs (Application log), you may also see the following errors:

Level: Error Event ID:  21209 Source: Microsoft Forefront TMG Control Description:  The Forefront TMG configuration agent was unable to upload the configuration to the forefornt TMG services. This could be due to a corrupt configuration.  The Forefront TMG configuration agent is reverting the configuration back to the last known configuration.

Level: Error Event ID: 14016 Source: Microsoft Forefront TMG Firewall Description:  Forefront TMG failed to load the firewall policy configuration

Level: Error Event ID: 21177 Source: Microsoft Forefront TMG Web Proxy Description:  The failed to reload its configuration.  If you recently applied changes to the configuration, verify that these changes are configured properly.

You may also get pop up warnings when selecting the different screens in the TMG Management Console, or if you are attempting to restore from a backup.

So what is broken?

Forefront TMG stores it configuration in an Active Directory Lightweight Directory Services (AD-LDS) database.  AD-LDS and its predecessor ADAM provides a directory very similar to Active Directory.  This database is a file located on the TMG server and there are also registry references to the directory.

From past experience it is usually a single entity that has become corrupt. Imagine a rule that was created and then deleted but the entry was not successfully purged from AD-LDS.  This causes a conflict that prevents the configuration database from being loaded.

Recovery Steps

Step 1 - Decide on course of action

The procedure that follows involves editing the registry and manually editing the AD-LDS database.  This should not be done unless all the actions are understood and the risk of further damage has been negated. Know that you are proceeding at your own risk.  It is also strongly advised that once the config is loading up, to restore from a last known good backup.

The alternative is to rebuild from scratch, then import an existing backup.

Step 2 - Backup exiting registry keys and database

Since we will be manually scratching around in the registry it is a good idea to export the following keys and sub keys before you start.

HKEY_LOCAL_MACHINEIsaStg_Eff1Policy HKEY_LOCAL_MACHINEIsaStg_Eff2Policy

Next, create a copy of the existing AD-LDS database.

• To do this you will first need to stop the ISASTGCTRL service.
• Then copy the adamntds.dit file from C:Program Files\Microsoft Forefront Threat Management Gateway\ADAMData
• Start the service again.

 Step 3 - Provoke an error pop up message

In one instance the following error would occur every time I selected the firewall policy. In another instance I had to attempt make a change to the array properties before the error would occur.

You should see an error similar to this:

The key here is to look for the faulty object.  Make a note of the "object", "class" and "scope" as you will need these later on.

Step 4 - Look up the object GUID

Open regedit and browse to HKEY_LOCAL_MACHINEIsaStg_Eff1Policy.  Now Find the "object" name from the error message above.

The object would be the value of the MSFPCname key (SSTP Publishing from our screen shot)

The object GUID is the parent key eg. {0298BBEA-4E05-427C-BFE5-D9079CFA25AE} from

HKEY_LOCAL_MACHINEIsaStg_Eff1PolicyPolicyRules**{0298BBEA-4E05-427C-BFE5-D9079CFA25AE}**

Step 5 - Connect to the AD-LDS with ADSI edit

Go to Administrative Tools | ADSI Edit (adsiedit.msc)

Connect with the following settings

• Name: TMG
• Connection Point: CN=FPC2
• Computer: localhost:2171

Browse down the directory till you find the GUID

Typically this would be under  CN=PolicyRules,CN=ArrayPolicy,CN={the array GUI},CN=Arrays,CN=Array-Root,CN=FPC2

Compared to the other object you may see that this one is empty.

Step 6 - Clear the corrupt entries

Delete the policy object in ADSI Edit (CN={GUID}

Delete the GUID key entry from the registry using Regedit.

Search for the GUID value again just ensure that all copies are removed.

Step 7 - Restart and check

It is possible to restart the individual services, but a full reboot is recommended.

If you have the same symptoms, check the alert detail and see if the object is the same or a different one. Repeat the process for the new object.

If everything is working again, take the time to restore your last backup and overwrite the existing configuration.

Conclusion

It is possible to manually edit the Forefront TMG configuration, but it is only as a last resort short of rebuilding.  The best advice is still to create and store regular backups. For more information on this, see my previous article - Forefront TMG Configuration Backup Scripts For Standalone and Enterprise Arrays