1dent1ty cHa0s

Based on a post by Richard Wakeman in the MIIS TechNet Forum last week, I decided to post an that I have made use of in the past to troubleshoot Kerberos authentication problems. While I tend to always insist on a thorough Active Directory Health Check before starting any AD dependent project you may still find yourself dealing with some type of Kerberos authentication issue in relation to MIIS.

Later on in the thread, Danny Alvares points out that in the Identity and Access Management Series 1.4 under Chapter 5: Identity Aggregation and Synchronization there are some recommendations when performing password synchronization to set the "KdcWaitTime" from the default 10 seconds to 30 seconds. This would give the MIIS Server some additional time before giving up on its authentication attempt. Presumably this would be evident in large distributed Active Directory deployments where network latency is very evident or the Domain Controllers themselves are taxed.

In my experience, there are two "common" Kerberos authentication issues that I have had to deal with and one rather rare one concerning replication:

1. Kerberos Time Skew - By default (you haven't altered the default Kerberos options in your Default Domain Policy GPO) all computers in the forest must have their clocks set to within 5 minutes of each other (time zones notwithstanding). The most important relationship is between the client and the authenticating Domain Controller - if the time is skewed by more than 5 minutes then the built-in security methods within the Kerberos protocol will consider the authentication requests to be compromised and deny the request. For this reason - ensure that the time on your MIIS Server and the time on the Domain Controllers in your site are all synchronized. A well built environment should have a skew of only a few seconds max. To confirm a time skew problem you can use the w32tm program from the command line:
   
   w32tm /monitor /domain:
   
   or to just check your local site DCs:
   w32tm /monitor /computers:[computer1],[computer2],...
   
   The NTP results should indicate an offset of less than 5 minutes. If you are seeing offsets greater than 5 minutes then you can force a manual reset by using the net time command:
   
   net time <\\computertosyncto> /SET
   
   To resolve this for good, consult the following KB article:
   How to configure an authoritative time server in Windows Server 2003
   
2. UDP Filtering - By default, all Kerberos authentication traffic utilizes UDP port 88; however issues with token size can increase the size of the packet and force it to use TCP port 88. If you are dealing with firewalls that are not enabled for both TCP and UDP port 88 or routers that are filtering UDP then you will have difficulty with authentication between distributed hosts. This can manifest itself through extended logon times (several to tens of minutes or longer). One method that I have used to troubleshoot whether or not the network is, in fact, filtering UDP requests is to force all Kerberos authentication traffic to use TCP instead of UDP. By setting the "MaxPacketSize" to 1 you can force Kerberos to use TCP.
   
   See the for the MaxPacketSize option which allows you to adjust the value from the default (2000 in Windows 2000, 1436 in Windows 2003).

The other "not so common" Kerberos issue I have run into has nothing to do with MIIS but AD replication itself. Suffice it to say, the DCs themselves have a secure channel with the domain in the form of the password set on its computer account. If the default 30 day password reset on the secure channel fails for whatever reason, the secure channel will no longer be valid which can cause all sorts of issues to pop up - one of which is an "Access Denied" on replication links between DCs. The really frustrating thing about troubleshooting this is that the error doesn't show up on the DC that has the problem, it shows up on a DC that is trying to replicate to the offending DC. So, you have to dig a bit to uncover this and identify the DC that is causing the problem. The resolution is rather drawn out so I will refrain from posting it here.

For some time now I've been grappling with the problem of trying to understand what skillset bases best identify someone who is likely to be proficient, or even interested, in learning MIIS. Given the relative demand for the Identity Management and the shortage of talent currently it is rather unlikely to come across people who already know MIIS or have worked with Identity Management tools or on related projects. Add that to the fact that Ensynch is not the only company trying to increase its stable of MIIS savvy talent and you have serious issue! So, our only other opportunity is to grow and mentor from within - but how exactly?

Formalized Training
For years now, Oxford Computer Group (through SQLSoft) has been the only company in the world capable and knowledgeable enough with the product to offer training for MIIS and they've done a superb job between their two classes (of which the 2731 course is now offered by Microsoft CTECS - but be forewarned about non-OCG instructors with little or no practical knowledge). Formalized training is of course the quickest way to obtain training, but is certainly the most expensive.

Mentored Training/OJT
I think many companies tend to rely on On the Job Training or training through a skilled mentor when available. This has mixed results, of course, as many variables can interfere and limit the effectiveness of this method, but certainly nothing beats OJT for shear exposure to process and problem solving. Even those with plenty of OJT can still benefit from some formalized training to increase breadth of knowledge, but we still haven't identified what types of skills provide an effective base for which to layer on formalized, mentored, or on the job training? Also, remember that introducing Identity Management to a new organization adds a tremendous amount of impact to its People, Process, and Technology.

MIIS Skillset Requirements
MIIS is certainly unlike most other Microsoft applications in that it demands knowledge of so many diverse skills. While traditional products like Active Directory and Exchange require deep administrative knowledge, MIIS requires the following:

• Knowledge of SQL - MIIS is a SQL application so knowing how to construct basic SQL queries is essential when dealing with SQL data sources and building simple SQL based reports. As you get more and more familiar with MIIS, you'll find that many of the more "advanced" tips and tricks are just SQL tricks that a DBA is likely to know.
• Development Skills - MIIS is built to be extended and customized, yet it is this rich extensibility that quickly leads any MIIS implementation deep into coding territory. Some background in scripting or coding is a must here! Experience with the .NET Framework is a serious plus.
• Knowledge of the Connected Directory - MIIS isn't much until you connect it to something. Usually you're feeding a series of file, DB, and LDAP data sources into a System Directory like Active Directory or eDirectory so it is vital that you have more than casual knowledge of the data sources you are provisioning to.
  

So, how many DBA/Developer/Admins do you know? Furthermore, which skills or aspects of those skills to you think are most important?

My Opinion
Coming from an NDS/NT4/AD background you could say I am heavily stilted towards the Directory Admin side and since I've always considered myself more of a scripter than a developer, I've had to pick up the SQL and DEV skills as I went along. I've also had the fortune of attempting to mentor pure DEV and SQL Admin types who were "forced" into supporting an MIIS implementation and while the whole forced aspect certainly tends to kill chances for success faster than anything, the biggest learning curve for both camps wasn't picking up MIIS, but understanding Active Directory. So, I firmly believe that the best skill set to start with is someone who has a experience with Directory Administration (and has felt the pain that is supporting a large deployment) and has an interest in scripting or development of some sorts. Admins who can write their own utilities or scripts to accomplish administrative tasks are a step above the rest!

So, I'm anxious to hear your input - what do you think is most important in selecting a potential MIIS convertee? What should you have before attending your first MIIS class or working on your first MIIS deployment?

Today Ensynch announced its win of a Microsoft Partner award in the Advanced Infrastructure Solutions category:

Tempe, Ariz. (July 18, 2006) — Ensynch Inc., the Southwest’s leading provider of IT infrastructure services, staffing, and data center solutions, announced the company is a recipient of a Microsoft Worldwide Partner Award. Ensynch is the only Arizona-based company to claim a prestigious worldwide award. It also is one of only nine honorees worldwide to be named in the Advanced Infrastructure category. Microsoft’s Worldwide Partner Awards are the most prestigious awards given to partners, indicative of Ensynch’s high standards of excellence and its continuing successful partnership with Microsoft.

Awards were presented in a number of categories to Microsoft partners from around the globe, with winners chosen from a pool of more than 2,300 entrants. Ensynch’s Identity Management Solution was selected as the winner for the Active Directory and Identity Management, Advanced Infrastructure Solutions category. The award was presented to Ensynch in Boston, Massachusetts at an awards ceremony at the Microsoft Worldwide Partner Conference, the evening of July 12, 2006.

This award is based on a lot of hard work by some very talented Technical Architects, Engineers, and Project Managers from my company. Some of the people assigned to this project spent more than a year working in Prague, CZ developing processes and procedures to facilitate a massive migration of resources of varying sources to Active Directory and Exchange 2003. Major kudos to:

• Pat O'Horo
• Greg Whitworth
• Sean Murphy
• Joel Fraboni
• Jake Ballecer
• Jeremy Muir
• Kim Essendrup
• Matt Henderson
• and anyone else I missed...
  

The full release can be found .