Recently we had cause to do a whole lot of research ourselves and end up calling Microsoft to get our implementation vetted and troubleshot (it was not working – all or almost all connections that should have been Kerberos connections were degrading back to NTLM).

Here are the salient notes and facts about troubleshooting and achieving the ultimate goal (having Kerberos working with our systems).

• The most detail-oriented troubleshooting involves use of Microsoft’s NetMon (apparently the new version is pretty sweet) or Wireshark. We used Wireshark. This captures all the network traffic your workstation or server is exposed to during the recording period. You can filter the network transactions according to protocol, port, originating and destination IP addresses, etc. It also lets you inspect HTTP packets and other protocols it knows how to decode.
• If you are following the TCP stream (right-click on an HTTP  TCP packet in the capture and choose “Follow TCP Stream”), what you are looking for are GET requests that include an Authentication or WWW-Authenticate header field. Further, in there, what you are looking for is a string that starts with “Negotiate: ” and is then followed with a semi-random string. If the string is a Kerberos string, it starts with the letter “Y”, and can be quite long (~1 page or window full of gibberish). That’s the packet where your client presents a Kerberos ticket to the server. Follow up troubleshooting from that point. If the server responds with a 401 (unauthorized), your client will start degrading the authentication, which means Kerberos is screwed up somehow for you. Other Negotiate strings can start with “T|RM” or “o”, etc.
• You’ll also find Kerbtray helpful. This is part of the Windows Server 2003 Resource Kit. Run it in the systray of the client/server you’re troubleshooting. This will tell you if the machine it’s running on (and the user account that’s current logged in) have any Kerberos tickets, and when they were granted. Unfortunately it doesn’t look like Kerbtray will show any tickets granted to any other accounts running via “Run As…” in your login session.
• Other tools that are useful: Active Directory Users and Computers (part of normal Windows 2003 administrative tools under program files for any Admin account you log into) – You’ll need sufficient rights to see the servicePrincipalName property on various accounts (service accounts, application pool IDs).
• Another: ADSI Edit, which is part of the Windows Server 2003 SP2 support tools, is a customized LDAP Browser which gives you another window into the AD and another way to look at service accounts. This is another way of getting to similar information that you can find in ADUC, above.
• Another: SetSPN, which you use on the command line to add SPNs to accounts and to list SPNs already assigned to accounts. Depending on how well or inconsistently your AD reflects your LDAP organization (of the same data), you may find you need to find the Distinguished Path (the string with all the CN= and OU= values) for your AD Object with a “setspn -l” command on the object and then find the object in LDAP with ADSI Edit based on the path output by “setspn -l”.
• Kerberos related activity in the IIS logs appear to be timestamped in UTC. Don’t freak out if that’s several hours in advance of your local timezone. Kerberos probably isn’t screwed up due to timesync errors.
• Another way of looking for Kerberos-authenticated sessions is by looking at your (Web Front End) server’s Security Events Log. Open it via the Event Viewer. You are looking for a Success event, number 540, where the User is the user you’re troubleshooting. If the session achieved a valid Kerberos login, the Logon Process in the Event Description should read “Kerberos” and the Authentication Package should also read “Kerberos”. Logon Type is probably 3. If the Logon Process and Authentication Package read “Negotiate” or “NTLM” instead, your Kerberos is probably not working. If you have more than one Web Front End that are load balanced, be sure to check all of them for the user you’re troubleshooting.
• Proper Kerberos setup is almost all about the SPN. If you are lost about what SPNs you should be using, you might want to look at the Excel worksheet that SharePoint from Scratch published (down as I write this post up – drop me a note and I can send you the one I did up myself).
• If your domain’s FQDN is domain.local and you have a web application with an FQDN in that domain of host.subdomain.domain.local, your SPNs may have to include:
  – HTTP/host.subdomain.domain.local
  – HTTP/host.subdomain
  – HTTP/host
  all set to the service account that runs the Application Pool for that web application. What’s at issue is whether the “host.subdomain” or simply the “host” SPN is the proper one in that context for the short SPN name. I’ll update this if I have more information.
• The service descriptor listed if you use “setspn -l” on a service account is sometimes capitalized and sometimes lower-case (i.e. the SPN is sometimes listed as “http/host.subdomain.domain.local” and sometimese as “HTTP/host.subdomain.domain.local”). It depends on how the SPN was created, and who created it. According to Microsoft IIS support, the capitalization does not matter, but the convention is to capitalize it.
• You may find SPNs missing for application servers. If you do a “setspn -l” on a server object in the active directory (e.g. “setspn -l domain\host$”) and you get no SPNs returned, you probably neet to use “setspn -a” on that object (with a domain admin account running the command) to add the FQDN and short name for the host to its SPNs. You may have to set these SPNs from a command shell running on the server in question. For a host whose FQDN in the domain is host.domain.local, the two SPNs that should exist for that host are:
  – HOST/host.domain.local (command would be “setspn -a HOST/host.domain.local domain\host$”)
  – HOST/host (command would be “setspn -a HOST/host domain\host$”)
• The overall, final fix for almost all of our problems actually laid in applying the hotfix that installs the Microsoft Office Servers Infrastructure Update: July 15, 2008. This was weird especially because the seminal guide for implementing Kerberos on SharePoint (on TechNet) seems to indicate that you only need this hotfix if you plan to implement Kerberos with your SSP internal farm transactions.