Helpful hints for resolving AD FS problems – Part 1

Hi everyone.

Over the past week I’ve been building a lab for an upcoming deep dive into Microsoft’s Web Application Proxy.  During the course of building the lab I ran into a few interesting issues with AD FS and the Web Application Proxy that I wanted to cover.  Some were similar to issues I’ve run into in production environments and some were new to me.

These issues are interesting in that there aren’t any obvious indicators of the problem in any of the typical logs.  Two out of three required some trial and error to determine root cause, while the third drove me quite insane for a good two weeks before getting an answer from an “official” source.  Over the course of this series of blogs I’ll cover each issue in detail with the hopes that it will help others troubleshoot these issues in the future.

Issue 1: AD FS Certificate authentication fails

I’m going to start with the problem that took me the longest to resolve and eventually required getting the answer directly from an official source.

For those of you that are unfamiliar, AD FS provides the capability to offer multi-factor authentication methods both native and third-party.  Out of the box, it supports certificate-based authentication as an option for a multi-factor or “step-up” authentication mechanism.

A few months back I wanted to take advantage of the certificate authentication feature to provide a two-factor authentication solution for applications integrated with AD FS.  Like a good engineer I did my Googling, read the Microsoft articles and various blogs out there to understand how the feature worked and what the requirements were.  I built a lab in Azure, setup an AD FS server, and ensured port 49443 was open in addition to the the typical ports required by AD FS.  I created my instance of AD CS, issued a user certificate containing the user’s UPN in the subject alternate name field, and setup a sample SAML app and configured it to require Certificate authentication.

How easy it all sounds right?  I navigated to the sample application and got the screen below…

Screen Shot 2017-06-04 at 9.29.35 PM

and I waited….  and waited…. and waited…  Ummm, what went wrong?  Well surely the AD FS log will tell me what happened.

Screen Shot 2017-06-04 at 9.34.03 PM.png

Well isn’t that odd.  No errors or warnings in the AD FS Admin log.  A quick check of the Application and System logs showed no errors either.  Maybe the AD FS Debug log would show me something?  I flipped on the log and attempted another authentication.

Screen Shot 2017-06-04 at 9.38.07 PM

Nothing as well?  Maybe the server can’t query the revocation lists designated in the certificates CDP?  Nope, not that either the server can successfully contact the CDP endpoints.  At this point I began to get quite frustrated and attempted packet captures, Fiddler captures, and anything and everything I could think of.  Nothing I tried revealed the answer.

I finally gave in (which I can tell you is incredibly challenging for me) and reached out to an “official” source.  We chatted back and forth and went through much of the same steps as outlined above to ensure I didn’t miss anything.  However, we ran into another dead end.  He then reached out to some other engineers he knew and eventually we got a hit.  We were told to check to see if there were any intermediary certificates stored within the trusted root certificate authorities store.  Sounds like an odd circumstance, but sure why not.

Upon opening up the certificates MMC, opening the machine store, and exploring the trusted root certificate authorities store low and behold I see an intermediary certificate within the store.  I deleted the certificate, restarted the AD FS server and attempted another login to the sample claim application and hit the screen below.

Screen Shot 2017-06-04 at 9.50.16 PM

Boom, I’m finally receiving the certificate prompt.  Clicking the OK button brings about the successful login below.

Screen Shot 2017-06-04 at 9.51.23 PM

So what was the issue?  Apparently AD FS certificate authentication fails without generating an error in any logical location (maybe nowhere at all?) if there is an intermediary certificate in the trusted root certificate authority machine store.  I’ve verified this is an issue in both AD FS 2012 R2 and AD FS 2016.  Now why this occurs is unknown to me.  It could be the underlining HTTPS.SYS driver that pukes and doesn’t report any errors to the event logs.  I didn’t get a straight answer as to why this occurs, just that it will due to some type of integrity check on the machine certificate store.  Odd right?

That completes the rundown of the first of three problems I’ll be outlining in this series of blogs.  Hopefully this helps save someone else some time and aggravation.

See you next post!



Citrix headaches

One of our clients is beginning to utilize the Citrix XenApp Fundamentals installation we put in a year ago, and we ran into an interesting error when attempting to setup a new user with access. As a bit of background, the client is using Citrix XenApp Fundamentals 3.0 with Windows Server 2008 SP 32-bit. and the user we granted Citrix access to has been an Active Directory user for over three years. There were only two users utilizing the Citrix connection and it had been working flawlessly.

We setup the new user with Citrix access and had them attempt to connect. Once the user selected an application, we ran into an interesting issue. Instead of receiving the streamed application like normal, the Explorer interface ended up popping up with the following error message:

“You have started Windows Explorer in your ICA Seamless session. This will obscure your local Explorer desktop.
to re-enable this ICA Seamless connection, please logoff and restart the Seamless remote application.”

We knew off the bat that the existing users were having no issues, so that meant it was either the user’s computer, the user’s profile, the user account, or an issue with adding new accounts to Citrix. We opted to eliminate the Citrix issue first and granted another AD user permission on the Citrix server and logged in without an issue. Next, we tried logging into Citrix using that new AD account from the affected user’s computer, again no issue. Lastly, we tried logging in with that user’s account from another computer and we received the same error. We now knew the problem was related to the user’s Active Directory account.

After a bit of Googling, we came across some KB articles from Citrix. Unfortunately, we couldn’t find an article that fit our issues. The Citrix community forums were a bit more helpful and we tried a few fixes suggested by community users such as the following:

  • Delete the profile for the affected user account.
  • Delete the registry key for the affected user from HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionProfileList
  • Restart the server
  • After removing the profile and the profile registry key, restart the computer, and copy the Default user profile from a working Windows Server 2008 install

None of the above fixes seemed to do the trick. After some more searching I ran across a blog post from James Scanlon. His fix applied to XenApp 5 running on a Server 2008 SP2 install, but we figured it was worth a try at this point. The fix involves clearing an attribute for the affected Active Directory account. We hopped on the domain controller, opened ADSI edit (ADSIEDIT.MSC), expanded domain and organizational unit nodes, and right-clicked on the user and choose to edit the properties. The culprit in this scenario was the userParameter attribute, which from what I understand is used to set user specific settings for terminal services. This value was either corrupt or previously set to a value that was no longer valid. After clearing the value we had the user try logging again, and lo and behold, the application was now streaming correctly.

It must not be a common occurrence, because I wasn’t able to find many references to it on a web. So either my Google skills were suffering due to lack of caffeine, or it’s not a common error. Whatever the case, the problem is solved and all is right in the world!