UPDATE 17-aug-2017: Well Citrix did what they promised to do. The Citrix Netscaler nCore version 12 build 51.24 was released on 14th of July. In the release notes it is mentioned in the Fixed Issues section that issue #0654092 is resolved. But when I tested it, the issue was not resolved that I described earlier in this post. So I raised a ticket at Citrix TAC and explained and demonstrated everything. Also I tested it on a fresh out-of-the-box VPX v12 build 51.24; also same result.
After Citrix TAC consulted there Dev Team, Citrix has raised a new Enhancement request number, that is #0692815. Hopefully it will be resolved in the near future, but I can imagine that this has quite far-stretching consequences because it should be addressed outside of the NetScaler OS.
UPDATE 23-jun-2017: Citrix Support confirmed to me that CTX215496 will be addressed/fixed in version 12.0 build 51. This build is sheduled for release on 30th of June 2017 if everything go's to plan.
One of the first things that come to my mind when publishing a website towards the public internet is; SECURITY.
A good start is getting it on HTTPS; but once on HTTPS, you must make sure that the known weaknesses of securing SSL connections is properly dealt with.
To test this, I usually use the SSL Server Test of Qualys SSL Labs; a free online service that performs a deep analysis of the configuration of any SSL web server facing the public internet.
By making sure that the correct Cipher Suites are used in a correct order, weak SSL protocols (like SSLv3) are turned off and known exploits like POODLE and POODLE_TLS are mitigated, you are a on a good path.
How to accomplish this on a NetScaler (one of my favorite load balancers) is by taking care of these things. You can read about it here on this Citrix Blog article.
Once done, you may test it on Qualys SSL Labs. If correctly followed, you will get a A+ rating; see figure 1 below.
Figure 1
But sometimes, you will only see an A-rating (see figure 2). You probably checked everything and compared one to another, to come to the conclusion everything is set OK... but still getting only a grade A; how is this possible comes to mind (with a few swear words :) )
I had also to deal with this issue; I noticed that directly published public internet facing websites get a A+ grade, but when a AAA-TM form of the NetScaler gets involved it did got better than a A-rating.
After looking closer to this, I noticed this only happens when a HTTP 302 of 301 is used (see figure 3).
Figure 2
Figure 3
When you visit a website terminated on a virtual server on the NetScaler, and the VIP itself also depends on a Authentication Server; the NetScaler will throw in a HTTP 302 so that you can authenticate first, before you can continue.
And here lies the problem. The NetScaler does not insert the Strict-Transport-Security value in the RESPONSE HEADER when it does a redirect via a HTTP 301 or 302 (see figure 4 and 5), only on a HTTP 200 (see figure 6). This is because the redirect takes place outside of the NetScaler, so the rewrite policy does not 'stick'. More on this issue here.
Figure 4
Figure 5
Figure 6
Hopefully Citrix will resolve this issue soon; the last time the Citrix edited the article was on 01 aug 2016; ; Citrix mentioned they put it on the Enhancement ticket for the Citrix Developers working on the NetScaler.
UPDATE 04-SEPT-2017: I discovered that by protecting Autodiscover with a 401-authentication; the Skype for Business client is not capable to utilizing this, resulting in not discovering EWS settings. Lack of EWS result in; no calendar information, no free/busy information etc. So my advice is not to protect Autodiscover with a 401-authentication if you are using Skype for Business. If not; leave it on; your Outlook Client can handle this.
If you try to search on 'Exchange with NetScaler' ; you will probably wind out on the following:
The thing is that these article are in my opinion not complete; it does what the title says it does; load balance Exchange with NetScaler. But if you want to do more with NetScaler like handling AAA… you are in a challenge; probably searching on the web for scraps of documentation how to accomplish a proper load balance configuration with use of the AAA function within NetScaler. The AAA function is a cool feature that you can use to offload the authentication mechanism at the NetScaler, while protection your front end of Exchange. In my opinion, FreeBSD does a better job on protecting you against HTTP/HTTPS/SSL exploits compared to Microsoft does for her Internet Information Services (IIS).
What you will find in this guide is complete walkthrough for configuring Exchange on your NetScaler. Although I have one server in my lab, you can do the same by just adding more servers to your service group(s) on your NetScaler; just like that. Also what you will find is a SSL A+ grade (more on this here on my other blog post) Outlook Web Access page with proper Cipher suites.
There are however prerequisites for this guide to have in place prior following it; my lab consists (and thus this guide is based on) of the following:
Exchange 2016 (probably 2013 as well)
Citrix NetScaler 11.1 with Enterprise license or higher
Windows Active Directory (Forrest and Domain level is not important for this guide)
Internet connection :)
(Good) coffee
Also this guide is ordered in a specific way because the some sections depend on the preceding action:
Create Cipher suite
Create SSL profiles
Create servers
Create Monitors
For AAA authentication server
Form SSO profile
Traffic Profile
Traffic policy
Session Policy
Session Profile
Authentication virtual server
LDAP servers
LDAP Policies
Create service groups
Create vServers
Create content switch
Cipher group
First we must create a proper Cipher suite to get a better security profile; it is important that the order of the ciphers as showed in figure 1 below, is maintained.
If you read this Citrix Blog; they also mention Cipher Name SSL3-DES-CBC3-SHA ; I removed that one, because it is classified as WEAK. Also SSL3 is disabled on the SSL profiles, so it is not worth mentioning it in your custom cipher group.
As per 26th of November, Qualys announced it will change its lab grading.
Figure 1
SSL profile
The SSL profile eases your administration; if you want to change something related to SSL to effect all; than this is what you want to use.
NOTE: If you want to use SSL Profiles, you must first enable default SSL profile (a link how to this). You must understand by enabling default SSL profile there is no way back, other than rebooting the NetScaler and ditching the changes. So once set and saved, you must revert the complete configuration to undo this change…
We will create in total 4 new cipher suites:
Backend communication to Microsoft Exchange
Backend communication to Windows IIS
Frontend communication for Content Switches
Front end communication for the Citrix vServers in the Load balancing section.
So go to System / Profile / SSL profile
Below you see I already created the necessary profiles, but i will walk you through them.
Checkmark the ns_default_SSL_profile_backend and click Add
Figure 2
Do the following:
Disable SSLv3
Disable TLSv11
Disable TLSv12
Select the Custom cipher suite we created earlier
Why disabling TLSv11 and TLSv12 ?
This is because if you are on prior NS 11.0 build 65.35 or earlier you must also disable TLSv11 and TLSv12 due to the incompatibility between IIS and NetScaler. For more details, check this Citrix article Back-End Connection on TLS 1.1/1.2 from NetScaler to IIS Server Breaks.
If you are on NS 12.0 build 41.16 you can enable TLSv11 and TLSv12; Citrix fixed this in April 2017.
Figure 3
Like I explained earlier; we are going to create a backend SSL profile for IIS servers, just like we did for Exchange above.
Do the following:
Disable SSLv3
Disable TLSv11
Disable TLSv12
Select the Custom cipher suite we created earlier
Figure 4
Next we will create the SSL profile for the front end connections. This profile will be bound to the Load balance vServers on the NetScaler that will serve the clients.
Change here the following:
Disable SSLv3
Enable TLSv1
Enable TLSv11
Enable TLSv12
Deny SSL Regeneration: FRONTEND_CLIENT
Configure the DH Key
Select the Custom Cipher suite, just as you did in the preceding SSL profile
Select ALL ECC curves
How do I setup a Diffie-Hellman key on NetScaler? To configure the DH key; you must walk through this Citrix Article
Figure 5
Next up is the SSL profile; it looks the same like the preceding front end SSL profile, only this one differs because the SNI is set to ENABLED. This is needed because when you use more than 1 certificate on your content switch; you have to make sure that SNI is enabled so that your vServer or vSwitch in this case can send the correct certificate to the correct request.
So configure the following here:
Disable SSLv3
Enable TLSv1
Enable TLSv11
Enable TLSv12
Deny SSL Regeneration: FRONTEND_CLIENT
Configure the DH Key
SNI Enable: ENABLED
Select the Custom Cipher suite, just as you did in the preceding SSL profile
Select ALL ECC curves
Figure 6
So were done here. Move forward to the Load balance servers
Create servers
Firstly we must create the servers we are landing on; in my case; I have only one Exchange server in my lab; so I will create only one. If you have more than one in your environment; also create appropriately; I will tell you later on how you must configure the load balance setting etc.
Go to Traffic Management / Load balancing / Servers
Figure 7
Create each real Exchange server you have as a server in the NetScaler.
Name: I would suggest to fill in its real FQDN; SERVER01.domain.tld IP address: the real IP of the server that NetScaler can reach.
The rest is left default.
Figure 8
Create monitor
Now create the Monitor's that you will use to monitor if your Exchange functions are healthy.
KEMP wrote a very nice how to manual for configuring it on KEMP (I must admit, it is more complete than the one created by Citrix )
To make things easy; make sure the following health URI's are used:
NOTE: In Exchange 2016, clients no longer interact with Exchange using RPC, it is all HTTPS now. The protocols that Outlook clients use to access their mailbox are MAPI over HTTP and Outlook Anywhere. In Exchange 2016 the default is MAPI/HTTP.
Outlook 2010 with KB2965295 or greater is required with Exchange 2016.
Figure 9
Go to Traffic management / Load balancing / Monitors
Create the first monitor that you we will use later for health checks on the Outlook Web Access and set everything like the below figures.
Figure 10
Now you must fill in: HTTP Request: (in this case) GET /owa/healthcheck.htm Response code: 200 SSL Profile: your SSL profile like in figure 4
Like I mentioned earlier in the SSL profile chapter, if you have NS prior 11.0 build 65.35, select a custom cipher suite that has TLSv11 and TLSv12 disabled.
If you are using NS11.0 build 65.35 or later; no worries here; leave it as it was. Same thing for NS 12.0 build 41.16.
Figure 11
Create Service Groups
Here you should create the Service Groups for all of the Exchange Services. You should create them separately instead on only 1 service or service group. That makes you more flexible in the future.
Figure 12
For every group pick a unique name for it so that you can recognize it later.
Also choose the Service Group Member(s). Choose the special created SSL profile for backend communication and last but not certainly least; choose the appropriate monitor you created earlier for the service
Figure 13
Create SSO profile form for OWA
One of the cool things that the NetScaler can do is that you can create a Single Sign On experience by letting the NetScaler 'fill in' a login form for you; how cool is that?! It is not literally filling in the username and password in the required fields like LastPass or a KeePass (plugin), but it sends the required fields in a POST action to the backend once the NetScaler validated the provided credentials to your LDAP or RADIUS environment; that doesn't mean that if your user is known via the LDAP query, it also has rights to access Exchange. It is my opinion very helpful that NetScaler validates the provided credentials, while protecting the front end of your Exchange environment. In my opinion NetScaler (and of course FreeBSD) can do a better job on security than Windows can do with IIS :)
But back to the configuration part; go to Security / AAA-Application Traffic / Policies / Traffic and click on the Form SSO profiles tab and click Add
Figure 14
Fill in like I did in the figure below: Name : as_exchange-owa_sso_form Action URL : /owa/auth.owa Username field: username Password field: password Success Criteria: HTTP.RES.SET_COOKIE("cadata").VALUE("cadata").LENGTH.GT(70) Name value pair : flags=4&trusted=0 Response size : 60000 (VERY IMPORTANT, AFTER A CUMULATIVE UPDATE OF EXCHANGE, THE RESPONSE SIZE NEEDED TO BE BUMPED UP A FEW BYTES.) Extraction : DYNAMIC Submitted method : POST
Figure 15
Click on the Traffic Profiles tab, we will create 2 traffic profiles.
One for Logging in Outlook Web Access
One for Logging out of Outlook Web Access when you punch the 'Log out' button in the top right corner ; we will use this one later in this guide
Figure 16
Fill in like I did in the figure below: Name : traffic_prof_exchange-owa_sso AppTimeout : 1 Single Sign On : ON Form SSO: as_exchange-owa_sso_form (The form you created a few moments earlier)
Figure 17
This traffic profile is the one that tells the NetScaler what to do when you hit the log off button and recognizes the URI ; it will log you also of the NetScaler by ending your session.
Figure 18
Now it is time to put the above SSO Form en two traffic profiles together in two traffic policies.
Figure 19
Name: traffic_pol_exchange-owa_sso Profile: traffic_prof_exchange-owa_sso (the one you created a few moments earlier) Expression: HTTP.REQ.URL.CONTAINS("/owa/auth/logon.aspx")
Figure 20
Name: traffic_pol_exchange-owa_sso_logout Profile: traffic_prof_exchange-owa_sso (the one you created a few moments earlier) Expression: HTTP.REQ.URL.CONTAINS("/owa/logoff.owa")
The NetScaler will recognize URI and close your session on the NetScaler when you hit the Logoff button in OWA; it will bring you back to your configured AAA webpage.
Figure 21
Create Session policy
Here we create a session policy that you will bind to you AAA server(s) you are going to use for Exchange.
Go to Security / AAA-Application Traffic / Policies / Session and click on Session Profiles and click Add
Figure 22
Name : as_ses_prof_exchange-owa_sso_001
Check mark the following to override global configuration: 2,3,4,5,6,7,8 (like the figure below :) )
Then set like wise the figure below; Default Authorization Actions : ALLOW Single Sign On : ON Credential Index: Primary Single Sign-ON Domain : VERY IMPORTANT; THIS IS YOUR PRE-WINDOWS 2000 DOMAIN NAME (!) HTTP only Cookie : No Persistent cookie validity : 30
Figure 23
Next create the Session Policy ; click add.
Figure 24
Name : ses_pol_sso_001 Request profile : guess what, this is the one you created earlier :) ) Expression: ns_true
Figure 25
Create Authentication Virtual Server(s)
Now it is time to create the AAA server(s) for Exchange.
I have two in my lab:
One for only AD username and Password
Second one for Multi factor; it requires also a random token generator like SecurID RSA or Azure Authenticator. If you want to know how to do this on a NetScaler 11.x or 12.x; go to my next thread about accomplishing this.
Go to Security / AAA - Application Traffic / Virtual Servers and click add
Figure 26
Here we create the AS virtual server.
Name: choose a name so that you can identify it later Authentication domain: set the authentication domain to your domain that you use. IP address and Port: your internal IP that you will connect to and set port to 443. Certificate: 1 server certificate; the one that you will use when you connect to the AS virtual IP; that can be a internal issue PKI certificate or a public issued one. It should match the FQDN that you will publish your AS virtual IP on. Basic Authentication: Select here the LDAP or RADIUS policy(s) that you should have created earlier. If not; go to next chapter for this. 401 Based Virtual Servers: This is where you can see where your Authentication Virtual server is configured on. When you create your AS VIP, it will not be connected to a virtual server. SSL Profile: Select your SSL profile that you hardened with earlier. Policies: select the exchange OWA SSO Session policy that you created earlier.
Figure 27
This is the Session Policy that you should bind to the AS Virtual VIP.
Figure 28
Like I said here above; you should have a correct LDAP policy in place to help your users to authenticate them selves; you can see the figure below how it looks like.
I have created 2 LDAP policies to accommodate this;
One is for the Pre-Windows 2000 username (SAMAccountName)
Second one is for the UPN username (UserPrincipleName)
Why two LDAP policies?
This is for cascaded login; if the first LDAP policy has failed, it will go to the second etc. This way the user can try his SAMAccountName or his UPN name.
Figure 29
Go to Security / AAA - Application Traffic / Policies / Authentication / Basic Policies / LDAP. Select Servers Tab and click on Add.
Figure 30
Fill in the following:
Name: LDAP_<IP of Domain Controller>_SAM_001 Server IP: the IP of your LDAP Server, usually a Windows Domain Controller Security: in my case i have LDAP Secure, so i chose SSL. If you don't have secured your LDAP, choose PLAIN TEXT
Connections Settings: Base DN: fill in the DN path of your users where the LDAP server should search in Administrator Bind DN: fill in the full CN of your AD user that the NetScaler will use when crawling your AD via LDAP. Administrator Password: you know; the AD user your filled in 'Administrator Bind DN'.
After wards you can verify it by clicking on Test Connection; it should look green. Server Logon Name Attribute: SAMAccountName Group Attribute: memberOf Sub Attribute Name: CN SSO Name Attribute: SAMAccountName (very important here; this will be the username the NetScaler will use then talking to the back-end of your configured VIP.
The rest is default; just compare it and configure it likewise.
Figure 31
The second LDAP server looks the same as the above LDAP server; only the Server Logon Name- and SSO name Attribute differ.
This one does a LDAP query for the UPN instead.
Figure 32
Now we have created the single factor AS virtual server. I you want to know how to configure dual factor authentication on NetScaler 11.1 or NetScaler 12.0; go to my next thread about how to do this.
Create Virtual Servers
Now we can create the Virtual servers
Figure 33
Then we look at the SPECIFIC SETTINGS and/or PERSISTENCY.
Name: rp_vs_exchange_owa_ssl_001 Services: the service group you created earlier for OWA. Certificate: choose the certificate you will use for OWA. SSL Profile: choose the correct SSL profile we created earlier. Authentication: in my setup i choose the dual factor setup, but you can choose the single factor AS server i used in this blog post. If you want to use the Dual Factor method, check my other blog post how to this. Until then, stick to the single factor AS server. :) Polices: we have 3 policies in total; 2 for traffic and 1 for enabling HSTS.
Figure 34
Here are both traffic policies for Logging in and logging out of OWA
Figure 35
And the rewrite policy for inserting HSTS in the header.
Figure 36
This next vServer is for RPC calls from the Outlook Client. You will notice this one is very simple put together compared to the one used for OWA.
RPC communication is important for Exchange version prior Exchange 2013. From Exchange 2013 and forward, Microsoft will gradually say goodbye to RPC over HTTP and hello to MAPI over HTTP communication. When using Exchange 2013 or higher, the NetScaler VIP for the RPC traffic will not have any hit registered. But for allowing Outlook Clients that don't have MAPI compatibility, the RPC vServer can come in handy.
So let's get going on the configuration:
Name: rp_vs_exchange_rpc_ssl_001 Service Group: the service group you specially made for this exchange service. Certificate: choose the correct certificate for this. Policy: choose the rewrite policy for HSTS.
Figure 37
This vServer is for Exchange Web Access.
Name: rp_vs_exchange_ews_ssl_001 Service Group: the service group you specially made for this exchange service. Certificate: choose the correct certificate for this. Policy: choose the rewrite policy for HSTS.
Figure 38
This vServer is for ActiveSync. This one has also 401-based authentication enabled on it.
Name: rp_vs_exchange_activesync_ssl_001 Service Group: the service group you specially made for this exchange service. Certificate: choose the correct certificate for this. Authentication: choose the AS server you created for this that uses the single factor. Policy: choose the rewrite policy for HSTS.
Figure 39
UPDATE 04-SEPT-2017: I discovered that by protecting Autodiscover with a 401-authentication; the Skype for Business client is not capable to utilizing this, resulting in not discovering EWS settings. Lack of EWS result in; no calendar information, no free/busy information etc. So my advice is not to protect Autodiscover with a 401-authentication if you are using Skype for Business. If not; leave it on; your Outlook Client can handle this.
This vServer is for Autodiscover. This one has also 401-based authentication enabled on it.
Name: rp_vs_exchange_autodiscover_ssl_001 Service Group: the service group you specially made for this exchange service. Certificate: choose the correct certificate for this. Authentication: choose the AS server you created for this that uses the single factor. Policy: choose the rewrite policy for HSTS.
Figure 40; just ignore the authentication settings; see my earlier annotations.
For the Outlook AddressBook.
Name: rp_vs_exchange_oab_ssl_001 Service Group: the service group you specially made for this exchange service. Certificate: choose the correct certificate for this. Policy: choose the rewrite policy for HSTS.
Figure 41
For MAPI over HTTPS connections
Name: rp_vs_exchange_mapi_ssl_001 Service Group: the service group you specially made for this exchange service. Certificate: choose the correct certificate for this. Authentication: none; the real server will handle this. As of time of this writing, NetScaler doesn't support the authentication between MAPI client and Exchange 2016. Policy: choose the rewrite policy for HSTS.
Figure 42
Create content vSwitch
No we have all the vServers in place, lets try hem all together in one content vSwitch.
Go to Traffic Management / Content Switching / Virtual Servers and click add.
Here we must pay close attention:
Name: cs_vs_www_ssl_001 Content switching Policy Binding: select all content switching policies in the correct order. Certificate: choose the correct certificate for this, you must enable SNI on both the SSL profile as on binding the certificate if you choose to use more than 1 FQDN on the content vSwitch. This is the case once you use Exchange 2013 and higher, Office Online Server can also be used by this content vSwitch :). Authentication: none; this will be handled by the individual vServer. Policy: choose the rewrite policy for HSTS.
Figure 43
This is the response policy for the content vSwitch.
Figure 44
Here we have all the bindings that bind all of your vServers to the content vSwitch
You can see that instead of the using the Expression mentioned in the Citrix guide; i used a different one. One that ignores CASES. Quite handy don't you think :)
Figure 45
First you must create each content switch action.
Each action must target is own virtual server.
Go to Traffic Management / Content Switching / Actions
Figure 46
After creating the action; it is now time for the Policies.
Go to Traffic Management / Content Switching / Policies
NOTE: if you wonder why I added the HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS("/cgi"), that is due to the following.
When you try to connect to OWA we get authenticated to AAA vServer and after that, the page gets stuck at https://<exchangeFQDN>/cgi/selfauth?<ecnrypted and encoded text>.
To solve this, you should add a extra binding on the contentswitch with this expression and with the same cs action on it as on the '/owa' expression.
Now we are almost done here. You must make sure that your Exchange servers that will face your users, have the correct authentication setting for OWA and ECP.
This must be set to Form based and UPN; otherwise SSO for OWA will not work.
Go to your Exchange Admin center (https://FQDN/ecp) / servers / virtual directories / owa
Figure 48
Figure 49
You can choose for the basic Authentication way or choose for the FBA (Form Based Authentication) ; the latter I would prefer and I will tell you why.
Before Exchange 2016 CU4, you could enable the Legacy Logoff string by editing the exchange config file, so that when you are using a 401 basic authentication when reverse proxying it; the load balancer would recognize the logoff button when pressed and thus properly closing the connection to the client; the user is now logged off. I wrote a blog post about how to do this on a KEMP load balancer.
Well after the release of Exchange 2016 CU4, Microsoft Exchange Team decided to roll back this feature. Thus breaking the logoff recognition when using 401 basic authentication.
I also checked at the Exchange team blog for an answer; the Exchange team confirmed that the logoff.owa is the new (or old way, its how you look at it) for logging off.
I broke my head about this because at first I thought something was wrong with my reverse proxy setup; every time I pressed the Log Off button only a popup would appear (like the one below). Even when I tried directly logging on the webmail without using the reverse proxy had the same result. So by testing directly I ruled out this was a reverse proxy issue; it is apparently by design.
Figure 50
So how to overcome this; ?
You can go for the FBA, but some load balancers cannot handle FBA properly with a nice AAA proxy page, like KEMP load balancer (as of time this writing of course). But the NetScaler can do a nice FBA login for you and use the authentication proxy at the same time! So NetScaler is in in de advantage here.
