Dave Stork's IMHO : High Availability

Simplifying the OWA URL with Citrix Netscaler

dmstork — Mon, 24 Dec 2012 12:16:00 GMT

Next to Content Switching (which I recently wrote a post about), Citrix Netscalers can also do URL Rewrites. This enables us to simplify the OWA URL.

First, be sure the Rewriting option is enabled by going into System, then Settings and choose Configure Basic Settings. Check the tick box for Rewrite

After this, first make an Rewrite Action by going to Rewrite>Actions and add an Action. Give it a comprehensive name and set the type to REPLACE. In the Expression the following should be used:

http.REQ.URL

In the String expression for replacement text, the following value should be used:

“/owa/”

Be sure to type it in and not copy it from this blog, otherwise it could not workd correctly. The screenshot below shows the value as mentioned before. Click Create to create the Rewrite Action and click Close to close the window.

Now you can create a Rewrite Policy by going to Rewrite>Policies and then click add…

Again, give it a sensible name and be sure the Action is set to the earlier created Rewrite Action (in the screenshot below Rewrite_Action_OWA).

For the Expression, use the following:

http.REQ.URL.EW(“/”)

Again, type it and do not copy and paste. Finally, press Create and Close. This Rewrite Policy now checks for URL's which use the root path / and will replace it with /owa/.

But in order to make it happen, the policy has to be enabled somewhere. In this case I bind it to a Load Balancing Virtual Server already previously made (see this blog post). This has to be the Virtual Server which is responsible for (at least) Outlook Web Access.

Open the Virtual Server, go to the Policies Tab and press the Rewrite (request) button. Right-click in the window and choose Insert Policy. Choose the previously made Rewrite Policy as shown below:

And voila! Now every user entering https://webmail.contoso.com/ will be directed to https://webmail.contoso.com/owa/ without a fuss! And because the policy triggers only on the root, directly using /owa, or /ecp for that matter, will also work.

How about HTTP to HTTPS redirection?

That is not done via Rewrites, but there are more ways than one. Make a Load Balancing Virtual Server, listening on port 80 and as IP address the Virtual IP used for OWA. You do NOT check any services. Instead go to the Advanced Tab and in the Redirect URL enter HTTPS:// with the virtual IP used for Webmail. Press Create and close. Do remember to enable traffic over TCP port 80 towards the Netscaler, otherwise this won’t work. This is also described in the Netscaler Deployment guide and depicted in the image below:

If you are also using Content Switching, you can also make a Content Switching Virtual Server accepting traffic on port 80 and again using the OWA Virtual IP. As a target the Load Balancing Virtual Server using port 443 should be used (that can be used multiple times as a target. Description how it was made in this blog post). This is shown in the image below:

You should make duplicate Content Switching policies, as they can only be used once. The Expression however, is exactly the same as the Content Switching Policy used in the Content Switching Virtual Server using SSL.

Now every user will be directed to the correct URL, whether they use http://webmail.contoso.com, https://webmail.contoso.com/ or http://webmail.contoso.com/owa/ .

Load balancing Exchange 2010 with Citrix Netscaler using Content Switching

dmstork — Fri, 21 Dec 2012 17:45:00 GMT

Next to F5, KEMP technologies and a lot of other network load balancing vendors there’s also Citrix with it’s Netscaler brand. Especially when an environment also has Citrix servers, it could mean that well scaled Netscaler devices are present and can also be used for other purposes next to Citrix Secure Gateway access. Obviously Exchange 2010 comes to mind.

Citrix already has a very helpful Netscaler Exchange 2010 deployment guide (PDF warning). But unfortunately that guide is not always something one can implement exactly. For instance, in the guide Citrix uses an unique IP address for each separate protocol, which is not always possible if these are limited.

However, all or most Netscalers also provide Content Switching and with this you only have to use one IP but also have optimized settings for persistence/affinity and time-out for all protocols using the same TCP port (HTTPS). For some background information around persistence for Exchange 2010, check this article.

First create the services as described in the Citrix Deployment guide. You make one per physical server for each specific service, like HTTP (Load Balancing>Services>Add>):

When that is done you can create a Virtual Server for each different protocol, meaning OWA, ActiveSync, OAB, EWS etc. (Load Balancing>Virtual Servers>Add>). In this example, the OWA Service is shown with the specific Load Balancing method and persistence options (note that COOKIEINSERT requires SSL Offloading).

But instead of entering an IP address, keep it emtpy and untick the “Directly Addressable” box.

Now you have to make sure Content Switching is enabled on you Netscaler. You can do that via System>Settings>Configure Basic Settings> Enable Content Switching.

After this you can create Content Switching (CS) Policies via Content switching>Policies>Add…. For OWA I would check whether the specific hostname is requested in the HTTP request: HTTP.REQ.HOSTNAME.CONTAINS("webmail.contoso.com")

You can build it with the expression builder via Configure… button and build the expression from there.

When you’ve made the CS Policies, you can now make Content Switching servers via Content switching>Virtual Servers>Add…

Now you can add the IP address the Netscaler has to respond to. This is also the Virtual IP (VIP) address you have to point your FQDN for OWA and other protocols towards.

In the CSW field (open per default), right click and choose “Insert Policy”. A drop down menu appears (as shown above), and every available CS policy is visible. Note that a policy can only be used once.

In this case the previously made webmail.contoso.com policy is selected. Now select the target field and the different Load Balancing Virtual Servers are listed, in this case only VIP_Exchange_OWA.

Select it and choose Yes in the corresponding question box,

Now every HTTP request on IP 172.16.0.205 with FQDN webmail.contoso.com will be directed to use the Load Balancing Virtual Service which uses two Client Access Servers previously defined as valid services.

If you want to make another Load Balancing services for other protocols with other persistence timeout values, but with the same VIP, make another Contents Switching Policy and add it to the same Content Switching Virtual Server. However, you will have to point them to other Load Balancing targets, namely those with the optimal settings.

For Autodiscover use the expression:

HTTP.REQ.HOSTNAME.CONTAINS("autodiscover.contoso.com")

For ActiveSync use the expression:

HTTP.REQ.HOSTNAME.CONTAINS("webmail.contoso.com") && HTTP.REQ.URL.PATH.TO_LOWER.STARTSWITH("/microsoft-server-activesync")

For EWS, OAB and Outlook Anywhere you can change the ActiveSync expression with the URL Paths /ews, /oab and /rpc. If you don’t specify these specifically, they would just use the OWA Content Switching policy (as it is agnostic about the path in this case) and thus the same persistent values as those specified for OWA. I found that it is sufficient most times.

Insert every CS Policy in the CS Virtual Server, and order them in the correct sequence. Note that Netscalers checks policies with a lower priority value first and works up to higher values (first 10 and then 100). The protocols which would trigger with specific paths in it should come first, otherwise they would be triggered by our first policy and will not get the optimized load balancing rules.

In the above example you can see the generic webmail.contoso.com policy has an OWA target and a priority of 100. Subsequent policies are ActiveSync (EAS), Autodiscover and Offline Address Book (OAB) each with a corresponding target and persistence settings.

After implementation you can check whether the rules are (correctly) being used by watching the Hits column.

So with Netscaler Content Switching you are able to still optimize persistence settings per protocol and still use one Virtual IP address for each HTTPS service.

For these screenshots I’ve used the Citrix Netscaler Free trail virtual appliance which can be downloaded from www.citrix.com. Note that for some of these settings you’ll also need SSL Offloading. The specific configuration and certificate selection (in the Content Switching Virtual Server for instance) is not shown.

Unable to change database path on Exchange 2010 in a DAG?

dmstork — Wed, 02 Nov 2011 18:19:00 GMT

If you find yourself unable to change the database path of an Exchange 2010 database, check whether it is part of a DAG. See the screenshot for the missing option:

If so, you cannot move it. The solution is to remove all copies except the active. Now you can move the DB and Transaction log paths.

Do note that you will have downtime! Changing the path requires a dismount and since you do not have the redundancy of a DAG, you will have to plan this in a service window.

After you are finished, you can add Mailbox Copies. Be sure you have made the same drive changes to the other DAG members!

Databases within a DAG need to be on the same drive letters and same unique path. It is tricky to change the location of the DB when you have to consider 15 other servers (DAG member maximum). The same drive configuration has to be present.

Of course it is better to have the database locations correct right from the start, but unfortunately we don’t live in an ideal world.

Exchange, Load balancers and recommendations

dmstork — Fri, 15 Jul 2011 16:07:00 GMT

This is a follow-up post to Differences in Exchange Load Balancing recommendations by Microsoft and vendors.

This post refers to issues I discovered and were discussed in that post. I suggest reading the previous article before reading this one. I also expect some experience with load balancers in combination with Exchange Server 2010.

In the previous post I mentioned several discrepancies in the Exchange deployment guides from several Load balancing companies. I contacted them and pointed out their apparent discrepancies. Their responses were very cooperative, kudos to them!

Due to time restrictions, I have not checked all deployment guides mentioned. Possibly later on

Kemp Technologies

I have had intensive contact with Kemp Technologies on mentioned discrepancies and other issues. This was during an implementation of two multi role Exchange Server 2010 boxes in a DAG with their virtual VLM-100, with clients using Outlook 2007 and Outlook 2010.

SNAT

I mentioned that they do not recommended SNAT as the setup for exchange. Unfortunately this was a misunderstanding in the terminology used. Microsoft Source NAT is not the same as current Kemp SNAT, it is however the same as L7 non-transparency. Kemp SNAT is when the source IP is changed to the VIP of the rule, not the real IP.

Static Ports

The wildcard rule, as mentions in the Kemp exchange deployment guide, did miss some additional clarification. Kemp agreed that it is a good practice to use static ports on the Exchange server. However, using a rule per port does have the disadvantage that clients can connect to different client access servers For all services needed by outlook MAPI RPC. A wildcard rule resolves this.
It is also important to note that specific port rule for the same VIP always has preference over the wildcard rule.

Idle Connection Timeout

Now, this wasn’t something I mentioned in my previous blog, but did have me busy for quite some time. It is also why it has taken a while to prepare this post.

In their deployment guide Kemp states that the Idle Session Timeout values, should be around 20 seconds. Kemp explains this as they want the failover value to be as close to the DAG failover time, as that’s would be within 30 seconds. If the value was high, the user had to wait for the idle timeout to expire. But if it was to low, the client would reconnect multiple times. So, that value would be a trade off.

Now, while deploying the load balancer with low values, users got warnings that their Outlook has re-established it’s connection to Exchange. Several times per minute. Next to that, the CPU of both Client Access Servers went to the roof and made the Exchange environment perform poorly. Increasing the value immediately showed improvement on performance.

Other issues with low values presented themselves when opening the Global Address book on Outlook Online mode. First it failed with the message that it hadn’t a connection to Exchange, while the user was working without (many) problems. A second attempt worked perfectly.
This is probably due to the different port used by Outlook. Although a wildcard rule was used for Outlook MAPI RPC the Kemp Load Balancer, the timeout value will be valid per port and is not based on source IP.

Kemp will address this in an upcoming update; the option will be available to kill not yet timed out connections whenever the real server has failed the Real Server check.

Having said that, I must confess that I didn’t have any real troubles with Outlook clients (in online mode) having to wait for the timeout to reconnect. When I disabled one Real Server CAS, after a while the clients just reconnected and worked perfectly within seconds to a few minutes.

So, the trade-off Kemp mentions could be a non-issue. At least in the case of Outlook 2010 RTM in Online Mode with a single VLM-100. But I can see no drawback from killing idle connections to a failed Real Server.

Note (edit 20/07/2011): Disabling a Real server is not the same as a failure. New connections to Real Server will not be set up and exsisting connections will drain and end after (default) 300 seconds.

In any case, Timeout values for MAPI connections (i.e. Outlook) should be high (as in hours).

Loadbalancer.org

In my last post I would contact Loadbalancer.org regarding their deployment guide. And their response came quickly and their updated Deployment Guide is already available.

Balancing Policy

They agreed with the Microsoft recommendation regarding using round robin instead of least weighed connection. They already have corrected their Exchange deployment guide, with a correct notation that it could take some time in order to let the load balancer evenly distribute the sessions.

Another interesting note: During my contact with them, they let me know that they are working with Microsoft to get their product Exchange load balancer qualified. I do not have any experience with them, but more certified choices are always a good thing.

Coyote point

In between this and the previous post I was made aware of the Coyote Point deployment Guide. Unfortunately they also had some discrepancies.

Balancing Policy

In their Exchange Deployment Guide v1.3 they mention on page 9 that round robin actually isn't recommended. They actually note that Microsoft recommends least connections as balancing policy. Obviously, Microsoft has since changed its stance on this.

Network deployment

On the subject of Source NAT, Default Gateway or Direct Server Return, they advise dg or static routes ( page10) Although Microsoft recommends Source NAT, the coyote point does not recommend this. The reason is because Exchange needs the client IP addresses. That however is false. You do lose client IP information, but in some or even most cases this is more preferable than a major change in your network architecture, IMHO.

Persistence

Pages 20/21 discuss persistence, they do not recommend specific values but mention examples. They are however in the range of several hours, where Microsoft almost always recommends 1 hour as a rule of thumb. But the explanation is correct.

They have responded to the issues I have raised and they are already in the progress of changing their recommendation to be in accordance with those from Microsoft. So if you use or are planning to use their products, keep this in mind.

Conclusion

Terminology, terminology, terminology.... Almost all companies have different definitions describing concepts and technology. This is quite unfortunate as it tends to elongate troubleshooting and support. I still would like to see that the Microsoft Load Balancing Qualification for Exchange contained the demand for explicitly defining terminology as used by Microsoft and by load balancing companies. Kemp technologies has stated that they are working to get the terminology in sync with Microsoft.

Also, I cannot find a complete Technet article describing the same recommendations that where made in the TechEd session EXL307.

I also would like to see some sort of requalification of some sorts, I would be surprised that some of the deployment guides were based on older recommendations from Microsoft who has since changed due to customer responses and support calls. This would make this qualification much more valuable and leads to better understanding of load balancers by admins and technical specialists and a higher quality of Exchange 2010 high available deployments.

And as a last recommendation: Don’t always believe what vendors say. Check, test and supply information to your load balancing vendor. I have had very good responses by all vendors I’ve contacted and they where very appreciative of my feedback.

Full disclosure

I have no legal or financial connection with the load balancing companies mentioned in this or the previous post. Kemp and possibly other vendors make unlimited trial load balancers available for personal testing purposes. These offers did not contain any strings attached.

Differences in Exchange Load Balancing recommendations by Microsoft and vendors

dmstork — Mon, 30 May 2011 09:40:00 GMT

*** See my followup here Exchange, Load balancers and recommendations:

At TechEd North America 2011 Andrew Ehrensing (Solution Architect form Microsoft) presented the session “Load Balancing with Exchange Server 2010” (EXL307) for which the video and slides can be found here. It was an excellent session, a lot of useful information even if you don’t have to load balance with Exchange per se.

For me it was an extra interesting session, as we just had implemented a load balancer with Exchange, which resulted in some issues (bad performance, third party monitoring software failed etc.). I already thought that our setup had some design flaws and was looking for configuration recommendations. Unfortunately, I couldn’t find any in-depth information on Microsoft TechNet, except for basic information.

I will not go into the session in detail in this post (perhaps in another), but the session basically had the recommendation that in an One-arm Load Balancing setup Source NAT (SNAT) should be used, even with clients or servers in the same subnet. Alternatively putting the Load Balancer as Default Gateway (LBDG) is also a valid option. Direct Server Return (DSR) wasn’t advised, as you have to add a loopback adapter with the Virtual IP (VIP) to all CAS Exchange Servers behind that specific Load Balancer. SNAT is least intrusive to the configuration of your Exchange Servers.

As I dug deeper in manuals and recommendations, I found discrepancies in the recommendations. Why is this important? It is a matter of getting adequate support. Do you get support from your vendor if you follow Microsoft recommendations, when they are different?

Kemp Technologies

In this specific case we were using a VLM-100 from Kemp Technologies and are quite happy with the value for money they supply. In previous situations we did not have any issues, but that were different environments.

Digging deeper in the Kemp Manual (PDF) I found that they primarily recommended a Load Balancer as Default Gateway with optional DSR configured. As we had poor experiences with this setup and in this case we would be required to implement LBDG with DSR, instead we opted for SNAT.

However, in the manual for basically the same scenario where Microsoft recommended SNAT, Kemp mentions SNAT does not make any sense and recommends LBDG and which in some cases implies DSR. See for references from the TechEd session video time code ~9:04 and ~19:00 and slides 7 and 15. Compare with page 12 of the Kemp Load Balancing manual.

I contacted Kemp for this difference in stance, pointed them to the TechEd video and slides and am currently awaiting a call from their Developer team. I will update this post when I receive new information.

During this mail exchange, they also pointed out that the static ports we configured could be changed to a wildcard Virtual Server (VS) on their load balancers. It would work better, was the motivation. Not very convincing IMHO, especially as Andrew in the same session pointed out that it is better to have static ports as this would putt less strain the load balancer (video time code 35m30s). This is also the configuration as Exchange MVP Henrick Walther has described in several blogposts, but specifically in this one.

Other Vendors

Curious how other vendors recommend their product in combination with Exchange, I stumbled upon the loadbalancer.org deployment guide for Exchange 2010 (PDF). In this guide they recommend “Least Connection” as distribution method as Microsoft recommends plain Round Robin (see page 12). Also with reason: Microsoft found issues when a CAS server is re-introduced in the CAS Array and as Load Balancing “node”, it could be overwhelmed when using any weighted, least connection or similar method. (video time code 34m09s). I will contact loadbalancer.org on this issue shortly.

I haven’t found any other discrepancies yet, but will update or follow up this post when an interesting difference comes to my attention. If you found one yourselves, feel free to mail or comment with specifics.

Conclusion

I did not check any of the other vendors and their recommendations. But this situation is very interesting as the products from Kemp Technologies are tested and qualified by Microsoft. Apparently that does not mean the recommendations have to correspond… I find that a bit disappointing and devalues the qualification IMHO.

Change in Exchange 2010 SP1 CAS static port configuration

dmstork — Thu, 28 Apr 2011 09:49:30 GMT

For those who are using Exchange 2010 DAG and a Network Load Balancer, note that there is a small difference between RTM and SP1. If you choose to use static ports for your Client Access Servers, for the Address Book Services you would edit the Microsoft.exchange.addressbook.service.exe.config located in: “C:\Program Files\Microsoft\Exchange Server\V14\Bin”

But with SP1 (directly installed or upgraded from RTM), the location has changed to the registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeRpc\ParametersSystem

You have to make a new REG_DWORD with the value “TCP/IP Port” and the port number you want to use.

The setting is not automatically changed when upgrading to SP1, the ports will always be dynamically again.

If you fail to change this, Outlook users fail to connect correctly to you Exchange environment and you could end up troubleshooting for some time (I know I have…).

A lot of documentation out there for configuring static ports, was made for RTM and isn’t adjusted for SP1. So keep this in mind.

Thanks to AlexisA for some checking and testing!

Public Folders and the DAG

dmstork — Tue, 23 Nov 2010 07:36:00 GMT

When realizing a High Available Exchange 2010 environment, you automatically going to use DAG (Database Availabilty Groups). It is a different approach with previous versions of Exchange, who leverage server redundancy. DAG supplies us with mailbox database redundancy.

When using a DAG the single point of entry for your clients and for all protocols (including MAPI RPC), is the Client Access Array (or CAS Array). Essentially a DNS record pointing to a Load Balancer.

A Hardware Load Balancer (HLB) is Microsoft's recommended way to enable redundancy and load balance this entry point (BTW: On TechEd 2010 Europe, the Microsoft Exchange team mentioned that it does not recommend it’s own Windows NLB solution anymore). So, if a database or server fails, a failover occurs and the load balancer(s) redirects the traffic to a remaining Client Access Server.

Wait, what? No Public Folder HA?

This solution gives us MAILBOX database redundancy. If you still need Public Folders (PF), you are in a pickle. Clients do not connect to the Public Folders Database via de CAS Array; they directly connect to the server with a Public Folder Database defined in the Default Public Folder attribute of a Mailbox Database.

You can have duplicate Public Folder Databases via PF Replication, but there is no automatic PF Server failover! This could be catastrophic for your SLA, as Outlook 2003 (supported in combination with Exchange 2010) requires Public Folders to operate correctly or operate at all! Luckily, Outlook 2007 & 2010 do operate, but it is possible that they don’t have Public Folder access during a failover.

So, what options do I have? Upgrading Outlook off course, which has a lot of other benefits. When your clients aren’t the bottleneck, move the Public Folder functionality to other products (SharePoint for instance).

Those pesky legacy apps…..

But what, if you are stuck with Outlook 2003? Or going to discontinue Public Folders, but just not at the moment? Or other reasons I didn’t think of?

You can script some form of HA into it, at least your users don’t have to wait for admin intervention. I have a customer who uses a DAG and has Outlook 2003, which cannot be upgraded quickly. In this case a coworker made a Exchange Management Shell script which checks every x minutes whether the Default Public Folder Attribute of a Mailbox Database is referring to the same server (i.e. the server on which the Active copy reside, is logically not in a failed state). This results that when a failover occurs, within defined minutes the attribute will be changed to a server which is still operational. It’s dirty, but it works.

I’ve added this script to this post for educational purposes, just scroll all the way down. Due to security reasons, the extention is now TXT. Just rename it to PS1. Furthermore this script currently only works with a two server DAG. This is published as is, use with caution and no guarantees are given etc. etc.. Use at your own risk.

But..

But before you get scared using this script, there is good news! During TechEd 2010 Europe Ross Smith (Microsoft Exchange team) announced that in Service Pack 1 Rollup Update 2 this issue will be addressed via an alternate server tag! The finer pointers are not clear yet and they didn’t mention a release date, but: there was much rejoicing.

PS1: Well, I already had prepared a draft blog post but decided to postpone publishing after TechEd…. None the less, this script could still be helpful for the time being.

PS2: thanks FrodoB for the script.

Dave Stork's IMHO : High Availability

Simplifying the OWA URL with Citrix Netscaler

Load balancing Exchange 2010 with Citrix Netscaler using Content Switching

Unable to change database path on Exchange 2010 in a DAG?

Exchange, Load balancers and recommendations

Kemp Technologies

SNAT

Static Ports

Idle Connection Timeout

Loadbalancer.org

Balancing Policy

Coyote point

Balancing Policy

Network deployment

Persistence

Conclusion

Full disclosure

Differences in Exchange Load Balancing recommendations by Microsoft and vendors

Kemp Technologies

Other Vendors

Conclusion

Further Reading

Change in Exchange 2010 SP1 CAS static port configuration

Further reading

Public Folders and the DAG

Wait, what? No Public Folder HA?

Those pesky legacy apps…..

But..

Further Reading