Identifying and Troubleshooting Firewall Access Issues

While working with server builds and configurations, one is frequently faced with a scenario where the system (as a client), needs to connect to a server at the other end and the process doesn’t work.  If one’s lucky then an intelligent message is returned by the interface, hinting about the problem but more often than not, the message says something on the lines of: “I failed for some unknown reason.  Please work out for yourself what went wrong.”

I’ve found that in a lot of cases, it’s due to some firewall blocking a particular kind of conversation.  Companies are quite security conscious these days (a good thing!) so there usually are multiple layers of protection in place, making such communication impossible unless it is specifically allowed at all layers.  So, is there an easy method of finding out if some firewall is in the way of you successfully making your connection?  Fortunately, I’ve been using a simple “out-of-the-box” technique for quite a few years now which has saved me loads of times over that period.  As I quite frequently find people who are still unaware of this easy technique, I thought I should share it here so that people can benefit from it.

Now, before I start, I am assuming that people will use this on Windows and *nix systems and we’re talking some TCP protocol here.  With that assumption in place, if I need to work out if a firewall is blocking my communication, I open a command-prompt (assuming a Windows system first) and type:

netstat -ano | findstr -i SYN_SENT

Initially, it shouldn’t return anything.  The next step is to run the process that was being attempted previously and simultaneously, run the command again.  If a firewall is blocking communication, an output will be returned, showing SYN_SENT status for the client:port/server:port pair (also with the ProcessID to help verify we’re looking at the right process).  Once this happens, you can see which protocol(s) is/are being blocked and then it could be passed on to the network administrator to decide upon and open if appropriate.  The equivalent command I use on *nix systems is:

netstat -ano | grep -i SYN_SENT

Explanation: This technique relies on the fact that every TCP connection starts with a three-way handshake.  The client sends a SYN packet and waits for an ACK packet from the server.  While it waits, the connection status is set to SYN_SENT.  If there is a firewall blocking the connection, the server won’t ever receive the SYN and therefore, will never reply.  This means, for the duration of the process i.e. until timeout, the client’s connection will remain in SYN_SENT status.  This is basically what we’re looking for when using the commands mentioned above.  As you can see, you have to be quick to run the command while the process is being run as SYN_SENT status disappears very quickly, once the connection attempt times out.

Bonus Tip: As all we’re doing above is running a command, piping the output into a string filter and searching for a particular string, the same technique can be used in various other scenarios where a lot of output is returned and you need to filter the output, to look for a particular string e.g. a target server or status.  For example, running:

dcdiag /v | findstr -i “test”

is much more efficient in displaying success and failure results than without the filter.

Hope this helps!

Exchange 2010 Installation issues (Part II)

You may find that your Exchange 2010 installation fails part way during the install, with the error message (taken from ExchangeSetupLog):

[Timestamp] [1] [ERROR] Cannot start service winrm on computer ‘.’.
[Timestamp] [1] [ERROR] The service cannot be started, either because it is disabled or because it has no enabled devices associated with it

Some organisations use customised hardened builds (like ours) and have services disabled until they are required.  In our case, WinRM is one such service.  The solution is pretty obvious in this case: Set the service to “Automatic” and re-run setup.  The service will be set to “Automatic (Delayed Start” by default, which is fine.  The installation will resume from where it was interrupted and will hopefully complete without any issues this time.

Another issue you might hit in rare cases is that while running Exchange 2010 setup, the install fails with the error message:

[Timestamp] [2] Process standard output: ERROR ( message:Cannot find APP object with identifier “Default Web Site/owa”. )

[Timestamp] [2] Process standard error:
[Timestamp] [2] [ERROR] Unexpected Error
[Timestamp] [2] [ERROR] Process execution failed with exit code 1168.

Some organisations harden IIS (Internet Information Services) installation in their Windows builds by changing default paths but also the default website name.  Exchange 2010 setup specifically looks for that name during the install process and fails if it can’t find it.

Solution: Go to IIS Manager.  Expand the tree until you get to the default website.  Right-click and rename the website to “Default Web Site” and restart IIS.  Confirm that the rename worked and IIS is running.  Re-run setup.  This time, it should continue from where it failed and should complete successfully.

Hope this helps!

Exchange 2010 Installation issues (Part I)

While installing Exchange 2010, you may find setup failing part way, complaining about having no more endpoint mappers (taken from ExchangeSetupLog):

[Timestamp] [1] 0. ErrorRecord: There are no more endpoints available from the endpoint mapper. (Exception from HRESULT: 0x800706D9)
[Timestamp] [1] 0. ErrorRecord: System.Runtime.InteropServices.COMException (0x800706D9): There are no more endpoints available from the endpoint mapper. (Exception from HRESULT: 0x800706D9)

Many organisations have their custom Windows builds which they generally automate, customise and harden.  If such builds have “Windows Firewall” and “Base Filtering Engine” services set to “Manual” or “Disabled”, it will very likely cause the failure mentioned above.  These services are required for a successful Exchange 2010 installation.

So, the solution is simple: Exit setup.  Set both services mentioned to “Automatic” and restart the server.  Restart setup.  Setup keeps track of installation progress all the way through so it will resume from where it was interrupted.  Just select the required roles again and follow general prompts to continue the install.

Additional Point: Enabling these services also prevent other problems later: Publishing Outlook Anywhere via TMG also requires these services to be enabled.  Otherwise, it also fails with a similar error message as mentioned above.

Hope this helps!

Exchange Search Indexing of Office 2007 documents

If you are running Exchange 2007 and find that content search for Office 2007 documents in Outlook doesn’t return proper results, you are not alone!  However, it’s not due to some shortcoming in Exchange.  It’s because you need to install and register IFilters with the Microsoft Windows Indexing Service on the server(s) running your Exchange services and then register them with Exchange, for content searching on those types of document to work.  Don’t feel bad if you missed this upon implementation as this process is not well-known and is not part of the standard install.  In fact, Exchange 2010 has introduced it as a pre-install task and even that enables content searching for the Office 2007 types and not Office 2010!  Generally, this issue is only discovered when one day you realise that you’re missing out on Office 2007 documents when searching your mailbox!

Fortunately, it’s not that difficult to implement.  Here are the main steps for it:

1. Download and install the Office 2007 Filter Pack  Make sure that you download the filter pack appropriate for your server architecture i.e. x86 or x64
2. Register the IFilters with Exchange using the instructions mentioned here  Tip: Use the VBScript to generate the reg file to run – it’s much easier and reliable!
3. Re-index the existing databases.  Either do it manually by stopping services, deleting the index and restarting the service or take the easy way out by running “.\ResetSearchIndex.ps1 <database name> ” in the “Scripts” folder under the Exchange installation path.

First two steps of the process are not service impacting.  The third step should be run with caution as rebuilding the index consumes a lot of CPU.  So, either do it out of hours or with degradation of service announced.  Obviously, impact will depend on resources in your environment and type/size of content within the mailboxes.  Fortunately, as the process can be run per database, impact can be distributed over time.  Consequence of not running it is that old Office 2007 documents already within mailboxes, won’t get indexed and therefore, people will get an inconsistent experience while searching.  All new content will get indexed and mailboxes will also get indexed if they move to another database after this process has been run.

This will enable content indexing of Office 2007 format documents for your Exchange environment (and also zip files!)

(Bonus Note: You could also enable content search for Adobe Acrobat format files in a similar way but it has always been unreliable.  Paul Robichaux has documented this in detail here.)