SSL certificates exist in several formats. The most common formats that you encounter when working with NetScaler are PEM and PFX. The NetScaler supports binding PEM certificates to virtual servers, and it is a common format among open source systems, but you will also often stumble on PFX, especially when moving from IIS. In the most recent version, version 11.0, you can directly import PFX certificates without having to convert them, but it is nevertheless handy to know how to convert them using OpenSSL (included on the NetScaler) if you have used an use an earlier version:

Converting PFX server certificate to PEM using OpenSSL

You can also examine PEM certificates on the NetScaler without having to export them to your PC first. Let's quickly discuss the various sections of a certificate, as this becomes important when troubleshooting:

PEM certificate with encrypted private key

In the screenshot, we see an encrypted private key followed by the certificate. The Proc-Type tells us the key is encrypted, and the DEK-Info tells us the encryption method used. The actual key then follows, demarcated by a BEGIN and END.

Note that, with the PEM format, the certificate and key can exist in the same file, but they don't necessarily have to. Also, the password protection that results in the encrypted form is optional. By contrast, in PFX, they always are in the same file and the file is always protected by a password.

You can run the same conversion with the –nodes option, in which case you will get a PEM file that doesn't have the encryption:

The following are the steps for SSL handshake:

Now, let's look at the version of the handshake that happens when the SSL session reuse is happening. Here, you see that the client already comes with a session ID from a previous session, and in the rather short SSL handshake that happens, the server certificate is not exchanged and no validation is done, and some computationally expensive steps are avoided by doing so:

Session reuse is a great SSL performance technique. This is why it is enabled by default.

While troubleshooting SSL/TLS issues, you will find yourself often needing to look beyond the handshake and at the HTTP, or at other layer 7 level information. You can again use Wireshark for this purpose. To decrypt the trace:

There is also a password field that needs to be filled in if the key is password protected. As an alternative, you can also use OpenSSL to decrypt the key even before getting to this step. The command is:

openssl rsa -in <encrypted key> -out < decrypted output filename>

The quickest way to tell if the decryption worked is by using the filter ssl && http, as you can see the decrypted HTTP is in green in the following screenshot:

Note that apart from an incomplete handshake being captured (due to a reused SSL session), a second case where Wireshark will fail to decrypt the trace is when protocols such as Diffie Hellman are used in the key exchange. In such cases, instead of the server's public and private keys, randomly generated keys are used that Wireshark doesn't have visibility into and thus the decryption fails.

If you are having trouble with decryption, look for clues in the SSL Debug file that you can choose under Edit | Preferences | SSL:

You will, on occasion, need to share the key with Citrix support for troubleshooting issues involving SSL. Citrix has strict measures in place around how this information is handled to avoid risk. However, sharing the key might not bode well with your security policy; luckily, it's easy to resolve this situation without adding to the risk by decrypting and exporting the session keys to share with Citrix.

With these session keys, Citrix (or any third-party you share the session keys with) can only decrypt a specific conversation. To do this using Wireshark first decrypt the trace as we've discussed. Then, navigate to File | Export SSL Session Keys.

The result will be a text file with the RSA Session ID and the Master Key that you can share with anyone that needs to be able to decrypt the session. The way this file will be used for decryption is by going to the SSL Protocol Preferences section of Wireshark and specifying it as the (Pre)-Master-Secret log filename.

Now with the key concepts out of the way, let's look at some ways to approach troubleshooting SSL issues.

Having verified that the HTTP version of the VIP works correctly, let's look at some potential failure areas:

Card failures are fortunately very rare. But in case one happens, you will see a number of telltale signs. A card failure will:

SSL is such a vital component for customers, especially those bound by compliance laws that, keeping on top of the issues, is a constant effort and requires regular handling, what with new vulnerabilities being reported on a regular basis. Consider the following best practices to ensure you are well protected: