Today we are going to setup SSL offload for a VS.
1) First lets check that CS is enabled.
> sh ns feature | grep 'SSL Offloading'
9) SSL Offloading SSL OFF
>
> enable ns feature SSL
Done
>
> sh ns feature | grep 'SSL Offloading'
9) SSL Offloading SSL ON
>
2) Add IPs, VLANs and Bind them together.
add ns ip 192.168.20.10 255.255.255.0 -type SNIP
add ns ip 172.16.100.10 255.255.255.0 -type SNIP
add vlan 20
add vlan 100
bind vlan 20 -ifnum 1/1
bind vlan 20 -ipaddress 192.168.20.10 255.255.255.0
bind vlan 100 -ifnum 1/2
bind vlan 100 -ipaddress 172.16.100.10 255.255.255.0
> sh ns ip
Ipaddress Traffic Domain Type Mode Arp Icmp Vserver State
--------- -------------- ---- ---- --- ---- ------- ------
1) 192.168.10.10 0 NetScaler IP Active Enabled Enabled NA Enabled
2) 192.168.20.10 0 SNIP Active Enabled Enabled NA Enabled
3) 172.16.100.10 0 SNIP Active Enabled Enabled NA Enabled
Done
>
> sh vlan
1) VLAN ID: 1
Link-local IPv6 addr: fe80::20c:29ff:feed:7bb5/64
Interfaces : 0/1 LO/1
2) VLAN ID: 20 VLAN Alias Name:
Interfaces : 1/1
IPs :
192.168.20.10 Mask: 255.255.255.0
3) VLAN ID: 100 VLAN Alias Name:
Interfaces : 1/2
IPs :
172.16.100.10 Mask: 255.255.255.0
Done
>
3) Add the real web servers which will be the pool for the VS.
add server pool1_srv1 172.16.100.1
add server pool1_srv2 172.16.100.2
add server pool1_srv3 172.16.100.3
add server pool1_srv4 172.16.100.4
> sh servers
1) Name: pool1_srv1 State:ENABLED
IPAddress: 172.16.100.1
2) Name: pool1_srv2 State:ENABLED
IPAddress: 172.16.100.2
3) Name: pool1_srv3 State:ENABLED
IPAddress: 172.16.100.3
4) Name: pool1_srv4 State:ENABLED
IPAddress: 172.16.100.4
Done
>
4) Create the service group, vserver, monitor and bind them and the real servers to the service group.
We will bind an SSL VS to the pool of HTTP servers we created above.
We will eventually terminate (When everything is working), the client SSL connection on the Netscaler and then forward/load balance the client connection to the pool of HTTP servers in clear text, thereby reducing the load on the servers as they don't have to decrypt the SSL.
add servicegroup SG_pool1 HTTP
add lb vserver VS_SSL_web SSL 192.168.20.100 443 -lbmethod ROUNDROBIN
add lb monitor MON_pool1 HTTP
bind lb vserver VS_SSL_web SG_pool1
bind servicegroup SG_pool1 pool1_srv1 80
bind servicegroup SG_pool1 pool1_srv2 80
bind servicegroup SG_pool1 pool1_srv3 80
bind servicegroup SG_pool1 pool1_srv4 80
bind servicegroup SG_pool1 -monitorName MON_pool1
Note the output below. The state is down as we haven't bound a cert to the VS yet.
> sh lb vserver VS_SSL_web
VS_SSL_web (192.168.20.100:443) - SSL Type: ADDRESS
State: DOWN[Certkey not bound]
Last state change was at Sat Apr 24 07:19:44 2021
Time since last state change: 0 days, 00:01:07.690
Effective State: DOWN
Client Idle Timeout: 180 sec
Down state flush: ENABLED
Disable Primary Vserver On Down : DISABLED
Appflow logging: ENABLED
No. of Bound Services : 4 (Total) 4 (Active)
Configured Method: ROUNDROBIN BackupMethod: NONE
Mode: IP
Persistence: NONE
Vserver IP and Port insertion: OFF
Push: DISABLED Push VServer:
Push Multi Clients: NO
Push Label Rule: none
L2Conn: OFF
Skip Persistency: None
Listen Policy: NONE
IcmpResponse: PASSIVE
RHIstate: PASSIVE
New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
Mac mode Retain Vlan: DISABLED
DBS_LB: DISABLED
Process Local: DISABLED
Traffic Domain: 0
TROFS Persistence honored: ENABLED
Retain Connections on Cluster: NO
Bound Service Groups:
1) Group Name: SG_pool1
1) SG_pool1 (172.16.100.1: 80) - HTTP State: UP Weight: 1
2) SG_pool1 (172.16.100.2: 80) - HTTP State: UP Weight: 1
3) SG_pool1 (172.16.100.3: 80) - HTTP State: UP Weight: 1
4) SG_pool1 (172.16.100.4: 80) - HTTP State: UP Weight: 1
Done
>
The back end is up.
> sh servicegroup SG_pool1
SG_pool1 - HTTP
State: ENABLED Effective State: UP Monitor Threshold : 0
Max Conn: 0 Max Req: 0 Max Bandwidth: 0 kbits
Use Source IP: NO
Client Keepalive(CKA): NO
TCP Buffering(TCPB): NO
HTTP Compression(CMP): NO
Idle timeout: Client: 180 sec Server: 360 sec
Client IP: DISABLED
Cacheable: NO
SC: OFF
SP: OFF
Down state flush: ENABLED
Monitor Connection Close : NONE
Appflow logging: ENABLED
ContentInspection profile name: ???
Process Local: DISABLED
Traffic Domain: 0
1) Monitor Name: MON_pool1 State: ENABLED Weight: 1 Passive: 0
1) 172.16.100.1:80 State: UP Server Name: pool1_srv1 Server ID: None Weight: 1
Last state change was at Sat Apr 24 07:20:13 2021
Time since last state change: 0 days, 00:04:19.60
Monitor Name: MON_pool1 State: UP Passive: 0
Probes: 52 Failed [Total: 0 Current: 0]
Last response: Success - HTTP response code 200 received.
Response Time: 2 millisec
2) 172.16.100.2:80 State: UP Server Name: pool1_srv2 Server ID: None Weight: 1
Last state change was at Sat Apr 24 07:20:13 2021
Time since last state change: 0 days, 00:04:19.60
Monitor Name: MON_pool1 State: UP Passive: 0
Probes: 52 Failed [Total: 0 Current: 0]
Last response: Success - HTTP response code 200 received.
Response Time: 4 millisec
3) 172.16.100.3:80 State: UP Server Name: pool1_srv3 Server ID: None Weight: 1
Last state change was at Sat Apr 24 07:20:13 2021
Time since last state change: 0 days, 00:04:19.40
Monitor Name: MON_pool1 State: UP Passive: 0
Probes: 52 Failed [Total: 0 Current: 0]
Last response: Success - HTTP response code 200 received.
Response Time: 4 millisec
4) 172.16.100.4:80 State: UP Server Name: pool1_srv4 Server ID: None Weight: 1
Last state change was at Sat Apr 24 07:20:13 2021
Time since last state change: 0 days, 00:04:19.30
Monitor Name: MON_pool1 State: UP Passive: 0
Probes: 52 Failed [Total: 0 Current: 0]
Last response: Success - HTTP response code 200 received.
Response Time: 4 millisec
Done
>
5) Create a CSR to generate a certificate to bind to our SSL VS.
Go to the Netscaler shell and then to /nsconfig/ssl.
> shell
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
root@ns# cd /nsconfig/ssl
Create the private key for your cert.
root@ns# openssl genrsa -out vs_ssl_web.key 2048
Generating RSA private key, 2048 bit long modulus
.......................................+++++
.........................................................................................................................................+++++
e is 65537 (0x10001)
root@ns# ls -l vs_ssl_web.key
-rw-r--r-- 1 root wheel 1679 Apr 25 02:54 vs_ssl_web.key
root@ns#
Create and copy to /nsconfig/ssl on the Netscaler, a cfg file "vs_ssl_web.cfg" for the CSR.
This is the contents of the .cfg file.
[req]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
[req_distinguished_name]
countryName = AU
countryName_default = AU
stateOrProvinceName = NSW
stateOrProvinceName_default = NSW
localityName = Sydney
localityName_default = Sydney
organizationalUnitName = IT
organizationalUnitName_default = IT
organizationName = Netscaler SSL Lab
organizationName_default = Netscaler SSl Lab
commonName = *.netscalerssllab.com.au
commonName_default = *.netscalerssllab.com.au
[req_ext]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.netscalerssllab.com.au
DNS.2 = netscalerssllab.com.au
root@ns# ls -l vs_ssl_web*
-rw-r--r-- 1 root wheel 721 Apr 25 03:02 vs_ssl_web.cfg
-rw-r--r-- 1 root wheel 1679 Apr 25 02:54 vs_ssl_web.key
root@ns#
Create the CSR by combining the private key and the config file.
root@ns# openssl req -new -key vs_ssl_web.key -config vs_ssl_web.cfg -batch -nodes -out vs_ssl_web.csr -sha512
root@ns#
root@ns# ls -l vs_ssl_web*
-rw-r--r-- 1 root wheel 721 Apr 25 03:02 vs_ssl_web.cfg
-rw-r--r-- 1 root wheel 1180 Apr 25 03:07 vs_ssl_web.csr
-rw-r--r-- 1 root wheel 1679 Apr 25 02:54 vs_ssl_web.key
root@ns#
6) Create the cert.
Send our CSR to the CA for cert creation.
In our case I will copy to my local CA.
Full credit to this site for showing what to do to create a fully functional internal CA.
https://jamielinux.com/docs/openssl-certificate-authority/introduction.html
$ scp vs_ssl_web.csr root@192.168.0.31:/root/ca
root@192.168.0.31's password:
vs_ssl_web.csr 100% 1180 461.1KB/s 00:00
Examine the CSR on the CA.
[root@localhost ca]# openssl req -text -in vs_ssl_web.csr
Certificate Request:
Data:
Version: 1 (0x0)
Subject: C = AU, ST = NSW, L = Sydney, OU = IT, O = Netscaler SSl Lab, CN = *.netscalerssllab.com.au
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:e6:6c:14:36:6a:61:b9:72:2d:c6:8d:e2:48:eb:
: :
c8:1b:b0:8b:1d:57:53:03:54:eb:3b:d9:5b:1b:5b:
ac:b3
Exponent: 65537 (0x10001)
Attributes:
Requested Extensions:
X509v3 Key Usage:
Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:*.netscalerssllab.com.au, DNS:netscalerssllab.com.au
Signature Algorithm: sha512WithRSAEncryption
d0:a8:7a:62:6e:98:85:3d:21:fa:17:28:b9:da:af:57:71:b5:
: :
44:35:0e:0c:4e:01:35:1b:6c:83:ac:8b:71:97:e2:fd:cb:83:
55:42:88:ed
-----BEGIN CERTIFICATE REQUEST-----
MIIDLTCCAhUCAQAweDELMAkGA1UEBhMCQVUxDDAKBgNVBAgMA05TVzEPMA0GA1UE
:
dflAbLOnCkJHWlJwIhRDlmRoOtYmLWPbB5Za5AfwbDtESanxwoOaXQJA62Gf2YO9
MiY0GKZXKhQq80NxncRXB8zw+aAqm0WVnyb3RDUODE4BNRtsg6yLcZfi/cuDVUKI
7Q==
-----END CERTIFICATE REQUEST-----
[root@localhost ca]#
Create the cert with the intermediate CA to sign the CSR.
The server_cert option tells OpenSSL to create a server certificate.
The other option is to say usr_cert to generate a client certificate which is not appropriate here.
[root@localhost ca]# openssl ca -config intermediate/openssl.cnf -extensions server_cert -days 750 -notext -md sha256 -in vs_ssl_web.csr -out intermediate/certs/vs_ssl_web.crt
Using configuration from intermediate/openssl.cnf
Enter pass phrase for /root/ca/intermediate/private/intermediate.key.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 4097 (0x1001)
Validity
Not Before: Apr 25 07:26:21 2021 GMT
Not After : May 15 07:26:21 2023 GMT
Subject:
countryName = AU
stateOrProvinceName = NSW
localityName = Sydney
organizationName = Netscaler SSl Lab
organizationalUnitName = IT
commonName = *.netscalerssllab.com.au
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
3C:61:52:F5:DE:DE:D0:93:AF:10:52:9B:5D:F8:5E:AE:0D:ED:4D:C4
X509v3 Authority Key Identifier:
keyid:0B:7F:84:21:7C:C4:A4:1D:5F:62:74:33:70:2B:5B:AE:8E:03:A2:6D
DirName:/C=AU/ST=NSW/L=SYDNEY/O=NETSCALERNOTES/OU=IT/CN=NETSCALERNOTES CA
serial:10:00
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:*.netscalerssllab.com.au, DNS:netscalerssllab.com.au
Certificate is to be certified until May 15 07:26:21 2023 GMT (750 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
[root@localhost ca]#
Check that our index file has a line referring to the new cert we just created.
[root@localhost ca]# nder your CA_default section in your openssl.cnf.
V 230515072621Z 1001 unknown /C=AU/ST=NSW/L=Sydney/O=Netscaler SSl Lab/OU=IT/CN=*.netscalerssllab.com.au
[root@localhost ca]#
Verfiy the cert and note that it has been issued by the Internediate CA and that it is a server type cert.
Make sure we have SANs present in the cert.
In my CA I had to add "copy_extensions = copy" under the CA_default section in openssl.cnf.
There are other ways to achieve the same outcome such as with an ext file.
root@localhost ca]# openssl x509 -noout -text -in intermediate/certs/vs_ssl_web.crt
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4097 (0x1001)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = AU, ST = NSW, O = NETSCALERNOTES, OU = IT, CN = NETSCALERNOTES Intermediate CA
Validity
Not Before: Apr 25 07:26:21 2021 GMT
Not After : May 15 07:26:21 2023 GMT
Subject: C = AU, ST = NSW, L = Sydney, O = Netscaler SSl Lab, OU = IT, CN = *.netscalerssllab.com.au
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:e6:6c:14:36:6a:61:b9:72:2d:c6:8d:e2:48:eb:
:
26:26:4c:e5:2c:5d:56:48:22:44:bf:90:c4:7a:bb:
c8:1b:b0:8b:1d:57:53:03:54:eb:3b:d9:5b:1b:5b:
ac:b3
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Cert Type:
SSL Server
Netscape Comment:
OpenSSL Generated Server Certificate
X509v3 Subject Key Identifier:
3C:61:52:F5:DE:DE:D0:93:AF:10:52:9B:5D:F8:5E:AE
X509v3 Authority Key Identifier:
keyid:0B:7F:84:21:7C:C4:A4:1D:5F:62:74:33:70:2B
DirName:/C=AU/ST=NSW/L=SYDNEY/O=NETSCALERNOTES/OU=IT/CN=NETSCALERNOTES CA
serial:10:00
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:*.netscalerssllab.com.au, DNS:netscalerssllab.com.au
Signature Algorithm: sha256WithRSAEncryption
45:61:6b:b9:54:60:e9:e6:66:58:57:71:48:10:ea:9e:69:48:
:
99:e0:b6:4e:92:d1:76:5a:7a:05:7e:27:e3:cc:26:0d:e1:ba:
69:01:12:8c:86:d5:67:5e
[root@localhost ca]#
Verify on our CA that our new cert has a valid chain of trust against the root/intermediate cert chain.
[root@localhost ca]# openssl verify -CAfile intermediate/certs/ca-chain-root-intermediate.cert.pem intermediate/certs/vs_ssl_web.crt
intermediate/certs/vs_ssl_web.crt: OK
[root@localhost ca]#
Grab the intermediate cert and our new cert from the CA and then copy to the Netscaler.
Grabbing...
user@TOWER ~
$ scp root@192.168.0.31:/root/ca/intermediate/certs/vs_ssl_web.crt vs_ssl_web.crt
root@192.168.0.31's password:
vs_ssl_web.crt 100% 2000 782.5KB/s 00:00
user@TOWER ~
$ scp root@192.168.0.31:/root/ca/intermediate/certs/intermediate.cert.pem intermediate.cert.pem
root@192.168.0.31's password:
intermediate.cert.pem 100% 2045 800.4KB/s 00:00
Post copy to NS.
root@ns# ls -l vs_ssl_web*
-rw-r--r-- 1 root wheel 721 Apr 25 03:02 vs_ssl_web.cfg
-rw-r--r-- 1 root wheel 2000 Apr 25 05:28 vs_ssl_web.crt
-rw-r--r-- 1 root wheel 1180 Apr 25 03:07 vs_ssl_web.csr
-rw-r--r-- 1 root wheel 1679 Apr 25 02:54 vs_ssl_web.key
root@ns#
root@ns# ls -l *.pem
-rw-r--r-- 1 root wheel 2045 Apr 25 05:29 intermediate.cert.pem
root@ns#
7) Now the the certs are copied to the Netscaler, add them into the config.
Add the Intermediate cert - we will reference it in the config as intermediate cert.
> add ssl certKey intermediate_cert -cert intermediate.cert.pem
Done
>
Add the cert for the SSL offloading on the VS. Associate it with it's private key. We will reference it in the config as vs_ssl_web_cert.
> add ssl certKey vs_ssl_web_cert -cert vs_ssl_web.crt -key vs_ssl_web.key
Done
>
8) Link the Intermediate cert and our new cert on Netscaler to create the cert chain on the Netscaler.
> link ssl certKey vs_ssl_web_cert intermediate_cert
Done
>
> sh ssl certLink -summary
--------------------------------------------------------------------
CertName CA CertName
--------------------------------------------------------------------
1 vs_ssl_web_cert intermediate_cert
Done
>
> sh ssl certChain vs_ssl_web_cert
Certificate chain details of certificate name vs_ssl_web_cert are:
1) Certificate Name: intermediate_cert linked; not a root certificate
2) Certificate Name: /C=AU/ST=NSW/L=SYDNEY/O=NETSCALERNOTES/OU=IT/CN=NETSCALERNOTES CA
Action: Add certificate with this subject name
Done
>
> sh ssl certChain vs_ssl_web_cert -summary
--------------------------------------------------------------------
CertName CA CertName
--------------------------------------------------------------------
1 vs_ssl_web_cert
Done
>
9) Bind the cert to the VS.
> bind ssl vserver VS_SSL_web -certkeyName vs_ssl_web_cert
Done
>
10) Check the state of the VS
Note the state is now UP as we have applied the cert to the VS.
> sh lb vserver
1) VS_SSL_web (192.168.20.100:443) - SSL Type: ADDRESS
State: UP
Last state change was at Sun Apr 25 05:57:45 2021
Time since last state change: 0 days, 00:00:43.110
Effective State: UP
Client Idle Timeout: 180 sec
Down state flush: ENABLED
Disable Primary Vserver On Down : DISABLED
Appflow logging: ENABLED
No. of Bound Services : 4 (Total) 4 (Active)
Configured Method: ROUNDROBIN BackupMethod: NONE
Mode: IP
Persistence: NONE
Vserver IP and Port insertion: OFF
Push: DISABLED Push VServer:
Push Multi Clients: NO
Push Label Rule: none
L2Conn: OFF
Skip Persistency: None
Listen Policy: NONE
IcmpResponse: PASSIVE
RHIstate: PASSIVE
New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
Mac mode Retain Vlan: DISABLED
DBS_LB: DISABLED
Process Local: DISABLED
Traffic Domain: 0
TROFS Persistence honored: ENABLED
Retain Connections on Cluster: NO
Done
>
11) Test from the client browser before importing the root cert into windows.
The cert for our site is seen as invalid as we dont have a full cert chain.
Due to the cert linking on the Netscaler, the client can see the intermediate cert that signed the VS cert but it has no knowledge of a trusted root cert that signed the intermediate cert.
12) Test from the client browser after importing the root cert into windows.
We could have also built the full chain on the Netscaler but it is a more common scenario to install a trusted cert into a client and that can be easily done on a mass scale via Group Policy.
Leave the Netscaler chain to cert + intermediate.
Generate pks12 file on the CA (cert + key) for importing into windows.
-r--------. 1 root root 3326 Apr 23 22:31 ca.key.pem
[root@localhost ca]# openssl pkcs12 -export -in certs/ca.cert.pem -inkey private/ca.key.pem -out ca.p12
Enter pass phrase for private/ca.key.pem:
Enter Export Password:
Verifying - Enter Export Password:
[root@localhost ca]#
Import into windows via MMC under "Console Root > Trusted Root Certification Authorities > Certificates" and browse again.
TRUSTED SITE!
The client trusting the root cert now and seeing the server cert and intermediate cert from the Netscaler can now trust the site as it has knowledge of the full cert chain.
Of course importing a pkcs12 file does create one big security issue, namely that we have unnecessarily imported the private key along with the needed public key.
Better just to get the root cert, ca.cert.pem onto windows then give it a .crt extension to become ca.cert.pem.crt. This way windows will see it as a cert file now and you can just double click on the cert and install it. Doing this you will just install the cert with its public key.
13) Check SSL stats.
Note the "back end" stats are all zero proving we are doing SSL offload - terminating and processing SSL connections on the front end only and passing unencrypted http to the back end servers.
> sh ssl stats
SSL Summary
# SSL cards present 0
# SSL cards UP 0
SSL engine status 1
SSL sessions (Rate) 0
System
Transactions Rate (/s) Total
SSL transactions 15 497
SSLv2 transactions 0 0
SSLv3 transactions 0 0
TLSv1 transactions 0 0
TLSv1.1 transactions 0 0
TLSv1.2 transactions 15 497
TLSv1.3 transactions 0 0
Front End
Sessions Rate (/s) Total
SSL sessions 0 336
SSLv2 sessions 0 0
SSLv3 sessions 0 0
TLSv1 sessions 0 0
TLSv1.1 sessions 0 0
TLSv1.2 sessions 0 336
TLSv1.3 sessions 0 0
New SSL sessions 0 384
SSL session misses 0 6
SSL session hits 15 226
Back End
Sessions Rate (/s) Total
SSL sessions 0 0
SSLv3 sessions 0 0
TLSv1 sessions 0 0
TLSv1.1 sessions 0 0
TLSv1.2 sessions 0 0
Session multiplex attempts 0 0
Session multiplex successes 0 0
Session multiplex failures 0 0
Encryption/Decryption statistics
Crypto Operation Rate (bytes/s) Total Bytes
Bytes encrypted 434786 12850960
Bytes decrypted 7335 228400
Done
>
> sh ver
NetScaler NS12.1: Build 61.19.nc, Date: Mar 8 2021, 09:45:59 (64-bit)
Done
>
