CUC (6) CUCM (26) Jabber (6) Python (2) Routing (3) Solarwinds Orion NPM (4) switching (1) Video (6) voice (2)

Thursday, 21 June 2018

ASA AnyConnect VPN 2 Factor authentication configuration

RADIUS and Symantec VIP.

I will use screenshots of ASDM, and at the end I will add the required CLI commands.  the diagram below show a diagram of the steps the FW goes through when using 2FA authentication:

Fig.1 -  2FA steps ASA

As you can see in Fig. 1 the first step in the authentication process is to connect to ISE which then connects to AD, you could configure it to go to AD directly.


Any Connect Connection Profile

  • enable Cisco Anyconnect acces on the outside interface.
    • choose to "Bypass interface access lists for inbound VPN sessions
Fig.2. - remote VPN connection profile

Now drill into the connection profile itself.  (Fig.2)

Fig -2 Main Anyconnect connection profile window
Fig.2. shows that the authentication is set to AAA, which is offloaded to ISE using RADIUS, which authenticates, on (very likely) AD credentials. I will address the ISE configuration part of this in a separate post. So pretty much the first factor is the RADIUS authentication.

Because 2FA, uses two authentication sources, as the name suggest, you will also need to add a secondary authentication method, this time I have used a server group called VIP (using Symantec's VIP service).

If you are using Symantec or any 3rd party 2FA provider, such as through MS Azure, then you can decide to point your secondary AAA server to either an on premise 2FA gateway or a cloud thingy. Either way, from an ASA point of view you will need a different IP address. Typically, you will connect on ports tcp/1812 for authentication and tcp/1813 for accounting.

Group Policy

Configure a group policy to assign to your connection profile. I prefer to create a separate group policy for each profile, even though I would inherit most of the parameters from the default policy. This makes it easier to make changes that do not impact other connection profiles using the same default values. Assign this group policy to the connection profile in the step above. If you are going to use the Anyconnect client. You would need to select SSL VPN client.

Anyconnect prompt customisation

you might decide to change the anyconnect login prompt to state  that the second authentication of a 2FA security code is required. for instance:

To do this, you will need to customize the client's language file:

Config > Remote Access VPN > Network (Client) Access > AnyConnect Customization/Localization > GUI Text and Messages. Edit the language file:

msgid "Second Password"
msgstr "VIP Access Security Code:"


group-policy AnyConnect_2FA attributes
 vpn-simultaneous-logins 2
 vpn-tunnel-protocol ssl-client 
  anyconnect profiles value Test_Client_Profile type user

 enable Internet
 anyconnect image disk0:/anyconnect-win-4.5.01044-webdeploy-k9.pkg 1
 anyconnect image disk0:/anyconnect-macos-4.5.01044-webdeploy-k9.pkg 2
 anyconnect profiles Test_Client_Profile disk0:/test_client_profile.xml
 anyconnect enable
 tunnel-group-list enable

tunnel-group 2FA_AnyConnect general-attributes
 address-pool Pool1
 authentication-server-group ISERADIUS
 secondary-authentication-server-group VIP use-primary-username
 default-group-policy 2FA_SSL

if you want to use alias for the vpn connection profile:

tunnel-group 2FA_AnyConnect webvpn-attributes
 group-alias Test_2FA enable

aaa-server VIPRADIUS (Inside) host
 timeout 60
 key *****
 authentication-port 1812
 accounting-port 1813
aaa-server VIPRADIUS (Inside) host
 timeout 60
 key *****
 authentication-port 1812

 accounting-port 1813


Monday, 18 June 2018

F5 Application Policy Manager Authentication using AD

Apart from a F5 BIG-IP being an awesome load balancer with all sorts of VIPs and SSL offloading capabilities, it is also capable of providing VPN and portal capabilities. These capabilities are configured under the Access Policy tab of your F5 and become available once the correct license is installed. To find out your licensed entitlements, go to SYSTEM > License.

Ok, so let's assume you want to give external users (or roaming users) access to some sort of web portal, that is facing the internet (for instance a website for your employees to enter their time sheets, or some ordering tool for your sales force, could be anything).

So the first thing you will need to do is set up your authentication, for this example we will use Active Directory as our source for authentication.

Authentication using Active Directory:

In your access policy tab, go to AAA servers > Active directory (please note that you can also chooose LDAP, RADIUS, TACACS, SAML and a few more).

Essentially, you will now add 1 or more AD servers to a pool that the access profile (next step) can point to. The F5 will use a service account to allow credentials checks on AD. This of course will need to be pre-provisioned in AD.

Access policy configuration.

Below is a screenshot of an access policy called SSO_portal.

Figure 1, access policy configuration

I have added two AAA active directory pools: aaa_ad and aaa_r.

Apart from the AAA servers, all the smarts lie in the actual policy itself. In figure 1, this policy can be view through the visual policy editor. So lets have a look at the actual policy

Figure 2. Visual policy editor
As you can see in the policy above, the first check is that of the IP address of the party that is trying to log in. Based on that WebAuth, or Kerberos is used and Webauth for all others (Fallback). After this the userID is checked, to see if it is part of a certain predefined AD group. If it is, the authentication is successful and the authentication is ALLOWED (figure 2 , right hand side)

Fig.3 SSO authentication domain

Apply the Access policy to the LTM VIP

of course the access policy will need to be applied to a VIP. because this will actually land the traffic on the F5 platform. I like to look at a VIP as some sort of socket that you can plug into the F5 so it can apply smarts.
Below is  a screenshot of part of the Virtual Server configuration in LTM, here the previously configured Access Profile "SSO_Portal" is assigned to the VIP

Access Policy Reporting.

In order to troubleshoot user not being able to log into your application for some reason, the access policy reporting page on the BIG-IP is a good resource for trying to figure out why. Son on your BIG-IP got to Access Policy > Reports >View Reports. As can be seen  in figure 5, my login with user ID "mink" failed

Figure 5 -Denied access
Please note that these reports are run per partition, so run the report for the partition, your VIP is running on. To view the details on how the access policy flow is applied against the login attempt; click on session ID details.

Thursday, 7 June 2018

Troubleshoot TLS using wireshark

Because you cant be a good network engineer if you do not know how to drive wireshark, i decided to put a post up on how to capture and analyse TLS negotiation.  For this purposes, I used  Before you do the capture, its good to do an nslookup for the domain so you can filter out relevant traffic (yes wireshark calls it 'ssl'). But really you can just use the public IP address on your BIG-IP F5 is that is what you want to analyse.  so hit your website, using https. Once pulled up, stop the capture.

Figure 1 - filter on ssl
So put a display filter in using 'ssl' as the syntax (sure if you are real smart you could have already used ssl as the capture filter).You might now have multiple TLS sessions t multiple destination, so the output needs to be more granular even. Choose the IP address you are interested in in this case (which is the IP address used by some wanky cloud provider that cnn uses). so let us follow a particular stream:

Fig.2 -Follow SSL Stream

Following the ssl stream will give you a clear picture of the whole TLS hand shake and exchange of public keys, cert up to the exchange of symmetric key used for further encryption (through for instance AES 256).

the first thing that always happens when connecting using https is the client (your browser) announcing its cipher capabilities, it basically tells the server you are connecting to what security algorithms you are capable of. This is shown in the screen shot below

Fig.3 Client Hello

This is a client Hello, using Chrome v 67, as you can see only Elliptic Curve Diffie Helman predominately. It is also interesting to see that the client attempts TLSv1.2. In the opposite direction a server hello is sent to the client:

Fig. 4 Server Hello
Basically the server has decided it will use the securest possible cipher set. in this case ECDH, AES128 and Sha256. This is why it is important to define cipher suites on your webserver/F5 so security cant be forced by the client into using lower security ciphers such as DES or 3DES. by simply removing it from the capability of the server.

So the next thing that would happen, is the server issuing its certificate. As can be seen in figure 5

Fig. 5 -server issuing certificate to client

In Figure 5 you can see the common name of the cert. and the CA (global Sign) and a stack of subject alternative names. After that the cert finalizes the Hello phase with a Hello done.  in short:

Figure 6-  Hello summary

the next thing that happens is the exchange of "change cipher specs". essentially this is where the negotiation of symmetric session encryption keys takes place that will be used for further encryption of traffic (for instance by means of AES). as can be seen in Figure 6, this happens straight after the server Hello done.

Ofcourse the symmetric encryption keys cannot be exchanged without being encrypted themselves. so in order to hide them from a man in the middle attach, Diffie Helman comes into play. So If i drill into the client key exchange packet, I can see in our example that Elliptic Curve Diffie Helman is being used:

Fig. 7- ECDHE Elliptic curve Diffie Helman

As you can see, all of this TLS exchange is aimed to provide confidentiality and integrity. The actuall exchange of the X509 cert. provides the authentication portion of this.


Monday, 19 March 2018

IPSEC Site to site VPN 101

Little post on IPSEC, also a good recap for myself. I will outline the components used in IPSEC when setting up a site  to site VPN for instance. I dont really want to drill down too much into the details of each protocol. This post is more around the components needed and steps/tips to trouble shoot.

IPSEC VPN is described in RFC 4301.

IPSEC is not a protocol, its is more similar to an architecture, that contains a number of protocols (mainly isakmp, AH and ESP)

IPSEC comprises of the following main elements:
  • IKE/IKEv2: which is used to negotiate tunnel parameters. These parameters are: (H.A.G.L.E)
    • Hashing for data integrity (md5/sha)
    • Authentication (certs or preshared key (PSK))
    • Group (Diffie-Hellman group)
    • Lifetime (how long will the tunnel be up for (seconds)
    • Encryption for data confidentiality (AES, 3DES, AES-256)
  • AH which is more or less become obsolete because of it inability to deal with NAT
  • ESP Encapsulation for encryption and authentication.

In order to have a fully functional site to site VPN, there are 2 types of SA's (Security Association) that need to be established:

  1. ISAKMP SA's (Phase 1), bidirectional SA, used as a management channel, to exchange keys between the peers for instance (similar to the B-channel on a BRI)
  2. IPSEC SA's (Phase 2), these are unidirectional, so you will need 2 IPSEC SA's. This happens once Phase 1 is successful. These IPSEC SA's are used to define how the end user data is protected, using transform sets
So for a functional IPSEC tunnel, a total of 3 SA's are required. These two types of SA's are also referred to as Phase 1 and Phase 2. Here is a visualization of this:

ESP (encapsulating Security Payload) and SPI

Once the SA's have been established, ESP does the hard work of protecting the traffic across the tunnel. ESP uses IP as its Layer 3 protocol and puts itself at layer 4. So if you were to Wireshark capture Tunneled traffic, you would not see a TCP port, but an ESP header containing an SPI (security Perimeter Index), a sequence number, followed by an encrypted payload.  

SPI are a 32bit random number that is associated with an IPSEC sa. As soon as a VPN endpoint receives an ESP encapsulated packet with a certain SPI, it knows exactly what transform set to apply to decrypt and integrity check the payload. If your firewall/router has multiple site to site IPSEC VPNs, you will have a multitude of SPI.

If you issue a sh crypto ipsec sa on a firewall or router, this will show you the associated SPI's. for example:

fw01/pri/act# sh cry ipsec sa
    <output omitted>
    inbound esp sas:
      spi: 0xE11468F9 (3776211193)
         SA State: active
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 223428608, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373768/1686)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
    outbound esp sas:
      spi: 0x859B5920 (2241550624)
         SA State: active
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 223428608, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373506/1686)
         IV size: 8 bytes
As you can see there are 2 IPSEC SA's one for inbound, one for outbound, each with their own SPI, on the remote end these values will be opposite of course.

You can also clearly see the applied transform sets of each SA, using ESP with Triple DES encryption and HMAC-MD5 for message authentication and integrity.

Remember every SA can have a different transform set, although this is not recommended.

Diagnostics on Cisco devices

Of course, when will you be most likely to set up a site to site VPN? Correct, when talking into some 3rd party systems, or when some 3rd party needs to talk into some of your systems.  What sort of Forewall/VPN concentrator does this party have? Really could be anything, Juniper, Palo alto, Sonic. I have had my fair share of pain getting VPN's set up inter vendor. And before you blame the other guy for not sending you the right transform set, make damn sure that you are right. So I think a few hints and tips could make a lot of difference.

There are really two commands available that should be your best friend when diagnosis IPSEC VPN issues:

  • sh crypto isakmp sa
  • sh crypto ipsec sa

fw01/pri/act# show crypto isakmp sa  detail
IKEv1 SAs:
   Active SA: 5
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 5
1   IKE Peer:
    Type    : L2L             Role    : responder
    Rekey   : no              State   : MM_ACTIVE
    Encrypt : 3des            Hash    : MD5
    Auth    : preshared       Lifetime: 28800
    Lifetime Remaining: 16894

As you can see above, there is your 'HAGLE' (hash, authentication,Group (well sort of), Lifetime, Encryption). What is important above is that the ISAKMP SA is established  (MM_ACTIVE). If you do not see your ISAKMP SA established, go no further, trouble shoot Phase1 first. Other states are:

MM_NO_STATE (typically cause by connectivity issue with the peer), MM_KEY_EXCH (could mean PSK mismatch), MM_SA_SETUP (peers agree on isakmp sa and will continue the process of negotiation).

If your Phase1 SA is not established, you will need to run debug commands to see what transform set are being attempted and what the remote end is sending you. for this you can use:

  • debug crypto isakmp
  • debug crypto isakmp error
  • debug crypto isakmp ha
Before you do this, you might want to consider, using conditional debugging, i.e. only capture debug information related to the failing VPN tunnel/peer. This because you might be running a large number of tunnels which would result in large amount of debug information.  To enable conditional crypto debugging:

debug crypto condition <cond-type> <cond-value>

as a condition value, you could choose the peer IP address for instance. To reset the condition use: debug crypto condition reset

Now the next test phase 2, if phase 1 is established, use the show cypto ipsec sa command. You would get something similar to the output below:

fw01/pri/act# sh crypto ipsec sa peer
peer address:
    Crypto map tag: outside_map, seq num: 2, local addr:
      access-list Internet_cryptomap_1 extended permit ip         
      local ident (addr/mask/prot/port): (
      remote ident (addr/mask/prot/port): (
      #pkts encaps: 359869, #pkts encrypt: 358126, #pkts digest: 358126
      #pkts decaps: 280949, #pkts decrypt: 279563, #pkts verify: 279563
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 359869, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0
     local crypto endpt.:, remote crypto endpt.:
      path mtu 1500, ipsec overhead 58(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 67EA5A7C
      current inbound spi : 08278FA3
    inbound esp sas:
      spi: 0x08278FA3 (136810403)
         SA State: active
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 223428608, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373709/1407)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
    outbound esp sas:
      spi: 0x67EA5A7C (1743411836)
         SA State: active
         transform: esp-3des esp-md5-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 2, IKEv1, }
         slot: 0, conn_id: 223428608, crypto-map: outside_map
         sa timing: remaining key lifetime (kB/sec): (4373391/1407)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

The SA will show you H.A.G.L.E for both inbound and outbound SA's. It will also show you what traffic is protected by the VPN Tunnel. 

Good luck and namaste!!

Tuesday, 6 March 2018

BGP Conditional Advertisement Feature

BGP Conditional Advertisement Feature

This is actually a pretty cool feature, i didn't even know it existed until I was looking for a solution to advertise a subnet (prefix in BGP talk), only if a certain condition existed. This is exactly what conditional advertisements does

"The Border Gateway Protocol (BGP) conditional advertisement feature provides additional control of route advertisement, depending on the existence of other prefixes in the BGP table."

I am assuming, for those who want to read this post, that you have some understanding of BGP and its use of prefix-lists and route maps, otherwise this post might be hard to understand. Mind you, conditional advertisement are part of the CCIE R&S exam.

So let me go straight to the scenario:

So the routers under my admin domain are BEN and IBM. My primary router is BEN and my public IP range I am advertising is 
  • My two ISPs are Telstra and Next. 
  • BEN has an eBGP neighbour with Telstra,
  • IBM has an eBGP peer with Next. 
  • Then BEN and IBM from an iBGP neighbourship.
Nothing new so far. Now I have found that when advertising out the same public IP address (prefix) towards 2 different providers, even with AS path prepend, trying to make one ISP more preferable over the other, is highly unpredictable. This is because some providers prefer other providers no matter how often you AS prepend the crap out of your public prefix. This can cause asynchronous routing where your exit path is the primary ISP and entry through your secondary router. So I was looking for another solution; only route my public IP addresses out to the backup provider (Next in my case), in the event the primary fails. Or even better; fail over when the primary ISP stops advertising a default route into my organisation through the primary router.

In order to put all this in place, most, if not all configuration is done on the secondary router; IBM, so lets dive in.

As you can see below, the secondary internet router (IBM) has 2 default gateways

IBM#sh ip bgp topology *

For address family: IPv4 Unicast

BGP table version is 26, local router ID is

Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,

              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,

              x best-external, a additional-path, c RIB-compressed,

Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
     Network          Next Hop            Metric LocPrf Weight Path
 *>i             0    200      0 3000 i
 *                         100      0 4000 i

The most preferred on comes from the BEN router, which in turn is being advertised by the Telstra Router ( Initially I was going to use ip sla tracking on the IBM router to advertise out if BEN lost the connection to Telstra, but this is not as fool proof as checking if the default gateway is still being advertised by BEN, because if my primary internet router no longer sends a default route to my secondary internet router, the either my primary router is down, the link to Telstra is down, or Telstra is for some other reason no longer advertising a default route.

OK so on my IBM i set up a conditional advertisement to my Next BGP peer:

router bgp 5000
address-family ipv4
 neighbor advertise-map ADVERTISE non-exist-map NON-EXIST

what this means is that route map ADVERTISE is being invoked when the condition in route map NON-EXIST no longer exists.

route-map NON-EXIST permit 10
 match ip address prefix-list TEST
 match community 1
So the ADVERTISE route map is the easy part, it constitutes our public IP prefix

access-list 60 permit
the NON-EXIST route map is the condition that needs checking, and has in fact two conditions in it; it checks the prefix for a certain community and it checks if the actual prefix is available in the BGP table:

ip prefix-list TEST seq 5 permit

The reason there are two conditions, is that  (refer to the sh ip bgp topology * output above), there are two prefixes in the table; one from each provider. Now I am only interested in checking one of them; namely the one that comes from BEN I though it would be easiest to add a check for a certain community in (although AS path would have worked as well).

ip community-list 1 permit 362000
So basically this second condition check to see if the route has 362000 as the community.
You can check the route to see if the community attribute is set and has the correct value. see below

IBM#sh ip bgp
BGP routing table entry for, version 25
Paths: (3 available, best #1, table default)
  Not advertised to any peer
  Refresh Epoch 1
  3000, (received & used) from (
      Origin IGP, metric 0, localpref 200, valid, internal, best
      Community: 36200
      rx pathid: 0, tx pathid: 0x0
  Refresh Epoch 1

So at this stage both conditions should be met; a) a default route in the BGP table and b)a route with community attribute 36200. So our public prefix should NOT be advertised out IBM to Next. To verify: 

IBM#sh ip bgp nei
BGP neighbor is,  remote AS 4000, external link
---<output omitted>
Condition-map NON-EXIST, Advertise-map ADVERTISE, status: Withdraw
---<output omitted>
As you can see the conditional advertisement states "withdraw" which means the condition to start advertising is not met; ie.e we have a valid default route coming from BEN. So let me break something to trigger the condition to change. For this I will shut the connection between Telstra and BEN. (Remember BEN does not originate, its receives it from Telstra and as soon as that link breaks, it should no longer receive a default route either).

when debugging routing on IBM:

*Mar  7 04:45:48.908: RT: updating bgp (0x0)  :
    via   0 1048577
*Mar  7 04:45:48.915: RT: closer admin distance for, flushing 1 routes
*Mar  7 04:45:48.919: RT: add via, bgp metric [20/0]

as you can see the from BEN gets purged from the bgp table. and consequently the conditional advertisement kicks in:

IBM#sh ip bgp nei
  Condition-map NON-EXIST, Advertise-map ADVERTISE, status: Advertise
To double check this, we check what routes the IBM router is sending to Next:

IBM#sh ip bgp nei advertised-routes
BGP table version is 13, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
     Network          Next Hop            Metric LocPrf Weight Path
 *>            0         32768 i

Any questions, drop me a line.