CUC (6) CUCM (28) Jabber (6) Python (2) Routing (3) Solarwinds Orion NPM (4) switching (1) Video (6) voice (3)

Sunday, 11 November 2018

Regenerate certificates on CUCM

Quick post on what to do when your certificates on cucm are about to expire, and when you have set up your cert monitor, you will get swamped with email alerts.

To check what certificates are expiring, go to cucm > OS administration > Security > Certificate management.

There are two types of certificates: self-signed and signed by a CA.  From a security point of view you should not use self signed certificates.  When installing CUCM, the certificate store gets populated with self signed certs, with a 5 year expiry period. In my experience,  usually  all but the tomcat certs are self signed. Which makes life a lot easier when regenerating new certs. There are a couple of types of certificate types:

1-Call Manager
3-IPSEC (used for DRF, backup)

As said, there is a big chance all these need to be regenerated because they were generated at the same time: during install.

The most important thing to keep in mind is to never regenerate both Callmanager.pem and TVS.pem certificates at the same time.  I suggest the following order, that served me well a couple of times:

1) Regenerate the CallManager.pem certificate on the publisher Call Manager followed by restart of CallManager, TVS and TFTP service on PUB

2) Regenerate the CallManager.pem certificate on the subscriber Call Manager followed by restart of CallManager, TVS and TFTP service    and repeat for every SUB in your cluster.

Steps 1 and 2 are impacting because restarting call manager service cause phones to fail over.
As a test after you performed steps 1 and 2, go to the certificate store and verify if all call managers now contain the newly regenerated certificate in their store. Note: there is no need to manually import certs, because replication will sync the certs between the call managers. Of course step when using CA signed certs, in step two, you will need to create a CSR, have it signed and import the cert back into ONLY the server on which the CSR was generated.

3) Regenerate the TVS.pem certificate followed by restart of TVS and TFTP service on the publisher Call Manager.

4) Regenerate the TVS.pem certificate followed by restart of TVS and TFTP service on the subscriber Call Manager

5) Regenerate the CAPF.pem certificate on the publisher CM server followed by regenerating it on the subscriber CM and then restart CAPF service only on publisher CM.

6) Regenerate the tomcat certificate on publisher Call Manager followed by regenerating it on the subscribers server as well

7) Restart the Cisco Tomcat on publisher Call Manager followed by subscriber Call Manager

8) regenerate IPSEC .pem on publisher, restart C: utils service restart Cisco DRF Local AND C: utils service restart Cisco DRF Master, then regenerate on SUBS  (restart DRF from SSH Console).

There is really not much to it, just follow the steps in the order above, and restart the services. When I do changes like this I keep RTMT open and monitor the registration of the phones while I go through then changes; Good luck

Thursday, 11 October 2018

F5 Big IP SAML configuration IdP

This post will discuss how you can configure your F5 Big-IP as a SAML identity provider (IdP).

In order to use your Big-IP as an IdP, there are 3 main parts to the configuration work:

1-configure Local IdP services
2-Configure external service Provider (SP) connectors.
3-assign SAML to the access policy

The External SP connectors (point 2),  are used so external services (such as, for instance Office 365, Webex or Sales Force) can successfully redirect the client that wants to access its services,  to the F5 to allow the User/client to connect to whatever server it needs to connect to.

Before I dive into the configuration part, below is a schematic representation of what SAML looks like.

Fig.1 SAML schematics
SAML is like an authentication proxy. For example you hit a web site, https, and it requires a log in, when that webpage is configured for SSO, it needs to know what(IdP) can actually do the authentication and echo the result on the SP's behalf. So essentially the IdP does all the hard work here, all the SP does is redirects the client to the IdP for authentication and if successful, the IdP will redirect the client back to the SP.

So this post will focus on the right hand part of the picture, the configuration and on how SAML requestors are allow to connect to the IdP to receive an assertion for future single sign on.  so Lets start.

On your Big-IP, go to Access Profile > SAML > BIG-IP as IdP.

1-Create the IdP

Fig.2. IdP General settings

the IdP entity ID is basically a URL that allows and SP to connect to that same URL/ID

Because SAML heavily relies on HTTP, it uses SSL for server authentication. So in the initial communications between SP and IdP, the IdP will sign the assertion (see Fig.1.) with its private key which the SP know how to decrypt using the IdP's public cert/key. (see below in Figure 3)

Fig.3. IdP security settings

So yeah, time to upload, some certs. remember, if you use like Cloud service SP's to use your on premise  F5 Big IP IdP to connect to, these certs will need to be  signed by a trusted CA, not your internal CA!!!

Next you need to configure the Assertion (see fig.4):

Fig.4 Assertion settings

These SAML assertions are key to single sign on, and contain security information, such as authentication state, security attributes and authorization. The assertion is sent from the IdP to the SP and contain security statements based on which the SP can make an access control decision on the access attempt of the client.

2 Create the SAML SP Connector

Of course the IdP is  no good just sitting there, of course SP's will need to be able to connect to is to obtain an assertion. on the other hand you dont just want every thing and every body to be able connect to it to perform proxy authentications.  So in short every SP that requires SAML to the IdP on the F5, will need to be explicitly defined. F5 call this External SP Connectors (they can be organization internal as well, they are called external because the are external to the device itself).
There are 3 key points the Big-IP will need to know in order to actually communicate with the external SP:

  1. the external SP Entity  ID, this is a URL that the SP presents when sending the GET to request a assertion from the IdP
  2. Assertion Consumer Service, also a URL
  3. the F5 will require the CA signed certificate of the SP in its certificate store. This to allow SSL to be set up and to allow the SP to authenticate itself against the IdP. 

Point 1 can be configured under General settings:

Point 2 can be configured under Endpoint settings:

Point 3 can be configured under Security settings:

Basically when the user first time tries to log into the SP, the SP actually redirects the login attempt to the IdP. after that is successful, the IdP sends an assertion to the SP and the SP send an authentication successful response to the client. (see Fig.1).

3-Use SAML in your access policy

In this part, we have to point the SP to the IdP that we configured in part 1 (name=saml_idp) to the access policy.

Fig. 5. Pointing the SP to the IdP
For this, you go to the access policy configuration under the "SSO/Auth Domains" tab you can point to the previously added IdP. (see Fig. 5).

To see this in the visual policy editor it would look something like this:

Here you have 2 types of authentication (Kerberos and web auth), split up in 3 AD groups; all using SAML.

To get a  better understanding of SAML, I suggest you install for instance the Chrome SAML tracer extension, which provides an indept insight of the contents of SAML  POST and GET messages, for example, where the IdP assertion is shown.

I know there is much more to write about SAML and I might in the future, but I wanted to focus on its configuration on the Big-IP's. Namaste

Thursday, 4 October 2018

packet capture on ASA

In order to do perform a packet capture on a Cisco ASA from the CLI, you will need to use the 'capture' command. by default the capture will use a 0.5MB buffer for a single capture session.

ON the CLI issue the capture command, enter and this will show you the current and historic captures that run/have run on the ASA.

Below are the options:

 capture Test ?

  access-list             Capture packets that match access-list
  buffer                          Configure size of capture buffer, default is 512 KB
  circular-buffer  Overwrite buffer from beginning when full, default is
  ethernet-type    Capture Ethernet packets of a particular type, default is IP
  headers-only     Capture only L2, L3 and L4 headers of packet without data in
  interface                  Capture packets on a specific interface
  match                         Capture packets matching five-tuple
  packet-length    Configure maximum length to save from each packet, default
                                           is 1518 bytes
  real-time                Display captured packets in real-time. Warning: using this
                                           option with a slow console connection may result in an
                                          excessive amount of non-displayed packets due to performance
  trace                          Trace the captured packets
  type                            Capture packets based on a particular type

So the 3 main ones are logging against an existing access-list, against an interface or a match statement like the one below,  capturing only traffic to, similar to a Wireshark capture filter.:

capture f5test5 interface IDMZ  match ip any host

once you are happy with the capture duration, or, once you have generated test traffic, view the capture:

view capture f5test5

Tuesday, 2 October 2018

Dialplan, Dialpeer, Digit manipulation conundrum

A quick post on the order of digit manipulation and dial peer matching, because every time  i am tasked to configure something IOS dialplan related, I have to remind myself.

Every call across a CME, H323 gateway, CUBE or whatever, has two parts:


INBOUND Dial peer matching

Inbound dial peer matching is done in the following order:
  1. Match the dialed number (DNIS) using the incoming called-number dial peer configuration
  2. Match the caller ID information (ANI) using the answer-address dial peer configuration
  3. Match the caller ID information (ANI) using the destination-pattern dial peer configuration command.
  4. Match an incoming POTS dial peer by using the port dial-peer configuration command
  5. If no match has been found using the previous four methods, use dial peer 0

Most times, to make this behavior controllable, and to catch it at the first step, particularly if there is no need for inbound digit manipulation add a dial peer like:

dial-peer voice 111111 voip
 description # Default VOIP Incoming DP #
 incoming called-number .
 voice-class codec 1

If none of your oher dialpeers have any more specific "incoming called number" statements, your dialpeer 11111 will get hit on the inbound.

OUTBOUND Dial peer matching

Digit manipulation order:

Num-exp does get hit first, it is configured globally and NOT on a dial peer level.

The second tier, is the automatic digit strip of a pots dial peer on explicitly matched digits. for example is you offer a called number of 038945666 and you have a pots dial peer with destination patter 0389..... the 0389 will get stripped, because its explicitly matched (applying no digit-stripforward digits all or prefix 0389 on the dial peer, would prevent this from happening).

Active calls poller Solarwinds SNMP

Use OID, (0)

which represents the call legs as a show call active voice compact would return

You could use  = Voice_ActiveCalls_E1  if you use E1's , but i would advise using  as this single OID can cover ISDN channels as well as SIP call legs.

Use a Transform rule to divide the call legs by 2, as this truly represents the number of active calls. because remember a single call consists of two call legs, from the perspective of the observing gateway; one inbound call leg and one outbound call leg.

Port channel between Cisco switch and ESXi host

I had this issue because the IP Hash load balancing configuration is not propagated to the Management PortGroup.
Try to follow the procedures described in this KB, it works in my case
Note: To reduce the chance of network connection loss please change the vSwitch load balancing to Route based on IP hash using this method
  1. Shut down all ports in the team from the physical switch leaving a single port as active. - (Management PortGroup e.g. Fa1/0/1)
  2. Change the load balancing to Route based on ip hash on the vSwitch and Management Portgroup.
  3. Configure the port channel on the physical switch. - (Fa1/0/1 - Fa1/0/4)
  4. Enable the ports on the physical switch. - (no shut Fa1/0/2 - Fa1/0/4)

PKI on an ISR for IWAN, certificate, convolutius maximus


1-Enable HTTP server on your CA

2-Enable NTP and sync with NTP server

After the clock is set the certificate server automatically turns to running status

3-configure the PKI server

crypto pki server IWAN-IOS-CA
 database level complete
 database archive pem password 7 12345678
 grant auto
 lifetime crl 24
 lifetime certificate 730
 lifetime ca-certificate 3650
 auto-rollover 14
 database url bootflash:/CA/

to verify issue show cry pki server:

Certificate Server IWAN-IOS-CA:
    Status: enabled
    State: enabled
    Server's configuration is locked  (enter "shut" to unlock it)
    Issuer name:
    CA cert fingerprint: E4A39166 37EEEEEEEE6 E231E131
    Granting mode is: auto
    Last certificate issued serial number (hex): 5
    CA certificate expiration timer: 11:51:29 AEDT Jan 22 2027
    CRL NextUpdate timer: 06:15:39 AEST DEC 3 2018
    Current primary storage dir: bootflash:/CA/
    Database Level: Complete - all issued certs written as <serialnum>.cer
    Auto-Rollover configured, overlap period 14 days
    Autorollover timer: 11:51:30 AEDT Jan 8 2027

If any issues with the status of your PKI server, go back to points 1 and 2; check http server, ntp and storage access.

4-RSA key pairs


#)crypto key generate rsa usage modulus 2048.

Choose the usage keys as this will generate two key pairs, one for encryption and one for signatures.
The CA server itself , automatically generates a 1024-bit Rivest, Shamir, and Adelman (RSA) key pair. You must manually generate an RSA key pair if you prefer a different key pair modulus. The certificate server uses a regular Cisco IOS RSA key pair as its CA key. This key pair must have the same name as the certificate server. If you do not generate the key pair before the certificate server is created on the router, a general-purpose key pair is automatically generated during the configuration of the certificate server.


show cry key mypubkey rsa

Key name:
Key type: RSA KEYS
 Storage Device: private-config
 Usage: Signature Key
 Key is not exportable. Redundancy enabled.
 Key Data:
  30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101
  7781F687 B16B290E 5D7079FC 322DB108 A48ACCB6 00EAE6B6 D113D74C 637CC203
  9D020301 0001
% Key pair was generated at: 21:45:30 AEST Aug 2 2018
Key name:
Key type: RSA KEYS
 Storage Device: private-config
 Usage: Encryption Key
 Key is not exportable. Redundancy enabled.
 Key Data:
  30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101
  FB1F1223 8B4400C0 8185679E BA645CEC 9A1C36BC 490134FF A414BD9A 268B1390

5-add the trustpoint:

A trustpoint, also known as the certificate authority (CA), manages certificate requests and issues certificates to participating network devices. To add a trust point:

crypto pki trustpoint IWAN-CA
 enrollment url
 subject-name cn=iwan-router
 revocation-check crl
 rsakeypair IWAN-RSA-KEYS
 auto-enroll 80 regenerate

6- authenticate the  trustpoint:

MPLS-HUB(config)#crypto pki authenticate IWAN-CA
Certificate has the following attributes:
       Fingerprint MD5: 63233948 B341560F E546B155 F4542BEA
      Fingerprint SHA1: 0F66B929 7F1D5B68 FFE9581C 3733326E 518BA792

% Do you accept this certificate? [yes/no]: yes
Trustpoint CA certificate accepted.

after doing this, in the log:

Jul 26 23:16:29.690: %PKI-6-CERTRENEWAUTO: Renewing the router certificate for trustpoint IWAN-CA
Jul 26 23:16:29.735: %CRYPTO-6-AUTOGEN: Generated new 512 bit key pair
Jul 26 23:16:29.842: %PKI-6-CERTRET: Certificate received from Certificate Authority
Jul 26 23:16:29.842: %PKI-4-NOAUTOSAVE: Configuration was modified.  Issue "write memory" to save new certificate
Jul 26 23:17:29.642: %PKI-6-CERTRENEWAUTO: Renewing the router certificate for trustpoint IWAN-CA
Jul 26 23:17:29.744: %PKI-6-CERTRET: Certificate received from Certificate Authority
Jul 26 23:17:29.744: %PKI-4-NOAUTOSAVE: Configuration was modified.  Issue "write memory" to save new certificate


Issue:  )#Crypto pki enroll IWAN-CA

this will request a certificate to be signed by the CA

MPLS-HUB(config)#crypto pki enroll IWAN-CA
Trustpoint IWAN-CA has already enrolled and has a router cert issued to it.
If you successfully re-enroll this trustpoint,the existing certificate will be replaced.

Do you want to continue with re-enrollment? [yes/no]:
Aug  2 21:47:30.240: PKI:get_cert IWAN-CA 0x10 (expired=0):
Aug  2 21:47:30.241: PKI:get_cert IWAN-CA 0x4 (expired=0):
Aug  2 21:47:30.241: PKI: our cert expires before the CA cert for IWAN-CAyes
% Start certificate enrollment ..
% Create a challenge password. You will need to verbally provide this
   password to the CA Administrator in order to revoke your certificate.
   For security reasons your password will not be saved in the configuration.
   Please make a note of it.

Aug  2 21:48:01.569: CRYPTO_PKI: locked trustpoint IWAN-CA, refcount is 1
Aug  2 21:48:01.638: %CRYPTO-6-AUTOGEN: Generated new 512 bit key pair
Aug  2 21:48:01.639: CRYPTO_PKI: unlocked trustpoint IWAN-CA, refcount is 0