Upgrading ISE?

Are you planning to upgrade Cisco ISE (Identity service Engine)? Then you should know about the upgrade readiness tool.

Cisco created an Upgrade Readiness Tool (URT) which simulates an upgrade of ISE to verify your ISE deployment prior to actually performing the upgrade.

The Upgrade Readiness Tool will check for possible obstacles during the upgrade, which could save you valuable time in your service window. The tool will also give you an estimate on how long the upgrade will take, which allow you to plan a suffient amount of time for your service window.

URT must run on an standalone ISE node or the secondary PAN node.

Currently the tool is available from ISE 2.0 and up.

The URT must run from the CLI. Here is a short guide to run the URT bundle:

1. Download the URT bundle from Cisco.com. The URT bundle is available, where you download the upgrade bundle and patches.
2. Copy the URT to the local disk - copy ftp://10.1.1.244/ise-urtbundle-2.4.0.x.SPA.x86_64.tar.gz disk:/
3. Initiate the install of the URT bundle - See below

ise-test-secondary/admin# application install ise-urtbundle-2.4.0.x.SPA.x86_64.tar.gz disk
Save the current ADE-OS running configuration? (yes/no) [yes] ?
Generating configuration…
Saved the ADE-OS running configuration to startup successfully
Getting bundle to local machine…
Unbundling Application Package…
Verifying Application Signature…
Initiating Application Install…
###########################################
Installing Upgrade Readiness Tool (URT)
##########################################

An example of a full succesful log is available in the following link:
https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/upgrade_guide/b_ise_upgrade_guide_24/b_ise_upgrade_guide_24_chapter_01.html

One of the issues I often encounter during upgrades is discovered by the URT in the output below:
Fri Jul 5 08:40:23 CEST 2019 : ########################################
Fri Jul 5 08:40:23 CEST 2019 : # Running Upgrade Readiness Tool (URT) #
Fri Jul 5 08:40:23 CEST 2019 : ########################################
Fri Jul 5 08:40:23 CEST 2019 : This tool will perform following tasks:
Fri Jul 5 08:40:23 CEST 2019 : 1. Pre-requisite checks
Fri Jul 5 08:40:23 CEST 2019 : 2. Clone config database
Fri Jul 5 08:40:23 CEST 2019 : 3. Copy upgrade files
Fri Jul 5 08:40:23 CEST 2019 : 4. Data upgrade on cloned database
Fri Jul 5 08:40:23 CEST 2019 : 5. Time estimate for upgrade
Fri Jul 5 08:40:23 CEST 2019 :

Fri Jul 5 08:40:23 CEST 2019 : Copying upgrade file fixes
Fri Jul 5 08:40:23 CEST 2019 : ==========================
Fri Jul 5 08:40:23 CEST 2019 : - N/A
Fri Jul 5 08:40:23 CEST 2019 :

Fri Jul 5 08:40:23 CEST 2019 : Pre-requisite checks
Fri Jul 5 08:40:23 CEST 2019 : ====================
Fri Jul 5 08:40:23 CEST 2019 : Reading config file…
Fri Jul 5 08:40:23 CEST 2019 : Running 1 of 6 (Diskspace check)
Fri Jul 5 08:40:23 CEST 2019 : Running DiskSpace_Sanity.sh /opt/CSCOcpm/logs/iseurt-20190705-083935.log /opt/urt
Fri Jul 5 08:40:23 CEST 2019 : Disk Space sanity check
Fri Jul 5 08:40:23 CEST 2019 : - Successful
Fri Jul 5 08:40:23 CEST 2019 : Running 2 of 6 (NTP sanity check)
Fri Jul 5 08:40:23 CEST 2019 : Running NTP_Sanity.sh /opt/CSCOcpm/logs/iseurt-20190705-083935.log /opt/urt
Fri Jul 5 08:40:23 CEST 2019 : NTP sanity
Fri Jul 5 08:40:44 CEST 2019 : - Successful
Fri Jul 5 08:40:44 CEST 2019 : Running 3 of 6 (Appliance or VM check)
Fri Jul 5 08:40:44 CEST 2019 : Running Check_Hardware.sh /opt/CSCOcpm/logs/iseurt-20190705-083935.log /opt/urt
Fri Jul 5 08:40:44 CEST 2019 : Appliance/VM compatibility
Fri Jul 5 08:40:44 CEST 2019 : VM appliance detected…
Fri Jul 5 08:40:44 CEST 2019 : -Checking VM for minimum hardware requirements
Fri Jul 5 08:40:44 CEST 2019 : - Successful
Fri Jul 5 08:40:44 CEST 2019 : Running 4 of 6 (Trust Cert check)
Fri Jul 5 08:40:44 CEST 2019 : Running Cert_Validation.sh /opt/CSCOcpm/logs/iseurt-20190705-083935.log /opt/urt TRUSTCERT
Fri Jul 5 08:40:45 CEST 2019 : Trust Cert Validation
Fri Jul 5 08:41:24 CEST 2019 : - Failed
Fri Jul 5 08:41:24 CEST 2019 : Running 5 of 6 (System Cert check)
Fri Jul 5 08:41:24 CEST 2019 : Running Cert_Validation.sh /opt/CSCOcpm/logs/iseurt-20190705-083935.log /opt/urt SYSTEMCERT
Fri Jul 5 08:41:24 CEST 2019 : System Cert Validation
Fri Jul 5 08:41:37 CEST 2019 : - Successful
Fri Jul 5 08:41:37 CEST 2019 : Running 6 of 6 (Invalid MDMServerName in Authorization Policies check)
Fri Jul 5 08:41:37 CEST 2019 : Running MDMAuthz_Validation.sh /opt/CSCOcpm/logs/iseurt-20190705-083935.log /opt/urt
Fri Jul 5 08:41:37 CEST 2019 : Invalid MDMServerNames in Authorization Policies check
Fri Jul 5 08:41:38 CEST 2019 : - Successful
Fri Jul 5 08:41:38 CEST 2019 : 5 out of 6 pre-requisite checks passed
Fri Jul 5 08:41:38 CEST 2019 : Passed scripts are not equal to total scripts
Fri Jul 5 08:41:38 CEST 2019 : Some pre-requisite checks have failed. Hence exiting…

This ISE deployment had an expired CA certificate, which meant running the URT before the upgrade saved 40 minutes in the service window.
I deleted the certificate and restarted the URT. The tool spent a loooong time on one step of the upgrade test. To figure out what was going on I turned to the logs.

The ISE logs can be found in the CLI under the show logging application command, but there is quite a few log files to look through, so I searched for a log, that was updated today using the below command:

ise-test-secondary/admin# show logg appl | i “Jul 05”
-output cut-
4571316 Jul 05 2019 14:38:30 dbupgrade-data-global-20190705-100146.log

I was upgrading this particular deployment from ISE 2.0 to ISE 2.4. Cisco changed the way Policy Sets are arranged in ISE 2.3, which means the upgrade of ISE includes a step where the policy is converted to the new policy set format. In this case, it was the conversion of the policy sets that was taking a long time to complete.

I accessed the log by using the following command.

An example of the conversion log is included below:
ise-test-secondary/admin# show logg application dbupgrade-data-global-20190705-100146.log tail
@@@ PsUpgrade: debug- :Reading Exception rules for Policy Set TEST
@@@ PsUpgrade: debug- :No Exception rules found for Policy Set :TEST
@@@ PsUpgrade: info- :Found a regular PS TEST. Upgrading the whole set.
@@@ PsUpgrade: info- :=== Upgrading legacy Radius Policy Set TEST
@@@ PsUpgrade: debug- :AN outer rule Default has allow protocol Default Network Access
@@@ PsUpgrade: debug- :There is one allow protocol Default Network Access for Policy Set TEST
@@@ PsUpgrade: info- :About to create a new Policy Set with outer authentication rules of Policy Set TEST

Notice the tail keyword, which allow me to view the changes in the log file as they happen.

Once the policy set conversion completed, so did the URT. Afterwhich the tool gave me an estimate of the time expected to do the upgrade.

Time estimate for upgrade
=========================
(Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes is not considered in calculating estimates)
Estimated time for each node (in mins):
ise-test-secondary(SECONDARY PAP,MNT):724
ise-test(PRIMARY PAP,MNT):184
Each PSN(3 if in parallel):134

General tips for ISE upgrade:

• Remember to follow the checklist and ‘Prepare for Upgrade’ list before the upgrade. https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/upgrade_guide/b_ise_upgrade_guide_24/b_ise_upgrade_guide_24_chapter_01.html

• Verify management access to all nodes - (SSH and HTTPS)

• Sync up the management node(s) and and admin nodes (Under Deployment -> Choose node -> Sync-Up) before the upgrade to ensure consitent configuration and MnT data.

• Make sure you have console access. If you loose VPN access or network connectivity to the ISE node, you can potentially loose all upgrade visibility.

• And run the Upgrade readiness tool… :)