Upgrade Exadata Machine Clusterware to 12.2.0.1

Hell All! It’s time to upgrade Exadata Machine infrastructure and database to Oracle 12c Release 2. There are some software and patch requirement you need to meet, before you can upgrade your Exadata machine to Oracle 12c Release 2. You might be able to perform most of the upgrade using rolling manner but database upgrade will require some down time. Ideally you would like to upgrade your infrastructure to 12c release 2 and install Oracle 12c release 2 software on separate mount with our impacting sexting database s running target Exadata machine. Later upgrade your databases to release 2 based on availability and downtime requirements. In any case, I strongly suggest making fool proof recovery plan for your Exadata machine and be prepared for a complete Exadata machine recovery. If possible, start with non-production environment and try to leverage DR Exadata Machine for production environments. This blog will provide you overview of upgrade process for your exadata machine to 12c release 2.

Caution : – I consider this a high risk activity and if you don’t have experience performing these kind of upgrades, hire someone who have experience with such upgrades.  

Software Requirements

Your current Exadata machine configuration should meet following requirement for Oracle 12c Release 2 upgrade:

  • Current Oracle Database and Grid Infrastructure version must be 11.2.0.3, 11.2.0.4, 12.1.0.1 or 12.1.0.2.
  • Upgrades from 11.2.0.1 or 11.2.0.2 directly to 12.2.0.1 are not supported.
  • Exadata Storage Server version 12.2.1.1.0 will be required for full Exadata functionality including  ‘Smart Scan offloaded filtering’, ‘storage indexes’ and’ I/O Resource Management’ (IORM).
  • When available: GI PSU 12.2.0.1.1 or later (which includes DB PSU 12.2.0.1.1). To be applied during the upgrade process, before running rootupgrade.sh on the Grid Infrastructure home, or after installing the new Database home, before upgrading the database.
  • Fix for bug 17617807 and bug 21255373 is required to successfully upgrade to 12.2.0.1 from 11.2.0.3.28, 11.2.0.4.160419 and 12.1.0.1.160419. The fix is already contained in 11.2.0.4.160419 and 12.1.0.2.160419.
  • Fix for bug 25556203 is required on top of the 12.2.0.1 Grid Infrastructure home before running rootupgrade.sh

Pre-Upgrade

Create temporary directory to hold all the software and patches

dcli -l oracle -g ~/dbs_group mkdir /u01/app/oracle/patchdepot

Download following software and patches from E-delivery

V840012-01.zip Oracle Database 12c Release 2 Grid Infrastructure

(12.2.0.1.0) for Linux x86-64 V839960-01.zip Oracle Database 12c Release

(12.2.0.1.0) for Linux x86-64 Exadata Storage Server Software 12.2.1.1.0

Patch 6880880 - OPatch latest update for 11.2, 12.1 and 12.2

Create the new Grid Infrastructure (GI_HOME) directory

(root)# dcli -g ~/dbs_group -l root mkdir -p /u01/app/12.2.0.1/grid
 (root)# dcli -g ~/dbs_group -l root chown oracle:oinstall /u01/app/12.2.0.1/grid

Install Grid Software using zip option , runinstaller is no longer supported

(oracle)$ unzip -q /u01/app/oracle/patchdepot/grid_home.zip -d /u01/app/12.2.0.1/grid

Obtain and Apply latest OPatch to the target 12.2.0.1 Grid Infrastructure home.

[oracle@exadb1 unzip -oq -d /u01/app/12.2.0.1/grid /u01/upgrade/p6880880_122010_Linux-x86-64.zip

[oracle@exadb1 OPatch]# cd /u01/app/12.2.0.1/grid/OPatch

[oracle@exadb1 OPatch]# ./opatch version

Validate Readiness for Oracle Clusterware upgrade using CVU

Before executing CVU as the owner of the Grid Infrastructure unset ORACLE_HOME, ORACLE_BASE and ORACLE_SID.

unset ORACLE_HOME ORACLE_BASE ORACLE_SID

(Oracle)$ /u01/app/12.2.0.1/grid/runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/12.2.0.1/grid -dest_version 12.2.0.1.0 -fixupnoexec –verbose


Verifying User Equivalence ...PASSED

Verifying /dev/shm mounted as temporary file system ...PASSED

Verifying File system mount options for path /var ...PASSED

Verifying zeroconf check ...PASSED

Verifying ASM Filter Driver configuration ...PASSED


Pre-check for cluster services setup was successful.

CVU operation performed:      stage -pre crsinst

Date:                         Apr 20, 2018 8:15:00 AM

CVU home:                     /u01/app/12.2.0.1/grid/

Backup /u01 filesystem

tar -cvf oracle_inventory_apr20.tar /u01/app/oraInventory

tar -cvf oracle_gi_home_apr20.tar /u01/app/12.1.0.2/grid

Note : Task need to be performed on all three nodes as root

Run Exachk report 

./exachk –a

Verify no active rebalance is running

SYS@+ASM1> select count(*) from gv$asm_operation;

COUNT(*)

----------

0

 Upgrade Clusterware

Upgrade Grid Infrastructure to 12.2.0.1 (Will also apply latest CPU)

(oracle)$ cd /u01/app/12.2.0.1/grid

(oracle)$ gridSetup.sh -applyPSU /u01/upgrade/26737266

Launching Oracle Grid Infrastructure Setup Wizard...

Upgrade wizard steps 

On "Select Configuration Options" screen, select "Upgrade Oracle Grid Infrastructure , and then click Next.

On "Grid Infrastructure Node Selection" screen, verify all database nodes are shown and selected, and then click Next.

On "Specify Management Options" screen, specify Enterprise Management details when choosing for Cloud Control registration.

On "Privileged Operating System Groups" screen, verify group names and change if desired, and then click Next. If presented with warning: IINS-41808, INS-41809, INS-41812 OSDBA for ASM,OSOPER for ASM, and OSASM are the same group Are you sure you want to continue? Click Yes

On "Specify Installation Location" screen, choose "Oracle Base" and change the software location. The GI_HOME directory cant be chosen. It shows software location: /u01/app/12.2.0.1/grid from where you started gridSetup.sh

If prompted "The Installer has detected that the specified Oracle Base location is not empty on this and remote servers!" Are you sure you want to continue? Click Yes

On "Root script execution" screen, do not check the box. Keep root execution in your own control

On "Prerequisite Checks" screen, resolve any failed checks or warnings before continuing.

Solaris only: Solaris only: Review Document <2186095.1> Oracle Solaris-specific guidelines for GI software installation prerequisite check failure.

On "Summary" screen, verify the plan and click 'Install' to start the installation (recommended to save a response file for the next time)

On "Install Product" screen monitor the install, until you are requested to run rootupgrade.sh (recommended to save a response file for the next time)

Before executing the last steps (rootupgrade.sh) of the installation process an additional step is required. rootupgrade.sh execution will happen after next steps.

When required relink oracle binary with RDS 

(oracle)$ dcli -l oracle -g ~/dbs_group /u01/app/12.2.0.1/grid/bin/skgxpinfo

 If the command does not return 'rds' relink as follows:

For Linux: as owner of the Grid Infrastructure Home on all nodes execute the steps as follows before running rootupgrade.sh:

(oracle)$ dcli -l oracle -g ~/dbs_group ORACLE_HOME=/u01/app/12.2.0.1/grid \ make -C /u01/app/12.2.0.1/grid/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle

Execute rootupgrade.sh on each database server

/u01/app/12.2.0/grid/rootupgrade.sh

Note:- Don't Run this script all at once on all nodes

Verify cluster status

(root)# /u01/app/12.2.0.1/grid/bin/crsctl check cluster -all
 **************************************************************
 node-1:
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 **************************************************************
 node-2:
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 **************************************************************

check, re-configuring for Cloud Control, and cleaning up the old, unused home areas.

Post-Upgrade

Disable Diagsnap for Exadata

NOTE: Due to unpublished bugs 24900613 25785073 and 25810099, Diagsnap should be disabled for Exadata.

(oracle)$ cd /u01/app/12.2.0.1/grid/bin

(oracle)$ ./oclumon manage -disable diagsnap

Advance COMPATIBLE.ASM diskgroup attribute

ALTER DISKGROUP RECOC1 SET ATTRIBUTE 'compatible.asm' = '12.2.0.1.0';

ALTER DISKGROUP DBFS_DG SET ATTRIBUTE 'compatible.asm' = '12.2.0.1.0';

ALTER DISKGROUP DATAC1 SET ATTRIBUTE 'compatible.asm' = '12.2.0.1.0';

De-Install GI HOME

Wait for few days before De-Install

oracle)$ cd $ORACLE_HOME/deinstall

(oracle)$ ./deinstall -checkonly

(oracle)$ ./deinstall

change ORACLE_HOME to grid home for previous grid home deinstall:

(root)# dcli -l root -g ~/dbs_group chown -R oracle:oinstall /u01/app/12.1.0.2

(root)# dcli -l root -g ~/dbs_group chmod -R 755 /u01/app/12.1.0.2


(oracle)$ ./deinstall -checkonly

(oracle)$ ./deinstall