Upgrade Exadata Machine Clusterware to

Hell All! It’s time to upgrade Exadata Machine infrastructure and database to Oracle 12c Release 2. There are some software and patch requirement you need to meet, before you can upgrade your Exadata machine to Oracle 12c Release 2. You might be able to perform most of the upgrade using rolling manner but database upgrade will require some down time. Ideally you would like to upgrade your infrastructure to 12c release 2 and install Oracle 12c release 2 software on separate mount with our impacting sexting database s running target Exadata machine. Later upgrade your databases to release 2 based on availability and downtime requirements. In any case, I strongly suggest making fool proof recovery plan for your Exadata machine and be prepared for a complete Exadata machine recovery. If possible, start with non-production environment and try to leverage DR Exadata Machine for production environments. This blog will provide you overview of upgrade process for your exadata machine to 12c release 2.

Caution : – I consider this a high risk activity and if you don’t have experience performing these kind of upgrades, hire someone who have experience with such upgrades.  

Software Requirements

Your current Exadata machine configuration should meet following requirement for Oracle 12c Release 2 upgrade:

  • Current Oracle Database and Grid Infrastructure version must be,, or
  • Upgrades from or directly to are not supported.
  • Exadata Storage Server version will be required for full Exadata functionality including  ‘Smart Scan offloaded filtering’, ‘storage indexes’ and’ I/O Resource Management’ (IORM).
  • When available: GI PSU or later (which includes DB PSU To be applied during the upgrade process, before running rootupgrade.sh on the Grid Infrastructure home, or after installing the new Database home, before upgrading the database.
  • Fix for bug 17617807 and bug 21255373 is required to successfully upgrade to from, and The fix is already contained in and
  • Fix for bug 25556203 is required on top of the Grid Infrastructure home before running rootupgrade.sh


Create temporary directory to hold all the software and patches

dcli -l oracle -g ~/dbs_group mkdir /u01/app/oracle/patchdepot

Download following software and patches from E-delivery

V840012-01.zip Oracle Database 12c Release 2 Grid Infrastructure

( for Linux x86-64 V839960-01.zip Oracle Database 12c Release

( for Linux x86-64 Exadata Storage Server Software

Patch 6880880 - OPatch latest update for 11.2, 12.1 and 12.2

Create the new Grid Infrastructure (GI_HOME) directory

(root)# dcli -g ~/dbs_group -l root mkdir -p /u01/app/
 (root)# dcli -g ~/dbs_group -l root chown oracle:oinstall /u01/app/

Install Grid Software using zip option , runinstaller is no longer supported

(oracle)$ unzip -q /u01/app/oracle/patchdepot/grid_home.zip -d /u01/app/

Obtain and Apply latest OPatch to the target Grid Infrastructure home.

[oracle@exadb1 unzip -oq -d /u01/app/ /u01/upgrade/p6880880_122010_Linux-x86-64.zip

[oracle@exadb1 OPatch]# cd /u01/app/

[oracle@exadb1 OPatch]# ./opatch version

Validate Readiness for Oracle Clusterware upgrade using CVU

Before executing CVU as the owner of the Grid Infrastructure unset ORACLE_HOME, ORACLE_BASE and ORACLE_SID.


(Oracle)$ /u01/app/ stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/ -dest_crshome /u01/app/ -dest_version -fixupnoexec –verbose

Verifying User Equivalence ...PASSED

Verifying /dev/shm mounted as temporary file system ...PASSED

Verifying File system mount options for path /var ...PASSED

Verifying zeroconf check ...PASSED

Verifying ASM Filter Driver configuration ...PASSED

Pre-check for cluster services setup was successful.

CVU operation performed:      stage -pre crsinst

Date:                         Apr 20, 2018 8:15:00 AM

CVU home:                     /u01/app/

Backup /u01 filesystem

tar -cvf oracle_inventory_apr20.tar /u01/app/oraInventory

tar -cvf oracle_gi_home_apr20.tar /u01/app/

Note : Task need to be performed on all three nodes as root

Run Exachk report 

./exachk –a

Verify no active rebalance is running

SYS@+ASM1> select count(*) from gv$asm_operation;




 Upgrade Clusterware

Upgrade Grid Infrastructure to (Will also apply latest CPU)

(oracle)$ cd /u01/app/

(oracle)$ gridSetup.sh -applyPSU /u01/upgrade/26737266

Launching Oracle Grid Infrastructure Setup Wizard...

Upgrade wizard steps 

On "Select Configuration Options" screen, select "Upgrade Oracle Grid Infrastructure , and then click Next.

On "Grid Infrastructure Node Selection" screen, verify all database nodes are shown and selected, and then click Next.

On "Specify Management Options" screen, specify Enterprise Management details when choosing for Cloud Control registration.

On "Privileged Operating System Groups" screen, verify group names and change if desired, and then click Next. If presented with warning: IINS-41808, INS-41809, INS-41812 OSDBA for ASM,OSOPER for ASM, and OSASM are the same group Are you sure you want to continue? Click Yes

On "Specify Installation Location" screen, choose "Oracle Base" and change the software location. The GI_HOME directory cant be chosen. It shows software location: /u01/app/ from where you started gridSetup.sh

If prompted "The Installer has detected that the specified Oracle Base location is not empty on this and remote servers!" Are you sure you want to continue? Click Yes

On "Root script execution" screen, do not check the box. Keep root execution in your own control

On "Prerequisite Checks" screen, resolve any failed checks or warnings before continuing.

Solaris only: Solaris only: Review Document <2186095.1> Oracle Solaris-specific guidelines for GI software installation prerequisite check failure.

On "Summary" screen, verify the plan and click 'Install' to start the installation (recommended to save a response file for the next time)

On "Install Product" screen monitor the install, until you are requested to run rootupgrade.sh (recommended to save a response file for the next time)

Before executing the last steps (rootupgrade.sh) of the installation process an additional step is required. rootupgrade.sh execution will happen after next steps.

When required relink oracle binary with RDS 

(oracle)$ dcli -l oracle -g ~/dbs_group /u01/app/

 If the command does not return 'rds' relink as follows:

For Linux: as owner of the Grid Infrastructure Home on all nodes execute the steps as follows before running rootupgrade.sh:

(oracle)$ dcli -l oracle -g ~/dbs_group ORACLE_HOME=/u01/app/ \ make -C /u01/app/ -f ins_rdbms.mk ipc_rds ioracle

Execute rootupgrade.sh on each database server


Note:- Don't Run this script all at once on all nodes

Verify cluster status

(root)# /u01/app/ check cluster -all
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online
 CRS-4537: Cluster Ready Services is online
 CRS-4529: Cluster Synchronization Services is online
 CRS-4533: Event Manager is online

check, re-configuring for Cloud Control, and cleaning up the old, unused home areas.


Disable Diagsnap for Exadata

NOTE: Due to unpublished bugs 24900613 25785073 and 25810099, Diagsnap should be disabled for Exadata.

(oracle)$ cd /u01/app/

(oracle)$ ./oclumon manage -disable diagsnap

Advance COMPATIBLE.ASM diskgroup attribute




De-Install GI HOME

Wait for few days before De-Install

oracle)$ cd $ORACLE_HOME/deinstall

(oracle)$ ./deinstall -checkonly

(oracle)$ ./deinstall

change ORACLE_HOME to grid home for previous grid home deinstall:

(root)# dcli -l root -g ~/dbs_group chown -R oracle:oinstall /u01/app/

(root)# dcli -l root -g ~/dbs_group chmod -R 755 /u01/app/

(oracle)$ ./deinstall -checkonly

(oracle)$ ./deinstall

Acceptable Hidden Parameters on Exadata Machine

We all have seem hidden parameters being used as a workaround to solve a specific problem, and should be removed once a system has been upgraded to a version level that contains the fix for the specific problem. So what happen when you migrate database to Exadata Machine, specially using physical migration methods? Most likely they are not removed during the migration process, even though the version level might contains the correct fix. Verifying the hidden database initialization parameter usage helps avoid hidden parameters being used any longer than necessary. Otherwise, use of hidden initialization parameters not recommended by Oracle development can introduce instability, performance problems, corruptions, and crashes to your Exadata environments.

Please verify hidden initialization parameter usage in each ASM and database instance using following sql.

select name,value from v$parameter where substr(name,1,1)=’_’;

All being said, there are some acceptable hidden parameters for Exadata Machine. Please review the list of acceptable hidden parameters based on their usage.

Generally Acceptable Hidden Parameters Table

  1. _file_size_increase_increment with possible value of 2143289344
  2. _enable_NUMA_support depend on database versions
  3. _asm_resyncckpt with value of 0 to Turns off resync checkpointing
  4. _smm_auto_max_io_size 1024 to permits 1MB IOs for hash joins that spill to disk
  5. _parallel_adaptive_max_users with value of 2
  6. _assm_segment_repair_bg as false for bug 23734075 work around
  7. _backup_disk_bufcnt as 64 (Only when ZFS based backups are in use)
  8. _backup_disk_bufsz as 1048576 (Only when ZFS based backups are in use)
  9. _backup_file_bufcnt as 64 (Only when ZFS based backups are in use)
  10. _backup_file_bufsz as 1048576 (Only when ZFS based backups are in use)