Virtualized Exadata Machine (Isolation vs Efficiency)

Virtualizing Exadata machine has become an important deployment decision for many Exadata customers and most of them like to explore or at least discuss virtualization to see if there is any benefit for them. Since I have already been part of those conversations, I decided to share my thoughts on this topic to help my readers.

Oracle started supporting Exadata virtualization while ago and it’s free. You might want to virtualize your Exadata Machine for many reasons (Consolidation, Security, and Compliance) and end result will be to achieve some level of Isolation. Isolation is probably one of the main reason to virtualized Exadata machine. And if you are planning to virtualize your Exadata, keep in mind that everything (CPU, memory, disk) will be hard partitioned. Even though you can over-provision CPU’s, Oracle recommend strongly against over provisioning any resources. With dedicated CPU’s, memory and disks you will be able to achieve great isolation but it will not be an efficient use of Exadata machine resources. For instance, virtualization will provide you opportunity to have different patching cycles for each Exadata VM cluster but not without maintenance overhead. I have worked with Exadata Rack with up to 3 VM’s and it was not fun patching them, imagine if you have multiple virtualize Exadata machines. Remember Oracle releases around 4 bundle patches a year and you need to apply at least 2 of them to be in compliance for Oracle Platinum Service. Additionally, since everything is hard partitioned in virtualized Exadata machine, you will not be able to use IDLE hardware resources from Other VM’s. Hence you are wasting very expensive Hardware and Software resources.

It’s also important to understand that there are many levels of isolation like physical level, OS level, Storage Level, cluster level or RDBMS level and you can still achieve some level of isolation without virtualizing Exadata machine. For example you can have multiple RDBMS homes, different ASM disk groups and isolated network using VLAN.  I am not against virtualizing Exadata Machine but you should have very good use case for it. I would suggest combining above mentioned isolation strategies with 12c multitenant option to achieve excellent efficiency. But again if you are required to isolate everything at OS level, virtualizing Exadata Machine using OVM is your only option. Even though Exadata VM’s are also great for consolidation but best strategy would be to combine VM’s with database native consolidation options like multitenant. Exadata VMs have good Isolation but have poor efficiency and higher maintenance. Virtualizing Exadata Machine should not be your standard built, you should always consider bear metal install over Virtualized Exadata install.

Managing Virtualized Exadata Machine

First thing you should know about managing Exadata VMs is that you can migrate Bare Metal Oracle cluster to OVM cluster. Conversion of Bear Metal to OVM can be achieve with 0 down time or minimum downtime, depend on various migration methods.

Memory: You can decrease or increase amount of memory allocation to user domain with proper planning. For example: if you want to decrease memory allocated to user domain , you should consider instance memory parameter and make sure you still have enough memory left for user domain to support SGA/PGA for all the running databases. Memory changes to user domain is not dynamic and will require restart of user domain.

 CPU: Similarly, you can increase and decrease number of vCPU’s assign to a user domain. You can also dynamically increase or decrease number of vCPU’s assign to user domain as long as it will exceed number of vCPU’s assign to that domain. Overprovisioning is possible but not recommended, it will require full understanding of workload on all the user domains.

Storage:  Like CPU and memory you can also increase size of underline storage for any user domains. You can add new logical volume, increase size of root file system and increase size of Oracle Grid or RDBMS files system. . You can even add a new mount, if you like to add another Oracle RDBMS home

Backup: With addition to all other backups you need to backup both management and user domain in virtualized Exadata environment. As a best practice backup destination should reside outside of the local machine.

Virtualizing Exadata Machine Using OVM

It’s been a while since Oracle started supporting OVM with Exadata Machine. It means you can virtualize your Exadata Machine using Oracle VM technology with no additional cost. Ideally, Oracle VM with Exadata machine should be use for database consolidation or isolation only but you are not restricted. You can use Exadata VMs to install other products like management or ETL tools but it’s not recommend to run major applications like SAP or EBS. Currently Exadata VM are only certified to Oracle enterprise Linux, may be in future Oracle will start supporting other operating systems. Please see below diagram to compare quarter rack Virtualized Exadata Machine with Bare Metal.

Virtualized Exadata 01

Virtualized Exadata machine architecture can be significantly different than the Bare Metal one. With Bare Metal install, you have one Oracle cluster for whole Exadata machine, unless you physically partition your Exadata machine. With Virtualized Exadata Machine you have 1 management domain (domO) and at least 1 user domain (domU) on each node, depending on the number of VM clusters being deployed. Management Domain (domO) is automatically created with 4 vCPU, 8 GB RAM, with no Oracle GRID and RDBMS software installed in it. Each user domain (domU) should be sized carefully based on the databases you are planning to host in future. With virtualized Exadata install, you only virtualize compute nodes and share storage between them. This also mean that all the VM clusters will have their own dedicated ASM disks groups and physical disks.


Exadata Install Process Got Much Simpler With Elastic Configuration

Recently I had performed an Exadata X-6 Bear Metal install and I thought it will be a good idea to share my experience with all of you. Exadata deployment is itself a mini project. You need to work with database /network  team to run OEDA configuration tool to generate all the configuration files. But for this blog, I will focus on the install part only. Before you can this proceed with Exadata Install process, you need to make sure you have run the check-IP script to verify all the DNS entries and IP addresses. It’s very important that you don’t start with install process without verify network configuration.

At this point you should have all the configuration and software files ready on a USB drive or on your laptop. Now you can use following steps to compete Exadata install.

Step 1: You can use Ethernet port 48 to connect to Exadata network. Mostly likely, there is going to blue Ethernet cable you can use for this step. Modify your local area network settings as per following configuration.

Capture 1

Step 2: Connection to Exadata node8 ( using ssh and run ibhosts. You should see all the nodes with their addresses and elasticNode keyword with all nodes.

Capture 2

Step 3: cd into /opt/oracle.supporTools and run –free –reclaim on node 8 ( and node 10 (

Capture 4

Step 4: run reclaim disk check on both nodes. ( –check) and make sure you get the following output.

Capture 5

Node: – Please make sure you run step 3, 4 on all Compute nodes (node8, node10).

Step 5: Plugin the USB and mount it using following procedure. This step is optional, if you are planning to copy software through ssh.

for x in `ls -1 /sys/block`; do udevadm info –attribute-walk 

–path=/sys/block/$x | grep \

 -iq ‘DRIVERS==”usb-storage”‘; if [ $? -eq 0 ] ; then echo /dev/${x}1; \

fi ; done

Capture 6

Step 6: Unzip OEDA tool to /opt/oracle.SupportTools/onecommand directory , then copy all the configuration files to /opt/oracle.SupportTools/onecommand/linux-x64.

Capture 11

Step 7 : Copy /opt/oracle.SupportTools/onecommand/ to node10 ( 2nd dbnode or all compute nodes )

Capture 19

Step 8: Apply Elastic configuration from node8. This is a very important step, you only need to run this step from 1 node only. This step will use your Exadata configuration file to assign new IP addresses and reboot all exadata nodes including storage nodes.

  • cd /opt/oracle.SupportTools/onecommand/linux-x64
  • ./ -cf customer_name-configFile.xml

Capture 12

Step 9: Connect to Exadata using new IP address. It’s important to understand that your machine has new ip addresses now. Change your local area settings and assign same IP/MASK as your PDU01 . You can find this information in your OEDA installation template.

Capture 13

Step 10: Connect through ssh and run ibhosts to see if new ip addresses has been assign. Its important that you don’t see any elasticNode keyword on the screen, otherwise your elastic configuration is not completed.

Capture 14

Step 11: Move onecommand directory under /u01 on both dbnodes.

Capture 15

Step 12: Copy all the required software and patches to (/u01/onecommand/linux-x64/WorkDir). You only have to do this on node1. Again you can find complete list of software in your Exadata installation template.

Capture 21

Step 13: Run check-IP script one more time to validate all the network settings.

/opt/oracle.SupportTools/onecommand/linux-x64/ -cf customer_name-configFile.xml

Step 14: If you don’t get any error during check-IP process, you can proceed with Exadata install. At this point, there are 19 steps to complete Exadata software install. You can go thought with them 1 by 1 or you can run all of them together. I will strongly recommend doing it 1 by 1, it will help you troubleshoot any issue during the install process. You can using following command to list all the steps.

  • ./ -cf customer_name-configFile.xml -l

Capture 16

Step 15: You can start going through each steps 1 by 1 using following command. I will only post screen shots for step 1, 19

(Step 1 Screen Shot)

Capture 17

(Step 19 Screen Shot)

Capture 19b

Final Step: Please login to all nodes (Compute & Storage) and change the root password.

Capture 20


Installing Additional RDBMS Home on Exadata Machine

As you may already know that you can have multiple RDBMS Versions/Homes running on Exadata machine. You can request to install multiple homes by Oracle ACS or Oracle Certified Partner with your original Exadata deployment. But in case you end up needing to add Oracle RDBMS on Exadata machine, you can follow below steps to add RDBMS home successfully. In my case I already had a home but I needed to add home.

Step 1 : Create new local storage mount for new Oracle RDBMS home. Steps to add local storage can be little different if your Exadata Machine is virtualized, please follow Exadata maintenance guide to create a local storage mount. You can also use existing storage mount but its best practice to have separate mount for maintenance and support reasons.

Upload 01

Step 2 : Identify RDBMS Software and Patches, You can use following 3 methods to get the complete list.

  • Open SR with Oracle
  • Get complete list from already installed home
  • Use OEDA utility

Upload 02

Step 3 : Download Software & Patches to Exadata Machine, I used latest OEDA utility to get the complete list of following software and patches.

  • RDBMS Software
  • Latest Opatch Utility
  • Latest Exadata Bundle Patche April 2016
  • ORACLE DATABASE Overlay Patch for Bug#23200778
  • Oracle JavaVM Component PSU

Upload 03

Step 4 :Install Oracle Software to new database mount. I will not get into details here, since most of you are already familiar with RDBMS Cluster install.

Upload 04

Step 5 : Implement RDS protocol over Infiniband network

  • Set ORACLE_HOME environment variable
  • cd $ORACLE_HOME/rdbms/lib
  • make -f $ORACLE_HOME/rdbms/lib/ ipc_rds ioracle


Step 6 :  Install latest Opatch Utility: There are many ways to you install latest Optach Utility, I simply created a text (dbs_groups) with all the dbnodes name in it and install opatch utility using dcli command. You do need to copy zip file to all the nodes.

  • Create dbs_group
  • Scp Node2: /u01/app/oracle/product/software/
  • dcli -l oracle -g dbs_group unzip -oq -d /u01/app/oracle/product/ /u01/app/oracle/product/software/

Upload 05

Step 7 : Check version of new dcli utility and verify if patch is installed successfully

  • dcli -l oracle -g dbs_group /u01/app/oracle/product/ version

Upload 06

Step 8 : Apply Bundle April patch 22899777: I am applying April bundle patch, so I can be consistent with my other database home. I already have GRID (12c) and RDBMS (12c) home running from this machine and I wanted to be at same patch level as them. Please follow below step to Install patch database home only.  You will be applying patch locally, so you need to copy patch to all the nodes and repeat same process for all nodes in cluster.

  • scp to other nodes
  • unzip
  • export ORACLE_HOME=/u01/app/oracle/product/
  • export PATH=$PATH:/u01/app/oracle/product/
  • $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/app/oracle/product/software/22899777/22738760
  • $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/app/oracle/product/software/22899777/22502549/custom/server/22502549
  • $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseDir /u01/app/oracle/product/software/22899777/22738760
  • $ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseDir /u01/app/oracle/product/software/22899777/22502549/custom/server/22502549
  • change Permision on following files :
  • chmod 775 /u01/app/oracle/product/software/22899777/22502549/custom/scripts/*
  • /u01/app/oracle/product/software/22899777/22502549/custom/scripts/ -dbhome $ORACLE_HOME
  • $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/app/oracle/product/software/22899777/22738760
  • $ORACLE_HOME/OPatch/opatch napply -oh $ORACLE_HOME -local /u01/app/oracle/product/software/22899777/22502549/custom/server/22502549
  • /u01/app/oracle/product/software/22899777/22502549/custom/scripts/ -dbhome $ORACLE_HOME
  • $ORACLE_HOME/OPatch/opatch lsinventory

upload 07

Step 9 : Install ORACLE DATABASE Overlay Patch for Bug#23200778 for Linux-x86-64 Platforms

  • unzip
  • cd 23200778
  • opatch prereq CheckConflictAgainstOHWithDetail -ph ./
  • cd 23200778
  • opatch apply
  • $ORACLE_HOME/OPatch/opatch lsinventory

upload 08

Step 10 : Install Oracle JavaVM Component Database PSU (Apr2016)

  • unzip p22674697_112040_Linux-x86-64
  • cd 22674697
  • opatch prereq CheckConflictAgainstOHWithDetail -ph ./
  • cd 22674697
  • opatch apply
  • $ORACLE_HOME/OPatch/opatch lsinventory

upload 09

Step 11 : List opatch Ispatches

upload 10