How to Obtain Exadata Serial Numbers

There might be time when you will need to extract Serial number information from Exadata Hardware. There are 6 major components in Exadata Machine, some organization needs serial number to maintain hardware inventory. All the serial numbers are present in Customer Information sheet generated by Oracle but sometime it is hard map them to particular hardware component.  You can using following blog to obtain Exadata serial number of following 6 components.

  1. Exadata Rack
  2. Compute Nodes
  3. Storage Cells
  4. InfiniBand Switches
  5. Ethernet Switch
  6. PDU’s

 

Notes: – if you have configured SSH equivalency between root users, you can run following command on all compute nodes or all storage nodes at the same time. For this blog I wanted to keep it simple, so I will be login into each components to obtain serial number.

 

Exadata Rack

 

You can login to any compute node to obtain Exadata Rack serial number and Exadata rack serial number is unique for each Exadata Machine, so you should get the same serial number from all the nodes.

[root@XXXX-db01 ~]# ipmitool sunoem cli "show /SP system_identifier"

 

Note: – if you have configured SSH equivalency between all compute nodes, you can run following command on all compute nodes using DCLI utility at the same time.

 

Compute Nodes

 

Each Compute node will have a unique serial number, you should run following command on each compute nodes. You can have anywhere between 2-8 Exadata Compute nodes, depending on your Exadata configuration.

[root@XXXX-db01 ~]# dmidecode -s system-serial-number

 

Note: – if you have configured SSH equivalency between all compute nodes, you can run following command on all compute nodes using DCLI utility at the same time.

 

Storage Cells

Each Storage Cell will have a unique serial number, you should run following command on each storage cell. You can have anywhere between 3-14 Exadata Storage cells, depending on your Exadata configuration.

[root@XXXX-cel01-adm ~]# dmidecode -s system-serial-number

 

Note: – if you have configured SSH equivalency between all storage cells, you can run following command on all storage cells using DCLI utility at the same time.

 

InfiniBand Switches

 

You have multiple InfiniBand switches in your Exadata Machine. You need login to them individually as root user to obtain Serial numbers.

 

[root@XXXX-sw-ibb01 ~]# showfruinfo

 

Ethernet switch

 

You can have multiple Ethernet switches in your Exadata Machine. You need login to them individually as admin user to obtain Serial numbers. Make to execute enable command before you run show module to obtain serial number.

[root@XXXX-db01 ~]# ssh admin@XXXX-sw-adm01
Password:
XXXX-sw-adm01>enable
Password:
XXXX-sw-adm01# show module

 

 

PDU

You have multiple PDU’s in your Exadata Machine. You need login to them individually as root user to obtain Serial numbers. You need to open PDU admin console using their IP addresses and click module info tab on left hand corner to obtain serial number.

 

Enable Oracle TDE for Production Databases with no Downtime

Oracle now offers offline in place conversion of datafiles to TDE. Given if you have a Data Guard in place for your production database, you can encryption your production database using TDE with minimum downtime. This feature is only available for 11.2.0.4 and 12.1.0.2 Oracle Database versions and you will need to apply a patch 23315889 to enable this functionality. Once installed, the patch enables offline, in-place TDE conversion of data files at a Data Guard standby with a DDL command instead of having to reload data which can be time consuming, tedious, and in some cases complex

This feature is very important if you have a huge production database requiring 24/7 availability, both unplanned outages and planned downtime is a significant concern. Also it is important to note that if you planning to move your production database to any public cloud, you might be required to encrypt your production database.

You can easily encrypt your production database with standby with no downtime using following 10 steps

  1. Make sure primary and standby databases are in sync.
  2. Create the encryption wallet, and set the master key.
  3. Copy the wallet files to all nodes in the configuration (Oracle RAC primary nodes and all standby nodes).
  4. Place the standby in a mounted state with recovery stopped.
  5. On the standby: Encrypt data files in-place and in parallel.
  6. On the standby: Restart redo apply and catch up.
  7. Execute a Data Guard switchover making the encrypted standby the new primary and the unencrypted primary the new standby.
  8. On the NEW standby: Place the new standby database in a mounted state with recovery stopped.
  9. On the NEW standby: Encrypt data files in-place and in parallel.
  10. On the NEW standby: Restart redo apply and catch up.

 

 

 

 

Importance of running checkip.sh during Exadata Deployment Process

Checkip.sh file is generated by OEDA as part Exadata Deployment process. Checkip.sh script is use to ensure there are no IP address conflicts between the existing network and Oracle Exadata Rack. You can never overestimate the importance of running checkip.sh during Exadata Deployment Process. If you are missing a DNS entry or even there a typo in your configuration_file, you will run into issues during Exadata deployment process. Make sure you run checkip script before you connect Exadata machine to your network, even if you have already ran it during the planning phase.

This script is designed to check following conditions.

  1. IP addresses should respond to ping request
  2. IP addresses should not respond to ping request
  3. Reserve IP addresses resolving to DNS entries

You can run checkip.sh script from any machine on the network as long as it’s on the same subnet as Exadata is deployed on. You can use following syntax to run checkip.sh scripts.

# ./checkip.sh -cf configuration_file

Latest Version of checkip consist of following 11 sections and you should see following contents in you checkip.out file.

  1. NAME
Processing section NAME
GOOD : Name Server 162.XX.XX.XXX is responding to resolve request for prefix-db01.domain.com
  1. NTP
Processing section NTP
GOOD : 10.XX.XX.XX is responding to time server query
  1. GATEWAY
Processing section GATEWAY
GOOD : 10.XX.XX.XX is responding to ping request
  1. SCAN
Processing section SCAN
GOOD : prefix-scan.domain.com is forward resolving to 3  IP addresses [10.XX.XX.XXX, 10.XX.XX.XXX, 10.XX.XX.XXX]
  1. VIP
Processing section VIP
GOOD : prefix-db02-vip.domain.com is forward resolving to 10.XX.XX.XXX
  1. Compute
Processing section COMPUTE
GOOD : prefix-db01.domain.com is forward resolving to 10.XX.XX.XXX
  1. BACKUP
Processing section BACKUP
GOOD : prefix-db01-bk.domain.com is forward resolving to 10.XX.XX.XX
  1. CELL
Processing section CELL
GOOD: prefix-cel01-adm.domain.com is forward resolving to 10.XX.XX.XX
  1. FACTORY
Processing section FACTORY
GOOD : 192.168.1.1  is not responding to ping request
  1. Switches
Processing section SWITCHES
GOOD : prefix-sw-adm01.domain.com is forward resolving to 10.XX.XX.XX
  1. ILOM
Processing section ILOMS
GOOD : prefix-db01-ilo.domain.com is forward resolving to 10.XX.XX.XX

 

Note : – If this Oracle Exadata rack is to be added to an existing installation of Exadata machine, then run the CheckIP utility from an existing machine so that it can identify in-use IP addresses in the fabric. Not identifying existing IP addresses may cause IP collisions after installation of the new rack.

Network / IP Address requirements for Exadata (Bare Metal)

If you have decided to buy Exadata Machine to run your Oracle Databases, then your Exadata deployment journey will start with network setup and configuration. For this blog, I would use Exadata ¼ Rack but same can apply for Exadata eight Rack. It is important to remember that there is a huge different between network / IP Address requirements for Bare Metal and virtualized Exadata Machine, this post only covers Bare Metal Exadata network configuration.

Basically, there are 4 different types of Networks in Exadata. You will need around 32+ ip address, depending on your backup network configuration and spine switch.  Subnet mask can be shared between Admin, Client Access and Backup network but Private InfiniBand needs to have a separate subnet mask

  1. Administration (Management) – 15 IP’s
  2. Client Access –  7 IP’s
  3. Private InfiniBand – 10 IP’s
  4. Additional ( Backup ) – 2 IP’s

 

Administration Network (Management Network) 

Each database server and Exadata Storage Server has two network interfaces for administration. One network provides management access to the operating system through the Ethernet interface, and the other network provides access to the Integrated Lights Out Manager (ILOM) through the ILOM Ethernet interface. All IP addresses and corresponding hostnames should be registered in DNS

 

Components Total IP’s
Database Server 2
Storage Server 3
Leaf Switch 2
PDU Switch 2
Admin Switch 1
ILOM Network 5
Total 15

Client Access Network

Applications access the database through the client Ethernet network using Single Client Access Name (SCAN) and Oracle RAC Virtual IP (VIP) addresses. All IP addresses and corresponding hostnames should be registered in DNS

 

Components Total IP’s
Database Server 2
VIP 2
SCAN IP 3
Total 7

Private InfiniBand Network

The InfiniBand network is the private network between database servers and storage servers. Oracle Database uses this network for Oracle RAC cluster interconnect traffic and for accessing data on Exadata Storage Servers. This non-routable network is fully contained in Oracle Exadata Database Machine, and does not connect to your existing network. This network is automatically configured during installation. All IP addresses and corresponding hostnames should be registered in DNS.

 

Components Total IP’s
Database Server 4
Storage Server 6
Total 10                            

Backup Network (Optional Network)

Database servers can connect to additional existing networks using the available open ports not used by the management and client networks.

 

Components Total IP’s
Database Server 2
Total 2                              

Conclusion

Network configuring is an important part of Exadata deployment, any mishaps or even a typo can cause delays and loss of revenue. Make sure you do it right the first time. Good Luck!

 

 

Removing Oracle RDBMS Software from Exadata Machine

It is important to understand that the procedure to uninstall Oracle RDBMS is same as if you uninstalling on non-exadata system. I decided to write this blog to provide some clarity around uninstall process and outlines steps to safely remove RDBMS home from Exadata Machine. There can be a rare situation when you are required to uninstall Oracle RDBMS on EXADATA machine. For example if have upgraded your databases from 11g to 12c and you no longer require old RDBMS software on the Machine.

 

As per Oracle

“Oracle recommends that you use the deinstallation tool to remove the entire Oracle home associated with the Oracle Database, Oracle Clusterware, Oracle ASM, Oracle RAC, or Oracle Database client installation. Oracle does not support the removal of individual products or components.”

In any case you can use following steps to uninstall RDBMS home from Oracle EXADATA machine.

 

  1. Set correct ORACLE_HOME and PATH variable to target database Home.

 

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export PATH=$PATH:/u01/app/oracle/product/11.2.0/dbhome_1/bin

 

  1. Check /etc/oratab for database homes and make sure there are no database running from that home

 

  1. Make sure listener is not running from Target RDBMS Home
srvctl config listener -a

 

  1. Check if there are no processes running from target ORACLE_HOME and ORACLE_HOME/bin
cd $ORACLE_HOME
/sbin/fuser -u *
cd $ORACLE_HOME/bin
/sbin/fuser -u *

Note : – If there are any processes running from target RDBMS location , you will see following output.

 

  1. Backup target RDBMS homes on all Exadata nodes using tar

 

tar -cvf /Exadata/software/backup_1120_node1 /u01/app/oracle/product/11.2.0

 

  1. Change to $ORACLE_HOME/deinstall and deinstall utility
cd $ORACLE_HOME/deinstall

./deinstall -o /XXXXXXXX/software/


 

 

  1. Remove leftover directory using rmdir

 

Running Oracle Real Application Clusters (RAC) in Public Clouds

Good News! Oracle now Support Oracle RAC on Third party Clouds. One of the first vendors who utilized this policy was Amazon with the Amazon Web Services (AWS). Deployments made under this policy enable vendors to run the Oracle Database as part of their Infrastructure-as-a-Service (IaaS) offering as long as the infrastructure meets the installation prerequisites for the Oracle product being offered. A vendor can choose to provide additional deployment support for the Oracle Database or any other Oracle product in those environments, which in general are neither tested nor certified by Oracle. Most Third-Party Cloud vendors therefore choose to collaborate with Oracle on such support in order to improve the quality of the service offered.

 

As Per Oracle guide line:

“Oracle RAC is supported on all cloud environments supported by the Oracle Database, as long as the environment is able to provide the hardware, storage, and networking requirements as specified in the Oracle RAC and Grid Infrastructure documentation. With the exception of the Oracle Cloud, Oracle has not tested nor certified Oracle RAC in these environments.”

 

Oracle RAC is supported under following assumptions:

  1. Hardware, storage and networking requirements as specified in the Oracle RAC and Grid Infrastructure documentations are met
  2. Cloud infrastructure needs to provide shared storage
  3. Cloud infrastructure needs to provide multiple networks and ability to create a private, dedicated network

 

Caution: – It is possible to use local or server-based storage and make it appear as shared storage to the Oracle Database or create multiple virtual networks, while de facto there is only one physical network been provisioned. Such technologies are generally discouraged because they can have adverse effect on performance and availability, although they might be supported under Oracle’s policy.