Monitor Oracle Cloud Databases Using DBaaS Monitor

Oracle DBaaS Monitor provides monitoring and management of the Oracle database and listener on Oracle Database Cloud Service. It is not to say that DBaaS Monitor is as good as Oracle OEM but still it does a great job monitoring databases and Compute nodes. Additionally , it also provide some management capabilities. Please see below complete list of DBaaS capabilities and how to access it.

Database Monitoring Capabilities:

  • Tablespace usage and storage information
  • Database Listener Information and status
  • The alert log, with log searching capabilities
  • Open user sessions information like last SQL statement, explain plan, waits, contention, etc
  • A list of initialization parameters, with the ability to change parameter values, both in memory and in the SPFILE.
  • Database Backup Information
  • Database waits information, with log searching capabilities

Compute Node Monitoring Capabilities:

  • CPU utilization
  • Memory Usage information
  • Storage Information
  • OS process information

Management Capabilities:

  • Start up and shut down the database instance
  • Open and close a pluggable database
  • Create and drop a pluggable database
  • Plug in and unplug a pluggable database
  • Clone a pluggable database
  • Start and stop the listener

How to Access DbaaS Monitor:  

  1. Make Sure httpssl port (443) has been unblocked

  1. Open the Oracle Database Cloud Service

  1. Select Open DBaaS Monitor Console.

 

 

  1. Enter dbaas_monitor as the user name and the password specified during the database deployment creation process.

 

 

Supported Machine Images for Oracle Compute Cloud Service

A machine image is a template of a virtual hard disk of a specific size with an installed operating system. You need a machine images to create virtual machine instances in Oracle Compute Cloud Service. Oracle Compute Cloud Service supported many operating system images, some of them are provided by Oracle itself.  Oracle partners also provide operating system images which are compatible with Oracle Compute Cloud Service. Please see below complete list of options and images, available to you to create Oracle compute instance.

Option 1: Oracle provided images

Operating System Versions
Oracle Linux Oracle Linux 6.4, 6.6, 6.8, 7.1 and 7.2
Microsoft Windows 2008 R2

2012 R2

Oracle Solaris Oracle Solaris 11.3

Option 2: Oracle Partner provided images.

Operating System Versions
Ubuntu Ubuntu Server 14.04-LTS amd64

Ubuntu Server 15.10 amd64

Ubuntu Server 12.04-LTS amd64

CentOS CentOS 6 and 7
Debian Debian 8 (Jessie)
SUSE SUSE Linux Enterprise Server 11

SUSE Linux Enterprise Server 12

 

Option 3: Build your own Machine Image

You can build private images using x86, 64-bit versions of the following operating systems:

Operating System Versions
Oracle Linux 6.4 UEK3 and UEK4

6.6 UEK3 and UEK4

6.7 UEK4 and UEK4

6.8 UEK3

7.1 UEK3

7.2 UEK3

Oracle Solaris Oracle Solaris 11.3

 

Exadata Deployment Error : CELL-02559

While working on deploying Exadata for one of my client, I got following errors during deployment step # 2. Error clearly states that there is communication error between MS and CELLSRV. In my case this error was somewhat miss leading. After investigating for few hours, I found that ( /opt/oracle/cell/cellsrv/deploy/config/cellinit.ora ) files on Exadata storage cells contain wrong IP addresses. You need to update cellinit.ora files with correct ip addresses and restart all cell services to resolve this error.  You can use following steps to verify and resolve this issue.

[root@XXXX-db01linux-x64]# ./install.sh -cf client.xml -s 2 
 Initializing 
 Executing Update Nodes for Eighth Rack 
 running setup on: XXXX-db02 
 Completed copying resourcecontrol file to XXXX-cel02-adm 
 running setup on: cvdp2-cel02-adm 
 XXX-cel02-adm needs total CPU cores set from 20 to 10 
 ERROR: 
 CELL-02559: There is a communication error between MS and CELLSRV. 
 Error: Errors occured... 
 Collecting diagnostics... 
 ERROR: 

  1. Verify cell services using (list cell detail) on all nodes.

Note : – As you can see cellsrv service is stopped and causing original error. Also make a note of IP addresses, we need to verify them with cellinit.ora file and private InfiniBand ip addresses.

  1. Restart all cell services

You can try re-starting all services and see you get any errors

  1. Check InfiniBand IP addresses using ifconfig command

  1. Check contents of cellinit.ora

Note : – Make sure these are correct InfiniBand private IP addresses. In my case they were wrong.

  1. Update cellinit.ora with correct InfiniBand IP addresses

Make sure to backup cellinit.ora file and Edit cellinit.ora file with correct IP addresses using vi editor.

  1. Restart all cell services

Now you can restart all services and hopefully , you will not run into any issues this time.

alter cell restart services all

 

 

 

How to Obtain Exadata Serial Numbers

There might be time when you will need to extract Serial number information from Exadata Hardware. There are 6 major components in Exadata Machine, some organization needs serial number to maintain hardware inventory. All the serial numbers are present in Customer Information sheet generated by Oracle but sometime it is hard map them to particular hardware component.  You can using following blog to obtain Exadata serial number of following 6 components.

  1. Exadata Rack
  2. Compute Nodes
  3. Storage Cells
  4. InfiniBand Switches
  5. Ethernet Switch
  6. PDU’s

 

Notes: – if you have configured SSH equivalency between root users, you can run following command on all compute nodes or all storage nodes at the same time. For this blog I wanted to keep it simple, so I will be login into each components to obtain serial number.

 

Exadata Rack

 

You can login to any compute node to obtain Exadata Rack serial number and Exadata rack serial number is unique for each Exadata Machine, so you should get the same serial number from all the nodes.

[root@XXXX-db01 ~]# ipmitool sunoem cli "show /SP system_identifier"

 

Note: – if you have configured SSH equivalency between all compute nodes, you can run following command on all compute nodes using DCLI utility at the same time.

 

Compute Nodes

 

Each Compute node will have a unique serial number, you should run following command on each compute nodes. You can have anywhere between 2-8 Exadata Compute nodes, depending on your Exadata configuration.

[root@XXXX-db01 ~]# dmidecode -s system-serial-number

 

Note: – if you have configured SSH equivalency between all compute nodes, you can run following command on all compute nodes using DCLI utility at the same time.

 

Storage Cells

Each Storage Cell will have a unique serial number, you should run following command on each storage cell. You can have anywhere between 3-14 Exadata Storage cells, depending on your Exadata configuration.

[root@XXXX-cel01-adm ~]# dmidecode -s system-serial-number

 

Note: – if you have configured SSH equivalency between all storage cells, you can run following command on all storage cells using DCLI utility at the same time.

 

InfiniBand Switches

 

You have multiple InfiniBand switches in your Exadata Machine. You need login to them individually as root user to obtain Serial numbers.

 

[root@XXXX-sw-ibb01 ~]# showfruinfo

 

Ethernet switch

 

You can have multiple Ethernet switches in your Exadata Machine. You need login to them individually as admin user to obtain Serial numbers. Make to execute enable command before you run show module to obtain serial number.

[root@XXXX-db01 ~]# ssh admin@XXXX-sw-adm01
Password:
XXXX-sw-adm01>enable
Password:
XXXX-sw-adm01# show module

 

 

PDU

You have multiple PDU’s in your Exadata Machine. You need login to them individually as root user to obtain Serial numbers. You need to open PDU admin console using their IP addresses and click module info tab on left hand corner to obtain serial number.

 

Enable Oracle TDE for Production Databases with no Downtime

Oracle now offers offline in place conversion of datafiles to TDE. Given if you have a Data Guard in place for your production database, you can encryption your production database using TDE with minimum downtime. This feature is only available for 11.2.0.4 and 12.1.0.2 Oracle Database versions and you will need to apply a patch 23315889 to enable this functionality. Once installed, the patch enables offline, in-place TDE conversion of data files at a Data Guard standby with a DDL command instead of having to reload data which can be time consuming, tedious, and in some cases complex

This feature is very important if you have a huge production database requiring 24/7 availability, both unplanned outages and planned downtime is a significant concern. Also it is important to note that if you planning to move your production database to any public cloud, you might be required to encrypt your production database.

You can easily encrypt your production database with standby with no downtime using following 10 steps

  1. Make sure primary and standby databases are in sync.
  2. Create the encryption wallet, and set the master key.
  3. Copy the wallet files to all nodes in the configuration (Oracle RAC primary nodes and all standby nodes).
  4. Place the standby in a mounted state with recovery stopped.
  5. On the standby: Encrypt data files in-place and in parallel.
  6. On the standby: Restart redo apply and catch up.
  7. Execute a Data Guard switchover making the encrypted standby the new primary and the unencrypted primary the new standby.
  8. On the NEW standby: Place the new standby database in a mounted state with recovery stopped.
  9. On the NEW standby: Encrypt data files in-place and in parallel.
  10. On the NEW standby: Restart redo apply and catch up.

 

 

 

 

Importance of running checkip.sh during Exadata Deployment Process

Checkip.sh file is generated by OEDA as part Exadata Deployment process. Checkip.sh script is use to ensure there are no IP address conflicts between the existing network and Oracle Exadata Rack. You can never overestimate the importance of running checkip.sh during Exadata Deployment Process. If you are missing a DNS entry or even there a typo in your configuration_file, you will run into issues during Exadata deployment process. Make sure you run checkip script before you connect Exadata machine to your network, even if you have already ran it during the planning phase.

This script is designed to check following conditions.

  1. IP addresses should respond to ping request
  2. IP addresses should not respond to ping request
  3. Reserve IP addresses resolving to DNS entries

You can run checkip.sh script from any machine on the network as long as it’s on the same subnet as Exadata is deployed on. You can use following syntax to run checkip.sh scripts.

# ./checkip.sh -cf configuration_file

Latest Version of checkip consist of following 11 sections and you should see following contents in you checkip.out file.

  1. NAME
Processing section NAME
GOOD : Name Server 162.XX.XX.XXX is responding to resolve request for prefix-db01.domain.com
  1. NTP
Processing section NTP
GOOD : 10.XX.XX.XX is responding to time server query
  1. GATEWAY
Processing section GATEWAY
GOOD : 10.XX.XX.XX is responding to ping request
  1. SCAN
Processing section SCAN
GOOD : prefix-scan.domain.com is forward resolving to 3  IP addresses [10.XX.XX.XXX, 10.XX.XX.XXX, 10.XX.XX.XXX]
  1. VIP
Processing section VIP
GOOD : prefix-db02-vip.domain.com is forward resolving to 10.XX.XX.XXX
  1. Compute
Processing section COMPUTE
GOOD : prefix-db01.domain.com is forward resolving to 10.XX.XX.XXX
  1. BACKUP
Processing section BACKUP
GOOD : prefix-db01-bk.domain.com is forward resolving to 10.XX.XX.XX
  1. CELL
Processing section CELL
GOOD: prefix-cel01-adm.domain.com is forward resolving to 10.XX.XX.XX
  1. FACTORY
Processing section FACTORY
GOOD : 192.168.1.1  is not responding to ping request
  1. Switches
Processing section SWITCHES
GOOD : prefix-sw-adm01.domain.com is forward resolving to 10.XX.XX.XX
  1. ILOM
Processing section ILOMS
GOOD : prefix-db01-ilo.domain.com is forward resolving to 10.XX.XX.XX

 

Note : – If this Oracle Exadata rack is to be added to an existing installation of Exadata machine, then run the CheckIP utility from an existing machine so that it can identify in-use IP addresses in the fabric. Not identifying existing IP addresses may cause IP collisions after installation of the new rack.

Network / IP Address requirements for Exadata (Bare Metal)

If you have decided to buy Exadata Machine to run your Oracle Databases, then your Exadata deployment journey will start with network setup and configuration. For this blog, I would use Exadata ¼ Rack but same can apply for Exadata eight Rack. It is important to remember that there is a huge different between network / IP Address requirements for Bare Metal and virtualized Exadata Machine, this post only covers Bare Metal Exadata network configuration.

Basically, there are 4 different types of Networks in Exadata. You will need around 32+ ip address, depending on your backup network configuration and spine switch.  Subnet mask can be shared between Admin, Client Access and Backup network but Private InfiniBand needs to have a separate subnet mask

  1. Administration (Management) – 15 IP’s
  2. Client Access –  7 IP’s
  3. Private InfiniBand – 10 IP’s
  4. Additional ( Backup ) – 2 IP’s

 

Administration Network (Management Network) 

Each database server and Exadata Storage Server has two network interfaces for administration. One network provides management access to the operating system through the Ethernet interface, and the other network provides access to the Integrated Lights Out Manager (ILOM) through the ILOM Ethernet interface. All IP addresses and corresponding hostnames should be registered in DNS

 

Components Total IP’s
Database Server 2
Storage Server 3
Leaf Switch 2
PDU Switch 2
Admin Switch 1
ILOM Network 5
Total 15

Client Access Network

Applications access the database through the client Ethernet network using Single Client Access Name (SCAN) and Oracle RAC Virtual IP (VIP) addresses. All IP addresses and corresponding hostnames should be registered in DNS

 

Components Total IP’s
Database Server 2
VIP 2
SCAN IP 3
Total 7

Private InfiniBand Network

The InfiniBand network is the private network between database servers and storage servers. Oracle Database uses this network for Oracle RAC cluster interconnect traffic and for accessing data on Exadata Storage Servers. This non-routable network is fully contained in Oracle Exadata Database Machine, and does not connect to your existing network. This network is automatically configured during installation. All IP addresses and corresponding hostnames should be registered in DNS.

 

Components Total IP’s
Database Server 4
Storage Server 6
Total 10                            

Backup Network (Optional Network)

Database servers can connect to additional existing networks using the available open ports not used by the management and client networks.

 

Components Total IP’s
Database Server 2
Total 2                              

Conclusion

Network configuring is an important part of Exadata deployment, any mishaps or even a typo can cause delays and loss of revenue. Make sure you do it right the first time. Good Luck!

 

 

Removing Oracle RDBMS Software from Exadata Machine

It is important to understand that the procedure to uninstall Oracle RDBMS is same as if you uninstalling on non-exadata system. I decided to write this blog to provide some clarity around uninstall process and outlines steps to safely remove RDBMS home from Exadata Machine. There can be a rare situation when you are required to uninstall Oracle RDBMS on EXADATA machine. For example if have upgraded your databases from 11g to 12c and you no longer require old RDBMS software on the Machine.

 

As per Oracle

“Oracle recommends that you use the deinstallation tool to remove the entire Oracle home associated with the Oracle Database, Oracle Clusterware, Oracle ASM, Oracle RAC, or Oracle Database client installation. Oracle does not support the removal of individual products or components.”

In any case you can use following steps to uninstall RDBMS home from Oracle EXADATA machine.

 

  1. Set correct ORACLE_HOME and PATH variable to target database Home.

 

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export PATH=$PATH:/u01/app/oracle/product/11.2.0/dbhome_1/bin

 

  1. Check /etc/oratab for database homes and make sure there are no database running from that home

 

  1. Make sure listener is not running from Target RDBMS Home
srvctl config listener -a

 

  1. Check if there are no processes running from target ORACLE_HOME and ORACLE_HOME/bin
cd $ORACLE_HOME
/sbin/fuser -u *
cd $ORACLE_HOME/bin
/sbin/fuser -u *

Note : – If there are any processes running from target RDBMS location , you will see following output.

 

  1. Backup target RDBMS homes on all Exadata nodes using tar

 

tar -cvf /Exadata/software/backup_1120_node1 /u01/app/oracle/product/11.2.0

 

  1. Change to $ORACLE_HOME/deinstall and deinstall utility
cd $ORACLE_HOME/deinstall

./deinstall -o /XXXXXXXX/software/


 

 

  1. Remove leftover directory using rmdir