Checking Exadata Image Info

Login to any storage cell & compute node using root user & run imageinfo command.

Checking Storage cell image :

login as: root
root@XX.XXX.XX.XX's password:
Last login: Mon Oct 16 17:13:57 2017 from cellnode1

[root@cellnode1 ~]# imageinfo

Kernel version: 4.1.12-61.47.1.el6uek.x86_64 #2 SMP Fri Jun 23 19:43:18 PDT 2017 x86_64
Cell version: OSS_12.2.1.1.2_LINUX.X64_170714
Cell rpm version: cell-12.2.1.1.2_LINUX.X64_170714-1.x86_64

Active image version: 12.2.1.1.2.170714
Active image kernel version: 4.1.12-61.47.1.el6uek
Active image activated: 2017-08-15 11:16:34 -0400
Active image status: success
Active system partition on device: /dev/md6
Active software partition on device: /dev/md8

Cell boot usb partition: /dev/sdm1
Cell boot usb version: 12.2.1.1.2.170714

Inactive image version: 12.2.1.1.1.170419
Inactive image activated: 2017-05-20 03:20:27 -0700
Inactive image status: success
Inactive system partition on device: /dev/md5
Inactive software partition on device: /dev/md7

Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
Inactive grub config for the rollback: /boot/grub/grub.conf.inactive
Inactive kernel version for the rollback: 4.1.12-61.33.1.el6uek.x86_64
Rollback to the inactive partitions: Possible

Checking compute node image :

[root@dbnode1 ~]# imageinfo

Kernel version: 4.1.12-61.47.1.el6uek.x86_64 #2 SMP Fri Jun 23 19:43:18 PDT 2017 x86_64
Image kernel version: 4.1.12-61.47.1.el6uek
Image version: 12.2.1.1.2.170714
Image activated: 2017-08-15 15:44:12 -0400
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1

Verify Exadata Machine Configuration

Hello Everyone! I was recently task to perform a 360 health review and decide to share my experience with my readers. Part of Exadata 360 review, I performed detail review of Exadata configuration by verifying following 50 items. Initial Exadata Deployment usually don’t require verify following item if you have old deployment of Exadata machine, you might want to review following items at least once a year.

 

  1. Primary and standby databases should NOT reside on the same IB Fabric
  2. Use hostname and domain name in lower case
  3. Verify ILOM Power Up Configuration
  4. Verify Hardware and Firmware on Database and Storage Servers
  5. Verify InfiniBand Cable Connection Quality
  6. Verify Ethernet Cable Connection Quality
  7. Verify InfiniBand Fabric Topology (verify-topology)
  8. Verify InfiniBand switch software version is 1.3.3-2 or higher
  9. Verify InfiniBand subnet manager is running on an InfiniBand switch
  10. Verify celldisk configuration on flash memory devices
  11. Verify there are no griddisks configured on flash memory devices
  12. Verify griddisk count matches across all storage servers where a given prefix name exists
  13. Verify griddisk ASM status
  14. Verify InfiniBand is the Private Network for Oracle Clusterware Communication
  15. Verify Oracle RAC Databases use RDS Protocol over InfiniBand Network.
  16. Verify Database and ASM instances use same SPFILE
  17. Configure Storage Server alerts to be sent via email
  18. Configure NTP and Timezone on the InfiniBand switches
  19. Verify NUMA Configuration
  20. Verify Exadata Smart Flash Log is Created
  21. Verify Exadata Smart Flash Cache is Created
  22. Verify Exadata Smart Flash Cache status is “normal”
  23. Verify database server disk controllers use writeback cache
  24. Configure NTP and Timezone on the InfiniBand switches
  25. Verify that “Disk Cache Policy” is set to “Disabled”
  26. Verify Management Network Interface (eth0) is on a Separate Subnet
  27. Verify Platform Configuration and Initialization Parameters for Consolidation
  28. Verify all datafiles have “AUTOEXTEND” attribute “ON”
  29. Verify all “BIGFILE” tablespaces have non-default “MAXBYTES” values set
  30. Ensure Temporary Tablespace is correctly defined
  31. Enable auditd on database servers
  32. Verify AUD$ and FGA_LOG$ tables use Automatic Segment Space Management
  33. Use dbca templates provided for current best practices
  34. Gather system statistics in Exadata mode if needed
  35. Verify Hidden Database Initialization Parameter Usage
  36. Verify bundle patch version installed matches bundle patch version registered in database
  37. Verify service exachkcfg autostart status
  38. Verify database server file systems have “Maximum mount count” = “-1”
  39. Verify database server file systems have “Check interval” = “0”
  40. Set SQLNET.EXPIRE_TIME=10 in DB Home
  41. Verify /etc/oratab
  42. Verify all Database and Storage Servers are synchronized with the same NTP server
  43. Verify there are no failed diskgroup rebalance operations
  44. Verify the CRS_HOME is properly locked
  45. Verify db_unique_name is used in I/O Resource Management (IORM) interdatabase plans
  46. Verify Database Server Quorum Disks are used when beneficial
  47. Verify Oracle Clusterware files are placed appropriately
  48. Verify “_reconnect_to_cell_attempts=9” on database servers which access X6 storage servers
  49. Verify Flex ASM Cardinality is set to “ALL”
  50. Verify no Oracle Enterprise Linux 5 (el5) rpms exist on database servers running Oracle Linux (ol6)

 

Reference: Oracle Sun Database Machine Setup/Configuration Best Practices (Doc ID 1274318.1)

 

Using Oracle ZFS Storage Appliance for OLTP Workload

As it mentioned earlier that ZFS appliance is best suited to OLAP workload but you can use ZFS Appliance for Non-critical OLTP workload. By nature, Online transaction processing (OLTP) workloads tend to go after a small number of rows per IO request. Imagine a busy OLTP system where thousands of random IO’s going after a small amount of data, will require shorter response time to maintain and achieve healthy response time. That means you should use utilize ZFS read & write flash SSD’s to get reasonable performance for your OLTP applications. Similar to OLAP database workload, it is recommended to use Bigfile tablespace and Oracle Direct NFS. But unlike OLAP workload its best to use Advance Row Compression to optimize IO response time and memory utilization.

 

Recommended OLAP Database Layout on ZFS Storage
Oracle Files Record Size Sync Write Bias Read Cache Comp

ression

Example Share Name
Datafiles 128K throughput do not use cache devices LZ4 data
Temp 128K latency do not use cache devices LZ4 temp
Archive Logs 1M throughput do not use cache devices LZ4 archive
RMAN Backup 1M throughput do not use cache devices LZ4 backup

Source: Oracle 

Again, let’s look into rest of the recommendation of mapping different databases files to their recommended location and record sizes. Data files, temp files, and archive logs can be placed ton ZFS share with respective record sizes 32K, 128K and 1M.  Similarly, it’s best practices to place online redo logs and control files in the default Fast Recovery Area (FRA) on the preconfigured Exadata ASM diskgroup.

 

Using Oracle ZFS Storage Appliance for OLAP Workload

OLAP database workloads are mostly SQL queries going after a large chunk of tables or batch process loading bulk of data overnight. You might be loading Terabytes of data to your data warehouse system but those will not be considered random DML statements, hence critical workload will be users only query supporting DSS. This very true nature of DDS can be a good fit for databases compression including HCC. When it comes to using database compression one must understand that HCC will require some maintenance work to keep database HCC compress and it does not support OLTP compression. You should only use compression if you are cloning or duplicating a databases environment which already uses database compression so it can be a true representation of source environment. Additionally, you have an option to both tablespace format traditional big file tablespace but it’s recommended to user big file tablespace to achieve better performance, flexibility, and capacity. Finally, remember to use Oracle Direct NFS as it’s not recommended to use Oracle Intelligent Storage Protocol (OISP) for OLAP workload.

 

Recommended OLTP Database Layout on ZFS Storage
Oracle Files Record Size Sync Write Bias Read Cache Compression Example Share Name Storage Profile
Datafiles 32K latency all data and metadata LZ4 data Mirrored (NSPF)
Temp 128K latency do not use cache devices LZ4 temp Mirrored (NSPF)
Archive logs 1M throughput do not use cache devices LZ4 archive Mirrored (NSPF)
RMAN Backup 1M throughput do not use cache devices LZ4 backup Mirrored (NSPF)

Source: Oracle

Now let’s look into rest of the recommendation mapping different databases files to their recommended location and record sizes. Data files, temp files, and archive logs can be placed ton ZFS share with respective record sizes 128K, 128K, and 1M.  Even though you have the option to place redo logs and control on ZFS share, it’s best practices to place online redo logs and control files in the default Fast Recovery Area (FRA) on the preconfigured Exadata ASM diskgroup. It is hard to justify placing these two type of sensitive database files on ZFS share as they will not take much space and require high latency.

What is so special about Oracle ZFS Storage Appliance?

  1. Oracle ZFS provides extreme network bandwidth with 10G and InfiniBand connection with built-in network redundancy.
  2. Oracle ZFS provides extreme network bandwidth with 10G and InfiniBand connection with built-in network redundancy.
  3. Oracle ZFS ensure data integrity using feature like copy-on-write, metadata check summing, detect silent data corruption and errors correction before it is too late.
  4. Oracle ZFS uses an Oracle Intelligent Storage Protocol (OISP) to uniquely identify different types databases IO request and help you effectivity addressing performance bottlenecks using built-in Analytics.
  5. Oracle ZFS is extremely easy to manage using native web management interface and provides integration option with Oracle Enterprise Manager.
  6. Oracle ZFS support full range to compression options and tightly integrated with Oracle database to provide Hybrid Columnar Compression (HCC) which is only available on Oracle storage.
  7. Oracle ZFS is tightly integrated with Oracle database which helps achieve extreme backup rate up to 42 TB/hr and restore rate up 55 TB/hr.
  8. Oracle ZFS storage supports highly redundant and scalable InfiniBand architecture which can be seamlessly integrated with Oracle Exadata Machine to provide cost-effective storage option.
  9. Oracle ZFS appliance is integrated with Oracle RMAN to provide up to 2000 concurrent threads evenly distributed across many channels spread across multiple controllers.
  10. Oracle ZFS uses Oracle Direct NFS to reducing CPU and memory overhead by bypassing the operating system and writing buffers directly to user space.
  11. Oracle ZFS support 1MP record size to reduce number of IOPS that are required to disk, preserves the I/O size from RMAN buffers to storage, and improves performance of large block sequential operations.