Verify Exadata Machine Configuration

Hello Everyone! I was recently task to perform a 360 health review and decide to share my experience with my readers. Part of Exadata 360 review, I performed detail review of Exadata configuration by verifying following 50 items. Initial Exadata Deployment usually don’t require verify following item if you have old deployment of Exadata machine, you might want to review following items at least once a year.

 

  1. Primary and standby databases should NOT reside on the same IB Fabric
  2. Use hostname and domain name in lower case
  3. Verify ILOM Power Up Configuration
  4. Verify Hardware and Firmware on Database and Storage Servers
  5. Verify InfiniBand Cable Connection Quality
  6. Verify Ethernet Cable Connection Quality
  7. Verify InfiniBand Fabric Topology (verify-topology)
  8. Verify InfiniBand switch software version is 1.3.3-2 or higher
  9. Verify InfiniBand subnet manager is running on an InfiniBand switch
  10. Verify celldisk configuration on flash memory devices
  11. Verify there are no griddisks configured on flash memory devices
  12. Verify griddisk count matches across all storage servers where a given prefix name exists
  13. Verify griddisk ASM status
  14. Verify InfiniBand is the Private Network for Oracle Clusterware Communication
  15. Verify Oracle RAC Databases use RDS Protocol over InfiniBand Network.
  16. Verify Database and ASM instances use same SPFILE
  17. Configure Storage Server alerts to be sent via email
  18. Configure NTP and Timezone on the InfiniBand switches
  19. Verify NUMA Configuration
  20. Verify Exadata Smart Flash Log is Created
  21. Verify Exadata Smart Flash Cache is Created
  22. Verify Exadata Smart Flash Cache status is “normal”
  23. Verify database server disk controllers use writeback cache
  24. Configure NTP and Timezone on the InfiniBand switches
  25. Verify that “Disk Cache Policy” is set to “Disabled”
  26. Verify Management Network Interface (eth0) is on a Separate Subnet
  27. Verify Platform Configuration and Initialization Parameters for Consolidation
  28. Verify all datafiles have “AUTOEXTEND” attribute “ON”
  29. Verify all “BIGFILE” tablespaces have non-default “MAXBYTES” values set
  30. Ensure Temporary Tablespace is correctly defined
  31. Enable auditd on database servers
  32. Verify AUD$ and FGA_LOG$ tables use Automatic Segment Space Management
  33. Use dbca templates provided for current best practices
  34. Gather system statistics in Exadata mode if needed
  35. Verify Hidden Database Initialization Parameter Usage
  36. Verify bundle patch version installed matches bundle patch version registered in database
  37. Verify service exachkcfg autostart status
  38. Verify database server file systems have “Maximum mount count” = “-1”
  39. Verify database server file systems have “Check interval” = “0”
  40. Set SQLNET.EXPIRE_TIME=10 in DB Home
  41. Verify /etc/oratab
  42. Verify all Database and Storage Servers are synchronized with the same NTP server
  43. Verify there are no failed diskgroup rebalance operations
  44. Verify the CRS_HOME is properly locked
  45. Verify db_unique_name is used in I/O Resource Management (IORM) interdatabase plans
  46. Verify Database Server Quorum Disks are used when beneficial
  47. Verify Oracle Clusterware files are placed appropriately
  48. Verify “_reconnect_to_cell_attempts=9” on database servers which access X6 storage servers
  49. Verify Flex ASM Cardinality is set to “ALL”
  50. Verify no Oracle Enterprise Linux 5 (el5) rpms exist on database servers running Oracle Linux (ol6)

 

Reference: Oracle Sun Database Machine Setup/Configuration Best Practices (Doc ID 1274318.1)

 

How to Perform a Detail Exadata Health Check

Exadata is a significant investment for any customer and one should make sure to maximize investment by configuring Exadata machine as per best practices and utilize all the features of engineered systems. Oracle has provided an array of tools for Exadata machine, but we see a gap between standard Exadata configuration vs a truly optimize Exadata machine. Exachk is a great tool provided by Oracle to validate Exadata configuration and Oracle best practices, but it’s designed as a standard tool for all Exadata machines. Exachk is not specific to a particular type of workload or application and doesn’t investigate enchantment opportunities to achieve extreme performance from Exadata machine.

That is why you should perform a detail Exadata health check of your Exadata machine which goes above and beyond Exachk validation and Oracle Enterprise Manager monitoring capabilities. The goal of this health check is to maximize the Exadata investment and reduce the number of incidents which can impact the availability of critical applications. Here is list of task you should perform to perform a detail Exadata Health check

  1. Review Exachk report to evaluate Exadata configuration, MAA Best practices, and database critical issues.
  2. Review various types of Exadata logs including Exawatcher, alert, trace, CRS, ASM, listener.
  3. Review Flash cache contents, verify smart flash log feature and check write-back cache functionality.
  4. Review Exadata feature usage like HCC Compression, Smart Scan, offloading, Storage Indexes
  5. Review Maximum Availability Architecture including backup of critical configuration files
  6. Review and validate Oracle Enterprise Manager Configuration of Exadata plugin.
  7. Review resource utilization at storage & database level and provide recommendations.
  8. Review AWR reports for contention and slow running processes.
  9. Review database parameter settings as per Oracle best practices including hidden parameters.
  10. Review log retention policy to optimize storage utilization and maintain historical data for troubleshooting any future issues.