Jonas Mason, Principal Architect
This blog covers the experiences House of Brick consultants have had with the Oracle Database Appliance (ODA) based on migrating to it, supporting it, and patching it. The title indicates our mixed feelings about this offering.
ODA’s Purpose
The ODA is an engineered system that includes hardware, networking, storage, and software preconfigured in an appliance. When it arrives you can immediately deploy a two-node RAC instance that is highly available and performant. The ODA market is for those companies that want these features, but do not have the human resources or time to build this type of system themselves.
The Good
Performance
We have always been pleased with the ODA’s performance for OLTP, reporting, or data warehouse based workloads. Migrating to and from other databases using Data Pump was fast and straightforward. Migrating single instances to two node RAC instances was seamless in terms of performance; there was never any issue or question around cluster waits, which can sometimes be a problem.
For one organization, we were able to configure over 10 two-node Oracle RAC production instances on a single ODA with a fraction of the CPU licensed. The ability to scale up licensing in these smaller increments for Enterprise Edition features can be attractive.
Database Templates
The templates are useful when deploying new instances and can assist with correct setup, per Oracle ODA best practices and guidelines. We found that the smallest templates were sufficient to support the instances migrated to them based on legacy instance performance analysis. However, it is not currently possible to create custom templates.
Fast Deployment
The ODA is fast to deploy in cases where high availability is required at the instance layer and clearly defined by your company’s SLA. It meets the objective of delivering a turnkey solution in this regard; however, changing choices after deployment requires a full re-deployment, erasing all existing configurations and database instances. Further, management is handled very differently than traditional Oracle deployments and uses a monolithic command line tool that must be run as the root user for many operations.
Snapshotting
The ability to snapshot storage, and therefore new database instances, is a benefit with the ACFS file system if you want to spin up non-production instances.
One Throat to Choke
One selling feature of the ODA is that you get all your support from one vendor, and therefore avoid the vendor finger pointing that often happens in troubleshooting scenarios.
The Bad
No Throat to Choke
If you have had issues with Oracle Support in the past, with more commoditized versions of its database software, you should be questioning whether you want to embrace support on bundled software and hardware that is less mainstream.
And why would you want to further exacerbate your organization’s vendor lock-in by adding hardware to the mix?
Non-Standard
Both an experienced and inexperienced Oracle DBA will need to learn the ways of the ODA prior to supporting it. Attempts to apply traditional management and troubleshooting practices often result in a less than ideal outcomes and can even confound the operation of the ODA.
Creating a new database instance requires the use of the Oracle Appliance Kit Command Line Interface (OAKCLI) as the root user. Using DBCA to create a new database as the oracle user is not supported, and neither are any of the graphical tools or OEM interfaces that most of us are familiar with.
The ODA Patch Bundle includes updates for BIOS, hardware drivers/firmware, OS, JDK, Oracle ILOM, PSU, and grid infrastructure. Applying individual patches for Oracle Grid, database, or Linux is not recommended, as the ODA Inventory will not be updated, and future patch bundles will not work.
Database Templates and Flash Caches
Prior to go-live with several instances on an ODA, we noticed 100% CPU utilization on one or more database writer processes associated with one or more instances. The associated SR with Oracle took six months to resolve, and was related to a bug associated with the flash_cache. The flash_cache was enabled by default by deploying an extra small 12c instance with an ODA database template. Was the flash cache required for this customer’s instances? Not really. Did troubleshooting the issue and restarting instances on a regular basis to clear out CPU utilization problems for months after go-live consume valuable resource time? Absolutely!
So by sticking with the ODA way and creating a database with OAKCLI and a database template, we inadvertently introduced an unneeded feature with a bug. The 11g instances created with OAKCL did not have this feature enabled, and hence did not encounter CPU issues with the DB writer processes.
Unfortunately, database templates deploy features that you sometimes don’t really need, and again it’s impossible to create custom templates.
The Ugly
Maintenance: The Patch Bundle
While the Patch Bundle seeks to insulate the DBA from complexity, the significant dependencies and higher risk associated with patching the entire stack at once puts the organization more at risk. Mitigating risk is an objective of a DBA and System Administrator that is often facilitated by introducing as few changes as possible at any given time at any layer of the stack.
Our experience with applying Patch Bundles on an ODA system is that it is time consuming, poorly supported, and very difficult to troubleshoot without help from Oracle Support.
Our most recent ODA patching experience took four days and two Severity 1 SRs to complete. A single ODA cannot be patched without downtime; so two ODAs in a Data Guard configuration are required to avoid lengthy downtimes and outages.
Oracle Support
Oracle support is not organized to efficiently handle the ODA. Instead, it is organized in silos by functional area (e.g.: TNS/Network, storage, RAC/Clusterware, patches, core database, etc.). Receiving support for an ODA first requires identifying the functional area where the problem is occurring. Unfortunately, because of the non-standard and highly integrated nature of the platform, where the symptom shows up may not have much to do with where the actual problem started. Your experience with Oracle support will depend largely on how well you guess when opening an SR. Finally, troubleshooting by Oracle support focuses on traditional means rather than a holistic ODA approach. In our experience, SRs for the ODA can drag on for weeks and even months bouncing from functional area to functional area with no resolution.
OEM / Database Console
There is no database console in the traditional sense, and limited support through OEM in general. The proprietary ODA console is also very limited, so you’re stuck with non-standard command line interfaces that often require root access. This lack of OEM integration was a problem with one client’s DBA, who had become familiar with monitoring and maintaining legacy instances with OEM. The inability to setup Data Guard between two ODAs with OEM was also an issue for this client.
The ODA Way – or the Highway
Because the platform is so integrated, you must follow the ODA way of doing things. DBAs will frequently encounter issues when applying their experience and knowledge. Our conclusion is that the management and administration of the ODA is poorly considered and immature – even after years and multiple generations of the product.
The Alternative: VMware, Cloud, or Hybrid
Given our mixed feelings about the ODA, what’s the alternative for an organization that wants a highly available and performant Oracle database instance? A lot depends on how tight your SLA is on availability. Very satisfactory availability and protection can be achieved further down the stack with virtualization technologies that are mature and performant.
I personally appreciate the simplicity, scalability, and robustness of a single Oracle instance running on a standard Linux VM hosted on a VMware cluster. Vertical scaling of VMs running Oracle database software is effortless when more CPU or memory is required. The HA features of VMware are excellent and proven. Human resources are readily available in the marketplace to support this stack. This proven solution positions your organization to pivot to the cloud provider of your choice in the future.
Conclusion
The ODA radically departs from traditional Oracle offerings. While delivering on performance and availability, it comes up short on support, maintenance, and administration. Given the standardized Oracle instance alternatives on VMware that are now commonly deployed, along with growing interest in the cloud as a solution, we see little argument to be made in favor of the ODA.
