Jonas Mason, Senior Consultant

In part 1, I introduced a story that illustrates the impact of internal dynamics on Oracle performance and licensing costs. I described how various actors, and the perspectives they bring from their specific role in the IT process, can have a huge impact on the story. In this second part, I will describe why the brownouts continued despite HA implementations, describe the company’s Oracle license review process, and offer insights regarding key takeaways and lessons learned.
Brown Outs Continue Despite HA Implementations

At this point in the story, the organization has RAC and DataGuard implemented for its primary production database. Despite adding resource capacity by scaling horizontally, performance problems have become worse and brown outs have once again become a regular occurrence. Long running Data Manipulation Language (DML) statements by sessions spread across the cluster, are now flooding the private interconnect between RAC nodes. Despite recommendations that Jumbo Frames be configured for the private interconnect, it was not completed properly, adding to the overhead of expensive DML. An application that was hostile to a single instance, became much more hostile to multiple RAC instances.

The DBA can’t explain why two servers aren’t performing twice as well as one server based on his assumptions and Oracle’s marketing on RAC. Shouldn’t scaling horizontally double the processing capacity of the production instance? The CTO is furious and calls upon the Developers to be heroes again, since the DBA just repeats the request for more CPU and memory. To the DBA, the load average is clearly significant, and the only way to resolve this is to add more CPU, given the SQL tuning roadblocks. He thinks adding more memory to the SGA will decrease physical reads. The Developers attempt to tune the long running DML, but despite their efforts, the problems persist. As a result, the CTO reluctantly agrees to license even more processors from Oracle to prevent the outages that are causing the CEO to ask questions. The CTO realizes it is best to give up hopes of a bonus that year, pay the extra cost and keep his job.

The Oracle License Review

Brown outs have quieted down in the past few months, and sales at Oracle have slowed down in the last quarter. An Oracle salesperson has the idea to offer the organization a friendly license review to best serve everyone needs. From the salesperson’s point of view, this is a win-win opportunity to discover licensing gaps, resolve the gaps with additional license sales, and make a pitch to use Oracle hardware, such as the Oracle Database Appliance or Exadata. Certainly the pitch on the ODA or Exadata could be made to seem cheap, especially when compared to possible remediation demands.

Oracle makes a number of discoveries during the review of production and non-production VMs on hypervisor hosts spread between its two datacenters. It asserts that the organization has to pay for Oracle licenses based on all the cores in its hypervisor cluster, as a result of the VMs migrating to these hosts, per the migration history captured.

The demand to the organization is in the millions of dollars to resolve the licensing gap, since physical cores throughout the enterprise had hosted Oracle database VMs.

Oracle offers to provide a discount on the licensing costs if the organization places its instances on the latest Exadata. The DBA recommends Exadata to get out of his poor performance predicament and further his career with additional certifications. The Technical Manager gets fired because he wasn’t properly managing Oracle licensing and the CTO needed a scapegoat. The CEO and CTO confer and wonder why they went with Oracle database technology to begin with, since it is so much slower and more expensive than they ever imagined.

Lesson Learned

In our story, we have a fairly small organization experiencing growing IT pains as a result of a successful business model that focused on meeting customer demands and requests. The company focused on its core strengths of delivering functionality, which left database performance and licensing as an afterthought. Despite the relative simplicity of this organization’s Oracle environment, their own personal narratives, agendas, viewpoints and skill-sets blinded the actors involved. They were not prepared for a license review, in part because they could not wrap their heads around a clear and consistent Oracle License narrative.

A larger organization with hundreds or thousands of Oracle database servers is faced with even more complexity and potential exposure.   Despite having top talent, this talent is often siloed and the disjointed story that transpires is difficult to follow and painful to watch. Given the internal challenges all organizations face, how does an organization come up with an Oracle licensing narrative that stands on firm technical and legal ground?

In this story a consultant was not involved. There could have been internal resistance from the organization’s staff to bring in a consultant, who could question or challenge the decisions they had made. But what if the consultant was able to independently confirm aspects of what the DBA, Developers and other staff each had to offer? What if the disjointed Oracle licensing narrative of the siloed departments could be smoothed over, polished, and prepared for prior to an Oracle audit?

Conclusion

Deference is sometimes given to Oracle on their interpretation of their contracts, since their expertise in this area is presumed. They wrote the contract after all. As we know, however, language can be confusing and open to interpretation. While Oracle continues to stretch the interpretation of contractual clauses, to the dismay of its customers, in the world of negotiating this aggressiveness pays off for them. Of course Oracle has the technical and legal resources to interpret their contracts in its favor and to push its narrative in pursuit of profits.

Below are steps the company could have taken to avoid their licensing outcome:

• Modify the release process to consider performance
• Track performance metrics over time and tie these back to the SQL statements and tables that were responsible
• Implement Data Lifecycle Management
• Schedule regular reviews of CPU counts and utilization for Oracle servers
• Present findings on CPU exposure and utilization to legal, on a scheduled basis
• Highlight expensive SQL statements and reports that contribute to CPU utilization and place a cost on these
• Create representative teams across silos to address performance issues
• Engage a consultant like House of Brick to provide independent review and analysis of each of these elements

Whether the story that is transpiring at your organization can be compared to a comedy or tragedy, this reality could be taken advantage of during an Oracle audit or license review. What may once have been limited to internal dysfunction with limited consequences, could quickly become a reality filled with Oracle licensing gaps, legal and consulting fees, and significant financial demands if not addressed proactively.