Jonas Mason, Senior Consultant
House of Brick consultants have a unique perspective into organizational cultures, their Oracle database environments, and the impact this has on the ability to emerge from an Oracle audit unscathed or wounded.
We understand the competing demands associated with the roles and duties of the DBA, System Administrator, Developer, Technical Manager, CTO and CEO; and how this internal dynamic impacts an organization’s relationships with Oracle.
In this blog post, I will present a story that illustrates the impact of these internal dynamics on Oracle performance and licensing costs.
Readers may associate with the various actors in the story based on their specific IT background. However, I would urge everyone involved to take a step back in order to appreciate the more holistic view of the organization, and the impact this has on the relationship with Oracle. Even if we cannot fully appreciate the perspective of the actors in roles we have not held, we must accept that our own subjective and narrow view can impact the organization negatively.
The Actors
Any story is comprised of actors with different interests and needs competing against each other for fulfillment of those needs. An organization is no different, and the kind of story that unfolds is contingent on how its culture resolves cooperation challenges. A separation of duties and specialties is required, since no resource can be an expert in all things. Despite this separation, all can agree that the end goal should be to meet the needs of the organization and contribute to profitability.
DBA
The Oracle DBA is tasked with items critical to an organization’s databases, including maintaining uptime, backing up data, monitoring performance, and planning for growth. These duties eclipse all other responsibilities, as failure to perform these tasks properly usually leads to termination. DBAs also need to stay current and improve their skills. Most DBAs would rather not get too involved in licensing issues, deferring this to a Technical Manager or Chief Technology Officer.
System Administrator
The System Administrator is tasked with maintaining the infrastructure that the Oracle database servers and instances reside on. An SA may not understand that routine maintenance operations for Oracle Database VMs have license implications. Sometimes hypervisor configuration decisions made by the SA, based on their training for implementing non-Oracle database servers, negatively impacts these VMs and instances, causing them to use resources less efficiently.
Technical Manager
The Technical Manager is the conduit that relays hardware and software recommendations made by the DBA and SA to the CTO. A Technical Manager moderates demands for improved infrastructure to meet growing system utilization and licensing costs. In some organizations, the Technical Manager is responsible for Oracle licensing.
Developers
Custom development is often required to meet the ongoing needs of an organization and its customers. Developers are tasked with implementing new functionality as quickly and efficiently as possible, and typically are more versed in the front-end platforms than they are in Structured Query Language (SQL). Most do not have a deep understanding of the impacts that their SQL statements have on the database but instead rely on their DBA to provide guidance and feedback in this area.
Chief Technical Officer (CTO)
The CTO is responsible for technology that supports the organization in its goal to support its core business profitably. As such, the CTO needs to make sure its requirements for software licensing are understood, communicated, maintained, and accounted for. If these licensing costs become onerous, it is the CTO’s duty to explore alternatives. The CTO has the most insight into the impact that licensing has on the organization’s bottom line and should as a rule, challenge the Technical Manager to justify their recommendations on software purchases and maintenance. Given the expense and liability associated with Oracle licensing, the CTO is typically engaged and aware of the issues.
Oracle Corporation
Against this backdrop of actors is Oracle Corporation, which is in the business of making a profit from companies that use its software and hardware. This should not come as a shock to anybody, though the aggressiveness in which Oracle asserts its contractual and non-contractual rights might. Without this aggressive actor, the drama that unfolds at this, or any other organization, would not be nearly as noteworthy or interesting.
The Consultant
The role of a consultant certainly varies depending on their expertise and the purpose of the engagement. In some cases, the consultant is an additional technical resource to augment technical skills that an organization may lack. In other cases, in-house technical expertise is excellent, and it is the unique experience and perspective of the consultant that is valuable. Based on this experience and perspective, the questions a consultant asks can be just as valuable as the answers they provide. Generating a problem solving dialogue is one potential benefit of engaging a consultant.
Oracle Licensing: A Short Story
In this story, we have a company that uses an Oracle database instance as a back end to its custom application to meet customer requirements.
The application is constantly being modified to meet customer requests, and minor releases happen weekly, if not daily. Developers are skilled in delivering functionality quickly, but are not database coders by training. They do not consider database performance as they deliver and move on to the next task. The organization has no process in place to consider the performance of new and existing code. The database is considered a black box from which to read and write records. If response times are adequate during testing, Developers don’t give their SQL code a second thought. Technical Managers are satisfied with their Developers because they are delivering functionality for the business and its customers, which drives revenue and satisfaction.
As months and years pass, tables are filled with an increasing number of records. Statements that executed without issue before, now consume more system resources. The statements with full table scans that once executed instantaneously, no longer do so as millions of records have been added to tables that are joined inefficiently. On top of table growth, each bit of new code released adds load to the single Oracle instance and the server it on which it resides.
The DBA is busy taking care of backups, keeping an eye on performance, and pointing out problems with the SQL hitting the database. Developers take note of the e-mails sent out by the DBA on performance, but are not tasked by their Technical Managers to address these performance concerns, as tuning statements does not result in functionality that a client can appreciate. The DBAs concerns are not prioritized, as time spent on these issues by Developers would detract from their ability to deliver functionality, which brings in revenue.
CPU utilization continues to increase, more SGA memory is required, and Disk IOPS and throughput utilization grow. More and faster storage is purchased.
Since business is booming, there are also more active sessions hitting the server performing more work. The more data collected, the more business managers want reports to help them respond better to client demands and expectations. What was once touted as a very efficient OLTP system is now burdened with both expensive SQL and real time reports that aggregate data from large tables that have never been purged or archived.
The DBA complains to his Technical Manager that the instance and server are approaching capacity and that despite his efforts to communicate the issues to the Developers, he is getting no traction. The DBA asks his technical manager to talk to the technical managers in Development to get a resource that can assist in tuning SQL. Despite these efforts, no progress is made given the focus on delivering functionality.
Instead of working together to resolve the performance problems, DBAs and Developers dig in, and Technical Managers on both teams struggle to reconcile the competing narratives.
Developers blame the DBA for a system that can’t handle their SQL, while the DBA blames the Developers for the SQL that his powerful server can’t handle efficiently.
Brown Out Events
The first in a series of brown outs starts to occur under the weight of inefficient SQL, expensive reports, a lack of data life cycle management, and a release process that never considered performance.
Response times suffer and the Oracle instance is brought to its knees at random times during the day. The CTO wants to know what is going on and what to do about it. The DBA recommends increasing the memory, and virtual and physical CPU allocated to the server, as other recommendations to address expensive SQL have been dismissed for some time.
Infrastructure confirms they can deliver this easily, as the server is virtualized, but the CTO is concerned about the increased Oracle licensing costs that would result. These costs have a significant impact on her budget, her company’s bottom line, and her bonus. Instead of adding more resources and incurring the additional Oracle license liability and cost, ad hoc attempts are made by Developers to improve SQL.
A few hero Developers come through with query rewrites that allow the instance and server to return to more normal significant loads, which just prevent the server from browning out. The post mortem occurs, the developers are rewarded, and the DBA’s recommendations are put off.
The DBA shrugs and instead sees an opportunity to push through an agenda that serves both the company and himself. Now that instance availability is getting more attention, he may be able to push through maximum availability features such as RAC and DataGuard, tools that Oracle sales representatives have been pushing. The DBA could then get RAC and DataGuard training, which combined with his implementation experience, will make him a more highly sought after resource. Instead of becoming a more easily replaced commodity DBA, he would have new skills that contribute to his job security or even get him a higher paying job.
The DBA recommends RAC and DataGuard to the Technical Manager, who in turn, due to the brown out fear that pervades, is able to convince the CTO that this is necessary to safeguard the company. The CTO pushes to have this implemented during the next budget cycle, but more importantly, fails to consider other DR alternatives. She fails to consider that the array-based replication of storage that protects other virtualized application servers could also be a viable option to protecting the Oracle server and instance.
In the meantime, the System Administrator decides to move the Oracle VM to a new hypervisor host with twice as many CPU cores, as they are seeing CPU contention with other VMs on the source host. The SA doesn’t think he has changed the licensing for the VM because he hasn’t changed the virtual CPU.
Conclusion
In part 2, discover why the brownouts continue, despite HA implementations, read about the company’s Oracle license review process, and gain an understanding of the key takeaways and lessons learned from this all too familiar licensing story.
