With over 500 Oracle audits defended at House of Brick, we have seen it all! We have had the virtualization battles, and the DR metric arguments; we have dealt with Oracle’s aggressive overreach, and unjustified compliance claims. In our experience, the single most risky part of an Oracle audit, however, is database feature usage findings that the customer never even knew about beforehand. We are very good at diffusing Oracle’s unjustified claims and eliminating those fees from the findings, but database feature usage has resulted in our customers having to pay more to resolve an audit than any other thing.
Why Oracle Database Feature Usage Is an Audit Trap
“It’s a trap!”, Admiral Ackbar
We love Oracle technology. It has been proven to be the workhorse of enterprise applications for generations. There are some things about Oracle that we don’t love, however. One of those is with their handling of Database Enterprise Edition options and management packs.
The Hidden Risk of Oracle Enterprise Edition Features
When an Oracle Enterprise Edition database is installed, the majority of the associated options, management packs, and their associated usage triggers (collectively referred to as database features) are automatically installed and left enabled at the same time. And the majority of them cannot be disabled. Even a Standard Edition (any version), Personal Edition, or Oracle Free Edition (formerly Express Edition or XE) of database has licensable features included with the installation. There is no technological connection between feature usage and the licenses you may or may not be entitled to apply for using those features. Due to this lack of connection, the software does not know whether your usage of a product or feature is licensed or not. There are exceptions to this statement. For example, an SE 2 (Standard Edition2) installation will not allow you to partition tables, or compress outputs, but it will allow you to invoke an enterprise feature like Diagnostics Pack or Tuning Pack.
The audit-trap is that it is very easy to inadvertently start using a licensable feature in your Oracle database that you don’t actually have a license to cover. There are no warnings or messages from the database when you use a feature for which you do not have a license. Once an installed and enabled feature is used, Oracle keeps a diligent and permanent record of that usage in each database. During an audit, this feature usage view is queried by Oracle audit scripts, and the findings are included in the audit report. This may be the first time that most customers are even aware that they were using an expensive feature. Oftentimes, an unlicensed feature can be enabled in one database, and then propagated to others as it is cloned and promoted into production or disaster recovery environments. These options and packs must be licensed for the same Processor or Named User Plus counts as the database, so cost can add up quickly.
Another risk that our customers face is that Oracle’s feature usage statistics do not do a great job of filtering out false-positives. For example, Oracle’s Enterprise Manager (OEM) will trigger usage of Partitioning Option because its own database partitions tables. That is not the responsibility of the customer to pay for, but the fact that it is showing up in a feature usage query can be misleading or disconcerting.
How Oracle Tracks Feature Usage in Database Audits
Oracle Database Feature Usage Logging Process
Oracle has an internal process on each database that checks which features have been used since the last check. This process runs once per week, typically on a Saturday. Some tool vendors may tell you that they have real-time checking of Oracle feature usage, but this is misleading. You can check the feature usage profile every sixty seconds if you want, but you will only see changes after the Saturday feature checking process has completed and updated a database view called dba_feature_usage_statistics (DBAFUS).
In the screenshot from the Opscompass feature usage tracking, you can see that Oracle records
The feature trigger that was invoked (e.g., Oracle Utility Datapump (Export)), the total number of uses (number of weeks that the usage was detected),
The date of the first detection of that feature trigger, and
The date of the last detection of that feature trigger.
To complicate things further, there are certain feature triggers that invoke the use of a licensable option or pack, but that are not recorded in the DBAFUS view. All of this data is queried by Oracle during an audit, analyzed and compiled, and then compared to license entitlement to determine a customer’s compliance status. Unlicensed usage of database features, even those occurring in the past, will typically result in a claim of non-compliance, and a demand for license purchases, or to transition into an Unlimited License Agreement (ULA).
Quantifying the Feature Usage Risk
The real risk that you face from unintentional use of licensable Oracle database features depends entirely on the scope and configuration of your database environment. But to illustrate the potential risk, we present the following real, but anonymized scenarios:
Scenario 1: The compressed datapump export
In this scenario, a new Oracle DBA was tasked with developing a script to periodically export all the data from the databases. This is a common and routine exercise for DBAs, but for this inexperienced DBA, a simple command line flag was a costly mistake. When putting the Oracle Utility Datapump (Export) command expdp into the script, this DBA saw that if he used the command option COMPRESS=DATA_ONLY or COMPRESS=ALL, that it would dramatically reduce the time for performing the export. Even though his company did not have licenses for Oracle Advanced Compression Option, each database the script was run on happily complied with the command, and compressed the exported data.
Oracle audited this customer a year and a half later, and discovered that unlicensed Advanced Compression Option usage had propagated throughout the entire database environment of 650 processor cores. Oracle claimed a list-price deficiency of almost $7.5M, but then kindly offered the customer a considerable discount. Total avoidable risk over three years: $6.2M. As you can see in the Opscompass screenshot, we would have caught this, and alerted the customer in time to completely avoid this audit penalty.
Scenario 2: Enterprise feature used on SE 2 database
In this scenario, a customer had a deployment of SE 2 databases across an environment of 30 servers. These high-powered 2-socket servers (96 cores per socket) required two SE 2 licenses each, which cost the customer about $630k in a discounted license purchase, with annual support of about $140k. Facing some performance issues, the CIO, who came from a bigger company running Oracle Enterprise Edition databases, asked the DBA to run AWR reports on all the databases. AWR reports use an enterprise feature called Diagnostics Pack. This action introduced a very expensive multi-factor impact of 1) the SE 2 databases could no longer be considered Standard Edition, and had to be upgraded to Enterprise Edition, 2) Unlike Standard Edition versions that license by the server socket, Enterprise Edition licenses by the number of processor cores, and 3) the enterprise features had to also be purchased.
The math is staggering. For 60 two-socket servers, with 96-core chips in each socket, the customer was facing a list-price deficiency of over $273M! Paying this license fee would have put this customer out of business. So Oracle presented them an option that they could work with – enter into a ULA costing $25M up front, and pay $5.5M per year in support fees. Total avoidable risk over three years: $41.5M! Again, Opscompass would have caught this issue so that it could have been addressed before it ever became a problem.
How to Manage the Risk
Numbers like this almost seem too hard to comprehend, but we see similar patterns time and time again. Regularly checking the DBAFUS view with detailed comparisons between data gathering events is a critical governance commitment that every Oracle customer should adhere to. This will show changes to the feature usage profile that may indicate a license deficiency. House of Brick has outlined how to do this using Oracle’s freely available script. This is a manual process, however. As you will see below, House of Brick is now offering our Opscompass premium automatic Oracle database feature usage monitoring capability for free!
If a customer finds an unlicensed feature in use, then certain questions should be asked:
- What event caused this feature to be used?
- Was it intentional, or inadvertent?
- Did we derive business value from the use of that feature?
- Do we intend to continue using that feature going forward?
- Can our governance procedures reduce the risk that this feature will be used again?
- Depending on your answers, you may need to consider purchasing the required licenses, discontinuing unlicensed usage, or some optimized combination of both.
Opscompass to the Rescue!
Opscompass tracks about 2,400 Oracle products, and 3,500 product features. Just for Enterprise Edition database, we are tracking about 250 features that trigger usage of licensable Enterprise Options or Management Packs. Using the power of House of Brick-trained artificial intelligence (AI) enhancements, Opscompass can ingest and automatically analyze and populate all available Oracle license entitlement. Comparing the usage of Oracle software to the available entitlement, we perform a dynamic, autonomous, secure, confidential compliance audit on your environment. Imagine having an email automatically delivered every Monday morning, telling you if your Oracle licenses are in compliance or not! That is the power of Opscompass.
Recognizing that many organizations worldwide are leaving themselves vulnerable to this audit trap of uncontrolled database feature usage, we have decided to offer our automated Oracle database feature usage monitoring and drift detection capability to the world, for free! Now you don’t have to wonder if you have manually run the scripts recently, or if your DBAs have done something that will inadvertently cost you millions of dollars (and likely your job too). Schedule a demo and start using Opscompass for free today, or talk to one of our experts, and we can discuss your database governance options with you.
