I have a Test database that is a 2-node RAC system. I am working towards the goal of getting the production database to Oracle 12.1.0.2 in about a month’s timeframe. This of course means that I have to get Grid Infrastructure upgraded prior to the db upgrade. I have upgraded GI on my standby cluster and on my Test database as well. The primary GI upgrade is scheduled for this evening.

Ever since I upgraded GI in Test a few weeks ago, I’ve been getting alerts from EM12c similar to the following:

Host=host01
Target type=Cluster
Target name=test-scan
Categories=Business
Message=Server is under elevated memory pressure and services on all instances on this server will be stopped
Severity=Warning
Event reported time=Jul 29, 2015 1:05:13 PM CDT
Operating System=Linux
Platform=x86_64
Event Type=Metric Alert
Event name=wlm_event:wlm_qosm_mpa_risk_state
Metric Group=QoS Events
Metric=Memory Pressure Analysis Risk State
Metric value=RED

Some of the alert details were removed for brevity.

So where is this coming from? Why does it mean to me?

This error is coming from Oracle’s Quality of Service (QoS) in Grid Infrastructure. It relies on Cluster Health Monitor (CHM) information. More specifically, this alert is coming from Memory Guard. For some information on Memory Guard, see this PDF, specifically the end of the second page.

Memory Guard is trying to save me from myself, and as we will see, it is doing a poor job of it. The idea is that when the server has memory pressure, Memory Guard will take any services on that node out-of-service. Allowing more connections would consume even more memory and could make the situation worse. New connection requests must go to another node in the cluster running that service. This is exactly what the Message value in the alert is telling me.

According to this EM 12c document, section 4.3.2, Memory Pressure Analysis Risk State, the alert text is supposed to contain the server name. Yet the message text above does not tell me which server is having the problem. Luckily for me, it’s only a 2-node RAC cluster, so I don’t have too many to examine.

When I do look at the CPU utilization, everything is fine. Swap usage is practically zero on both nodes. Free memory is more than 25% on both nodes. Curious…why the alert in the first place?

Everytime I get this alert, I can another email that says the condition is cleared up within a few minutes. So the issue is short lived. Yet the alerts keep coming.

It turns out, after some investigation, that Oracle made a change to Memory Guard in Grid Infrastructure 12.1.0.2. In earlier versions, Memory Guard only looked after policy-managed databases. In GI 12.1.0.2, Memory Guard started looking after admin-managed databases as well. And my RAC databases are typically admin-managed, which is one reason why I’m seeing this now.

To further add to the issue, apparently, GI 12.1.0.2 has known Bug 1582630 where the amount of free memory if calculated incorrectly. Note 1929994.1 lists a workaround and there is a patch as well. I applied the workaround and it resolved my problem. I’ll get the patch applied to Test before I proceed to production in the not-too-distant future.

Thankfully, I discovered this before my production GI upgrade later tonight. Otherwise I would have had upset end users that may have experienced issues connecting to the database. This is just one more example of why I have a good test platform with which to discover and resolve the issues before the change is made in production.

I am upgrading a 2-node Oracle RAC testbed from Oracle 11.2.0.4 to Oracle 12.1.0.2. This is a simple testbed and the database is virtually empty. I’m using the DBUA to perform the upgrade. Should be simple, right?

After the database is upgraded, the DBUA is trying to move the RAC configuration to the new home. Early on, I get a “file exists error”. I check the $ORACLE_BASE/cfgtoollogs/dbua/upgrade1 directory and in the PostUpgrade.log file, I get this curious error:

[oracle@nau-rac01 upgrade4]$ cat PostUpgrade.log

create spfile=’/u01/app/oracle/product/11.2.0.4/dbs/spfileresp1.ora’ FROM pfile=’/u01/app/oracle/product/12.1.0.2/dbs/initresp.ora’

*

ERROR at line 1:

ORA-27038: created file already exists

Linux-x86_64 Error: 17: File exists

I’m not sure why 12.1.0.2 is trying to create a SPFILE in the old home for the upgrade. But for me, this is a simple fix. Remove file from this directory and re-run the upgrade. This isn’t an issue because this is an Oracle RAC database and the true SPFILE location is on shared disk, not in this unshared home, and is denoted in the OCR. Removing the file the DBUA was trying to create resolved my problem.

I’m trying to branch out a bit and craft a few videos. This one is my first attempt and I discuss the new 12c IDENTITY column performance, specifically for Oracle RAC databases.

Link to video on YouTube: https://youtu.be/aW_K_rEFkrI

Note: I don’t focus my energies on the planning side much. So these videos are not rehearsed. I will prepare a few scripts so that I don’t have to type as much, but otherwise, these videos will just be me being me.

So I’ve been having a tough time of it lately on things I should know better. We’ve all been there before.

I’m trying to recreate a 2-node RAC testbed on my laptop. This will be Oracle 12.1.0.2 on Oracle Linux 6.5 with VirtualBox 5.0. Should be pretty simple, right? I even wrote instructions on how I did this back in April of this year. I’m following the same exact steps in my document on the same exact laptop but yet I still have problems. The only two things that have changed is I am going directly to 12.1.0.2 (my doc was 12.1.0.1 I believe) and I now have VB 5.0 on my laptop.

I have my 2 virtual nodes created and ready to go. I fire up the OUI to start my installation of Grid Infrastructure. When the OUI gets to the linking phase, I get an error message.

Error in invoking target ‘irman ioracle’ of makefile. 

We’ve all been at a crossroads in life. You have a choice to make. Do I go left or do I go right? Unfortunately for me, I turned the wrong direction and wasted a few weeks of my spare time. At this point, I had a decision to make. Do I do exactly as the error said or do I rely on my experience? I blindly ignored the error and relied on my experience. Silly me.

Just two months ago I had the exact same error when I couldn’t get GI 12.1.0.2 to compile on a testbed. That testbed was running on VMWare ESX hosts. I filed a SR with Oracle Support and they let me know that my compile issue was because I did not have enough RAM devoted to each virtual machine. Being a virtual environment, this was easy enough to fix. I had my SysAdmin bump up the RAM and swap space on the virtual machines and GI 12.1.0.2 compiled exactly as promised. So I naturally assumed that I was running into the same issue here. On my laptop, I kept bumping up RAM. I expanded swap space. I even went so far as to rebuild the nodes from scratch. I spent the last two weeks down the road of experience and I found it to be bumpy, scratchy, dusty and very unpleasant.

The road I should have taken was to do explicitly as the pop-up box said…read the log file for more details. When I finally got past my stubbornness in thinking I knew the answer, I read the log file. I found the following messages towards the end.

INFO: - Linking Oracle

INFO: rm -f /u01/app/crs12.1.0.2/rdbms/lib/oracle

INFO: /usr/bin/ld: cannot find -ljavavm12
collect2: ld returned 1 exit status

INFO: make: *** [/u01/app/crs12.1.0.2/rdbms/lib/oracle] Error 1

Well its completely obvious now! A library file is missing. A quick check on the Internet easily led me to the solution. I’m not alone in this issue, but for some reason the OUI is not copying libjavavm12.a to $GRID_HOME/lib as it should be. With that pop-up box still sitting there, I issue the following on the node.

[oracle@host01 ~]$ export GRID_HOME=/u01/app/crs12.1.0.2
[oracle@host01 ~]$ cp $GRID_HOME/javavm/jdk/jdk7/lib/libjavavm12.a $GRID_HOME/lib

I then went back to the pop-up box and pressed the RETRY button. Sure enough, the damn thing compiled and the OUI finished up its work.

UPDATE: I had the same issue when installing the RDBMS 12.1.0.2 software on the cluster. I used the same exact fix to get oracle to compile correctly for the database software as well.

But I was not done. As anyone who has installed Grid Infrastructure knows, they need to run the $GRID_HOME/root.sh script on all nodes later on in the process. When I attempted this on the first node, I received a “segmentation fault” error. Turns out, there is a problem (and again, I’m not alone here) with perl in GI 12.1.0.2 installations. Even the following will receive a segmentation fault:

cd $GRID_HOME/perl/bin

./perl -v

The solution is to re-install Perl in $GRID_HOME. I found a blog entry by Laurent Leturgez which describes exactly how this is done. I never would have figured this out on my own. After re-installing Perl on all my nodes, the root.sh script ran just fine.

UPDATE: I had the same issue when installing the RDBMS 12.1.0.2 software on the cluster. I used the same exact fix to get perl to run without a segmentation fault.

Like all of us, I rely on my experience to save me lots of time. Things I learned a month or a year ago are applied today. Sometimes, experience gets in our way and takes us down a road we wished remained untravelled.

Many of us now have Oracle RAC installed in virtual environments on our laptops and workstations. This blog post shows how I did just that on my MacBook Pro. Recently, I was re-installing a new RAC testbed on my laptop when I ran into a few issues that took me awhile to sort out. After I had everything up and running, I figured the proper course of action would be to take a good backup of my work. The video below shows how to backup RAC testbeds that use Virtual Box.

https://youtu.be/TK8kosRfup0

I’m in the middle of testing for my company’s Oracle 12c production upgrades coming in September. Thankfully, I have a lot of good testbeds to fine-tune the upgrade process before proceeding to production. So far, things have been pretty smooth, even much better than my 11.2.0.3 to 11.2.0.4 upgrades a year ago. I thought I’d outline a few things I’ve dealt with during our testing in case this helps someone else down the road.

• New for me this year is that our Test database environment is now Oracle RAC. Previously, my only non-production RAC machines were empty databases on  VMs. At least now I can test out GI upgrades with our data. This is huge and led me to the next bullet point.
• Ran into Bug 19582630 in my Grid Infrastructure 12.1.0.2 installation. The new Memory Guard would incorrectly detect an out-of-memory condition which I blogged about here. A workaround was employed until such time as the bug’s patch could be installed.
• WM_CONCAT is no longer with us. This broke 4 packages for application code. It was easy to use LISTAGG as a replacement.
• Oracle 12c has deprecated the SEC_CASE_SENSITIVE_LOGON parameter. Changing to case sensitive passwords could break our application. I now have a utility to make application passwords case sensitive and I can now move forward.
• Hit Bug 20369415 in which an ACL with a timezone caused upgrade issues. The solution was to drop the ACL and recreate it without the timezone information.
• Whomever installed this database initially installed many unused database options like APEX, OLAP Catalog, and the EM Repository. I finally got around to removing these prior to the upgrade.
• CVU reports are not being purged. See Note 1964482.1 for details. I’ll get a cron job setup to remove these on a regular basis.
• Oracle 12c’s new security model with respect to Invoker’s Rights procedures required us to GRANT INHERIT ANY PRIVILEGES to a specific schema user.

That seems to be it for me, at least so far.  I’ll be rolling through the rest of our non-production databases soon before upgrading prod to 12c in a few weeks. My fingers are crossed that I’ve handled all of the upgrade issues.

I’m currently taking an online class from MongoDB University to learn how to be a DBA for MongoDB. My company doesn’t have any plans for MongoDB at this time. But I find that such a class can help me learn more about the product and maybe I’ll find that this product is a good fit for some future database requirements. For those interested in the class, here is a link. I highly recommend it and the best part is the cost!

As I’m nearing the end of the 7 week course, I started contemplating the benefits of database administrators learning multiple database systems. Some DBA’s are charged with only one type of database system. My first DBA job was such. I only used Oracle there. In my second DBA job, I was hired for my Oracle expertise. Shortly after I started with the company, they needed to deploy SQL Server and wanted to send someone from the DBA team to a class on the subject. I jumped at the chance. Most of us get involved with the database systems used by the company that employs us, and that’s where it ends. Sometimes, the company will get a new database product in the door and the DBA gets to stretch their wings with the new product. And then there are companies that have a large database footprint and can enjoy the luxury of hiring specialists. So they will hire some Oracle DBA’s and some SQL Server DBA’s, etc. In those companies, the DBA often gets put into a silo.

There are benefits of specializing in a specific product. You get to be real good with it. You get to learn all of the nuances of that product. And that can make you very beneficial to your employer. Some people don’t get to specialize, which brings up the phrase that says something like “he’s a jack-of-all-trades and master of none”. This can be beneficial as well until there is a sticky problem that needs a level of expertise the jack-of-all-trades has not had time to learn because they were too busy with their fingers in all those different types of database systems.

It is obvious that my expertise lies with Oracle. But I’ve worked with mySQL, Postgres, SQL Server, Adabas, IMS, and many others. Unless you’ve been hiding under a database rock somewhere, you’ve been hearing of the new wave of database engines…MongDB, Hadoop, Cassandra, Neo4j, and others. The database market is certainly changing.

As IT professionals, we have to learn to adapt to the changes. My Oracle expertise will still be needed for the foreseeable future. But what changes await me more distant in the future? Well if I could answer that question, I’d make a killing betting on sports! We can’t see the future. We can only prepare for it the best way we can, armed with the knowledge we have today, and prepared to adjust that knowledge as we learn more tomorrow.

Change is constant in IT and the new database engines being implemented today are further proof. It is human nature for Fear, Uncertainty, and Doubt (FUD) to creep in when thinking about this change. Will these new database systems mean the end of my career? Will I be replaced for another DBA that knows one of these products? I say to embrace the new technology. I still remember the day I was asked if I wanted to go to SQL Server training. I was excited for the opportunity. As time has passed, I’ve come to grips with the reasons why I like learning about database systems other than Oracle, which brings me to the point of this article. What are the benefits of database administrators learning multiple database systems? I’ve compiled a list of the benefits from my viewpoint. Other’s may have things to add to the list, so feel free to send me a comment or two. In no particular order, here are the benefits I see.

* When I learn a new database system, I try to go back to database theory. The type of theory they love to espouse in academia. This isn’t real-world stuff, but purely theoretical. Not practical at all. But the theory is important. And having a solid understand of database theory helps me quickly learn a new system. For example, I know the theory behind locking for concurrency control. When I am learning a new database system, I don’t have to learn transactional control from scratch. I leverage my existing knowledge, relying on that theoretical foundation, and figure out how that specific system implements it. In this way, I can appreciate the strengths of that particular system and know to stay away from its weaknesses. Learning new database systems reinforces the core database theory I know. It also gets me thinking about implementation of that theory in a new way that I may not have thought of before.
* From a career standpoint, I don’t think its a great idea to be put into a silo where I know only one thing. As I alluded to before, the only constant in the IT business is change. If I’m in a silo, what do I do when the only thing I know is gone? If all I knew was IMS, my job prospects would be pretty slim. For a career longevity standpoint, knowing multiple systems keeps me employed for the long haul.
* Again from a career standpoint, knowing multiple systems helps my company be more flexible with their database deployments. The business may have a requirement to host data in more than one database system. If I can’t support more than just Oracle, then I’m probably not the person for the job. Keeping flexible with the database systems I support makes the business more flexible and makes me more valuable to the business. The more valuable I am, the more job security I have and the more money I’ll make.
* New technology is cool! Like many in the IT business, I’m drawn to computer systems in general. We all like new tech. That new shiny smartphone. Now that smartphones are commonplace, we’re looking to Fitbits and now smart watches. We always like the new applications. I’ll upgrade to newest version of MS Office, not for the new features, but because I’m tired of seeing that same look and feel for the last 3 years. Being that I love working with data, nothing is cooler than that shiny new database engine I’ve never seen before.
* A new database system breaks up the monotony. We can all get complacent and bored with things from time to time. That’s human nature. Oracle likes to break up that monotony for me by introducing a new version with all of its new features that I need to learn. But sometimes I like to just learn a new product.
* A new database system may make you appreciate your old one! Just because the system is the newest thing around, doesn’t mean its better than the tried and true system that I’ve been working on for decades. It might be time to reinforce that and learn why I still use that old system.
* A new system may force you to think about your old system in a different light. It may get you to think outside the proverbial box, which can sometimes lead to different solutions. When learning the new system, you may go back and find a different way to implement something in the old system.
* Trolls exist everywhere! They don’t live solely under bridges any more. We’ve all been hit with the psuedo-IT professional that read some article somewhere and thinks they know more about the latest/greatest technology than you do. This video is hilarious and reminds me of similar conversations (Note: video does contain foul language) I’ve been unfortunate to have over my career. As database professionals, we must be able to talk intelligently on what a database system can and can’t do well. If we don’t learn about those systems, how can we be an advocate for the proper technology to meet the business requirements? In addition, my company is counting on me to provide solid advice for our corporate database architecture. I don’t want to be the guy that deployed a specific database engine only to find out a year or so down the road that we deployed the wrong thing. Knowledge is key in defending against trolls and helping the business make solid decisions.
* You never know when something new will come along and intrigue you more than what you’ve been working on. I know of a number of Oracle DBAs who are no longer working with the Oracle product and have decided they like something else better. How will you know if the grass is greener on the other side if you don’t go over there for a visit? I’m not necessarily advocating looking for the greener pasture, but I don’t like to close doors before I’ve stepped through them either. So have a look. Take a peek. Kick the tires a bit. It can’t hurt.
As a DBA and as an IT practitioner, I’ll still be looking at new database systems as they mature and become relevant. I’m sure that I’ll keep this up throughout the remainder of my career. I find benefit, even in my current job duties, to learning something new, even if that new thing doesn’t have any direct correlation to what I’m working on today.

There may be readers who disagree with some of the things I’ve said above. That’s ok. We aren’t talking about facts here, just opinions so there is no right or wrong. Feel free to leave comments and discuss in more detail if you like.

The Oracle Database Developer Choice Awards are open for voting. You can cast your votes until October 15th. The candidates are all well deserving of the award. So vote for your favorite today!

This video shows a few tips on how I use SQL Developer when tuning a SQL statement. I hope it helps someone.

Link to YouTube: https://youtu.be/MCOhtc_ubqc

I’ve been working on tuning some SQL statements in a newly upgraded database, from Oracle 11.2.0.4 to 12.1.0.2. As I typically do, I like to use SQL Developer to see the Explain Plan. I was surprised to see some gray text in the Explain Plan output, which you can see below.

My first reaction was to freak out and wonder why Oracle has both a Hash Join and a Nested Loops join for joining two tables. Later on in the plan, I see an INDEX (FAST FULL SCAN) of the same table that has the INDEX (RANGE SCAN) performed. Why two accesses of the same table? What is Oracle 12c doing?

All of this is part of the new 12c Optimizer’s Adaptive Query Optimization. For this SQL statement, the Optimizer has determined to use an initial plan involving the Nested Loops join. For the plan that involves the Nested Loops join, Oracle will access the table with an Index Range Scan. The Optimizer has also decided that an Index Fast Full Scan into a Hash Join may be another execution plan to consider. Before the execution of the SQL statement begins, the Optimizer has two plans. One plan is the default plan and the other plan is the adaptive plan. Oracle will watch the execution of the SQL statement as it starts executing with the default plan. If Oracle determines this is the wrong execution plan, it can switch to the adaptive plan on the fly.

For more info on Adaptive Plans, see this white paper from Oracle. Pages 3-5 illustrate Adaptive Join Plans in action.

The gray text in the Explain Plan above are simply the adaptive plan.  SQL Developer 4.1 shows both the default plan and the adaptive plan.

Further down in the Explain Plan is the Other XML section, which can be seen below. We can see my db version is 12.1.0.2. We can also the info type shows this is an Adaptive Plan.

Note the section titled “rows”. We can see each row of the Explain Plan. Those lines where “skp=0″ are part of the default plan. The lines where “skp=1″ are the adaptive plan, which we now know appear as gray text in SQL Developers.

Over the years, I’ve seen people ask questions on where they can access a demo Oracle database so that they can practice and learn SQL or PL/SQL. Until today, I never knew of any available Oracle testbed for public consumption. Now, Oracle Corp has provided us with LiveSQL.oracle.com and the only thing you need to access the site is a free OTN account.

Live SQL lets you access a SQL Worksheet where you can enter SQL and PL/SQL commands. You also have read-only access to the HR and SH sample schemas.

You can even create your own database objects.

A schema browser is included.

Can you can save a library of scripts. Live SQL also lets you access Community Code, a growing repository of other’s shared code.

So for those of you that want access to a demo environment, Oracle now provides one. That being said, I still think there is huge value in downloading the Oracle RDBMS from OTN and installing it in your own private testbed. DBA’s especially, will learn so much more from this more traditional route than Live SQL.  For me, Live SQL will be great for those times when I just need to demo something and don’t have easy access to an Oracle database. But then this site probably wasn’t thinking of users like me as their target audience.

I recently upgraded a critical production system from Oracle 11.2.0.4 to 12.1.0.2. This is also a 3-node Oracle RAC database. I ran into a few issues which needed resolution after the upgrade. Hopefully this helps out others in their upgrades.

The first issue was related to a Global Temp Table that is central to our custom app. The GTT was involved in too many Cartesian Products. The solution was to leverage Oracle 12c’s new Session Stats for GTT’s. This AskTom article discusses the new session stats. Tim Hall has another great article on the subject as well. So to solve the problem, I just added an appropriate call to DBMS_STATS in our package code where the GTT was populated with data. Unfortunately, I used GATHER_TABLE_STATS similar to the examples in the articles I linked above. In an environment with a high degree of concurrency, this led to too many ‘library cache pin’ issues. I modifed the package code to use SET_TABLE_STATS instead…much faster. All I needed to do was compute the number of rows, and with the knowledge of what a typical average row length would be, the rest of the stats were easy to compute. This alleviated the pin contention and that issue was resolved.

The next issue I had to fix was due an ORA-22990 error: “LOB Locators cannot span transactions”. A dev in our shop fixed the issue so I never determined the exact root cause. But something in 12c changed because the package with this code has been unchanged for at least the last 7 years. I have two guesses as to the root cause…either a supplied package is now issuing a COMMIT that wasn’t there before or a supplied package was spawning a new transaction. The dev just changed where we initialize the LOB locator to be a spot later in the code.

I’ve had Oracle 12c in our test pipleline since August, or about 2 months before we hit production. Its a fair question to ask why testing didn’t capture these issues prior to the prod upgrade. For the GTT table issue, we simply did not have enough load on the system with people hitting the same data points as our end users do. For the LOB Locator issue, the part of the application receiving this error is only executed during mid-to-late October, as I later found out. So it was never executed during our testing period. This is why we missed it during testing.

As I was fixing a few other minor performance issues (all resolved by updating table/index stats) and addressing the issues above, I learned that Oracle changed their stance on the free Extended Support for Oracle 11.2.0.4. I upgraded last Wednesday evening and on Friday, Oracle announced that free ES for 11.2.0.4 would be extended until May of 2017! One of the main drivers for me upgrading to 12c was to ensure that our mission-critical database would be supported and the initial free ES for 11.2.0.4 was to expire in January, just 4 months away. As I’m working through upgrade issues and I learned of this ES extension, I was starting to wonder if the announcement had a message (reading between the lines) that 12.1.0.2 wasn’t ready for prime time. I actually started the investigation on what it would take for me to downgrade this database. But I was able to work through all of the issues and today, life seems to be just great on 12.1.0.2.

SQL Developer 4.1.2 was just released today. The new version can be downloaded from the usual place on OTN. I had no problems installing it on my MacBook Pro.

Today at Oracle Open World 2015, I learned that Oracle12cR2 is now in beta. I attended a session from Oracle VP Andy Mendelsohn where he highlighted a few of the new features found in 12.2. This is not an exhaustive list, but contains some highlights from that session.

• Sharding for web-scale applications
• Table and column names can be longer than 30 bytes, now up to 128 bytes.
• AWR reports on Active Data Guard standby databases.
• In-Memory operations can run on Active Data Guard standby databases.
• Application Container
• PDB Hot Clones
• PDB Refresh is just a click away.
• PDB Relocate with zero downtime

Here is the official press release.  https://www.oracle.com/corporate/pressrelease/db12r2-beta-oow-102615.html

I attended a session today at Oracle Open World 2015 by Markus Michalewicz which highlighted some new features we can look forward to for Oracle Grid Infrastructure 12.2, currently in beta.  There were lots of new features, but a few stood out as major changes to the architecture.

Oracle 12.1 introduced Flex Clusters about two years ago. With Flex Clusters, you can have Hub Nodes and Leaf Nodes, the primary difference being that Leaf Nodes do not need access to the shared storage of the cluster. When this was announced, many people assumed you could run an Oracle RAC instance on a Leaf Node, but this was not the case. Not until Oracle 12.2 anyway. With 12.2, you can now run an Oracle RAC instance on a Leaf Node. There is one catch though. Since the Leaf Node does not have access to shared storage, there is no access to Online Redo Logs which means the instances on the Leaf Nodes are read-only to the data. This is a great way to offload reporting activities similar to the way people have been leveraging Active Data Guard. I haven’t seen this in action, but my initial thoughts would be that the DBA needs to weigh the performance carefully. Surely offloading reporting capabilities has its merits but it comes at the cost of additional latency when an instance on a Leaf Node needs to contact an instance on a Hub Node and transfer the blocks via Cache Fusion. After all, the Leaf Node has no direct access to disk so it must need to get the data block via a Hub Node.

The other big change I wanted to discuss is in relation to the GIMR database. The Grid Infrastructure Management Repository is a database to help collect performance metrics, introduced in Oracle 12.1.0.1. The 12.1.0.2 version made the GIMR mandatory. This database runs on only one node of the Oracle RAC cluster. In GI 12.2, we can now have a centralized GIMR on one cluster that other GI clusters can leverage. There are a whole host of ramifications for those that get to enjoy many RAC clusters in their enterprise.

While watching the presentation, it didn’t seem like these were major announcements. It almost sounded low-key. But I think that as we get to play with the 12.2 version, we’ll get a better handle of how much Oracle GI/RAC features just got a lot more expansive, which some might read as more complicated.

I’ve been working with MongoDB a bit and one of the things I like about it is how sharding helps the platform achieve high performance. MongoDB is not the only database platform that uses sharding, just one of the more mainstream ones. While I’ve been working with MongoDB and admiring sharding and the ability to handle JSON documents, I’ve also been troubled by a few other things. This is not meant as a total trashing of MongoDB as many database platforms, especially the new players on the block, suffer from the same things. This isn’t an exhaustive list, but here are some of the things I wish I could find in MongoDB.

• ACID compliance – The idea of eventually consistent never sits well with the DBA in me that wants to protect critical corporate data.
• Security – I come from Oracle, the most secure database around. No other DB platform has the level of security features found in Oracle.
• Tools – I’m used to Oracle Enterprise Manager for monitoring/managing my Oracle environment. I rely heavily on all that the Diagnostics Pack provides. My new favorite tool is Lighty for Oracle. Being a performance tuning specialist, I appreciate how well instrumented the Oracle kernel is and how many things I can leverage to resolve database performance issues. No other database comes close.
• Backup and Recovery – RMAN automates everything and takes all the hard work out of the picture when backing up and recovering the database.

Well if I’m looking for web-scale, Oracle is now going to provide it with 12cR2. Oracle 12c brought native JSON support, which is great. Oracle 12.2 will now let me create a sharded table and enjoy that web-scale-ability that MongoDB provides. My applications can connect to an Oracle database and enjoy sharded and non-sharded tables with the same database connection. I don’t lose any of the other stuff I’ve grown so fond of.

I heard a few other people walking around talking about 12.2’s new sharding ability. To me, this is a major point and could probably be at the top of the list of highlights from OOW15.

After our recent upgrades to Oracle 12.1.0.2, we noticed a few non-production databases that were filling up the DIAGNOSTIC_DEST. There were huge trace files in there which contained this type of information:

----- Cursor Obsoletion Dump sql_id=6aca4drrt76x3 -----
Parent cursor obsoleted 1 time(s).

It appears this is a known bug in 12.1.0.2. Oracle turned on some diagnostic info and forgot to turn it off. But you can turn it off in your environments with the following:

alter system set "_kks_obsolete_dump_threshold" = 0;

I’m leaving Open World 2015 a day early. I managed to pack in an awful lot of activity into the three days I was here and I met up with some very nice people. I expect that my OOW experience was similar to many others. There was so much to do and not enough time. But I’m happy to be going home as well.

I spent most of my Monday attending sessions and learning about the exciting new changes coming out in Oracle 12cR2 (due to be released in 2016).There was a lot of talk about cloud, Cloud and more CLOUD, which is no surprise. “On-Prem” is now a new word I learned this week. I blogged about 12.2’s new Sharding capabilities, which I think will be a big deal down the road. The ability to add sharding to the world’s #1 database will only make the engine more competitive. There are a number of new RAC features coming as well. Some of these made my head spin but I’m sure I’ll make sense of it sooner or later. One of the sessions I attended today even discussed some new Data Guard features.

I never once visited any vendor booth on the exhibition floor, which was a first for me. I never saw any vendors that I really wanted to talk to and part of it is because I was so busy doing other things. In the Exhibition Hall, I did spend some time on the Demo Grounds. I like these because, as I tweeted out, this gives me a chance to ask pointed questions out a specific feature. Most people like to use the Demo Grounds to get a little demo of that feature. But typically for me, I’ve already seen the feature or been exposed to it, but I like to learn just a bit more about that feature and the expert is right there just waiting for me to ask.

I met up with a number of people whom I’ve never seen before in person but had virtual conversations with in some social media environment in the past. Its always good to put a face with the name. And there were some people I wanted to meet up with but our collective schedules didn’t quite allow it. I was invited to a meetup event where I was able to interact with people that I know from the OTN or MOSC communities. All the people there I knew from these forums over the years but had never had a face to face discussion with so it was nice to talk them in person. The collection of people in the room spanned a number of countries all over the world and their expertise covered many different areas of the forums, so we had quite the diverse group.

I wanted to get to more of the Oak Table World events, but I only got to one: Christian Antognini’s discussion about Adaptive Dynamic Sampling. Next time I’ll try to get a few more of these as I’m sure all of them are excellent.

And then there are all of the things that I did not do. Oh well..maybe next time! I’m not sure if I’ll make it back for OOW16, but if not, then OOW17 for sure.

After a recent upgrade to 12.1.0.2, I’ve been working on a number of performance issues. Many such issues are related to poor SQL and a number of issues I’ve resolved I’ve proven were issues in the old 11.2.0.4 release. This just means it’s always been an issue. But people are taking the opportunity of the upgrade to get me to fix things that have been broken for quite a while.

While looking at performance issues, I’ve come across two SQL statements that are becoming pigs in our system.  Here is a screenshot of the two SQL statements as seen in Lighty:

We can see that the first SQL statement (SQL ID 4b4wp0a8dvkf0) is consuming CPU and waiting on reads from the Control File. The second SQL statement (SQL ID frjd8zfy2jfdq) is using lots of CPU and has a number of other wait events as well. Here is the SQL text of these statements.

SQL ID: frjd8zfy2jfdq

SELECT

    executions

    ,end_of_fetch_count

    ,elapsed_time / px_servers elapsed_time

    ,cpu_time / px_servers cpu_time

    ,buffer_gets / executions buffer_gets

  FROM

    (

      SELECT

          SUM (executions) AS executions

          ,SUM (

            CASE

              WHEN px_servers_executions > 0

              THEN px_servers_executions

              ELSE executions

            END

          ) AS px_servers

          ,SUM (end_of_fetch_count) AS end_of_fetch_count

          ,SUM (elapsed_time) AS elapsed_time

          ,SUM (cpu_time) AS cpu_time

          ,SUM (buffer_gets) AS buffer_gets

        FROM

          gv$sql

        WHERE

          executions > 0

          AND sql_id = : 1

          AND parsing_schema_name = : 2

SQL ID: 4b4wp0a8dvkf0

SELECT

    executions

    ,end_of_fetch_count

    ,elapsed_time / px_servers elapsed_time

    ,cpu_time / px_servers cpu_time

    ,buffer_gets / executions buffer_gets

  FROM

    (

      SELECT

          SUM (executions_delta) AS EXECUTIONS

          ,SUM (

            CASE

              WHEN px_servers_execs_delta > 0

              THEN px_servers_execs_delta

              ELSE executions_delta

            END

          ) AS px_servers

          ,SUM (end_of_fetch_count_delta) AS end_of_fetch_count

          ,SUM (elapsed_time_delta) AS ELAPSED_TIME

          ,SUM (cpu_time_delta) AS CPU_TIME

          ,SUM (buffer_gets_delta) AS BUFFER_GETS

        FROM

          DBA_HIST_SQLSTAT s

          ,V$DATABASE d

          ,DBA_HIST_SNAPSHOT sn

        WHERE

          s.dbid = d.dbid

          AND bitand (

            nvl (

              s.flag

              ,0

            )

            ,1

          ) = 0

          AND sn.end_interval_time > (

            SELECT

                systimestamp AT TIME ZONE dbtimezone

              FROM

                dual

          ) - 7

          AND s.sql_id = : 1

          AND s.snap_id = sn.snap_id

          AND s.instance_number = sn.instance_number

          AND s.dbid = sn.dbid

          AND parsing_schema_name = : 2

    )

    )

Both of these are part of the new Adaptive Query Optimization features now in 12c. More specifically, these relate to the Automatic Dynamic Statistics portion of this feature. SQL ID frjd8zfy2jfdq is Oracle obtaining information on SQL statement performance from GV$SQL. SQL ID 4b4wp0a8dvkf0 is Oracle obtaining the same information on SQL statement performance from the Active Session History tables.

Bertand Drouvot discusses this on his blog here: https://bdrouvot.wordpress.com/2014/10/17/watch-out-for-optimizer_adaptive_features-as-it-may-have-a-huge-negative-impact/

Additionally, I sat in on a session by Christian Antognini at Oak Table World 2015 where he mentioned these SQL statements. His slides from OTW are pretty much the same as these:

http://www.soug.ch/fileadmin/user_upload/SIGs/SIG_150521_Tuning_R/Christian_Antognini_AdaptiveDynamicSampling_trivadis.pdf

Those links above and the MOS Notes I reference below provided much of the basis of the information I present here.

All of the Adaptive Query Optimization features are supposed to make the DBA’s life better. They are supposed to help the Optimizer make better decisions, even after a SQL statement has started executing. In this specific case, these queries are supposed to help the CBO obtain better statistics, even if the stats are missing. This can help improve SQL performance, but as I found in my case, it is hampering overall system performance.

For more on Adaptive Query Optimization, see Note 2031605.1. This will lead you to other notes, but in particular to this discussion, Note 2002108.1 Automatic Dynamic Statistics.

Making matters worse is that my production system seeing this behavior is Oracle RAC. When SQL ID frjd8zfy2jfdq is performed on Oracle RAC systems, parallel query is used which is obvious from my screen shot by the enq: PS – contention and “PX%” wait events.

We can turn of dynamic sampling as follows:

alter system set optimizer_dynamic_sampling=0 scope=both;

The default value of this parameter is 2.

For me, these queries are consuming resources and impacting overall database performance. Yet these features are designed to improve performance. There is always a risk that if I turn off the feature to help performance in one area, it will hurt performance in another area. But since optimizer_dynamic_sampling<>11 for me, I’m not using that feature to its fullest so I’m not obtaining all of the benefit I could be. Also, our code is not relying on dynamic sampling to occur. So it’s safe for me to turn this off.

After changing the parameter, I could see an immediate change as shown below.

The red line indicates the time I made the change. It is clear that the ASH version of this query is no longer executing. The V$SQL version is still executing but no longer seeing parallel query wait events. It’s mostly just consuming CPU now. I consider this progress, but not a full resolution.

As a side note, I could have turned off all Adaptive Query Optimization features with the following:

alter system set optimizer_adaptive_features=false scope=both;

But I do know that I have queries “enjoying” Adaptive Join Optimization and I don’t want to turn it all off, just dynamic sampling.

So now what to do with SQL ID frjd8zfy2jfdq? Let’s see if we can resolve the remaining issue. From one of the MOS Notes I linked above, I know that we can set this hidden parameter:

alter system set "_optimizer_dsdir_usage_control"=0 scope=both;

The default value for this hidden parameter was 126 in my 12.1.0.2 system. I found the default setting with the following:

select a.ksppinm name, b.ksppstvl value, b.ksppstdf deflt,

from

  sys.x$ksppi a,

  sys.x$ksppcv b

where  a.indx = b.indx

  and  a.ksppinm like '\_%' escape '\'

  and  a.ksppinm like '%dsdir%'

order by name;

This value is important in case I want to set it back without having to take the parameter out of the SPFILE, which would require downtime.

I can now proceed with changing this hidden parameter and setting it to zero. Here is how that SQL statement looks in Lighty after the change:

Looks to be “mission accomplished” in stopping those two SQL statements from executing.

Those running 12.1.0.2 on Oracle RAC may want to verify that these two SQL statements are not causing performance problems of their own.

This seems to be one of those cases where a feature that is supposed to help performance actually does the opposite.

Today I was trying to add a new node to an existing Oracle RAC cluster. Grid Infrastructure is version 12.1.0.2 running on Oracle Linux 5.11.

I ran through the usual Cluster Verification Utility (CVU) checks and all checks passed. To add the node, I issued the following command from the $GRID_HOME/addnode directory.

./addnode.sh -silent "CLUSTER_NEW_NODES={newnode}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={newnode-vip}"

The operation would fail. The log file showed the following error:

INFO: *********************************************
INFO: Task resolv.conf Integrity: This task checks consistency of file /etc/resolv.conf file across nodes
INFO: Severity:CRITICAL
INFO: OverallStatus:OPERATION_FAILED

I know for certain that there are no issues with resolv.conf. It has been working just fine on the two existing nodes. Everything checks out. This hasn’t been my first issue with resolv.conf over the years. Normally, I know that the OUI gets the check wrong and I ignore the condition and continue.

I did find some help via Google to set the following environment variable before running addnode.sh:

export IGNORE_PREADDNODE_CHECKS=Y

This sounded promising. However, it made no difference and I received the same error. Since addnode.sh is calling the OUI in silent mode, I figured I would try the directive to ignore prerequisites.

./addnode.sh -silent "CLUSTER_NEW_NODES={drorp05}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={drorp05-vip}" -ignoreSysPrereqs

I still received the same error. So how do I get addnode.sh to get past the failed check? The solution was too obvious to be seen initially.

The addnode.sh script just calls the OUI. So the answer is to run just the following:

./addnode.sh

Run the script with no parameters. This fires up the GUI version of the OUI. The next screen asks for the new host name and its VIP, which were previously parameters to the script. The checks were run and of course, a failure was seen. But this time, I could check the box to Ignore the failures and continue with the installation.