Moving Grid Infrastructure Components in Oracle 18c

posted February 21, 2019, 10:51 AM by

by Jeff Stonacek, Principal Architect

I recently ran into an interesting issue with Oracle 18c Grid Infrastructure (GI).  We were building several three-node clusters for a client and on one of the clusters, the ASM diskgroup used for GI was named differently than all of the other clusters.  Obviously, the diskgroup name is arbitrary, and has no effect on the operation of Clusterware.  However, we decided it was a good idea to fix the problem so that all of the GI diskgroups had the same name (which worked for me since I can be slightly OCD).

There are a couple of ways to approach this change.  The simplest way is to deconfigure Grid Infrastructure on all nodes, rebuild the diskgroup with the correct name, and then reconfigure Grid Infrastructure.  This method requires a bit of downtime however, so we decided to see what we could do to avoid downtime as much as possible.

Another option is to move all of the Grid Infrastructure components to a new diskgroup, which is the method I describe in this blog.

Move Grid Infrastructure

This blog covers moving version 18c GI components from one diskgroup to another.  Every version of Oracle introduces more components than the previous version.  Oracle 18c however, is very similar to 12c.  The following components were moved during this exercise:

  • Oracle Cluster Registry (OCR)
  • Voting Disks
  • ASM spfile
  • ASM password file
  • Grid Infrastructure Management Repository (GIMR) database

 

The obvious first step in this process is to create a new ASM diskgroup with the correct name.  Creating an ASM diskgroup is a routine operation and does not need any further elaboration.  Just create a new diskgroup as you would any other.

In the following examples, the original diskgroup name is CRS.  The new diskgroup name is OCR.

Oracle Cluster Registry (OCR)
Oracle provides utilities to move the Cluster Registry.  The ocrcheck and ocrconfig commands do the work for you.  First check the status of the OCR with ocrcheck.  Then login as root to manage the OCR and set the Oracle environment to the GI Home.

# export ORACLE_HOME=/u01/grid/1830
# export PATH=$ORACLE_HOME/bin:$PATH

# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          4
Total space (kbytes)     :     491684
Used space (kbytes)      :      84388
Available space (kbytes) :     407296
ID                       : 1573228169
Device/File Name         :       +CRS

 

Now add a copy of the OCR to the +OCR diskgroup and remove the original copy from the +CRS diskgroup.

# ocrconfig -add +OCR
# ocrconfig -delete +CRS

 

The OCR is now located in the +OCR diskgroup.

# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          4
Total space (kbytes)     :     491684
Used space (kbytes)      :      84372
Available space (kbytes) :     407312
ID                       : 1573228169
Device/File Name         :       +OCR
Device/File integrity check succeeded

 

Voting Disks
Next, we’ll tackle the voting disks.  Unlike the ocrconfig command, these commands can be completed as the GI Home owner.

First, query the registry to see where the existing voting disks are located.

$ export ORACLE_HOME=/u01/grid/1830
$ export PATH=$ORACLE_HOME/bin:$PATH

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
  1.ONLINE 7902fa0a196d4f13bf08dcd022d37b26 (/dev/asm/ASM_CRS01) [CRS]
Located 1 voting disk(s).

 

Then run the crsctl command to move the voting disk.

$ crsctl replace votedisk +OCR

 

The voting disks are now located in the +OCR diskgroup.

$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1.ONLINE a6ad595bd0a04f92bf3fbeef8d672e9b (/dev/asm/ASM_OCR01) [OCR]
Located 1 voting disk(s).

 

ASM Password File
Check to see where the password file is located for the ASM instance.

srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

 

Use the pwcopy command to move the password files in ASM.

pwcopy +CRS/orapwASM +OCR/orapwASM
pwcopy +CRS/orapwASM_backup +OCR/orapwASM_backup

 

Modify the ASM instance to set the password file location.

srvctl modify asm -pwfile +OCR/orapwASM
srvctl modify asm -pwfilebackup +OCR/orapwASM_backup

 

The password file is now on the new diskgroup.

srvctl config asm
ASM home: <CRS home>
Password file: +OCR/orapwASM
Backup of Password file: +OCR/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM

 

ASM Spfile
When creating an 18c cluster, the default is to put the spfile for the ASM instance in ASM.  However, you can run the following steps to move the ASM spfile to the new diskgroup.

Using SQLPlus, create a pfile from the running ASM instance and then create a spfile in the new diskgroup.  In the GI Home, connect to the ASM instance.

. oraenv
+ASM1

sqlplus / as sysasm

create pfile='/tmp/initasm.ora' from spfile;
create spfile='+OCR' from pfile='/tmp/initasm.ora';

exit

 

Now, as the Grid owner, run the gpnptool to update the GPnP profile with the new spfile.

gpnptool get

 

At this point, we need to restart cluster services in order for the ASM spfile change take effect.  This can be done in a rolling fashion.  For each node in the cluster, stop and then start cluster services.  Do this one node at a time so the cluster services remain available on the surviving nodes during the restart.

crsctl stop crs

 

Once Grid Infrastructure is down, start things back up.

crsctl start crs

 

Check the status of the restart with the following command.

crsctl stat res -t

 

Grid Infrastructure Management Repository (GIMR) Database
The last component that needs to be moved is the GIMR database.  First, the GIMR database needs to be located.

$ srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node rac182

 

Step one in moving the GIMR database is to stop the ora.crf service on each node of the cluster.  Login to each node, as root, and run the following commands from the GI Home.

# export ORACLE_HOME=/u01/grid/1830
# export PATH=$ORACLE_HOME/bin:$PATH

# ctl stop res ora.crf -init
bash: ctl: command not found...
[root@rac181 ~]# crsctl stop res ora.crf -init
CRS-2673: Attempting to stop 'ora.crf' on 'rac181'
CRS-2677: Stop of 'ora.crf' on 'rac181' succeeded

# crsctl modify res ora.crf -attr ENABLED=0 -init

 

Connect to the node where the GIMR database is running, as the GI Home owner.  From the GI Home, run the following dbca command to delete the database.

$ dbca -silent -deleteDatabase -sourceDB -MGMTDB
[WARNING] [DBT-19202] The Database Configuration Assistant will delete the Oracle instances and datafiles for your database. All information in the database will be destroyed.
Prepare for db operation
32% complete
Connecting to database
35% complete
39% complete
42% complete
45% complete
48% complete
52% complete
65% complete
Updating network configuration files
68% complete
Deleting instance and datafiles
84% complete
100% complete
Database deletion completed.
Look at the log file "/u01/gridbase/cfgtoollogs/dbca/_mgmtdb/_mgmtdb0.log" for further details.

 

Now, recreate the GIMR container database using the new diskgroup.

dbca -silent -createDatabase -createAsContainerDatabase true \
-templateName MGMTSeed_Database.dbc \
-sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName +OCR \
-datafileJarLocation /u01/grid/1830/assistants/dbca/templates -characterset AL32UTF8 \
-autoGeneratePasswords -skipUserTemplateCheck

Prepare for db operation
10% complete
Registering database with Oracle Grid Infrastructure
14% complete
Copying database files
43% complete
Creating and starting Oracle instance
45% complete
49% complete
54% complete
58% complete
62% complete
Completing Database Creation
66% complete
69% complete
71% complete
Executing Post Configuration Actions
100% complete
Database creation complete. For details check the logfiles at:
/u01/gridbase/cfgtoollogs/dbca/_mgmtdb.
Database Information:
Global Database Name:_mgmtdb
System Identifier(SID):-MGMTDB
Look at the log file "/u01/gridbase/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details.

 

Next, create the GIMR pluggable database inside the container.

mgmtca -local

 

Finally, login to each node of the cluster as root and re-enable the ora.crf service.

# crsctl modify res ora.crf -attr ENABLED=1 -init
# crsctl start res ora.crf -init
CRS-2672: Attempting to start 'ora.crf' on 'rac181'
CRS-2676: Start of 'ora.crf' on 'rac181' succeeded

 

Cleanup
At this point, all components have been removed from the old +CRS diskgroup.  The diskgroup can now be removed from ASM and the disk devices decommissioned from the server.

Conclusion

In this blog, we have shown how to relocate all Grid Infrastructure components from one ASM diskgroup to another. These instructions can be used to move individual components, or to move all components and evacuate the entire diskgroup.  This method is much less invasive than deconfiguring Grid Infrastructure and rebuilding, as there is only a short, rolling outage required to restart Grid Infrastructure one node at a time.

Share with your networkTweet about this on Twitter
Twitter
Share on LinkedIn
Linkedin
Share on Facebook
Facebook
Digg this
Digg
Email this to someone
email

Leave a Reply

Your email address will not be published. Required fields are marked *

Icon URL Target
1

This site uses Akismet to reduce spam. Learn how your comment data is processed.

WANT TO LEARN MORE?