当RAC安装遇到问题需要重新安然或者集群重要组件OCR损坏等场景,需要重装集群环境时,需要对集群配置进行重新配置(Re-configure);本文对此过程进行测试整理,如下:
参考文档:
How to Configure or Re-configure Grid Infrastructure With config.sh/config.bat (文档 ID 1354258.1)
How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (文档 ID 1377349.1)
测试环境为LINUX64位+两节点RAC 11.2.0.4.160719.
OCR/voting与数据在同一个ASM磁盘组–DATA.
如下记录Deconfigure/Reconfigure的过程。
1.检查并记录当前集群配置–输出省略
crsctl stat res -t
crsctl stat res -p
crsctl query css votedisk
ocrcheck
oifcfg getif
srvctl config nodeapps -a
srvctl config scan
srvctl config asm -a
srvctl config listener -l listener -a
srvctl config database -d ludarac -a
2.解除整个集群的配置
如果 OCR 和 Voting Disk 没有在 ASM 上面,或者在 ASM 上面且为单独的磁盘组–无业务数据
在所有远程节点,使用 root 执行:
# <$GRID_HOME>/crs/install/rootcrs.pl -deconfig -force -verbose
一旦上面命令在所有节点执行完毕,在本地节点,使用 root 用户执行:
# <$GRID_HOME>/crs/install/rootcrs.pl -deconfig -force -verbose -lastnode
如果 OCR 或 Voting Disks 在 ASM 并且有用户数据此磁盘组中:
如果 GI 版本是 11.2.0.3 并且 bug 13058611 和 bug 13001955 被安装,或者 GI 版本是 11.2.0.3.2 GI PSU 或者更高:
在所有远程节点,使用 root 执行:
# <$GRID_HOME>/crs/install/rootcrs.pl -deconfig -force -verbose
一旦上面命令在所有节点执行完毕,在本地节点,使用 root 用户执行:
# <$GRID_HOME>/crs/install/rootcrs.pl -deconfig -force -verbose -keepdg -lastnode
如下为解除配置过程:
节点1:
[root@luda1 ~]# /u01/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
Network exists: 1/192.168.57.0/255.255.255.0/eth0, type static
VIP exists: /luda1-vip/192.168.57.216/192.168.57.0/255.255.255.0/eth0, hosting node luda1
VIP exists: /luda2-vip/192.168.57.218/192.168.57.0/255.255.255.0/eth0, hosting node luda2
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2613: Could not find resource ‘ora.registry.acfs’.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘luda1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘luda1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘luda1’
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘luda1’
CRS-2673: Attempting to stop ‘ora.ludarac.db’ on ‘luda1’
CRS-2677: Stop of ‘ora.ludarac.db’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘luda1’
CRS-2677: Stop of ‘ora.oc4j’ on ‘luda1’ succeeded
CRS-2672: Attempting to start ‘ora.oc4j’ on ‘luda2’
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘luda1’
CRS-2677: Stop of ‘ora.asm’ on ‘luda1’ succeeded
CRS-2676: Start of ‘ora.oc4j’ on ‘luda2’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘luda1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘luda1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘luda1’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘luda1’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘luda1’
CRS-2677: Stop of ‘ora.ctssd’ on ‘luda1’ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘luda1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘luda1’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘luda1’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘luda1’
CRS-2677: Stop of ‘ora.cssd’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.crf’ on ‘luda1’
CRS-2677: Stop of ‘ora.crf’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘luda1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘luda1’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘luda1’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘luda1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘luda1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
节点2:
[root@luda2 ~]# crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
ONLINE ONLINE luda2
ora.LISTENER.lsnr
ONLINE ONLINE luda2
ora.asm
ONLINE ONLINE luda2 Started
ora.gsd
OFFLINE OFFLINE luda2
ora.net1.network
ONLINE ONLINE luda2
ora.ons
ONLINE ONLINE luda2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE luda2
ora.luda2.vip
1 ONLINE ONLINE luda2
ora.ludarac.db
1 ONLINE OFFLINE
2 ONLINE ONLINE luda2 Open
ora.cvu
1 ONLINE ONLINE luda2
ora.oc4j
1 ONLINE ONLINE luda2
ora.scan1.vip
1 ONLINE ONLINE luda2
[root@luda2 ~]# /u01/11.2.0/grid/crs/install/rootcrs.pl -deconfig -force -verbose -keepdg -lastnode
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
CRS resources for listeners are still configured
Network exists: 1/192.168.57.0/255.255.255.0/eth0, type static
VIP exists: /luda2-vip/192.168.57.218/192.168.57.0/255.255.255.0/eth0, hosting node luda2
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2613: Could not find resource ‘ora.registry.acfs’.
CRS-4000: Command Stop failed, or completed with errors.
CRS-2613: Could not find resource ‘ora.registry.acfs’.
CRS-4000: Command Delete failed, or completed with errors.
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘luda2’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘luda2’
CRS-2673: Attempting to stop ‘ora.ludarac.db’ on ‘luda2’
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘luda2’
CRS-2677: Stop of ‘ora.ludarac.db’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘luda2’
CRS-2677: Stop of ‘ora.oc4j’ on ‘luda2’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘luda2’
CRS-2677: Stop of ‘ora.asm’ on ‘luda2’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘luda2’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘luda2’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘luda2’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘luda2’
CRS-2677: Stop of ‘ora.ctssd’ on ‘luda2’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘luda2’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘luda2’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘luda2’
CRS-2677: Stop of ‘ora.cssd’ on ‘luda2’ succeeded
CRS-2613: Could not find resource ‘ora.drivers.acfs’.
CRS-4000: Command Modify failed, or completed with errors.
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘luda2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘luda2’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘luda2’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘luda2’
CRS-2676: Start of ‘ora.diskmon’ on ‘luda2’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘luda2’ succeeded
CRS-4611: Successful deletion of voting disk +DATA.
ASM de-configuration trace file location: /tmp/asmcadc_clean2017-03-18_02-47-04-PM.log
ASM Clean Configuration START
ASM Clean Configuration END
ASM with SID +ASM1 deleted successfully. Check /tmp/asmcadc_clean2017-03-18_02-47-04-PM.log for details.
CRS-2613: Could not find resource ‘ora.drivers.acfs’.
CRS-4000: Command Delete failed, or completed with errors.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘luda2’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘luda2’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘luda2’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘luda2’
CRS-2677: Stop of ‘ora.ctssd’ on ‘luda2’ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘luda2’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘luda2’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘luda2’
CRS-2677: Stop of ‘ora.cssd’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.crf’ on ‘luda2’
CRS-2677: Stop of ‘ora.crf’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘luda2’
CRS-2677: Stop of ‘ora.gipcd’ on ‘luda2’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘luda2’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘luda2’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘luda2’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Removing Trace File Analyzer
Successfully deconfigured Oracle clusterware stack on this node
至此解除集群配置完成。
3.在节点1上运行重新配置脚本
grid用户:节点1图形界面运行 $GRID_HOME/crs/config/config.sh
分别在两个节点运行root.sh脚本:
[root@luda1 ~]# sh /u01/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization – successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to oracle-ohasd.conf
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘luda1’
CRS-2676: Start of ‘ora.mdnsd’ on ‘luda1’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘luda1’
CRS-2676: Start of ‘ora.gpnpd’ on ‘luda1’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘luda1’
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘luda1’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘luda1’ succeeded
CRS-2676: Start of ‘ora.gipcd’ on ‘luda1’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘luda1’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘luda1’
CRS-2676: Start of ‘ora.diskmon’ on ‘luda1’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘luda1’ succeeded
Disk Group DATA mounted successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 47b91b4201274f50bfd80f57ed0b29d7.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 47b91b4201274f50bfd80f57ed0b29d7 (/dev/asm-diskb) [DATA]
Located 1 voting disk(s).
CRS-2672: Attempting to start ‘ora.asm’ on ‘luda1’
CRS-2676: Start of ‘ora.asm’ on ‘luda1’ succeeded
CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘luda1’
CRS-2676: Start of ‘ora.DATA.dg’ on ‘luda1’ succeeded
Preparing packages for installation…
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster … succeeded
[root@luda2 ~]# cd /u01/11.2.0/grid/
[root@luda2 grid]# sh root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization – successful
Adding Clusterware entries to oracle-ohasd.conf
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node luda1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Preparing packages for installation…
cvuqdisk-1.0.9-1
Configure Oracle Grid Infrastructure for a Cluster … succeeded
[root@luda2 grid]# crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.asm
ONLINE ONLINE luda1 Started
ONLINE ONLINE luda2 Started
ora.gsd
OFFLINE OFFLINE luda1
OFFLINE OFFLINE luda2
ora.net1.network
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.ons
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.registry.acfs
ONLINE ONLINE luda1
ONLINE ONLINE luda2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE luda1
ora.luda1.vip
1 ONLINE ONLINE luda1
ora.luda2.vip
1 ONLINE ONLINE luda2
ora.cvu
1 ONLINE ONLINE luda1
ora.oc4j
1 ONLINE ONLINE luda1
ora.scan1.vip
1 ONLINE ONLINE luda1
4.添加数据库、监听等资源
添加监听
[grid@luda1 ~]$ srvctl start listener -l listener -n luda1
[grid@luda1 ~]$ srvctl start listener -l listener -n luda2
添加数据库
[oracle@luda1 ~]$ srvctl add database -d ludarac -o /u01/app/oracle/product/11.2.0/dbhome_1 -p +DATA/ludarac/spfileludarac.ora
[oracle@luda1 ~]$ srvctl add instance -d ludarac -i ludarac1 -n luda1
[oracle@luda1 ~]$ srvctl add instance -d ludarac -i ludarac2 -n luda2
[oracle@luda1 ~]$ srvctl config database -d ludarac
Database unique name: ludarac
Database name:
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/ludarac/spfileludarac.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ludarac
Database instances: ludarac1,ludarac2
Disk Groups:
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@luda1 ~]$ srvctl start database -d ludarac
———————————
添加DB时的小问题:
[oracle@luda1 ~]$ srvctl add database -d ludarac -o /u01/app/oracle/product/11.2.0/dbhome_1 -p +DATA/ludarac/spfileludarac.ora
PRCS-1007 : Server pool ludarac already exists
PRCR-1086 : server pool ora.ludarac is already registered
解决:
[grid@luda1 ~]$ crsctl delete serverpool ora.ludarac
————————
检查资源状态:
[root@luda2 grid]# crsctl stat res -t
——————————————————————————–
NAME TARGET STATE SERVER STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.LISTENER.lsnr
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.asm
ONLINE ONLINE luda1 Started
ONLINE ONLINE luda2 Started
ora.gsd
OFFLINE OFFLINE luda1
OFFLINE OFFLINE luda2
ora.net1.network
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.ons
ONLINE ONLINE luda1
ONLINE ONLINE luda2
ora.registry.acfs
ONLINE ONLINE luda1
ONLINE ONLINE luda2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE luda1
ora.luda1.vip
1 ONLINE ONLINE luda1
ora.luda2.vip
1 ONLINE ONLINE luda2
ora.ludarac.db
1 ONLINE ONLINE luda1 Open
2 ONLINE ONLINE luda2 Open
ora.cvu
1 ONLINE ONLINE luda2
ora.oc4j
1 ONLINE ONLINE luda2
ora.scan1.vip
1 ONLINE ONLINE luda1
此时检查集群、数据库版本,均无异常,均为解除配置前的,包括打的PSU均正常。
[grid@luda1 ~]$ opatch lspatches
22502505;ACFS Patch Set Update : 11.2.0.4.160419 (22502505)
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
23054359;Database Patch Set Update : 11.2.0.4.160719 (23054359)
[oracle@luda1 ~]$ opatch lspatches
23054319;OCW Patch Set Update : 11.2.0.4.160719 (23054319)
23054359;Database Patch Set Update : 11.2.0.4.160719 (23054359)
[grid@luda1 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
SQL> select action_time,action,id,version,comments from dba_registry_history;
ACTION_TIME ACTION ID VERSION COMMENTS
—————————— ————— ———- ————— ——————————
24-AUG-13 12.03.45.119862 PM APPLY 0 11.2.0.4 Patchset 11.2.0.2.0
10-NOV-16 10.27.27.622013 AM APPLY 0 11.2.0.4 Patchset 11.2.0.2.0
10-NOV-16 01.20.24.929806 PM APPLY 160719 11.2.0.4 PSU 11.2.0.4.160719