IMP: login as root for startup and shutdown
To Start CRS
Go to $CRS_HOME/bin
# ./crsctl start crs
Go to $CRS_HOME/bin
# ./crsctl start crs
Or
# /etc/init.d/init.crs start
Check CRSD process
ps -ef|grep crsd.bin
To Stop CRS
Go to $CRS_HOME/bin
# ./crsctl stop crs
Go to $CRS_HOME/bin
# ./crsctl stop crs
Or
# /etc/init.d/init.crs stop
CRS enable/disable
Go to $CRS_HOME/bin
crsctl enable crs
crsctl enable crs
crsctl disable crs # ensures that the services do not come up after a reboot.
To find out the value of your CLUSTER_NAME from CRS (OCR),
cd $CRS_HOME/bin
./cemutlo -n
lists the version of CRS software installed
crsctl query crs softwareversion []
lists the CRS software operating version
crsctl query crs activeversion
Status of CRS services
Go to $CRS_HOME/bin
bash-3.00$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.cmwdb1.gsd application ONLINE ONLINE cmwdb1
ora.cmwdb1.ons application ONLINE ONLINE cmwdb1
ora.cmwdb1.vip application ONLINE ONLINE cmwdb1
ora.cmwdb2.gsd application ONLINE ONLINE cmwdb2
ora.cmwdb2.ons application ONLINE ONLINE cmwdb2
ora.cmwdb2.vip application ONLINE ONLINE cmwdb2
Check current OCR file
bash-3.00$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 102188
Used space (kbytes) : 2140
Available space (kbytes) : 100048
ID : 617463612
Device/File Name : /dev/vx/rdsk/racdg/rac_srvconfig
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
Check current CRS file
bash-3.00$ crsctl query css votedisk
0. 0 /dev/vx/rdsk/racdg/crs
located 1 votedisk(s).
Files need to backed for CRS
CRS HOME and startup scripts
$CRS_HOME
/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/var/opt/oracle
/etc/inittab
OCR file
bash-3.00$ ocrconfig -showbackup
cmwdb1 2010/09/27 12:17:02 /oracle/crs/cdata/crs
cmwdb1 2010/09/27 08:17:01 /oracle/crs/cdata/crs
cmwdb1 2010/09/27 04:17:00 /oracle/crs/cdata/crs
cmwdb1 2010/09/26 12:16:57 /oracle/crs/cdata/crs
cmwdb1 2010/09/19 08:16:08 /oracle/crs/cdata/crs
CRS file
bash-3.00$ crsctl query css votedisk
0. 0 /dev/vx/rdsk/racdg/crs
dd if=/dev/vx/rdsk/racdg/crs of=/oracle/crs/data/crs/crs_backup.dmp bs=4k
Go to $CRS_HOME/bin
bash-3.00$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.cmwdb1.gsd application ONLINE ONLINE cmwdb1
ora.cmwdb1.ons application ONLINE ONLINE cmwdb1
ora.cmwdb1.vip application ONLINE ONLINE cmwdb1
ora.cmwdb2.gsd application ONLINE ONLINE cmwdb2
ora.cmwdb2.ons application ONLINE ONLINE cmwdb2
ora.cmwdb2.vip application ONLINE ONLINE cmwdb2
Check current OCR file
bash-3.00$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 102188
Used space (kbytes) : 2140
Available space (kbytes) : 100048
ID : 617463612
Device/File Name : /dev/vx/rdsk/racdg/rac_srvconfig
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
Check current CRS file
bash-3.00$ crsctl query css votedisk
0. 0 /dev/vx/rdsk/racdg/crs
located 1 votedisk(s).
Files need to backed for CRS
CRS HOME and startup scripts
$CRS_HOME
/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/var/opt/oracle
/etc/inittab
OCR file
bash-3.00$ ocrconfig -showbackup
cmwdb1 2010/09/27 12:17:02 /oracle/crs/cdata/crs
cmwdb1 2010/09/27 08:17:01 /oracle/crs/cdata/crs
cmwdb1 2010/09/27 04:17:00 /oracle/crs/cdata/crs
cmwdb1 2010/09/26 12:16:57 /oracle/crs/cdata/crs
cmwdb1 2010/09/19 08:16:08 /oracle/crs/cdata/crs
CRS file
bash-3.00$ crsctl query css votedisk
0. 0 /dev/vx/rdsk/racdg/crs
dd if=/dev/vx/rdsk/racdg/crs of=/oracle/crs/data/crs/crs_backup.dmp bs=4k
CRS Log files location
$CRS_HOME/log/hostname
cluster verification utlity
cluvfy comp crs -n all -verbose
cluvfy comp ocr -n all -verbose
cluvfy comp nodecon -n all -verbose
olsnodes
cluvfy comp -list
cluvfy stage -list
cvuqdisk
cluvfy stage -post crsinst -n all -verbose # as oracle
Do you want to check the health of the Clusterware?
crsctl check crs
crsctl check evmd
crsctl check cssd
crsctl check crsd
# OEM User Defined OS Metric.
Script is called by OEM to monitor the health of CRSd on a particular node.
# UserDefinedHostScripts/checkCRS.sh
The script runs the "crsctl check crsd" command. The normal output
is "CRS appears healthy". Any other response will trigger an alert.
The alert is configured to send an email and an snmp trap to Unicenter.
#!/usr/bin/ksh
if [[ $(/ora01/oracle/product/crs_1020/bin/crsctl check crsd) = "CRS appears healthy" ]]
then print "em_result=0\nem_message=CRSd is healthy\n"
else print "em_result=1\nem_message=CRSd not responding\n"
fi
exit
CRS reboot issue
Whenever a node is having issues joining the cluster back post reboot, here is a quick check list :
* /var/log/messages
* ifconfig
* ip route
* /etc/hosts
* /etc/sysconfig/network-scripts/ifcfg-eth*
* ethtool
* mii-tool
* cluvfy
* $CRS_HOME/log/hostname
To find the root cause, please provide the following from all the nodes to Oracle Support.
1)- OSWatcher output from all the nodes
2)- All log files under $CRS_HOME/log. You can use diagcollection.pl to get all log files.
3)- System OS logs "/var/log/messages".
4)- OPROCD log files under /etc/oracle/oprocd or /var/opt/oracle/oprocd.
5) Capture output of the cluvfy command.
Execute the following commands as oracle:
>ORACLE_SID=crs;ORAENV_ASK=NO;. oraenv;ORAENV_ASK=YES
>HOST=$(hostname)
>cd /opt/oracle/admin/SR
>script $HOST.cluvfy.log
>cluvfy stage -post crsinst -n all -verbose
>exit
Upload the files to Oracle in support of your SR.
Using the diagcollection.pl procedure
Run the following commands on each node. These commands must be run as root.
>export ORA_CRS_HOME=/ora01/oracle/product/crs_1020
>export ORACLE_BASE=/ora01/oracle
>export ORACLE_HOME=/ora01/oracle/product/crs_1020
>export HOSTNAME=$(hostname)
>cd $ORA_CRS_HOME/bin
>script diag.log
>env
>id
>./diagcollection.pl -collect
>exit
This creates the
diag.log file in $ORA_CRS_HOME/bin and
these files crsData_.tar.gz,
ocrData_.tar.gz,
oraData_.tar.gz and
basData_.tar.gz
If you get a prompt to overwrite existing tar.gz files, go ahead and overwrite.
This is a perl script, so /bin/perl has to be in your PATH.
