Important files:

odmget HACMPlogs shows where are the log files
odmget HACMPcluster shows cluster version
odmget HACMPnode shows info from nodes (cluster version)
(changing the location of the log files: C-SPOC > Log Viewing and Management)
/etc/es/objrepos HACMP ODM files
HACMP Logs: (location differs on newer and older HACMP versions)

/var/hacmp/adm/log/cluster.log main PowerHA log file (errors,events,messages … /usr/es/adm/cluster.log)
/var/hacmp/adm/history/cluster.mmddyy shows only the EVENTS, generated daily (/usr/es/sbin/cluster/history/cluster.mmddyyyy)
/var/hacmp/log/hacmp.out similar to cluster.log, but more detailed (with all the output of the scripts)

/var/hacmp/clcomd/clcomd.log contains every connect request between the nodes and return status of the requests
/var/hacmp/log/clinfo.log records the activity of the clinfo daemon
/var/hacmp/clverify shows the results of the verifications (verification errors are logged here)
/var/hacmp/log/clutils.log summary of nightly verification

/tmp/clstrmgr.debug debug info about the cluster (clstrmg.debug.long also exists) IBM support using these
/tmp/cspoc.log shows more info of the smitty c-spoc command (good place to look if a command fails)
/tmp/clconvert.log after cluster upgrade shows info about the results

RSCT Logs:
/var/ha/log RSCT logs are here
/var/ha/log/nim.topsvcs… the heartbeats are logged here (comm. is OK between the nodes)
clRGinfo Shows the state of RGs (in earlier HACMP clfindres was used)
clRGinfo -p shows the node that has temporarily the highest priority (POL)
clRGinfo -t shows the delayed timer information
clRGinfo -m shows the status of the application monitors of the cluster
resource groups state can be: online, offline, acquiring, releasing, error, unknown

cldump (or clstat -o) detailed info about the cluster (realtime, shows cluster status) (clstat requires a running clinfo)
cldisp detailed general info about the cluster (not realtime)
cltopinfo Detailed information about the network of the cluster (this shows the data in DCD not in ACD)
cltopinfo -i good overview, same as cllsif: this also lists cluster inetrfaces, it was used prior HACMP 5.1
cltopinfo -m shows heartbeat statistics, missed heartbeats
clshowres Detailed information about the resource group(s)
cllsserv Shows which scripts will be run in case of a takeover

clrgdependency -t PARENT_CHILD -sl shows parent child dependencies of resource groups

clshowsrv -v shows status of the cluster daemons (very good overview!!!)
lssrc -g cluster lists the running cluster daemons

lssrc -ls clstrmgrES shows if cluster is STABLE or not, cluster version, Dynamic Node Priority (pgspace free, disk busy, cpu idle)
ST_STABLE: cluster services running with resources online
NOT_CONFIGURED: cluster is not configured or node is not synced
ST_INIT: cluster is configured but not active on this node
ST_JOINING: cluster node is joining the cluster
ST_VOTING: cluster nodes are voting to decide event execution
ST_RP_RUNNING: cluster is running a recovery program
RP_FAILED: recovery program event script is failed
ST_BARRIER: clstrmgr is in between events waiting at the barrier
ST_CBARRIER: clstrmgr is exiting a recovery program
ST_UNSTABLE: cluster is unstable usually due to an event error
lssrc -ls topsvcs shows the status of individual diskhb devices, heartbeat intervals, failure cycle (missed heartbeats)
lssrc -ls grpsvcs gives info about connected clients, number of groups)
lssrc -ls emsvcs shows the resource monitors known to the event management subsystem)
lssrc -ls snmpd shows info about snmpd
halevel -s shows PowerHA level (from 6.1)

cl_ping pings all the adapters of the given list (e.g.: cl_ping -w 2 aix21 aix31 (-w: wait 2 seconds))
cldiag HACMP troubleshooting tool (e.g.: cldiag debug clstrmgr -l 5 cldiags vgs -h nodeA nodeB

——————————————————
/usr/es/sbin/cluster/utilities/get_local_nodename shows the name of this node within the HACMP
/usr/es/sbin/cluster/utilities/clexit.rc this script halt the node if the cluster manager daemon stopped incorrectly
——————————————————

Advertisements