用户的环境是aix版本的11.2.0.2集群,数据库实例hang,看到gcs resource,第一时间就反应是drm和lmon,结合hang前的awr也发现等待事件集中在gcs resource directory to be unfrozen,这个时候一般集中检查和gcs相关的信息:数据库告警日志,lmon trace,lms trace。整个过程是DRM触发了,但是并没有切换资源,导致实例hang住,根本原因是过大的buffer cache导致,根据lmon的信息和官方bug 12879027吻合,打上11.2.0.2.7的psu(DB和GI),后续继续观察
LMON进程trace可见如下:
*** 2014-08-14 21:13:51.87 CGS recovery timeout = 85 sec Begin DRM(231) (swin 1) * drm quiesce *** 2014-08-14 21:17:06.782 * Request pseudo reconfig due to drm quiesce hang 2012-07-14 21:17:03.752735 : kjfspseudorcfg: requested with reason 5(DRM Quiesce step stall) *** 2014-08-14 21:17:04.911 kjxgmrcfg: Reconfiguration started, type 6 CGS/IMR TIMEOUTS: CSS recovery timeout = 31 sec (Total CSS waittime = 65) IMR Reconfig timeout = 75 sec CGS rcfg timeout = 85 sec kjxgmcs: Setting state to 70 0. - AWR Top waits are "gcs resource directory to be unfrozen" & "gc remaster"
官方文档:
Bug 12879027 LMON gets stuck in DRM quiesce causing intermittent pseudo reconfiguration
This note gives a brief overview of bug 12879027.
The content was last updated on: 15-OCT-2013
Click here for details of each of the sections below.
Affects:
Product (Component) Oracle Server (Rdbms) Range of versions believed to be affected Versions BELOW 12.1 Versions confirmed as being affected Platforms affected Generic (all / most platforms affected)
Fixed:
The fix for 12879027 is first included in Interim patches may be available for earlier versions – click here to check.
Symptoms: |
Related To: |
Description
This bug is only relevant when using Real Application Clusters (RAC)
LMON process can get stuck in the DRM quiesce step triggering pseudo reconfiguration eventually. Rediscovery Notes: DRM quiesce step hangs and triggers pseudoreconfiguration especially in single window DRM and when the buffer cache is very large. Workaround None
Getting a Fix Use one of the "Fixed" versions listed above (for Patch Sets / bundles use the latest version available as contents are cumulative - the "Fixed" version listed above is the first version where the fix is included) or You can check for existing interim patches here: Patch:12879027
Please note: The above is a summary description only. Actual symptoms can vary. Matching to any symptoms here does not confirm that you are encountering this problem. For questions about this bug please consult Oracle Support. |
References
Bug:12879027 (This link will only work for PUBLISHED bugs)
Note:245840.1 Information on the sections in this article