今天在生产库解决一个ORA-01555 on Active Physical standby问题：
Applies to:

Oracle Server – Enterprise Edition – Version: 11.2.0.2.0 and later [Release: 11.2 and later ]
Information in this document applies to any platform.
Primary and Standby databases at 11.2.0.2
Symptoms

After upgrading to 11.2.0.2 we see this ORA-1555 on the Standby Database. It is not possible to logon with system, dbsnmp or other db-users. Only sys as sysdba is working.
It’s like we are running on system’s undo Tablespace, – and not the one which has been restored..

查看系统alert日志发现:
ORA-01555 caused by SQL statement below (SQL ID: 1f9apznp767fc, Query Duration=0 sec, SCN: 0x0006.62ab3687):
select object_name synonym_name, object_name, owner object_owner, status, object_type
from sys.all_objects o
where o.owner = object_owner
and o.object_type in (‘TABLE’, ‘VIEW’, ‘MATERIALIZED VIEW’, ‘SEQUENCE’, ‘FUNCTION’, ‘PROCEDURE’, ‘PACKAGE’, ‘TYPE’)
and o.object_name not like ‘BIN$%’

order by o.object_type, o.object_name

一个简单的查询 都导致了ora-01555 undo回滚段出现严重问题，同时出现如下问题：

ORA-00313: open failed for members of log group 15 of thread 1
ORA-00312: online log 15 thread 1: ‘/data/oracle/oradata/edw1/redo15.log’
ORA-27037: unable to obtain file status

客户端报如下错误：

同时发现mutex 等待 以及大量library cache lock 等待

通过查找metalink 发现与以下描述极其相似

This problem is introduced in 11.2.0.2 by the fix for bug 9214531

A self-hang / spin can occur when getting a library cache lock
leading to a wider hang scenario with other sessions waiting
for the blocked lock.

Rediscovery Notes:
A process spins in kglLock(),
other processes waiting for the spinning session
ORA-1555 on Active data Guard

解决方案：通过打p10018789_112020_Linux-x86-64解决此问题

打patch 过程过遇到很多问题，下面一一阐述：

opatch lsinventory raise error code 104

OPatch cannot find a valid oraInst.loc file to locate Central Inventory.
OPatch failed with error code 104

通过了解发现此RDBMS是通过tar包过来直接relink的，缺少全局的 Inventory 所以我们需要手动创建Inventory
首先要编辑一个oraInst.loc文件，使之指向我们要创建全局inventory的目录

inventory_loc=/data/oracle/oraInventory
inst_group=oinstall

cd $ORACLE_HOME/oui/bin
./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”” ORACLE_HOME_NAME=””

我们这里采用的是 ./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”/data/oracle/product/11.2.0/db1″ ORACLE_HOME_NAME=”oracle11g”

The inventory pointer is located at /etc/oraInst.loc

The inventory is located at /data/oracle/oraInventory

‘AttachHome’ was successful.

针对rac的Inventory重建 我们需要指定crs_home 和oracle_home

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”<10g Ora_Crs_Home Path>” ORACLE_HOME_NAME=”” LOCAL_NODE=’node1′ CLUSTER_NODES=node1,node2 CRS=true

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”<10g Oracle_Home Path>” ORACLE_HOME_NAME=”” LOCAL_NODE=’node1′ CLUSTER_NODES=node1,node2

下面的过程就显得很简单了：

解压缩补丁文件

unzip p10018789_112020_Linux-x86-64.zip

重新生成oraInventory

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=”/data/oracle/product/11.2.0/db1″ ORACLE_HOME_NAME=”oracle11g”

设置环境变量

export PATH=$ORACLE_HOME/OPatch:$PATH

停止redo apply

alter database recover managed standby database cancel;

关闭数据库

shutdownn immediate

打补丁

cd 10018789

opatch apply

打开数据库并开始redo应用

startup mount

alter database open read only

alter database recover managed standby database disconnect from session using current logfile;

观察日志 发现问题解决