一、故障描述
Oracle 12C PDBS（PDB NAME = WRYPC） 在note1节点上是MOUNTED状态，在note2节点上是READ WRITE状态。note1节点上启动该PDB长时间无反应hang住，并且伴有日志报错。
//note1 SQL> select DBID,NAME,LOG_MODE,OPEN_MODE from v$database; DBID NAME LOG_MODE OPEN_MODE ---------- --------- ------------ -------------------- 2140168445 EIC1 ARCHIVELOG READ WRITE SQL> select instance_name,status from v$instance; INSTANCE_NAME STATUS ---------------- ------------ eic12 OPEN SQL> select DBID,NAME,LOG_MODE,OPEN_MODE from v$database; DBID NAME LOG_MODE OPEN_MODE ---------- --------- ------------ -------------------- 2140168445 EIC1 ARCHIVELOG READ WRITE SQL> select con_id,dbid,guid,name,open_mode from v$pdbs; CON_ID DBID GUID NAME OPEN_MODE ---------- ---------- -------------------------------- --------------------------------- ---------- 2 3257667166 5B3C58220E311678E0532CF2640A4165 PDB$SEED READ ONLY 3 2473517642 5B3C7CE162C953F6E0532CF2640A910E EIC1PDB READ WRITE 4 1620899158 6A1F75EB99FE3131E0532DF2640A95C0 PDB_NAME READ WRITE 5 493040432 6A1F75EB9A003131E0532DF2640A95C0 ERWUPU MOUNTED 6 2865274021 6A224E09130541B9E0532DF2640AAE5F EWP2 MOUNTED 7 2198790855 6B3463863B9C1D6BE0532DF2640A61C5 ODS READ WRITE 8 2689821097 703D1085A55C3376E0532DF2640A8A3E EWPSHIDIANSHENG MOUNTED 9 1167482729 703DEDEFF33FDE66E0532DF2640A8E9F EWPSHIDIANGJ MOUNTED 10 1008500739 711B1C42A2F34FA9E0532DF2640AC62B ERWUPUMAP MOUNTED 11 148782956 A401C78AC106DC34E0532DF2640A8AA5 WRYPC READ WRITE 12 1161538067 75BEA3722224E4DAE0532CF2640AD0F6 ERWUPUMAP_XJ MOUNTED 13 1728756282 75BD0918B56DB368E0532CF2640A9170 WRYPCNEI2018W MOUNTED 14 1579638091 75BEE5B84C9E23A3E0532CF2640A7E2E ERWUPUMAP_XZ MOUNTED 15 2065390671 75BF0ECA028E34A4E0532CF2640AFFC3 ERWUPUMAP_BT MOUNTED 16 493305058 75C309E9051CCE0CE0532CF2640ABF80 WRYPCNEI2018Z MOUNTED 17 222405979 75CD5BD04A3846F9E0532CF2640AAE77 WRYPCWAI54W MOUNTED 18 855842903 75CDC85DE82DB0CCE0532CF2640ABD02 WRYPCWAI54Z MOUNTED 19 727403284 75CE11270B33EE13E0532CF2640ADC8A WRYPCWAI65W MOUNTED 20 1858334523 75CE387EA16712E1E0532CF2640AE409 WRYPCNEI65Z MOUNTED 21 2081645259 75CE6B8F25483504E0532CF2640AC72A WRYPCNEI66Z MOUNTED 22 3702096573 78057D27A412F5BEE0532CF2640A3B24 HLWZWFWPT READ WRITE 23 4268531149 7805BC8E63A35B78E0532CF2640A3FFA HLWZWFWPTER READ WRITE 24 331720291 7841DABFA10E7C55E0532CF2640AC927 EGOVA READ WRITE 25 744750214 78CE617BF1D4F540E0532CF2640AF51E HJXXYZT READ WRITE 26 1668454202 78421DAF6EF3D301E0532CF2640AB57D STHJXXYZT READ WRITE 27 3378782880 796DCEB39C43282FE0532CF2640A26C7 IWHEREORCL READ WRITE 28 3623836438 7C2FEDCF7B4291E1E0532CF2640A4072 ODSTEST READ WRITE 29 3912941240 7CE2F2FE4E47514DE0532CF2640A5754 ESERVICE READ WRITE 30 3016044583 A6874758312BDF72E0532CF2640A2F9F DECWRYPC READ WRITE 29 rows selected. SQL> //note2 SQL> set linesize 200 SQL> select DBID,NAME,LOG_MODE,OPEN_MODE from v$database; DBID NAME LOG_MODE OPEN_MODE ---------- --------- ------------ -------------------- 2140168445 EIC1 ARCHIVELOG READ WRITE SQL> select con_id,dbid,guid,name,open_mode from v$pdbs; CON_ID DBID GUID NAME OPEN_MODE ---------- ---------- -------------------------------- -------------------------------------------------- -------------------- 2 3257667166 5B3C58220E311678E0532CF2640A4165 PDB$SEED READ ONLY 3 2473517642 5B3C7CE162C953F6E0532CF2640A910E EIC1PDB READ WRITE 4 1620899158 6A1F75EB99FE3131E0532DF2640A95C0 PDB_NAME READ WRITE 5 493040432 6A1F75EB9A003131E0532DF2640A95C0 ERWUPU MOUNTED 6 2865274021 6A224E09130541B9E0532DF2640AAE5F EWP2 MOUNTED 7 2198790855 6B3463863B9C1D6BE0532DF2640A61C5 ODS READ WRITE 8 2689821097 703D1085A55C3376E0532DF2640A8A3E EWPSHIDIANSHENG MOUNTED 9 1167482729 703DEDEFF33FDE66E0532DF2640A8E9F EWPSHIDIANGJ MOUNTED 10 1008500739 711B1C42A2F34FA9E0532DF2640AC62B ERWUPUMAP MOUNTED 11 148782956 A401C78AC106DC34E0532DF2640A8AA5 WRYPC MOUNTED 12 1161538067 75BEA3722224E4DAE0532CF2640AD0F6 ERWUPUMAP_XJ MOUNTED CON_ID DBID GUID NAME OPEN_MODE ---------- ---------- -------------------------------- -------------------------------------------------- -------------------- 13 1728756282 75BD0918B56DB368E0532CF2640A9170 WRYPCNEI2018W MOUNTED 14 1579638091 75BEE5B84C9E23A3E0532CF2640A7E2E ERWUPUMAP_XZ MOUNTED 15 2065390671 75BF0ECA028E34A4E0532CF2640AFFC3 ERWUPUMAP_BT MOUNTED 16 493305058 75C309E9051CCE0CE0532CF2640ABF80 WRYPCNEI2018Z MOUNTED 17 222405979 75CD5BD04A3846F9E0532CF2640AAE77 WRYPCWAI54W MOUNTED 18 855842903 75CDC85DE82DB0CCE0532CF2640ABD02 WRYPCWAI54Z MOUNTED 19 727403284 75CE11270B33EE13E0532CF2640ADC8A WRYPCWAI65W MOUNTED 20 1858334523 75CE387EA16712E1E0532CF2640AE409 WRYPCNEI65Z MOUNTED 21 2081645259 75CE6B8F25483504E0532CF2640AC72A WRYPCNEI66Z MOUNTED 22 3702096573 78057D27A412F5BEE0532CF2640A3B24 HLWZWFWPT READ WRITE 23 4268531149 7805BC8E63A35B78E0532CF2640A3FFA HLWZWFWPTER READ WRITE CON_ID DBID GUID NAME OPEN_MODE ---------- ---------- -------------------------------- -------------------------------------------------- -------------------- 24 331720291 7841DABFA10E7C55E0532CF2640AC927 EGOVA READ WRITE 25 744750214 78CE617BF1D4F540E0532CF2640AF51E HJXXYZT READ WRITE 26 1668454202 78421DAF6EF3D301E0532CF2640AB57D STHJXXYZT READ WRITE 27 3378782880 796DCEB39C43282FE0532CF2640A26C7 IWHEREORCL READ WRITE 28 3623836438 7C2FEDCF7B4291E1E0532CF2640A4072 ODSTEST READ WRITE 29 3912941240 7CE2F2FE4E47514DE0532CF2640A5754 ESERVICE READ WRITE 30 3016044583 A6874758312BDF72E0532CF2640A2F9F DECWRYPC READ WRITE 29 rows selected. SQL> SQL> alter pluggable database WRYPC open; ^C alter pluggable database WRYPC open * ERROR at line 1: ORA-12152: TNS:unable to send break message // alert_eic11.log --日志报错 2020-11-23T12:44:29.311665+08:00 alter pluggable database WRYPC open 2020-11-23T12:44:29.490214+08:00 WRYPC(11):Recovering pdb: pdbid=11 WRYPC(11):JIT: pid 192727 requesting stop 2020-11-23T12:46:37.858454+08:00 DBRM (ospid: 58076) waits for event 'latch: shared pool' for 18 secs. DBRM (ospid: 58076) waits for latch 'shared pool' for 18 secs. 2020-11-23T12:46:49.379038+08:00 Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_lmhb_58243.trc (incident=2049040) (PDBNAME=C DB$ROOT): ORA-29771: process USER (OSID 192727) blocks DBRM (OSID 58076) for more than 70 seconds Incident details in: /u01/app/oracle/diag/rdbms/eic1/eic11/incident/incdir_2049040/eic11_lmhb_58243_i2049040.t rc 2020-11-23T12:46:58.068303+08:00 USER (ospid: 192727) is blocking DBRM (ospid: 58076) in a wait LMHB (ospid: 58243) kills USER (ospid: 192727). Please check LMHB trace file for more detail. 2020-11-23T12:47:01.204284+08:00 // alert_eic11.log --多次出现的错误 2020-11-23T05:49:10.563954+08:00 Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_ora_186547.trc: ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685 ORA-27300: OS system dependent operation:open failed with status: 13 ORA-27301: OS failure message: Permission denied ORA-27302: failure occurred at: sskgmsmr_7 2020-11-23T05:49:10.608316+08:00 Errors in file /u01/app/oracle/diag/rdbms/eic1/eic11/trace/eic11_ora_186547.trc: ORA-17503: ksfdopn:2 Failed to open file +DATA/EIC1/PASSWORD/pwdeic1.256.957086685 ORA-27300: OS system dependent operation:open failed with status: 13 ORA-27301: OS failure message: Permission denied ORA-27302: failure occurred at: sskgmsmr_7