12c pdb backup, "corrupt", restore, recover

backup pluggable database hilo;


Spooling started in log file: backupCdb.txt

Recovery Manager12.1.0.1.0

RMAN> backup pluggable database hilo;
Starting backup at 02-SEP-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=7 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=68 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00046 name=+DATA/POLOLU/DATAFILE/system.278.824993643
input datafile file number=00050 name=+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/ts.263.824999673
channel ORA_DISK_1: starting piece 1 at 02-SEP-13
channel ORA_DISK_2: starting full datafile backup set
channel ORA_DISK_2: specifying datafile(s) in backup set
input datafile file number=00047 name=+DATA/POLOLU/DATAFILE/sysaux.277.824993597
input datafile file number=00048 name=+DATA/POLOLU/DATAFILE/users.260.824993699
channel ORA_DISK_2: starting piece 1 at 02-SEP-13
channel ORA_DISK_1: finished piece 1 at 02-SEP-13
piece handle=+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/BACKUPSET/2013_09_02/nnndf0_tag20130902t212258_0.267.825110579 tag=TAG20130902T212258 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:02:16
channel ORA_DISK_2: finished piece 1 at 02-SEP-13
piece handle=+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/BACKUPSET/2013_09_02/nnndf0_tag20130902t212258_0.265.825110581 tag=TAG20130902T212258 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time: 00:02:15
Finished backup at 02-SEP-13

Starting Control File and SPFILE Autobackup at 02-SEP-13
piece handle=+DATA/KILAUEA/AUTOBACKUP/2013_09_02/s_825110715.257.825110719 comment=NONE
Finished Control File and SPFILE Autobackup at 02-SEP-13

RMAN>
RMAN> quit

Recovery Manager complete.



[oracle@waipio1 KILAUEA rman]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Mon Sep 2 22:22:23 2013

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

[grid@waipio1 ~]$ asmcmd ls +DATA/POLOLU/DATAFILE
SYSAUX.277.824993597
SYSTEM.278.824993643
UNDOTBS1.280.824993699
USERS.260.824993699
[grid@waipio1 ~]$ asmcmd rm -rf +DATA/POLOLU/DATAFILE
ORA-29261: bad argument
ORA-00600: internal error code, arguments: [ORA_NPI_ERROR], [600], [ORA-00600: internal error code, arguments: [KFCHK_ERRRET], [kfa.c], [3025], [], [], [], [], [], [], [], [], []
], [], [], [], [], [], [], [], [], []
ORA-06512: at line 4 (DBD ERROR: OCIStmtExecute)

SQL> startup
ORACLE instance started.

Total System Global Area 4993982464 bytes
Fixed Size                  2691856 bytes
Variable Size            1090522352 bytes
Database Buffers         3892314112 bytes
Redo Buffers                8454144 bytes
Database mounted.
Database opened.
SQL> alter pluggable database kona open;

Pluggable database altered.


SQL> alter pluggable database hilo open;
alter pluggable database hilo open
*
ERROR at line 1:
ORA-01157: cannot identify/lock data file 48 - see DBWR trace file
ORA-01110: data file 48: '+DATA/POLOLU/DATAFILE/users.260.824993699'



alter pluggable database hilo open
Mon Sep 02 22:28:14 2013
Errors in file /app/oracle/diag/rdbms/kilauea/KILAUEA/trace/KILAUEA_dbw0_637.trc:
ORA-01157: cannot identify/lock data file 48 - see DBWR trace file
ORA-01110: data file 48: '+DATA/POLOLU/DATAFILE/users.260.824993699'
ORA-17503: ksfdopn:2 Failed to open file +DATA/POLOLU/DATAFILE/users.260.824993699
ORA-15012: ASM file '+DATA/POLOLU/DATAFILE/users.260.824993699' does not exist
Mon Sep 02 22:28:14 2013
Errors in file /app/oracle/diag/rdbms/kilauea/KILAUEA/trace/KILAUEA_dbw0_637.trc:
ORA-01157: cannot identify/lock data file 47 - see DBWR trace file
ORA-01110: data file 47: '+DATA/POLOLU/DATAFILE/sysaux.277.824993597'
ORA-17503: ksfdopn:2 Failed to open file +DATA/POLOLU/DATAFILE/sysaux.277.824993597
ORA-15012: ASM file '+DATA/POLOLU/DATAFILE/sysaux.277.824993597' does not exist
Mon Sep 02 22:28:14 2013
Errors in file /app/oracle/diag/rdbms/kilauea/KILAUEA/trace/KILAUEA_dbw0_637.trc:
ORA-01157: cannot identify/lock data file 46 - see DBWR trace file
ORA-01110: data file 46: '+DATA/POLOLU/DATAFILE/system.278.824993643'
ORA-17503: ksfdopn:2 Failed to open file +DATA/POLOLU/DATAFILE/system.278.824993643
ORA-15012: ASM file '+DATA/POLOLU/DATAFILE/system.278.824993643' does not exist
Mon Sep 02 22:28:14 2013
Pluggable database HILO hit error 1157 during open read write
and will be closed.
ALTER SYSTEM: Flushing buffer cache inst=0 container=3 local
ORA-1157 signalled during: alter pluggable database hilo open...
Mon Sep 02 22:28:17 2013
Checker run found 3 new persistent data failures


Spooling started in log file: restorepdb.hilo.txt

Recovery Manager12.1.0.1.0

RMAN> restore pluggable database hilo;
Starting restore at 02-SEP-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=178 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=8 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00046 to +DATA/POLOLU/DATAFILE/system.278.824993643
channel ORA_DISK_1: restoring datafile 00050 to +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/ts.263.824999673
channel ORA_DISK_1: reading from backup piece +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/BACKUPSET/2013_09_02/nnndf0_tag20130902t212258_0.267.825110579
channel ORA_DISK_2: starting datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
channel ORA_DISK_2: restoring datafile 00047 to +DATA/POLOLU/DATAFILE/sysaux.277.824993597
channel ORA_DISK_2: restoring datafile 00048 to +DATA/POLOLU/DATAFILE/users.260.824993699
channel ORA_DISK_2: reading from backup piece +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/BACKUPSET/2013_09_02/nnndf0_tag20130902t212258_0.265.825110581
channel ORA_DISK_1: piece handle=+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/BACKUPSET/2013_09_02/nnndf0_tag20130902t212258_0.267.825110579 tag=TAG20130902T212258
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:02:26
channel ORA_DISK_2: piece handle=+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/BACKUPSET/2013_09_02/nnndf0_tag20130902t212258_0.265.825110581 tag=TAG20130902T212258
channel ORA_DISK_2: restored backup piece 1
channel ORA_DISK_2: restore complete, elapsed time: 00:02:25
Finished restore at 02-SEP-13

RMAN>
RMAN> quit

Recovery Manager complete.


Checker run found 3 new persistent data failures
Mon Sep 02 22:31:22 2013
Full restore complete of datafile 48 +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/users.278.825114675.  Elapsed time: 0:00:02
  checkpoint is 3246348
  last deallocation scn is 3
Mon Sep 02 22:31:22 2013
Full restore complete of datafile 50 +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/ts.263.824999673.  Elapsed time: 0:00:08
  checkpoint is 3246348
  last deallocation scn is 1720084
Mon Sep 02 22:31:28 2013
Checker run found 1 new persistent data failures
Mon Sep 02 22:33:28 2013
Full restore complete of datafile 47 +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/sysaux.260.825114673.  Elapsed time: 0:02:13
  checkpoint is 3246348
  last deallocation scn is 1854094
Mon Sep 02 22:33:28 2013
Full restore complete of datafile 46 +DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/system.280.825114671.  Elapsed time: 0:02:16
  checkpoint is 3246348
  last deallocation scn is 3028184
  Undo Optimization current scn is 695278


Spooling started in log file: recoverpdb.hilo.txt

Recovery Manager12.1.0.1.0

RMAN> recover pluggable database hilo;
Starting recover at 02-SEP-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=362 device type=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: SID=419 device type=DISK

starting media recovery
media recovery complete, elapsed time: 00:00:19

Finished recover at 02-SEP-13

RMAN> quit

Recovery Manager complete.

Mon Sep 02 22:40:10 2013
alter database recover datafile list clear
Completed: alter database recover datafile list clear
alter database recover if needed
 datafile 46 , 47 , 48 , 50
Mon Sep 02 22:40:10 2013
Media Recovery Start
Mon Sep 02 22:40:10 2013
Serial Media Recovery started
Mon Sep 02 22:40:11 2013
Recovery of Online Redo Log: Thread 1 Group 1 Seq 151 Reading mem 0
  Mem# 0: +DATA/KILAUEA/ONLINELOG/group_1.364.824921903
  Mem# 1: +DATA/KILAUEA/ONLINELOG/group_1.681.824921909
Mon Sep 02 22:40:21 2013
Recovery of Online Redo Log: Thread 1 Group 2 Seq 152 Reading mem 0
  Mem# 0: +DATA/KILAUEA/ONLINELOG/group_2.674.824921913
  Mem# 1: +DATA/KILAUEA/ONLINELOG/group_2.666.824921917
Mon Sep 02 22:40:23 2013
Recovery of Online Redo Log: Thread 1 Group 3 Seq 153 Reading mem 0
  Mem# 0: +DATA/KILAUEA/ONLINELOG/group_3.365.824921923
  Mem# 1: +DATA/KILAUEA/ONLINELOG/group_3.543.824921927
Mon Sep 02 22:40:29 2013
Media Recovery Complete (KILAUEA)
Completed: alter database recover if needed
 datafile 46 , 47 , 48 , 50


RMAN> alter pluggable database hilo open;

using target database control file instead of recovery catalog
Statement processed

alter pluggable database hilo open
Mon Sep 02 22:43:09 2013
Opening pdb HILO (3) with no Resource Manager plan active
Pluggable database HILO opened read write
Completed: alter pluggable database hilo open


RMAN> select name from v$datafile where con_id = 3;



NAME
--------------------------------------------------------------------------------

+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/system.280.825114671


+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/sysaux.260.825114673


+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/users.278.825114675


+DATA/KILAUEA/E55656EBC9CD1972E0437901A8C02245/DATAFILE/ts.263.824999673