Datafile in a PDB added but not backed up

(I think I've covered this with a pre-12c non-PDB example earlier, but this demo is with a 12c PDB)

What happens if you add a datafile to a PDB and then lose it without having a backup of the datafile ?

Here is a quick demo

$sqlplus '/ as sysdba'

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jun 15 22:10:19 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> show parameter db_create_file_dest

NAME         TYPE  VALUE
------------------------------------ ----------- ------------------------------
db_create_file_dest       string
SQL> alter session set db_create_file_dest='/u01/app/oracle/oradata';

Session altered.

SQL> create pluggable database newpdb admin user newpdb_adm identified by newpdb_adm;

Pluggable database created.

SQL> 
SQL> alter pluggable database newpdb open;

Pluggable database altered.

SQL> alter session set container=newpdb;

Session altered.

SQL> create user hemant_newpdb identified by hemant_newpdb;

User created.

SQL> grant dba to hemant_newpdb;

Grant succeeded.

SQL> 

SQL> !vi $ORACLE_HOME/network/admin/tnsnames.ora  --- added the entry for newpdb

SQL> 
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
$sqlplus hemant_newpdb/hemant_newpdb@newpdb

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jun 15 22:16:29 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> 
SQL> alter session set db_create_file_dest='/u01/app/oracle/oradata';

Session altered.

SQL> create tablespace my_user_data;

Tablespace created.

SQL> select file_name from dba_data_files
  2  where tablespace_name  = 'MY_USER_DATA'
  3  /

FILE_NAME
------------------------------------------------------------------------------------------------------------------------------------
/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf

SQL> select file_id, file_name
  2  from dba_data_files
  3  order by 1
  4  /

   FILE_ID
----------
FILE_NAME
------------------------------------------------------------------------------------------------------------------------------------
 41
/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_system_hgh0ddlg_.dbf

 42
/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_sysaux_hgh0ddlw_.dbf

 43
/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_undotbs1_hgh0ddlw_.dbf

 44
/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf


SQL> create table my_user_data_tbl tablespace my_user_data as select * from dba_objects;

Table created.

SQL> select count(*) from my_user_data_tbl;

  COUNT(*)
----------
     72623

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

So, I now have a new PDB, a custom tablespace in the PDB and a table with data in that Tablespace.

I have NOT yet taken a backup of the PDB.

For the purpose of this demo, I will corrupt the datafile.

$cat  >/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf
abcdef 
junk data
overwriting the datafile
$ls -l /u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf
-rw-r----- 1 oracle oinstall 43 Jun 15 22:22 /u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf
$cat /u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf
abcdef 
junk data
overwriting the datafile
$

Any attempt to read or write the Datafile will fail.

$sync;sync
$sqlplus '/ as sysdba'

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jun 15 22:23:38 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> alter system flush buffer_cache;

System altered.

SQL> quit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
$sqlplus hemant_newpdb/hemant_newpdb@newpdb

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jun 15 22:23:59 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Last Successful login time: Mon Jun 15 2020 22:16:29 +08:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select count(*) from my_user_data_tbl;
select count(*) from my_user_data_tbl
                     *
ERROR at line 1:
ORA-01115: IO error reading block from file  (block # )
ORA-01110: data file 44: '/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf'
ORA-27072: File I/O error
Additional information: 4
Additional information: 130


SQL> exit 
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
$      

The sync and flush buffer_cache commands ensure that Oracle does not try to read the table blocks from memory but actually has to attempt to read the Datafile.

Now it reports that the Datafile has an error.

However, I have not taken a backup of the Datafile.

I go ahead to use RMAN to "restore" and "recover" the Datafile.

$rman target /

Recovery Manager: Release 12.2.0.1.0 - Production on Mon Jun 15 22:30:32 2020

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL12C (DBID=768045447)

RMAN> list failure;

using target database control file instead of recovery catalog
Database Role: PRIMARY

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
171        HIGH     OPEN      15-JUN-20     One or more non-system datafiles are corrupt

RMAN> advise failure;

Database Role: PRIMARY

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
---------- -------- --------- ------------- -------
171        HIGH     OPEN      15-JUN-20     One or more non-system datafiles are corrupt

analyzing automatic repair options; this may take some time
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=267 device type=DISK
analyzing automatic repair options complete

Mandatory Manual Actions
========================
no manual actions available

Optional Manual Actions
=======================
1. Automatic repairs may be available if you shutdown the database and restart it in mount mode

Automated Repair Options
========================
Option Repair Description
------ ------------------
1      Restore and recover datafile 44  
  Strategy: The repair includes complete media recovery with no data loss
  Repair script: /u01/app/oracle/diag/rdbms/orcl12c/orcl12c/hm/reco_3798852352.hm

RMAN> 
RMAN> quit


Recovery Manager complete.
$cat /u01/app/oracle/diag/rdbms/orcl12c/orcl12c/hm/reco_3798852352.hm
   # restore and recover datafile
   sql 'NEWPDB''alter database datafile 44 offline';
   restore ( datafile 44 );
   recover datafile 44;
   sql 'NEWPDB''alter database datafile 44 online';
$
$rman target /

Recovery Manager: Release 12.2.0.1.0 - Production on Mon Jun 15 22:31:44 2020

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL12C (DBID=768045447)

RMAN> sql 'NEWPDB''alter database datafile 44 offline';

using target database control file instead of recovery catalog
sql statement: alter database datafile 44 offline

RMAN> restore ( datafile 44 );

Starting restore at 15-JUN-20
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=289 device type=DISK

creating datafile file number=44 name=/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf
restore not done; all files read only, offline, excluded, or already restored
Finished restore at 15-JUN-20

RMAN> recover datafile 44;

Starting recover at 15-JUN-20
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 15-JUN-20

RMAN> sql 'NEWPDB''alter database datafile 44 online';

sql statement: alter database datafile 44 online

RMAN> exit


Recovery Manager complete.
$

Note that the message "restore not done; all files read only, offline, excluded, or already restored" can be misleading.  This occurs when RMAN actually *creates* a Datafile because there is no backup of the Datafile.

I can now query the data.

$sqlplus hemant_newpdb/hemant_newpdb@newpdb

SQL*Plus: Release 12.2.0.1.0 Production on Mon Jun 15 22:34:40 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Last Successful login time: Mon Jun 15 2020 22:24:00 +08:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select count(*) from my_user_data_tbl;

  COUNT(*)
----------
     72623

SQL> quit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
$
$cd /u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile
$ls -l
total 829540
-rw-r----- 1 oracle oinstall 104865792 Jun 15 22:32 o1_mf_my_user__hgh0qyh7_.dbf
-rw-r----- 1 oracle oinstall 377495552 Jun 15 22:23 o1_mf_sysaux_hgh0ddlw_.dbf
-rw-r----- 1 oracle oinstall 262152192 Jun 15 22:40 o1_mf_system_hgh0ddlg_.dbf
-rw-r----- 1 oracle oinstall  67117056 Jun 15 22:14 o1_mf_temp_hgh0ddlw_.dbf
-rw-r----- 1 oracle oinstall 104865792 Jun 15 22:40 o1_mf_undotbs1_hgh0ddlw_.dbf

Here are the interesting messages from the alert log file :

2020-06-15T22:31:53.166607+08:00
NEWPDB(4):alter database datafile 44 offline
NEWPDB(4):Completed: alter database datafile 44 offline
2020-06-15T22:32:01.797043+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl12c/orcl12c/trace/orcl12c_m000_5900.trc:
ORA-01110: data file 44: '/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf'
ORA-01565: error in identifying file '/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf'
ORA-27048: skgfifi: file header information is invalid
Additional information: 2
2020-06-15T22:32:03.419839+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcl12c/orcl12c/trace/orcl12c_m000_5900.trc:
ORA-01122: database file 44 failed verification check
ORA-01110: data file 44: '/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf'
ORA-01565: error in identifying file '/u01/app/oracle/oradata/ORCL12C/A821006A1F6B14B3E0530100007FE3FA/datafile/o1_mf_my_user__hgh0qyh7_.dbf'
ORA-27048: skgfifi: file header information is invalid
Additional information: 2
2020-06-15T22:32:15.922471+08:00
alter database recover datafile list clear
Completed: alter database recover datafile list clear
alter database recover
 if needed datafile 44
2020-06-15T22:32:15.939087+08:00
Media Recovery Start
2020-06-15T22:32:15.947485+08:00
Serial Media Recovery started
2020-06-15T22:32:16.148455+08:00
Recovery of Online Redo Log: Thread 1 Group 2 Seq 71 Reading mem 0
  Mem# 0: /u01/app/oracle/oradata/orcl12c/redo02.log
2020-06-15T22:32:16.536560+08:00
Media Recovery Complete (orcl12c)
Completed: alter database recover
 if needed datafile 44
2020-06-15T22:32:22.582888+08:00

Unfortunately, the alert log does not show the "creating datafile" of the RESTORE command from RMAN.  It does show that the RECOVER command actually read from the Online Redo Log file.  In fact, if there had been multiple ArchiveLogs generated since the creation of the Datafile and the attempt to RECOVER the datafile, it would read from all those ArchiveLogs at this phase.  (Of course, if any ArchiveLog was missing, the RECOVER phase would fail).

So, if you lose any Datafile, ensure that you have all the ArchiveLogs generated since the Datafile was created and RMAN would recreate and recover the datafile for you.