Search In The Blog

Friday 19 October 2018

How to forcefully open database using _allow_resetlogs_corruption - ORA-1113 signalled during: alter database open resetlogs


Force database open : _allow_resetlogs_corrution


In real world scenario many times DBA comes to a point where he can’t open a database even with resetlogs. Or when he lost current redolog file. or due to some corruption.

In many situations - recreating controlfile and open resetlogs failed even though the chaeckpoint_change# in all datafile header is having unique value.

many situations where we need to tweak more after first run. but almost all time we opened a database using below method. This is the last way to open the database as per my knowledge.

Force open steps are:

1)      Backup the database while the database is closed.

THE INSTRUCTIONS HERE ARE DESTRUCTIVE. YOU ARE STRONGLY ADVISED TO BACKUP THE
DATABASE BEFORE PROCEEDING. IF YOU DO NOT DO THIS YOU MAY LOSE THE CHANCE TO
TRY OTHER OPTIONS.

2) Disable this database from EM if running. We also need to disable RAC to avoid Auto restart.

3) If your datafiles are from different points in time, it is best to try to
use system tablespace datafiles at a similar timestamp to the OLDEST files
you have online in the database. This reduces the chance that you will get
problems during the bootstrap phase of opening the database.

4) Edit your init.ora file to change undo_management and add two parameters.

* Change UNDO_MANAGEMENT=AUTO to
UNDO_MANAGEMENT=MANUAL
* Remove or comment out UNDO_TABLESPACE and UNDO_RETENTION.

* Add
CLUSTER_DATABASE=FALSE
JOB_QUEUE_PROCESSES=0
_ALLOW_RESETLOGS_CORRUPTION = TRUE

* If you only have a spfile available, you can from the closed, nomount or the
mount stage create an init.ora file as follows:

SQL> CREATE PFILE FROM SPFILE;

Do NOT edit the SPFILE.

5) Invoke SQL*Plus, startup mount, check that correct init.ora was used and
all datafiles are in the status of online or system.

$ sqlplus "/as sysdba"

SQL> startup mount pfile = (full path / file name to init.ora)
Confirm that the hidden parameters from step 3 were used:

SQL> show parameters corrupt

You should see both hidden parameters listed. If not, the wrong init.ora
may have been modified. Do not continue until "show parameters corrupt" shows
both hidden parameters.

SQL> show parameters undo

You should see undo management set to manual. If not, the wrong init.ora
may have been modified. Do not continue until "show parameters undo" shows
undo management as manual.

Check that all files you want to open with are listed as ONLINE or as SYSTEM.

SQL> select name, file#, status from v$datafile where status not in
('SYSTEM', 'ONLINE');

If any rows are returned from the query above, bring the file(s) online with:

SQL> ALTER DATABASE DATAFILE file# ONLINE;

and repeat until there are no files in an OFFLINE status. If any file remains or
changes into "recover" status after you try to online the file proceed to step 6.

6) Perform a fake incomplete recovery then open the database with resetlogs.

SQL> recover database using backup controlfile until cancel;

WHEN PROMPTED FOR AN ARCHIVELOG FILE TYPE cancel THEN PRESS ENTER.

SQL> ALTER DATABASE OPEN RESETLOGS;

7) If the database opens try selecting from a table. For example:

SQL> SELECT SYSDATE FROM DUAL;

If you get a row back the database is open and "functional". If you wish, you
may try to select from a other tables to make sure the database is functional
enough for the required export.

With the database open and functional you should attempt to export the database
IMMEDIATELY. Since database is unstable, don't try another shutdown/startup unless needed.
Once you have an export the database MUST be recreated from scratch.
This means dropping and deleting ALL datafiles and creating a new database from
scratch.

A database which has been opened in this way but not rebuilt will not be
supported by Oracle. Any delay in exporting the contents or any attempt to
use the system may cause irreparable damage.

NOTE: BE SURE TO REVERSE / REMOVE THE INIT.ORA PARAMETERS ADDED IN STEP 3
OTHERWISE YOU MAY ACCIDENTALLY CORRUPT ANY NEW DATABASE CREATED USING THE SAME
INIT.ORA FILE.

8) If the instance crashed in the open phase of step 5, check for trace files
in the background dump destination. If you find a trace file, check to see if
the trace file has an ORA-00600 [2662] or ORA-00600 [4000] error in it.
Either of these errors may also be seen in the alert.log file.

If you see the ORA-00600 [2662] or ORA-00600 [4000] error, provide Oracle Support
Services the full error message. Oracle Support Services will provide steps to advance
the SCN using a hidden parameter.



NOTE: BE SURE TO REVERSE / REMOVE THE INIT.ORA PARAMETERS ADDED IN STEP 3
OTHERWISE YOU MAY ACCIDENTALLY CORRUPT ANY NEW DATABASE CREATED USING THE SAME
INIT.ORA FILE.

*************************************************************************
* *
* CAUTION: Once the database is open, it is imperative that you export, *
* rebuild the database, and import. *
* *
* By forcing open the database in this fashion, there is a strong *
* likelihood of logical corruption, possibly affecting the data *
* dictionary. Oracle does not guarantee that all of the data will be *
* accessible nor will it support a database that has been opened by *
* this method and that the database users will be allowed to continue *
* work. All this does is provide a way to get at the contents of the *
* database for extraction, usually by export. It is up to you to *
* determine the amount of lost data and to correct any logical *
* corruption issues. *
* *
*************************************************************************





useful script to collect the database recovery information:


spool /tmp/current_status_info_new_2.txt
set pagesize 20000
set linesize 180
set pause off
set serveroutput on
set feedback on
set echo on
set numformat 999999999999999
alter session set nls_date_format = 'dd-mon-yyyy hh24:mi:ss';
archive log list;
select name,dbid,controlfile_type,open_mode,log_mode,checkpoint_change#,archive_change# from v$database;
select * from v$database_incarnation;
col name for a75
select * from v$restore_point;
select flashback_on from v$database;
select parallel from v$instance;
select protection_level from v$database;
select * from dba_streams_administrator;
select file#,name,status,checkpoint_change#,enabled from v$datafile;
select file#,name,status,enabled from v$tempfile;
select TS#,NAME,INCLUDED_IN_DATABASE_BACKUP,FLASHBACK_ON from v$tablespace;
select * from v$recover_file;
select * from v$backup;
select * from v$log;
select * from v$logfile;
select file#,name,recover,fuzzy,checkpoint_change#,creation_change#,checkpoint_time,creation_time,RESETLOGS_TIME,status from v$datafile_header;
select status, to_char(checkpoint_change#), to_char(checkpoint_time, 'DD-MON-YYYY HH24:MI:SS') as checkpoint_time,
count(*) from v$datafile_header
group by status, checkpoint_change#, checkpoint_time order by status, checkpoint_change#, checkpoint_time;
select count(*),fhsta from x$kcvfh group by fhsta;
select count(*),fhrba_seq from x$kcvfh group by fhrba_seq;
select count(*),fhscn from x$kcvfh group by fhscn;
select count(*),fhafs from x$kcvfh group by fhafs;
select min(FHSCN) "LOW FILEHDR SCN" , max(FHSCN) "MAX FILEHDR SCN", max(FHAFS) "Min PITR ABSSCN" from X$KCVFH ;
select fhdbn,fhdbi,hxfil,fhsta,fhscn,fhafs,fhrba_seq,fhtnm tbs_name from x$kcvfh;
select name, status , sequence#, thread#,
TO_CHAR(first_change#, '999999999999999999') as first_change#,
TO_CHAR(next_change#, '999999999999999999') next_change#,
to_char(completion_time,'DD-MON-YYYY HH24:MI:SS') completion_time
from v$archived_log where (select min(checkpoint_change#) from v$datafile_header) between first_change# and next_change#;
spool off
exit 




I hope this helps. Happy recovery :)


Sunday 26 August 2018

How to uninstall Oracle Home 12c on Exadata step by step


How to uninstall Oracle Home 12c on Exadata step by step

Owner of the oracle Home is oradbxp
Oracle Home is /u01/app/oracle/product/12.1.0.2/oradbxp_dbhome
This is Exadata machine and three node RAC

Steps to uninstall Oracle Home on exadata and normal server is same. below steps uninstall Oracle home from all nodes.

[oradbxp@myexa01 database]$
[oradbxp@myexa01 database]$ cd -
/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome/deinstall
[oradbxp@myexa01 deinstall]$
[oradbxp@myexa01 deinstall]$
[oradbxp@myexa01 deinstall]$ pwd
/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome/deinstall
[oradbxp@myexa01 deinstall]$
[oradbxp@myexa01 deinstall]$ cd -
/u01/app/oracle/Oeda/Software/12.1.0.2/database
[oradbxp@myexa01 database]$
[oradbxp@myexa01 database]$
[oradbxp@myexa01 database]$ ls -lrt
total 36
-rwxr-xr-x  1 oracle oinstall  500 Feb  7  2013 welcome.html
-rwxr-xr-x  1 oracle oinstall 8533 Jul  7  2014 runInstaller
drwxr-xr-x  2 oracle oinstall 4096 Jul  7  2014 rpm
drwxrwxr-x  2 oracle oinstall 4096 Jul  7  2014 sshsetup
drwxrwxr-x  2 oracle oinstall 4096 Jul  7  2014 response
drwxr-xr-x  4 oracle oinstall 4096 Jul  7  2014 install
drwxr-xr-x 14 oracle oinstall 4096 Nov 13 16:06 stage
-rw-------  1 oracle oinstall    0 Nov 23 18:16 nohup.out
[oradbxp@myexa01 database]$
[oradbxp@myexa01 database]$ ./runInstaller -deinstall -home /u01/app/oracle/product/12.1.0.2/oradbxp_dbhome
Checking for required space in /tmp directory ...
Space check on /tmp directory passed
Bootstrapping the deinstall components
Please wait ...
Bootstrapping completed
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DECONFIG TOOL START ############


######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##


Checking for existence of the Oracle home location /u01/app/oracle/product/12.1.0.2/oradbxp_dbhome
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: /u01/app/oracle
Checking for existence of central inventory location /u01/app/oraInventory
Checking for existence of the Oracle Grid Infrastructure home /u01/app/12.1.0.2/grid
The following nodes are part of this cluster: myexa01,ctsddbadm02,ctsddbadm03
Active Remote Nodes are ctsddbadm02,ctsddbadm03
Checking for sufficient temp space availability on node(s) : 'myexa01,ctsddbadm02,ctsddbadm03'

## [END] Install check configuration ##


Network Configuration check config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_check2017-11-29_03-57-07-PM.log

Specify all RAC listeners (do not include SCAN listener) that are to be de-configured. Enter .(dot) to deselect all. [LISTENER_XLHYPPRD]:

Network Configuration check config END

Database Check Configuration START

Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_check2017-11-29_03-57-37-PM.log

Use comma as separator when specifying list of values as input

Specify the list of database names that are configured in this Oracle home [xlhypprd1]:

###### For Database 'xlhypprd1' ######

Specify the type of this database (1.Single Instance Database|2.Oracle Restart Enabled Database|3.RAC Database|4.RAC One Node Database) [1]: 3
Specify the list of nodes on which this database has instances [myexa01]: myexa01,ctsddbadm02,ctsddbadm03
Specify the list of instance names [xlhypprd1]: xlhypprd1,xlhypprd2,xlhypprd3
Specify the local instance name on node   [xlhypprd1]:
Specify the diagnostic destination location of the database [/u01/app/oracle/diag/rdbms/xlhypprd_dr]:
Specify the storage type used by the Database ASM|FS []: ASM


Database Check Configuration END
Oracle Configuration Manager check START
OCM check log file location : /u01/app/oraInventory/logs//ocm_check9896.log
Oracle Configuration Manager check END

######################### DECONFIG CHECK OPERATION END #########################


####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: /u01/app/12.1.0.2/grid
The following nodes are part of this cluster: myexa01,ctsddbadm02,ctsddbadm03
Active Remote Nodes are ctsddbadm02,ctsddbadm03
The cluster node(s) on which the Oracle home deinstallation will be performed are:myexa01,ctsddbadm02,ctsddbadm03
Oracle Home selected for deinstall is: /u01/app/oracle/product/12.1.0.2/oradbxp_dbhome
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Following RAC listener(s) will be de-configured: LISTENER_XLHYPPRD
The following databases were selected for de-configuration : xlhypprd1
Database unique name : xlhypprd1
Storage used : ASM
Checking the config status for CCR
myexa01 : Oracle Home exists with CCR directory, but CCR is not configured
ctsddbadm02 : Oracle Home exists with CCR directory, but CCR is not configured
ctsddbadm03 : Oracle Home exists with CCR directory, but CCR is not configured
CCR check is finished
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-11-29_03-57-01-PM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-11-29_03-57-01-PM.err'

######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: /u01/app/oraInventory/logs/databasedc_clean2017-11-29_03-59-07-PM.log
Database Clean Configuration START xlhypprd1
This operation may take few minutes.
Database Clean Configuration END xlhypprd1

Network Configuration clean config START

Network de-configuration trace file location: /u01/app/oraInventory/logs/netdc_clean2017-11-29_03-59-26-PM.log

De-configuring RAC listener(s): LISTENER_XLHYPPRD

De-configuring listener: LISTENER_XLHYPPRD
    Stopping listener: LISTENER_XLHYPPRD
    Listener stopped successfully.
    Unregistering listener: LISTENER_XLHYPPRD
    Listener unregistered successfully.
Listener de-configured successfully.

De-configuring Listener configuration file on all nodes...
Listener configuration file de-configured successfully.

De-configuring Naming Methods configuration file on all nodes...
Naming Methods configuration file de-configured successfully.

De-configuring Local Net Service Names configuration file on all nodes...
Local Net Service Names configuration file de-configured successfully.

De-configuring Directory Usage configuration file on all nodes...
Directory Usage configuration file de-configured successfully.

De-configuring backup files on all nodes...
Backup files de-configured successfully.

The network configuration has been cleaned up successfully.

Network Configuration clean config END

Oracle Configuration Manager clean START
OCM clean log file location : /u01/app/oraInventory/logs//ocm_clean9896.log
Oracle Configuration Manager clean END

######################### DECONFIG CLEAN OPERATION END #########################


####################### DECONFIG CLEAN OPERATION SUMMARY #######################
Successfully de-configured the following database instances : xlhypprd1
Following RAC listener(s) were de-configured successfully: LISTENER_XLHYPPRD
Cleaning the config for CCR
As CCR is not configured, so skipping the cleaning of CCR configuration
CCR clean is finished
#######################################################################


############# ORACLE DECONFIG TOOL END #############

Using properties file /tmp/deinstall2017-11-29_03-56-52PM/response/deinstall_2017-11-29_03-57-01-PM.rsp
Location of logs /u01/app/oraInventory/logs/

############ ORACLE DEINSTALL TOOL START ############





####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-11-29_03-57-01-PM.out'
Any error messages from this session will be written to: '/u01/app/oraInventory/logs/deinstall_deconfig2017-11-29_03-57-01-PM.err'

######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to myexa01
Setting REMOTE_NODES to ctsddbadm02,ctsddbadm03
Setting CLUSTER_NODES to myexa01,ctsddbadm02,ctsddbadm03
Setting CRS_HOME to false
Setting oracle.installer.invPtrLoc to /tmp/deinstall2017-11-29_03-56-52PM/oraInst.loc
Setting oracle.installer.local to false

## [END] Preparing for Deinstall ##

Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START

Detach Oracle home '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' from the central inventory on the local node : Done

Delete directory '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' on the local node : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on local node. The directory is in use by Oracle Home '/u01/app/12.1.0.2/grid'.

Detach Oracle home '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' from the central inventory on the remote nodes 'ctsddbadm03,ctsddbadm02' : Done

Delete directory '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' on the remote nodes 'ctsddbadm02,ctsddbadm03' : Done

The Oracle Base directory '/u01/app/oracle' will not be removed on node 'ctsddbadm03'. The directory is in use by Oracle Home '/u01/app/12.1.0.2/grid'.

The Oracle Base directory '/u01/app/oracle' will not be removed on node 'ctsddbadm02'. The directory is in use by Oracle Home '/u01/app/12.1.0.2/grid'.

Oracle Universal Installer cleanup was successful.

Oracle Universal Installer clean END


## [START] Oracle install clean ##

Clean install operation removing temporary directory '/tmp/deinstall2017-11-29_03-56-52PM' on node 'myexa01'
Clean install operation removing temporary directory '/tmp/deinstall2017-11-29_03-56-52PM' on node 'ctsddbadm02,ctsddbadm03'

## [END] Oracle install clean ##


######################### DEINSTALL CLEAN OPERATION END #########################


####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' from the central inventory on the local node.
Successfully deleted directory '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' on the local node.
Successfully detached Oracle home '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' from the central inventory on the remote nodes 'ctsddbadm03,ctsddbadm02'.
Successfully deleted directory '/u01/app/oracle/product/12.1.0.2/oradbxp_dbhome' on the remote nodes 'ctsddbadm02,ctsddbadm03'.
Oracle Universal Installer cleanup was successful.

Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################


############# ORACLE DEINSTALL TOOL END #############

[oradbxp@myexa01 database]$

[oradbxp@myexa01 database]$

Wednesday 20 June 2018

TNS-12564: TNS:connection refused


TNS-12564: TNS:connection refused


We are using 12c six node primary. For this we have configured three node standby database. We validate the things from the configuration perspective and we are good.

Post this we configured data guard broker for the same and post data guard configiguration below error is frequently reporting to the alert of standby database.


standby alert log
=============
Fatal NI connect error 12521, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=XX.XXX.XXX.XX1)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XX.XXX.XXX.XX2)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XX.XXX.XXX.XX3)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XX.XXX.XXX.XX4)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XX.XXX.XXX.XX5)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=XX.XXX.XXX.XX6)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=dba_DGB)(INSTANCE_NAME=dba4)(CID=(PROGRAM=oracle)(HOST=ctsttbadm03.cts.com)(USER=orahcm))))
  VERSION INFORMATION:
        TNS for Linux: Version 12.1.0.2.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
  Time: 19-FEB-2018 22:02:26
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused



Cause :


This is caused by incorrect DGConnectIdentifier property in the broker's configuration (DGMGRL)

When Primary is a RAC database, the Standby Single Instance's DGConnectIdentifier property in the broker/dgmrl should be set to the SCAN name of the cluster database.
In this scenario, the DGConnectIdentifier was configured to use the cluster's VIP address, caused the connection problem from the Standby to the Primary.


Solution:

Check the database configuration in the broker using DGMGRL, as in example below:
For the Standby Database - "dba", it shows it as:
DGMGRL> show database verbose 'dba';

Properties:
DGConnectIdentifier = ' DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<hostname>)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dba)(SERVER=DEDICATED)))'
  

For the Primary Cluster Database - "dbaa", it shows it as:
DGMGRL> show database verbose 'dbaa';

Properties:
DGConnectIdentifier  = ' DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<vip hostname>)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dbaa)(SERVER=DEDICATED)))'
  


The ALERT LOG from the STANDBY continuously reports TNS-12521 while connecting to the (HOST=<vip hostname>)
Fatal NI connect error 12521, connecting to:
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<vip hostname>)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=dbaa_DGB)(INSTANCE_NAME=dba2)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=<hostname>)(USER=oracle))))
  

It should not use the VIP address 'vip hostname' as DGConnectIdentifier but the SCAN address for the PRIMARY cluster database instead.

Fix the DGConnectIdentifier property in the Broker configuration to use SCAN name of the cluster database
for example:
DGMGRL>edit database dbaa set property DGConnectIdentifier = '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = <scan host address>)(PORT = 1521))(CONNECT_DATA = (SERVICE_NAME = dbaa)))'


Wednesday 14 February 2018

Duplicate database filed with ORA-19870 ORA-19504 ORA-27044 Linux-x86_64 Error27 File too large



Duplicate database filed with ORA-19870 ORA-19504 ORA-27044 Linux-x86_64 Error27 File too large


I was trying with duplicate database from one of the standby database to UAT environment as a business requirement. Target server is UAT environment.

Using below command to duplicate database from target server –

RMAN> run
2> {
3> allocate auxiliary channel ch1 device TYPE disk;
4> allocate auxiliary channel ch2 device TYPE disk;
5> allocate auxiliary channel ch3 device TYPE disk;
6> allocate auxiliary channel ch4 device TYPE disk;
7> allocate auxiliary channel ch5 device TYPE disk;
8> allocate auxiliary channel ch6 device TYPE disk;
9> allocate auxiliary channel ch7 device TYPE disk;
10> allocate auxiliary channel ch8 device TYPE disk;
11> allocate auxiliary channel ch9 device TYPE disk;
12> set newname for database to '/data/MYUAT/DATAFILE/%b';
13> DUPLICATE DATABASE TO MYUAT noredo
14> PFILE='/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/initMYUAT.ora'
15> BACKUP LOCATION '/backupshare/ML_PUNE_DR_Golden_offbkp/'
16> Logfile '/data/MYUAT/LOGFILE/redo1.log' size 100M,
17> '/data/MYUAT/LOGFILE/redo2.log' size 100M,
18> '/data/MYUAT/LOGFILE/redo3.log' size 100M
19> ;
20> }

Duplicate database filed with below error –
Rman log for duplicate database -

channel ch4: starting datafile backup set restore
channel ch4: specifying datafile(s) to restore from backup set
channel ch4: restoring datafile 00012 to /data/MYUAT/DATAFILE/ptaudit.3092.947766809
channel ch4: restoring datafile 00023 to /data/MYUAT/DATAFILE/eocmlrg.3094.947766809
channel ch4: restoring datafile 00042 to /data/MYUAT/DATAFILE/eotplrg.3093.947766809
channel ch4: restoring datafile 00051 to /data/MYUAT/DATAFILE/psimage2.2098.947752747
channel ch4: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58900_1
channel ch2: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58908_1 tag=TAG20180115T021026
channel ch2: restored backup piece 1
channel ch2: restore complete, elapsed time: 00:13:12
channel ch2: starting datafile backup set restore
channel ch2: specifying datafile(s) to restore from backup set
channel ch2: restoring datafile 00013 to /data/MYUAT/DATAFILE/ptcmstar.2090.947766099
channel ch2: restoring datafile 00040 to /data/MYUAT/DATAFILE/eolarge.2100.947766099
channel ch2: restoring datafile 00045 to /data/MYUAT/DATAFILE/lmwork.2081.947766099
channel ch2: restoring datafile 00047 to /data/MYUAT/DATAFILE/psimage.2105.947752747
channel ch2: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58896_1
channel ch2: ORA-19870: error while restoring backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58896_1
ORA-19504: failed to create file "/data/MYUAT/DATAFILE/psimage.2105.947752747"
ORA-27044: unable to write the header block of file
Linux-x86_64 Error: 27: File too large
Additional information: 4

channel ch2: starting datafile backup set restore
channel ch2: specifying datafile(s) to restore from backup set
channel ch2: restoring datafile 00025 to /data/MYUAT/DATAFILE/eocuapp.3095.947766809
channel ch2: restoring datafile 00026 to /data/MYUAT/DATAFILE/eoculrg.3098.947766811
channel ch2: restoring datafile 00028 to /data/MYUAT/DATAFILE/eodslrg.3099.947766811
channel ch2: restoring datafile 00044 to /data/MYUAT/DATAFILE/lmlarge.2103.947752747
channel ch2: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58899_1
channel ch3: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58903_1 tag=TAG20180115T021026

Analysis:
t the initial look itself I found it OS related error for space issue or the file size is high which was failed to create file at OS level. So I asked OS team to check from the OS side.
As usual OS team come back to saying we are all good at OS side no issue found. ðŸ˜Š
Again I start investigating –
I checked and found that the file restoration failed is belongs to bigfile tablespace but there are other files which got successfully restored are also bigfile tablespace.
Then I checked side of this file- it’s 3.5 TB. Wow!!!
This file is 3.5 TB and all database physical size is 5 TB. Then I am sure that issue is with file size only and there MUST be limitation from OS side or filysystem side.
I started digging on filesystem – this /data created using ext3 file system. Below are the limitation for the filesystem level.








Solution –
So in ext3 individual file can be max 2TB, then OS team remounted the /data mount point with ext4 file system and issue got resolved. As on ext4 file system having individual file up to 16TB on RHEL6.

ORA-01119 ORA-27044 HPUX-ia64 File Too Large When Creating Tablespace (Doc ID 438853.1)
It's for different OS but relevant. 


Thanks for reading.



Complete error log for your ref –
Recovery Manager: Release 11.2.0.4.0 - Production on Wed Jan 17 03:16:21 2018
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to auxiliary database: MYUAT (not mounted)

RMAN> run
2> {
3> allocate auxiliary channel ch1 device TYPE disk;
4> allocate auxiliary channel ch2 device TYPE disk;
5> allocate auxiliary channel ch3 device TYPE disk;
6> allocate auxiliary channel ch4 device TYPE disk;
7> allocate auxiliary channel ch5 device TYPE disk;
8> allocate auxiliary channel ch6 device TYPE disk;
9> allocate auxiliary channel ch7 device TYPE disk;
10> allocate auxiliary channel ch8 device TYPE disk;
11> allocate auxiliary channel ch9 device TYPE disk;
12> set newname for database to '/data/MYUAT/DATAFILE/%b';
13> DUPLICATE DATABASE TO MYUAT noredo
14> PFILE='/u01/app/oracle/product/11.2.0.4/dbhome_1/dbs/initMYUAT.ora'
15> BACKUP LOCATION '/backupshare/ML_PUNE_DR_Golden_offbkp/'
16> Logfile '/data/MYUAT/LOGFILE/redo1.log' size 100M,
17> '/data/MYUAT/LOGFILE/redo2.log' size 100M,
18> '/data/MYUAT/LOGFILE/redo3.log' size 100M
19> ;
20> }
21>


allocated channel: ch1
channel ch1: SID=1710 device type=DISK

allocated channel: ch2
channel ch2: SID=5 device type=DISK

allocated channel: ch3
channel ch3: SID=573 device type=DISK

allocated channel: ch4
channel ch4: SID=1142 device type=DISK

allocated channel: ch5
channel ch5: SID=1711 device type=DISK

allocated channel: ch6
channel ch6: SID=6 device type=DISK

allocated channel: ch7
channel ch7: SID=574 device type=DISK

allocated channel: ch8
channel ch8: SID=1143 device type=DISK

allocated channel: ch9
channel ch9: SID=1712 device type=DISK

executing command: SET NEWNAME

Starting Duplicate Db at 17-JAN-18

contents of Memory Script:
{
   sql clone "create spfile from memory";
}
executing Memory Script

sql statement: create spfile from memory

contents of Memory Script:
{
   shutdown clone immediate;
   startup clone nomount;
}
executing Memory Script

Oracle instance shut down

connected to auxiliary database (not started)
Oracle instance started

Total System Global Area   19241058304 bytes

Fixed Size                     2261368 bytes
Variable Size               9596571272 bytes
Database Buffers            9596567552 bytes
Redo Buffers                  45658112 bytes
allocated channel: ch1
channel ch1: SID=1705 device type=DISK
allocated channel: ch2
channel ch2: SID=5 device type=DISK
allocated channel: ch3
channel ch3: SID=573 device type=DISK
allocated channel: ch4
channel ch4: SID=1142 device type=DISK
allocated channel: ch5
channel ch5: SID=1710 device type=DISK
allocated channel: ch6
channel ch6: SID=6 device type=DISK
allocated channel: ch7
channel ch7: SID=574 device type=DISK
allocated channel: ch8
channel ch8: SID=1143 device type=DISK
allocated channel: ch9
channel ch9: SID=1711 device type=DISK

contents of Memory Script:
{
   sql clone "alter system set  db_name =
 ''MLPRD'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   sql clone "alter system set  db_unique_name =
 ''MYUAT'' comment=
 ''Modified by RMAN duplicate'' scope=spfile";
   shutdown clone immediate;
   startup clone force nomount
   restore clone primary controlfile from  '/backupshare/ML_PUNE_DR_Golden_offbkp/MLprd_golden.ctl';
   alter clone database mount;
}
executing Memory Script

sql statement: alter system set  db_name =  ''MLPRD'' comment= ''Modified by RMAN duplicate'' scope=spfile

sql statement: alter system set  db_unique_name =  ''MYUAT'' comment= ''Modified by RMAN duplicate'' scope=spfile

Oracle instance shut down

Oracle instance started

Total System Global Area   19241058304 bytes

Fixed Size                     2261368 bytes
Variable Size               9596571272 bytes
Database Buffers            9596567552 bytes
Redo Buffers                  45658112 bytes
allocated channel: ch1
channel ch1: SID=1705 device type=DISK
allocated channel: ch2
channel ch2: SID=5 device type=DISK
allocated channel: ch3
channel ch3: SID=573 device type=DISK
allocated channel: ch4
channel ch4: SID=1142 device type=DISK
allocated channel: ch5
channel ch5: SID=1710 device type=DISK
allocated channel: ch6
channel ch6: SID=6 device type=DISK
allocated channel: ch7
channel ch7: SID=1143 device type=DISK
allocated channel: ch8
channel ch8: SID=574 device type=DISK
allocated channel: ch9
channel ch9: SID=1712 device type=DISK

Starting restore at 17-JAN-18

channel ch2: skipped, AUTOBACKUP already found
channel ch3: skipped, AUTOBACKUP already found
channel ch4: skipped, AUTOBACKUP already found
channel ch5: skipped, AUTOBACKUP already found
channel ch6: skipped, AUTOBACKUP already found
channel ch7: skipped, AUTOBACKUP already found
channel ch8: skipped, AUTOBACKUP already found
channel ch9: skipped, AUTOBACKUP already found
channel ch1: restoring control file
channel ch1: restore complete, elapsed time: 00:00:22
output file name=/data/MYUAT/CONTROLFILE/current.01
output file name=/data/MYUAT/CONTROLFILE/current.02
Finished restore at 17-JAN-18

database mounted

contents of Memory Script:
{
   set newname for datafile  1 to
 "/data/MYUAT/DATAFILE/system.3097.947766811";
   set newname for datafile  2 to
 "/data/MYUAT/DATAFILE/sysaux.2076.947766099";
   set newname for datafile  3 to
 "/data/MYUAT/DATAFILE/undotbs1.2095.947765199";
   set newname for datafile  4 to
 "/data/MYUAT/DATAFILE/psdefault.2085.947766809";
   set newname for datafile  5 to
 "/data/MYUAT/DATAFILE/undotbs2.2102.947752747";
   set newname for datafile  6 to
 "/data/MYUAT/DATAFILE/undotbs3.2089.947752747";
   set newname for datafile  7 to
 "/data/MYUAT/DATAFILE/undotbs4.2071.947765631";
   set newname for datafile  8 to
 "/data/MYUAT/DATAFILE/psimgr.3100.947766811";
   set newname for datafile  9 to
 "/data/MYUAT/DATAFILE/ptamsg.3096.947766809";
   set newname for datafile  10 to
 "/data/MYUAT/DATAFILE/ptapp.3114.947766845";
   set newname for datafile  11 to
 "/data/MYUAT/DATAFILE/ptappe.3105.947766811";
   set newname for datafile  12 to
 "/data/MYUAT/DATAFILE/ptaudit.3092.947766809";
   set newname for datafile  13 to
 "/data/MYUAT/DATAFILE/ptcmstar.2090.947766099";
   set newname for datafile  14 to
 "/data/MYUAT/DATAFILE/ptlock.3091.947766809";
   set newname for datafile  15 to
 "/data/MYUAT/DATAFILE/ptprc.2150.947766809";
   set newname for datafile  16 to
 "/data/MYUAT/DATAFILE/ptprjwk.3113.947766837";
   set newname for datafile  17 to
 "/data/MYUAT/DATAFILE/ptrpts.2151.947766809";
   set newname for datafile  18 to
 "/data/MYUAT/DATAFILE/pttbl.2152.947766809";
   set newname for datafile  19 to
 "/data/MYUAT/DATAFILE/pttlrg.2072.947766809";
   set newname for datafile  20 to
 "/data/MYUAT/DATAFILE/pttree.3111.947766815";
   set newname for datafile  21 to
 "/data/MYUAT/DATAFILE/ptwork.3107.947766811";
   set newname for datafile  22 to
 "/data/MYUAT/DATAFILE/eocmapp.3104.947766811";
   set newname for datafile  23 to
 "/data/MYUAT/DATAFILE/eocmlrg.3094.947766809";
   set newname for datafile  24 to
 "/data/MYUAT/DATAFILE/eocmwrk.322.947766809";
   set newname for datafile  25 to
 "/data/MYUAT/DATAFILE/eocuapp.3095.947766809";
   set newname for datafile  26 to
 "/data/MYUAT/DATAFILE/eoculrg.3098.947766811";
   set newname for datafile  27 to
 "/data/MYUAT/DATAFILE/eodsapp.3112.947766815";
   set newname for datafile  28 to
 "/data/MYUAT/DATAFILE/eodslrg.3099.947766811";
   set newname for datafile  29 to
 "/data/MYUAT/DATAFILE/eoecapp.2074.947766809";
   set newname for datafile  30 to
 "/data/MYUAT/DATAFILE/eoeclrg.2069.947766809";
   set newname for datafile  31 to
 "/data/MYUAT/DATAFILE/eoecwrk.3103.947766811";
   set newname for datafile  32 to
 "/data/MYUAT/DATAFILE/eoeiapp.3106.947766811";
   set newname for datafile  33 to
 "/data/MYUAT/DATAFILE/eoeilrg.3108.947766811";
   set newname for datafile  34 to
 "/data/MYUAT/DATAFILE/eoewapp.3109.947766813";
   set newname for datafile  35 to
 "/data/MYUAT/DATAFILE/eoewlrg.2083.947766099";
   set newname for datafile  36 to
 "/data/MYUAT/DATAFILE/eoewwrk.2077.947766099";
   set newname for datafile  37 to
 "/data/MYUAT/DATAFILE/eoiuapp.3110.947766815";
   set newname for datafile  38 to
 "/data/MYUAT/DATAFILE/eoiulrg.2099.947790101";
   set newname for datafile  39 to
 "/data/MYUAT/DATAFILE/eoiuwrk.2113.947766099";
   set newname for datafile  40 to
 "/data/MYUAT/DATAFILE/eolarge.2100.947766099";
   set newname for datafile  41 to
 "/data/MYUAT/DATAFILE/eotpapp.342.947766809";
   set newname for datafile  42 to
 "/data/MYUAT/DATAFILE/eotplrg.3093.947766809";
   set newname for datafile  43 to
 "/data/MYUAT/DATAFILE/lmimage.3115.947766855";
   set newname for datafile  44 to
 "/data/MYUAT/DATAFILE/lmlarge.2103.947752747";
   set newname for datafile  45 to
 "/data/MYUAT/DATAFILE/lmwork.2081.947766099";
   set newname for datafile  46 to
 "/data/MYUAT/DATAFILE/pvapp.3116.947766857";
   set newname for datafile  47 to
 "/data/MYUAT/DATAFILE/psimage.2105.947752747";
   set newname for datafile  48 to
 "/data/MYUAT/DATAFILE/lmapp.2096.947752747";
   set newname for datafile  49 to
 "/data/MYUAT/DATAFILE/psindex.2104.947752747";
   set newname for datafile  50 to
 "/data/MYUAT/DATAFILE/users.2068.947766809";
   set newname for datafile  51 to
 "/data/MYUAT/DATAFILE/psimage2.2098.947752747";
   set newname for datafile  52 to
 "/data/MYUAT/DATAFILE/psmatvw.3101.947766811";
   set newname for datafile  53 to
 "/data/MYUAT/DATAFILE/psgtt01.3102.947766811";
   set newname for datafile  54 to
 "/data/MYUAT/DATAFILE/audit_data.2070.947766809";
   set newname for datafile  55 to
 "/data/MYUAT/DATAFILE/undotbs5.2092.947752747";
   set newname for datafile  56 to
 "/data/MYUAT/DATAFILE/undotbs6.2080.947765411";
   restore
   clone database
   ;
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 17-JAN-18

channel ch1: starting datafile backup set restore
channel ch1: specifying datafile(s) to restore from backup set
channel ch1: restoring datafile 00001 to /data/MYUAT/DATAFILE/system.3097.947766811
channel ch1: restoring datafile 00018 to /data/MYUAT/DATAFILE/pttbl.2152.947766809
channel ch1: restoring datafile 00019 to /data/MYUAT/DATAFILE/pttlrg.2072.947766809
channel ch1: restoring datafile 00030 to /data/MYUAT/DATAFILE/eoeclrg.2069.947766809
channel ch1: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58907_1
channel ch2: starting datafile backup set restore
channel ch2: specifying datafile(s) to restore from backup set
channel ch2: restoring datafile 00002 to /data/MYUAT/DATAFILE/sysaux.2076.947766099
channel ch2: restoring datafile 00015 to /data/MYUAT/DATAFILE/ptprc.2150.947766809
channel ch2: restoring datafile 00029 to /data/MYUAT/DATAFILE/eoecapp.2074.947766809
channel ch2: restoring datafile 00050 to /data/MYUAT/DATAFILE/users.2068.947766809
channel ch2: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58908_1
channel ch3: starting datafile backup set restore
channel ch3: specifying datafile(s) to restore from backup set
channel ch3: restoring datafile 00003 to /data/MYUAT/DATAFILE/undotbs1.2095.947765199
channel ch3: restoring datafile 00021 to /data/MYUAT/DATAFILE/ptwork.3107.947766811
channel ch3: restoring datafile 00034 to /data/MYUAT/DATAFILE/eoewapp.3109.947766813
channel ch3: restoring datafile 00046 to /data/MYUAT/DATAFILE/pvapp.3116.947766857
channel ch3: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58903_1
channel ch4: starting datafile backup set restore
channel ch4: specifying datafile(s) to restore from backup set
channel ch4: restoring datafile 00004 to /data/MYUAT/DATAFILE/psdefault.2085.947766809
channel ch4: restoring datafile 00009 to /data/MYUAT/DATAFILE/ptamsg.3096.947766809
channel ch4: restoring datafile 00017 to /data/MYUAT/DATAFILE/ptrpts.2151.947766809
channel ch4: restoring datafile 00054 to /data/MYUAT/DATAFILE/audit_data.2070.947766809
channel ch4: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58909_1
channel ch5: starting datafile backup set restore
channel ch5: specifying datafile(s) to restore from backup set
channel ch5: restoring datafile 00005 to /data/MYUAT/DATAFILE/undotbs2.2102.947752747
channel ch5: restoring datafile 00014 to /data/MYUAT/DATAFILE/ptlock.3091.947766809
channel ch5: restoring datafile 00024 to /data/MYUAT/DATAFILE/eocmwrk.322.947766809
channel ch5: restoring datafile 00041 to /data/MYUAT/DATAFILE/eotpapp.342.947766809
channel ch5: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58901_1
channel ch6: starting datafile backup set restore
channel ch6: specifying datafile(s) to restore from backup set
channel ch6: restoring datafile 00006 to /data/MYUAT/DATAFILE/undotbs3.2089.947752747
channel ch6: restoring datafile 00020 to /data/MYUAT/DATAFILE/pttree.3111.947766815
channel ch6: restoring datafile 00027 to /data/MYUAT/DATAFILE/eodsapp.3112.947766815
channel ch6: restoring datafile 00037 to /data/MYUAT/DATAFILE/eoiuapp.3110.947766815
channel ch6: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58902_1
channel ch7: starting datafile backup set restore
channel ch7: specifying datafile(s) to restore from backup set
channel ch7: restoring datafile 00007 to /data/MYUAT/DATAFILE/undotbs4.2071.947765631
channel ch7: restoring datafile 00010 to /data/MYUAT/DATAFILE/ptapp.3114.947766845
channel ch7: restoring datafile 00016 to /data/MYUAT/DATAFILE/ptprjwk.3113.947766837
channel ch7: restoring datafile 00043 to /data/MYUAT/DATAFILE/lmimage.3115.947766855
channel ch7: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58906_1
channel ch8: starting datafile backup set restore
channel ch8: specifying datafile(s) to restore from backup set
channel ch8: restoring datafile 00008 to /data/MYUAT/DATAFILE/psimgr.3100.947766811
channel ch8: restoring datafile 00022 to /data/MYUAT/DATAFILE/eocmapp.3104.947766811
channel ch8: restoring datafile 00032 to /data/MYUAT/DATAFILE/eoeiapp.3106.947766811
channel ch8: restoring datafile 00055 to /data/MYUAT/DATAFILE/undotbs5.2092.947752747
channel ch8: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58905_1
channel ch9: starting datafile backup set restore
channel ch9: specifying datafile(s) to restore from backup set
channel ch9: restoring datafile 00011 to /data/MYUAT/DATAFILE/ptappe.3105.947766811
channel ch9: restoring datafile 00052 to /data/MYUAT/DATAFILE/psmatvw.3101.947766811
channel ch9: restoring datafile 00053 to /data/MYUAT/DATAFILE/psgtt01.3102.947766811
channel ch9: restoring datafile 00056 to /data/MYUAT/DATAFILE/undotbs6.2080.947765411
channel ch9: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58904_1
channel ch4: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58909_1 tag=TAG20180115T021026
channel ch4: restored backup piece 1
channel ch4: restore complete, elapsed time: 00:12:46
channel ch4: starting datafile backup set restore
channel ch4: specifying datafile(s) to restore from backup set
channel ch4: restoring datafile 00012 to /data/MYUAT/DATAFILE/ptaudit.3092.947766809
channel ch4: restoring datafile 00023 to /data/MYUAT/DATAFILE/eocmlrg.3094.947766809
channel ch4: restoring datafile 00042 to /data/MYUAT/DATAFILE/eotplrg.3093.947766809
channel ch4: restoring datafile 00051 to /data/MYUAT/DATAFILE/psimage2.2098.947752747
channel ch4: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58900_1
channel ch2: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58908_1 tag=TAG20180115T021026
channel ch2: restored backup piece 1
channel ch2: restore complete, elapsed time: 00:13:12
channel ch2: starting datafile backup set restore
channel ch2: specifying datafile(s) to restore from backup set
channel ch2: restoring datafile 00013 to /data/MYUAT/DATAFILE/ptcmstar.2090.947766099
channel ch2: restoring datafile 00040 to /data/MYUAT/DATAFILE/eolarge.2100.947766099
channel ch2: restoring datafile 00045 to /data/MYUAT/DATAFILE/lmwork.2081.947766099
channel ch2: restoring datafile 00047 to /data/MYUAT/DATAFILE/psimage.2105.947752747
channel ch2: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58896_1
channel ch2: ORA-19870: error while restoring backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58896_1
ORA-19504: failed to create file "/data/MYUAT/DATAFILE/psimage.2105.947752747"
ORA-27044: unable to write the header block of file
Linux-x86_64 Error: 27: File too large
Additional information: 4

channel ch2: starting datafile backup set restore
channel ch2: specifying datafile(s) to restore from backup set
channel ch2: restoring datafile 00025 to /data/MYUAT/DATAFILE/eocuapp.3095.947766809
channel ch2: restoring datafile 00026 to /data/MYUAT/DATAFILE/eoculrg.3098.947766811
channel ch2: restoring datafile 00028 to /data/MYUAT/DATAFILE/eodslrg.3099.947766811
channel ch2: restoring datafile 00044 to /data/MYUAT/DATAFILE/lmlarge.2103.947752747
channel ch2: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58899_1
channel ch3: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58903_1 tag=TAG20180115T021026
channel ch3: restored backup piece 1
channel ch3: restore complete, elapsed time: 00:14:05
channel ch3: starting datafile backup set restore
channel ch3: specifying datafile(s) to restore from backup set
channel ch3: restoring datafile 00031 to /data/MYUAT/DATAFILE/eoecwrk.3103.947766811
channel ch3: restoring datafile 00033 to /data/MYUAT/DATAFILE/eoeilrg.3108.947766811
channel ch3: restoring datafile 00035 to /data/MYUAT/DATAFILE/eoewlrg.2083.947766099
channel ch3: restoring datafile 00048 to /data/MYUAT/DATAFILE/lmapp.2096.947752747
channel ch3: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58898_1
channel ch7: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58906_1 tag=TAG20180115T021026
channel ch7: restored backup piece 1
channel ch7: restore complete, elapsed time: 00:14:04
channel ch7: starting datafile backup set restore
channel ch7: specifying datafile(s) to restore from backup set
channel ch7: restoring datafile 00036 to /data/MYUAT/DATAFILE/eoewwrk.2077.947766099
channel ch7: restoring datafile 00038 to /data/MYUAT/DATAFILE/eoiulrg.2099.947790101
channel ch7: restoring datafile 00039 to /data/MYUAT/DATAFILE/eoiuwrk.2113.947766099
channel ch7: restoring datafile 00049 to /data/MYUAT/DATAFILE/psindex.2104.947752747
channel ch7: reading from backup piece /backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58897_1
channel ch8: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58905_1 tag=TAG20180115T021026
channel ch8: restored backup piece 1
channel ch8: restore complete, elapsed time: 00:14:04
channel ch9: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58904_1 tag=TAG20180115T021026
channel ch9: restored backup piece 1
channel ch9: restore complete, elapsed time: 00:14:04
channel ch1: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58907_1 tag=TAG20180115T021026
channel ch1: restored backup piece 1
channel ch1: restore complete, elapsed time: 00:14:05
channel ch6: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58902_1 tag=TAG20180115T021026
channel ch6: restored backup piece 1
channel ch6: restore complete, elapsed time: 00:14:40
channel ch5: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58901_1 tag=TAG20180115T021026
channel ch5: restored backup piece 1
channel ch5: restore complete, elapsed time: 00:15:00
channel ch4: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58900_1 tag=TAG20180115T021026
channel ch4: restored backup piece 1
channel ch4: restore complete, elapsed time: 00:54:35
channel ch2: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58899_1 tag=TAG20180115T021026
channel ch2: restored backup piece 1
channel ch2: restore complete, elapsed time: 00:56:33
channel ch3: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58898_1 tag=TAG20180115T021026
channel ch3: restored backup piece 1
channel ch3: restore complete, elapsed time: 01:15:48
channel ch7: piece handle=/backupshare/ML_PUNE_DR_Golden_offbkp/FULL_BACKUP_MLPRD_58897_1 tag=TAG20180115T021026
channel ch7: restored backup piece 1
channel ch7: restore complete, elapsed time: 01:29:28
failover to previous backup

creating datafile file number=13 name=/data/MYUAT/DATAFILE/ptcmstar.2090.947766099
restarting auxiliary database without server parameter file
Oracle instance started

Total System Global Area   19241058304 bytes

Fixed Size                     2261368 bytes
Variable Size               9529462408 bytes
Database Buffers            9663676416 bytes
Redo Buffers                  45658112 bytes
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 01/17/2018 05:01:02
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
RMAN-06136: ORACLE error from auxiliary database: ORA-01119: error in creating database file '+ndata_my'
ORA-17502: ksfdcre:4 Failed to create file +ndata_ctsp
ORA-15001: diskgroup "NDATA_MY" does not exist or is not mounted
ORA-15077: could not locate ASM instance serving a required diskgroup
ORA-29701: unable to connect to Cluster Synchronization Service
RMAN-06956: create datafile failed; retry after removing /data/MYUAT/DATAFILE/ptcmstar.2090.947766099 from OS

Recovery Manager complete.