Wednesday 31 August 2011

RMAN Catalog Deadlock errors


Applies to:

Oracle Server - Enterprise Edition - Version: 11.1.0.6
This problem can occur on any platform.

Symptoms

RMAN backups that use catalog database fails with following deadlock errors.

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: ===========================================================
RMAN-03002: failure of backup plus archivelog command at 11/26/2008 22:00:43
RMAN-03014: implicit resync of recovery catalog failed
RMAN-03009: failure of partial resync command on default channel at 11/26/2008 22:00:43
ORA-00060: deadlock detected while waiting for resource


Dead lock trace file contains the following statement

DELETE FROM CKP WHERE DBINC_KEY = :B1 AND CKP_KEY IN (SELECT CKP_KEY1 FROM
(SELECT CKP_KEY CKP_KEY1 FROM CKP WHERE DBINC_KEY = :B1 ) CKP1, (SELECT :B2
CKP_KEY2 FROM DUAL UNION SELECT :B3 FROM DUAL UNION SELECT NVL(MAX(CKP_KEY),0) F
ROM CKP WHERE DBINC_KEY=:B1 UNION SELECT START_CKP_KEY FROM TSATT WHERE DBINC_KE
Y = :B1 UNION SELECT NVL(END_CKP_KEY,0) FROM TSATT WHERE DBINC_KEY = :B1 UNION S
ELECT START_CKP_KEY FROM DFATT WHERE DBINC_KEY = :B1 UNION SELECT NVL(END_CKP_KE
Y,0) FROM DFATT WHERE DBINC_KEY = :B1 UNION SELECT START_CKP_KEY FROM TFATT WHER
E DBINC_KEY = :B1 UNION SELECT NVL(END_CKP_KEY,0) FROM TFATT WHERE DBINC_KEY = :
B1 ) CKP2 WHERE CKP_KEY1 = CKP_KEY2(+) AND CKP_KEY2 IS NULL)



Cause

This issue is due to Bug 6830296
Abstract: RMAN-3014, ORA-60 RUNNING BACKUPS FOR 2 DATABASES USING SAME CATALOG

Fixed version :- 11.1.0.6

Solution

Workaround :-
In Rman catalog database create the following indexes :-

CREATE INDEX tfatt_i_sck on tfatt(start_ckp_key)
CREATE INDEX tfatt_i_eck on tfatt(end_ckp_key)
Or
Check for availability of one off patch using the following link Patch 6830296

Monday 15 August 2011

Track ASM Extents by X$KFFXP

I heard about the X$ fixed table (X$KFFXP). It keeps position of striped and mirrored extents... and etc. If You connect to ASM Instance, You can check and retrieve data from it.

$ sqlplus / as sysasm

SQL>
desc X$KFFXP

Name Null? Type
----------------------------------------------------------------------------------- -------- ----------
ADDR RAW(8)
INDX NUMBER
INST_ID NUMBER
GROUP_KFFXP NUMBER
NUMBER_KFFXP NUMBER
COMPOUND_KFFXP NUMBER
INCARN_KFFXP NUMBER
PXN_KFFXP NUMBER
XNUM_KFFXP NUMBER
LXN_KFFXP NUMBER
DISK_KFFXP NUMBER
AU_KFFXP NUMBER
FLAGS_KFFXP NUMBER
CHK_KFFXP NUMBER
SIZE_KFFXP NUMBER

Some columns in X$KFFXP table:
GROUP_KFFXP = V$ASM_DISKGROUP.GROUP_NUMBER
NUMBER_KFFXP = V$ASM_FILE.FILE_NUMBER
COMPOUND_KFFXP = V$ASM_FILE.COMPOUND_INDEX
INCARN_KFFXP = V$ASM_FILE.INCARNATION
XNUM_KFFXP = Extent Number
LXN_KFFXP = Logical Extent Number (0 = primary, 1 = mirrored copy, 2 = mirrored copy[High redundancy])
DISK_KFFXP = V$ASM_DISK.DISK_NUMBER
AU_KFFXP = V$ASM_DISKGROUP.ALLOCATION_UNIT_SIZE

So, I used X$KFFXP table for testing (just tested)

select dg.name, x.NUMBER_KFFXP "FILE NUMBER", x.XNUM_KFFXP "EXTENT NUMBER", decode(x.LXN_KFFXP,0,'P',1,'M','MM') "EXTENT TYPE", d.path from x$kffxp x, v$asm_disk d, v$asm_diskgroup dg where x.GROUP_KFFXP = d.GROUP_NUMBER and x.DISK_KFFXP = d.DISK_NUMBER and x.GROUP_KFFXP=dg.GROUP_NUMBER and x.NUMBER_KFFXP in (select file_number from v$asm_alias where name like 'USERS%') order by x.NUMBER_KFFXP, x.XNUM_KFFXP, x.LXN_KFFXP
/

P = primary
M = mirrored copy
MM = mirrored copy[High redundancy]

- On ASM, that uses External redundancy

SQL> select dg.name, x.NUMBER_KFFXP "FILE NUMBER", x.XNUM_KFFXP "EXTENT NUMBER", decode(x.LXN_KFFXP,0,'P',1,'M','MM') "EXTENT TYPE", d.path from x$kffxp x, v$asm_disk d, v$asm_diskgroup dg where x.GROUP_KFFXP = d.GROUP_NUMBER and x.DISK_KFFXP = d.DISK_NUMBER and x.GROUP_KFFXP=dg.GROUP_NUMBER and x.NUMBER_KFFXP in (select file_number from v$asm_alias where name like 'USERS%') order by x.NUMBER_KFFXP, x.XNUM_KFFXP, x.LXN_KFFXP
/

NAME FILE NUMBER EXTENT NUMBER EXTENT TYPE PATH
----- ----------- ------------- --------------- ---------------
DATA 316 0 P /dev/sdb
DATA 316 1 P /dev/sdb
DATA 316 2 P /dev/sdb
DATA 316 3 P /dev/sdb
DATA 316 4 P /dev/sdb
DATA 316 5 P /dev/sdb

- On ASM, that uses Normal redundancy

SQL> select dg.name, x.NUMBER_KFFXP "FILE NUMBER", x.XNUM_KFFXP "EXTENT NUMBER", decode(x.LXN_KFFXP,0,'P',1,'M','MM') "EXTENT TYPE", d.path from x$kffxp x, v$asm_disk d, v$asm_diskgroup dg where x.GROUP_KFFXP = d.GROUP_NUMBER and x.DISK_KFFXP = d.DISK_NUMBER and x.GROUP_KFFXP=dg.GROUP_NUMBER and x.NUMBER_KFFXP in (select file_number from v$asm_alias where name like 'USERS%') order by x.NUMBER_KFFXP, x.XNUM_KFFXP, x.LXN_KFFXP
/

NAME FILE NUMBER EXTENT NUMBER EXTENT TYPE PATH
------------------------------ ----------- ------------- --------------- ------------------------------
DATA 261 0 P ORCL:DATA10
DATA 261 0 M ORCL:DATA02
DATA 261 1 P ORCL:DATA03
DATA 261 1 M ORCL:DATA07
DATA 261 2 P ORCL:DATA09
DATA 261 2 M ORCL:DATA05
DATA 261 3 P ORCL:DATA05
DATA 261 3 M ORCL:DATA07
DATA 261 4 P ORCL:DATA08
DATA 261 4 M ORCL:DATA04
DATA 261 5 P ORCL:DATA04
DATA 261 5 M ORCL:DATA06

- On ASM, that uses High redundancy

SQL> select dg.name, x.NUMBER_KFFXP "FILE NUMBER", x.XNUM_KFFXP "EXTENT NUMBER", decode(x.LXN_KFFXP,0,'P',1,'M','MM') "EXTENT TYPE", d.path from x$kffxp x, v$asm_disk d, v$asm_diskgroup dg where x.GROUP_KFFXP = d.GROUP_NUMBER and x.DISK_KFFXP = d.DISK_NUMBER and x.GROUP_KFFXP=dg.GROUP_NUMBER and x.NUMBER_KFFXP in (select file_number from v$asm_alias where name like 'USERS%') order by x.NUMBER_KFFXP, x.XNUM_KFFXP, x.LXN_KFFXP
/

NAME FILE NUMBER EXTENT NUMBER EXTENT TYPE PATH
------------------------------ ----------- ------------- --------------- --------------------------------------------------
DATA 917 0 P o/192.168.99.6/DATA_EXA_CD_11_exacel02
DATA 917 0 M o/192.168.99.11/DATA_EXA_CD_02_exacel07
DATA 917 0 MM o/192.168.99.7/DATA_EXA_CD_00_exacel03
DATA 917 1 P o/192.168.99.9/DATA_EXA_CD_01_exacel05
DATA 917 1 M o/192.168.99.10/DATA_EXA_CD_11_exacel06
DATA 917 1 MM o/192.168.99.7/DATA_EXA_CD_06_exacel03
DATA 917 2 P o/192.168.99.9/DATA_EXA_CD_11_exacel05
DATA 917 2 M o/192.168.99.10/DATA_EXA_CD_01_exacel06
DATA 917 2 MM o/192.168.99.8/DATA_EXA_CD_02_exacel04
DATA 917 3 P o/192.168.99.6/DATA_EXA_CD_04_exacel02
DATA 917 3 M o/192.168.99.5/DATA_EXA_CD_10_exacel01
DATA 917 3 MM o/192.168.99.11/DATA_EXA_CD_10_exacel07
DATA 917 4 P o/192.168.99.8/DATA_EXA_CD_09_exacel04
DATA 917 4 M o/192.168.99.10/DATA_EXA_CD_09_exacel06
DATA 917 4 MM o/192.168.99.9/DATA_EXA_CD_03_exacel05
DATA 917 5 P o/192.168.99.6/DATA_EXA_CD_06_exacel02
DATA 917 5 M o/192.168.99.7/DATA_EXA_CD_10_exacel03
DATA 917 5 MM o/192.168.99.8/DATA_EXA_CD_07_exacel04

That was example to use it. However, you can use X$KFFXP to join with other views(V$ASM_*)

Saturday 13 August 2011

Converting SCN to a TimeStamp

I saw the nice function below while i was reading about locking and latches section of Mr Thomas Kytes Book (Expert Oracle Database Architecture 9i and 10g Programming Techniques and Solutions) and i said to myself i have to share it for the ones like me :)

If you know the SCN (system change number) you can get its timestamp value (within about +/–3 seconds) by the function scn_to_timestamp. After looking to the manual for more info i saw two other nice functions about scn. They are all under DBMS_FLASHBACK package and not available for the releases prior to 10g. I found these functions useful for dataguard issues recovery issues and flashback issues.

GET_SYSTEM_CHANGE_NUMBER: for getting the current system change number of the database.

SQL> select DBMS_FLASHBACK.GET_SYSTEM_CHANGE_NUMBER from dual;

GET_SYSTEM_CHANGE_NUMBER
————————
884871

SCN_TO_TIMESTAMP: for converting given scn to timestamp value ;

SQL> select scn_to_timestamp(884871) as timestamp from dual;

TIMESTAMP
—————————————————————————
09/03/2007 14:52:02,000000000

TIMESTAMP_TO_SCN : For getting SCN equivalent of the given timestamp value. You must do to_timestamp convertion for the character value.

SQL> select timestamp_to_scn(to_timestamp(’08/03/2007 14:24:54′,’DD/MM/YYYY HH24:MI:SS’)) as scn from dual;

SCN
———-
845396

Thursday 11 August 2011

Deconfigure EM Database Control

emca -deconfig dbcontrol db -repos drop -HOST OraDRStby -PORT 1522 -SID gridem -SYS_PWD oracle -SYSMAN_PWD oracle -DBSNMP_PWD oracle

-----------------------------------------------------------------------------------------------------
[oracle@OraDRStby:gridem]$ . oraenv
ORACLE_SID = [gridem] ?
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@OraDRStby:gridem]$ emca -deconfig dbcontrol db -repos drop -HOST OraDRStby -PORT 1522 -SID gridem -SYS_PWD oracle -SYSMA N_PWD oracle -DBSNMP_PWD oracle

STARTED EMCA at Aug 11, 2011 12:19:22 PM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.

Enter the following information:

Do you wish to continue? [yes(Y)/no(N)]: y
Aug 11, 2011 12:19:30 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/gridem/emca_2011_08_11_12_19_22.log.
Aug 11, 2011 12:19:30 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Aug 11, 2011 12:19:54 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Aug 11, 2011 12:22:13 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Aug 11, 2011 12:22:19 PM

Tuesday 9 August 2011

RMAN-06024 during duplicate database for standby

If you get RMAN-06024 when running "duplicate target database for standby" and you are sure you have run a "backup current controlfile for standby" then try doing a log switch on the target database to fix the issue.

RMAN-06024: no backup or copy of the control file found to restore

Oracle 10g - Manually Create a Physical Standby Database Using Data Guard

Step-by-step instructions on how to create a Physical Standby Database on Windows and UNIX servers, and maintenance tips on the databases in a Data Guard Environment.

Oracle 10g Data Guard is a great tool to ensure high availability, data protection and disaster recovery for enterprise data. I have been working on Data Guard/Standby databases using both Grid control and SQL command line for a couple of years, and my latest experience with Data Guard was manually creating a Physical Standby Database for a Laboratory Information Management System (LIMS) half a year ago. I maintain it daily and it works well. I would like to share my experience with the other DBAs.

In this example the database version is 10.2.0.3.. The Primary database and Standby database are located on different machines at different sites. The Primary database is called PRIM and the Standby database is called STAN. I use Flash Recovery Area, and OMF.

I. Before you get started:

1. Make sure the operating system and platform architecture on the primary and standby systems are the same;

2. Install Oracle database software without the starter database on the standby server and patch it if necessary. Make sure the same Oracle software release is used on the Primary and Standby databases, and Oracle home paths are identical.

3. Test the Standby Database creation on a test environment first before working on the Production database.

II. On the Primary Database Side:

1. Enable forced logging on your primary database:
SQL> ALTER DATABASE FORCE LOGGING;

2. Create a password file if it doesn’t exist.
1) To check if a password file already exists, run the following command:
SQL> select * from v$pwfile_users;

2) If it doesn’t exist, use the following command to create one:
- On Windows:
$cd %ORACLE_HOME%\database
$orapwd file=pwdPRIM.ora password=xxxxxxxx force=y
(Note: Replace xxxxxxxxx with the password for the SYS user.)

- On UNIX:
$Cd $ORACLE_HOME/dbs
$Orapwd file=pwdPRIM.ora password=xxxxxxxx force=y
(Note: Replace xxxxxxxxx with your actual password for the SYS user.)

3. Configure a Standby Redo log.
1) The size of the standby redo log files should match the size of the current Primary database online redo log files. To find out the size of your online redo log files:
SQL> select bytes from v$log;

BYTES
----------
52428800
52428800
52428800

2) Use the following command to determine your current log file groups:
SQL> select group#, member from v$logfile;

3) Create standby Redo log groups.

Standby no of redo log group ==> (maximum number of logfiles for each thread + 1) * maximum number of threads
My primary database had 3 log file groups originally and I created 4 standby redo log groups using the following commands:
SQL>ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 SIZE 50M;
SQL>ALTER DATABASE ADD STANDBY LOGFILE GROUP 5 SIZE 50M;
SQL>ALTER DATABASE ADD STANDBY LOGFILE GROUP 6 SIZE 50M;

SQL>ALTER DATABASE ADD STANDBY LOGFILE GROUP 7 SIZE 50M;

or

ALTER DATABASE ADD STANDBY LOGFILE GROUP 4 '/u01/app/oracle/oradata/PRIM/redostd04.log' SIZE 52100k ;

ALTER DATABASE ADD STANDBY LOGFILE GROUP 5 '/u01/app/oracle/oradata/PRIM/redostd05.log' SIZE 52100k ;

ALTER DATABASE ADD STANDBY LOGFILE GROUP 6 '/u01/app/oracle/oradata/PRIM/redostd06.log' SIZE 52100k ;

ALTER DATABASE ADD STANDBY LOGFILE GROUP 7 '/u01/app/oracle/oradata/PRIM/redostd07.log' SIZE 52100k ;

4) To verify the results of the standby redo log groups creation, run the following query:
SQL>select * from v$standby_log;

4. Enable Archiving on Primary.
If your primary database is not already in Archive Log mode, enable the archive log mode:
SQL>shutdown immediate;
SQL>startup mount;
SQL>alter database archivelog;
SQL>alter database open;
SQL>archive log list;

5. Set Primary Database Initialization Parameters
Create a text initialization parameter file (PFILE) from the server parameter file (SPFILE), to add the new primary role parameters.

1) Create pfile from spfile for the primary database:
- On Windows:
SQL>create pfile=’\database\pfilePRIM.ora’ from spfile;
(Note- specify your Oracle home path to replace ‘’).

- On UNIX:
SQL>create pfile=’/dbs/pfilePRIM.ora’ from spfile;
(Note- specify your Oracle home path to replace ‘’).

2) Edit pfilePRIM.ora to add the new primary and standby role parameters: (Here the file paths are from a windows system. For UNIX system, specify the path accordingly)

db_name=PRIM
db_unique_name=PRIM
LOG_ARCHIVE_CONFIG='DG_CONFIG=(PRIM,STAN)'
LOG_ARCHIVE_DEST_1=
'LOCATION=F:\Oracle\flash_recovery_area\PRIM\ARCHIVELOG
VALID_FOR=(ALL_LOGFILES,ALL_ROLES)
DB_UNIQUE_NAME=PRIM'
LOG_ARCHIVE_DEST_2=
'SERVICE=STAN LGWR ASYNC
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)
DB_UNIQUE_NAME=STAN'
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=ENABLE
LOG_ARCHIVE_FORMAT=%t_%s_%r.arc
LOG_ARCHIVE_MAX_PROCESSES=30
remote_login_passwordfile='EXCLUSIVE'
FAL_SERVER=STAN
FAL_CLIENT=PRIM
STANDBY_FILE_MANAGEMENT=AUTO
# Specify the location of the standby DB datafiles followed by the primary location;
DB_FILE_NAME_CONVERT='E:\oracle\product\10.2.0\oradata\STAN\DATAFILE','E:\oracle\product\10.2.0\oradata\PRIM\DATAFILE'
# Specify the location of the standby DB online redo log files followed by the primary location LOG_FILE_NAME_CONVERT=’E:\oracle\product\10.2.0\oradata\STAN\ONLINELOG’,’E:\oracle\product\10.2.0\oradata\PRIM\ONLINELOG’,’F:\Oracle\flash_recovery_area\STAN\ONLINELOG’,’F:\Oracle\flash_recovery_area\PRIM\ONLINELOG’

6. Create spfile from pfile, and restart primary database using the new spfile.
Data Guard must use SPFILE. Create the SPFILE and restart database.
- On windows:
SQL> shutdown immediate;
SQL> startup nomount pfile=’\database\pfilePRIM.ora’;
SQL>create spfile from pfile=’\database\pfilePRIM.ora’;
-- Restart the Primary database using the newly created SPFILE.
SQL>shutdown immediate;
SQL>Startup;
(Note- specify your Oracle home path to replace ‘’).

- On UNIX:
SQL> shutdown immediate;
SQL> startup nomount pfile=’/dbs/pfilePRIM.ora’;
SQL>create spfile from pfile=’/dbs/pfilePRIM.ora’;
-- Restart the Primary database using the newly created SPFILE.
SQL>shutdown immediate;
SQL>Startup;
(Note- specify your Oracle home path to replace ‘’).

III. On the Standby Database Site:

1. Create a copy of Primary database data files on the Standby Server:
On Primary DB:
SQL>shutdown immediate;

On Standby Server (While the Primary database is shut down):
1) Create directory for data files, for example, on windows, E:\oracle\product\10.2.0\oradata\STAN\DATAFILE.
On UNIX, create the directory accordingly.

2) Copy the data files and temp files over.

3) Create directory (multiplexing) for online logs, for example, on Windows, E:\oracle\product\10.2.0\oradata\STAN\ONLINELOG and F:\Oracle\flash_recovery_area\STAN\ONLINELOG.
On UNIX, create the directories accordingly.

4) Copy the online logs over.

2. Create a Control File for the standby database:
On Primary DB, create a control file for the standby to use:
SQL>startup mount;
SQL>alter database create standby controlfile as ‘STAN.ctl;
SQL>ALTER DATABASE OPEN;

3. Copy the Primary DB pfile to Standby server and rename/edit the file.

1) Copy pfilePRIM.ora from Primary server to Standby server, to database folder on Windows or dbs folder on UNIX under the Oracle home path.

2) Rename it to pfileSTAN.ora, and modify the file as follows. : (Here the file paths are from a windows system. For UNIX system, specify the path accordingly)

*.audit_file_dest='E:\oracle\product\10.2.0\admin\STAN\adump'
*.background_dump_dest='E:\oracle\product\10.2.0\admin\STAN\bdump'
*.core_dump_dest='E:\oracle\product\10.2.0\admin\STAN\cdump'
*.user_dump_dest='E:\oracle\product\10.2.0\admin\STAN\udump'
*.compatible='10.2.0.3.0'
control_files='E:\ORACLE\PRODUCT\10.2.0\ORADATA\STAN\CONTROLFILE\STAN.CTL','F:\ORACLE\FLASH_RECOVERY_AREA\STAN\CONTROLFILE\STAN.CTL'
db_name='PRIM'
db_unique_name=STAN
LOG_ARCHIVE_CONFIG=’DG_CONFIG=(PRIM,STAN)’
LOG_ARCHIVE_DEST_1=
‘LOCATION=F:\Oracle\flash_recovery_area\STAN\ARCHIVELOG
VALID_FOR=(ALL_LOGFILES,ALL_ROLES)
DB_UNIQUE_NAME=STAN’
LOG_ARCHIVE_DEST_2=
‘SERVICE=PRIM LGWR ASYNC
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)
DB_UNIQUE_NAME=PRIM’
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=ENABLE
LOG_ARCHIVE_FORMAT=%t_%s_%r.arc
LOG_ARCHIVE_MAX_PROCESSES=30
FAL_SERVER=PRIM
FAL_CLIENT=STAN
remote_login_passwordfile='EXCLUSIVE'
# Specify the location of the primary DB datafiles followed by the standby location
DB_FILE_NAME_CONVERT=’E:\oracle\product\10.2.0\oradata\PRIM\DATAFILE’,’E:\oracle\product\10.2.0\oradata\STAN\DATAFILE’
# Specify the location of the primary DB online redo log files followed by the standby location
LOG_FILE_NAME_CONVERT=’E:\oracle\product\10.2.0\oradata\PRIM\ONLINELOG’,’E:\oracle\product\10.2.0\oradata\STAN\ONLINELOG’,’F:\Oracle\flash_recovery_area\PRIM\ONLINELOG’,’F:\Oracle\flash_recovery_area\STAN\ONLINELOG’
STANDBY_FILE_MANAGEMENT=AUTO

(Note: Not all the parameter entries are listed here.)

4. On Standby server, create all required directories for dump and archived log destination:
Create directories adump, bdump, cdump, udump, and archived log destinations for the standby database.

5. Copy the standby control file ‘STAN.ctl’ from primary to standby destinations ;

6. Copy the Primary password file to standby and rename it to pwdSTAN.ora.
On Windows copy it to \database folder, and on UNIX copy it to /dbs directory. And then rename the password file.

7. For Windows, create a Windows-based services (optional):
$oradim –NEW –SID STAN –STARTMODE manual

8. Configure listeners for the primary and standby databases.

1) On Primary system: use Oracle Net Manager to configure a listener for PRIM and STAN. Then restart the listener.
$lsnrctl stop
$lsnrctl start

2) On Standby server: use Net Manager to configure a listener for PRIM and STAN. Then restart the listener.
$lsnrctl stop
$lsnrctl start

9. Create Oracle Net service names.
1) On Primary system: use Oracle Net Manager to create network service names for PRIM and STAN. Check tnsping to both services:
$tnsping PRIM
$tnsping STAN

2) On Standby system: use Oracle Net Manager to create network service names for PRIM and STAN. Check tnsping to both services:
$tnsping PRIM
$tnsping STAN

10. On Standby server, setup the environment variables to point to the Standby database.

Set up ORACLE_HOME and ORACLE_SID.

11. Start up nomount the standby database and generate a spfile.
- On Windows:
SQL>startup nomount pfile=’\database\pfileSTAN.ora’;
SQL>create spfile from pfile=’\database\pfileSTAN.ora’;
-- Restart the Standby database using the newly created SPFILE.
SQL>shutdown immediate;
SQL>startup mount;

- On UNIX:
SQL>startup nomount pfile=’/dbs/pfileSTAN.ora’;
SQL>create spfile from pfile=’/dbs/pfileSTAN.ora’;
-- Restart the Standby database using the newly created SPFILE.
SQL>shutdown immediate;
SQL>startup mount;
(Note- specify your Oracle home path to replace ‘’).

12. Start Redo apply
1) On the standby database, to start redo apply:
SQL>alter database recover managed standby database disconnect from session;

If you ever need to stop log apply services:
SQL> alter database recover managed standby database cancel;

13. Verify the standby database is performing properly:
1) On Standby perform a query:
SQL>select sequence#, first_time, next_time from v$archived_log;

2) On Primary, force a logfile switch:
SQL>alter system switch logfile;

3) On Standby, verify the archived redo log files were applied:
SQL>select sequence#, applied from v$archived_log order by sequence#;

14. If you want the redo data to be applied as it is received without waiting for the current standby redo log file to be archived, enable the real-time apply.

To start real-time apply:
SQL> alter database recover managed standby database using current logfile disconnect;

15. To create multiple standby databases, repeat this procedure.

IV. Maintenance:

1. Check the alert log files of Primary and Standby databases frequently to monitor the database operations in a Data Guard environment.

2. Cleanup the archive logs on Primary and Standby servers.

I scheduled weekly Hot Whole database backup against my primary database that also backs up and delete the archived logs on Primary.

For the standby database, I run RMAN to backup and delete the archive logs once per week.
$rman target /@STAN;
RMAN>backup archivelog all delete input;

To delete the archivelog backup files on the standby server, I run the following once a month:
RMAN>delete backupset;

3. Password management
The password for the SYS user must be identical on every system for the redo data transmission to succeed. If you change the password for SYS on Primary database, you will have to update the password file for Standby database accordingly, otherwise the logs won’t be shipped to the standby server.

Refer to section II.2, step 2 to update/recreate password file for the Standby database.

Reference:
Oracle Data Guard Concepts and Administration 10g Release 2 (10.2), B14239-04

Also good Metalink for Switchover

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&doctype=REFERENCE&id=751600.1


Friday 5 August 2011

Unnamed data file with standby_file_management=manual

Tried to solve the following problem the other day on my standby database:

File #7 added to control file as 'UNNAMED00007' because
the parameter STANDBY_FILE_MANAGEMENT is set to MANUAL
The file should be manually created to continue.
MRP0: Background Media Recovery terminated with error 1274
Fri Apr 8 10:00:22 2005
Some recovered datafiles maybe left media fuzzy
Media recovery may continue but open resetlogs may fail

I had done this before, but - couldn't remember how to do it. The same problem can occur when you are recovering your database and a data file has been added to it which was not part of your backup set.

So, here is the solution! I came close to it while looking up the Oracle SQL reference:

On the standby, use "ALTER DATABASE CREATE DATAFILE" as shown in this example:

  1. RECOVER MANAGED STANDBY DATABASE CANCEL;
  2. alter system set standby_file_management = 'manual';
  3. select file#,name from v$datafile where name like '%UNNAMED%';
  4. ALTER DATABASE CREATE DATAFILE 'unnamed datafile' AS 'proper filename';
    An example would be:

    alter database create datafile
    '/u01/app/oracle/product/11.2.0.2/dbhome_1/dbs/UNNAMED00005'
    as '/u02/oradata/STDBY/data007.ora';

  5. alter database set standby_file_management = 'auto';

"Unnamed data file" in this conectxt is not the fully qualified file name, only UNNAMED00007.

Done! Enable recovery again:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE [USING CURRENT LOGFILE]
2 DISCONNECT FROM SESSION;

If your database is on ASM, then please use the following syntax instead in step 2:

SQL> alter database create datafile 'unnamed datafile' as '+TargetDiskGroupName' size xM;

It's important to specify the size, or the command will fail. An example should clarify this:

SQL> ALTER DATABASE CREATE DATAFILE
2 '/u01/app/oracle/product/11.2.0/dbhome_1/dbs/UNNAMED00005'
3 AS '+DATA' size 500M;

NFS Mount on Linux

You have:

192.168.0.137 = your server (Standby -- needs  Primay node's mount point)
192.168.0.126 = linux client (Primary)

*On the Primary server (192.168.0.126)*
1. Check nfs service == service nfs status
2. edit /etc/exports --make sure it contains just the line (with a blank line after it):
(Primary nfs directory) (standbyIP)(Permissions)
/u01/app/oracle/flash_recovery_area/FWLIVE 192.168.0.137(rw,sync,no_root_squash)
other e.g. cat /etc/exports
/data/backup frameworkprod(rw)
/logs/backup/dbbackup tpk-efr-005(rw,sync,no_root_squash)
/logs/backup 10.254.249.12(ro,sync,no_root_squash) 10.254.249.22(ro,sync,no_root_squash) tpk-efr-002(ro,no_root_squash)
/logs/archivelogs/FWLIVE tpk-efr-005(rw,sync,no_root_squash)
3. service nfs restart

*On the Standby Server (192.168.0.137)*
4. edit /etc/fstab and add/edit the line(you may need to restart server to effect /etc/fstab):

OraDRProd:/u01/app/oracle/flash_recovery_area/FWLIVE /u01/app/oracle/flash_recovery_area/FWLIVE nfs rw,bg,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp 0 0
Or
192.168.0.126:/u01/app/oracle/flash_recovery_area/FWLIVE /u01/app/oracle/flash_recovery_area/FWLIVE nfs rw,bg,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp 0 0


5. make sure /u01/app/oracle/flash_recovery_area/FWLIVE exists (mkdir –p /u01/app/oracle/flash_recovery_area/FWLIVE )
6. mount -t nfs OraDRProd:/u01/app/oracle/flash_recovery_area/FWLIVE /u01/app/oracle/flash_recovery_area/FWLIVE
or
mount -t nfs -o rw,bg,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp OraDRProd:/u01/app/oracle/flash_recovery_area/FWLIVE /u01/app/oracle/flash_recovery_area/FWLIVE


If it does mount up ok, then you can unmount and change /etc/fstab to mount the directory to wherever you wanted it in the first place. Dont forget to remove the /u01/app/oracle/flash_recovery_area/FWLIVE mount point with rmdir.

NFS Mount Options

You must mount NFS volumes used for storing database files with special mount options on the host where the database server is running. When mounting an NFS file system, Oracle recommends that you use the same mount point options that your NAS vendor used when certifying the device. Refer to your device documentation or contact your vendor for information about recommended mount-point options.
In general, most vendors recommend that you use the NFS mount options listed in the following table.
OptionRequirementDescription
hardMandatoryGenerate a hard mount of the NFS file system. If the connection to the server fails or is temporarily lost, connection attempts are made until the NAS device responds.
bgOptionalTry to connect in the background if connection fails.
tcpOptionalUse the TCP protocol rather than UDP. TCP is more reliable than UDP.
nfsvers=3OptionalUse NFS version 3. Oracle recommends that you use NFS version 3 where available, unless the performance of version 2 is higher.
suidOptionalAllow clients to run executables with SUID enabled. This option is required for Oracle software mount points.
rsizeMandatoryThe number of bytes used when reading from the NAS device. This value should be set to the maximum database block size supported by this platform. A value of 8192 is often recommended for NFS version 2 and 32768 is often recommended for NFS version 3.
wsizeMandatoryThe number of bytes used when writing to the NAS device. This value should be set to the maximum database block size supported by this platform. A value of 8192 is often recommended for NFS version 2 and 32768 is often recommended for NFS version 3.
nointr (orintr)OptionalDo not allow (or allow) keyboard interrupts to kill a process that is hung while waiting for a response on a hard-mounted file system.
Note: Different vendors have different recommendations about this option. Contact your vendor for advice.
actime=0 or noacMandatoryDisable attribute caching.
Note: You must specify this option for NFS file systems where you want to install the software. If you do not use this option, Oracle Universal Installer will not install the software in the directory that you specify.

The mandatory mount options comprise the minimum set of mount options that you must use while mounting the NFS volumes. These mount options are essential to protect the integrity of the data and to prevent any database corruption. Failure to use these mount options may result in the generation of file access errors. Refer to your operating system or NAS device documentation for more information about the specific options supported on your platform.
https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=359515.1

Thursday 4 August 2011

Changing IP and hostname in Application Server 10g

Because we moved our new OEM10gr2 server to the datacenter, the IP address and the hostname changed. When added to the datacenter I had to start all the services again, but first I had to change the opmn configuration. Instead of changing all the config files by hand, I noticed a shell script in the $OMS_HOME/chgip/scripts, called chgiphost.sh

This script changes all the config files and replaces the ip and hostname. It will ask you for the source and target ip/hostname...

Task : Run the chgiphost Command

Follow these steps for each middle-tier instance on your host. Be sure to complete the steps entirely for one middle-tier instance before you move on to the next.

  1. Log in to the host as the user that installed the middle-tier instance.

  2. Ensure your ORACLE_HOME environment variable is set to the middle-tier Oracle home.

  3. Run the following commands in the middle-tier Oracle home:

    cd ORACLE_HOME\chgip\scripts chgiphost.bat -mid
    ORACLE_HOME\opmn\bin\opmnctl stopall
    ORACLE_HOME\opmn\bin\opmnctl startall
--
Saved my lot of time..



Useful Scripts for Data Guard


For Data Guard environments, I find the following scripts very useful.
Script to Collect Data Guard Primary Site Diagnostic Information
Note:241374.1
Script to Collect Data Guard Physical Standby Diagnostic Information
Note:241438.1
Script to Collect Data Guard Logical Standby Table Information
Note:269954.1

CHECKLIST BEFORE SWITCHOVER OPERATION

CHECKLIST BEFORE SWITCHOVER OPERATION
=======================================
Please note

log_archive_dest_state_3 :(archive dest for standby database)
log_archive_dest_state_1 :(archive dest for primary database)

SELECT PROTECTION_MODE FROM V$DATABASE;
SQL> select name,value from v$parameter where name in ('log_archive_dest_1','log_archive_dest_state_1','log_archive_dest_3','log_archive_dest_state_3);

Make sure 'lgwr' and not 'arch' is mentioned here, otherwise new primary database will not open after switchover (if the protection_mode is Maximum availability).

-- on standby database--
SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PHYSICAL STANDBY

-- on primary --
SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PRIMARY

SQL>select name,value from v$parameter where name in ('log_archive_dest_1','log_archive_dest_state_1', 'log_archive_dest_3','log_archive_dest_state_3);
log_archive_dest_state_3
------------------
DEFER --->DO NOT PROCEED.
issue this command : sql> alter system set log_archive_dest_state_3='ENABLE' scope=both;


---- on primary ----
SQL> select distinct status from v$datafile;

STATUS
——-
ONLINE
SYSTEM

If any file(s) are in RECOVER status, DO NOT PROCEED with the switchover


-- on standby database--
SQL> select distinct open_mode from v$database;

OPEN_MODE
———-
MOUNTED --->PROCEED with switchover.



SQL> select distinct open_mode from v$database;

OPEN_MODE
———-
READ ONLY --->DO NOT PROCEED with switchover.

To confirm that logs are being shipped and more importantly the LAST LOG archived has been applied…

---- on primary ----
SQL> select max(SEQUENCE#) “LAST_LOG_GENERATED” FROM V$ARCHIVED_LOG WHERE THREAD# =1 and ARCHIVED=’YES’;

LAST_LOG_GENERATED
——————
3988

-- on standby database--
SQL> select max(SEQUENCE#) “LAST_LOG_APPLIED” FROM V$ARCHIVED_LOG WHERE THREAD# = 1 and APPLIED=’YES’;

LAST_LOG_APPLIED
—————-
3988

---> PROCEED

Thanks

Monday 1 August 2011

How much recovery is enough recovery

Long ago in one of the non-technical group discussions we had to decide ‘How much money is enough money?’ Obviously no answer is enough answer. Fortunately in Oracle, we know how much recovery is just sufficient to OPEN the DB.

During the recovery of the DB, when the archives are being applied one after other and another, is it possible to definitively confirm that the most recently applied archive was the last one required to be applied and it is safe thereafter to open the DB in RESETLOGS?

There should be several ways of identifying where to stop, and here are three that I can think of:

  1. Alert log – If the backup was hot, finding out the last occurrence of END BACKUP and its corresponding SEQ# should tell us where to stop. Usually we stop after applying the above identified SEQ# and the DB should then be good to OPEN RESETLOGS.
  2. RMAN log – If the backup was taken using RMAN, its log also shows the SEQ# where the backup ends. We can stop applying archives once we reach the max SEQ#
  3. Query the Data Dictionary – Query the FHSTA column in X$KCVFH table during recovery to know if the recovery was sufficient or does it need more archives.

As I said, there must be other ways too.
I prefer option (3) because the information is right there in the DB to be queried for, anytime.

Consider a typical scenario where a user, say Lucy, is recovering the DB from the hot backup of the source database. She has already applied 30 archives and swears that she hasn’t seen another hungrier database. She has no access to the alert log of the source database to check how many more archives are to be applied. After each archive log has been applied, she CANCELs the recovery and gets the below error:

Specify log: {=suggested | filename | AUTO | CANCEL}
CANCEL
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: ‘/u11/data/PRD/PRD_SYSTEM_01.DBF’

The better way to confirm if the recovery was sufficient or not is to query the data dictionary. The SQL is:

SELECT hxfil FILENUMBER, fhsta STATUS, fhscn SCN, fhrba_seq SEQUENCE FROM x$kcvfh;

SQL> select hxfil FILENUMBER, fhsta STATUS, fhscn SCN, fhrba_seq SEQUENCE from x$kcvfh


The output will look like above.

The STATUS (or fhsta) column has the value of 1 which means the datafile with FILENUMBER needs more recovery. This is the time to apply more archives.

The STATUS is 0 when the recovery is sufficient and it is safe to OPEN the database.

The fhsta can have one of the below values:

0 – DB is consistent. No more recovery required.
1 – DB needs more recovery. It’s time to apply more archives.
4 – DB is in a FUZZY state. Was the backup good?
8192 -
8196 -

I have come across 8192 and 8196 statuses too, but I don’t know their significance yet.

When the archives have been sufficiently applied, the fhsta column will be zero and looks like below:


At this stage, it is possible to cancel the recovery and bring up the DB in RESETLOGS.

Specify log: {=suggested | filename | AUTO | CANCEL}
CANCEL
Media recovery cancelled.
SQL>
SQL> alter database open resetlogs;
Database altered.

The table X$KCVFH is a fixed table in Oracle. It stores the file headers along with their statuses. The name is derived from:

K – Kernel layer
C – Cache layer
V – RecoVery component
FH - File Header

Hope this post was useful.