Sunday, May 23, 2010

Nid revert

NID-00111: Oracle error reported from target database while executing
begin dbms_backup_restore.nidprocessdf(:fno, :istemp, :skipped, :idchged, :nmchged); end;
ORA-01116: error in opening database file /gp_utyeb05/d001/temp01.dbf
ORA-27037: unable to obtain file status
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 3
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 6185
ORA-06512: at line 1



Change of database ID failed.
Must finish change or REVERT changes before attempting any database operation.
DBNEWID - Completed with errors.

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.1.0.7
This problem can occur on any platform.

Symptoms

NID failing due to missing tempfiles:
ORA-01116: error in opening database file /ora/u10/oradata/eu0289p/temp01.dbf
ORA-27037: unable to obtain file status
Linux Error: 2: No such file or directory
Additional information: 3
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 6164
ORA-06512: at line 1

Cause

Bug 5861994

Issue reported matches those of bug 5861994, thus justifies cause. This bug is fixed in Oracle 11.2.

Solution

To implement the solution, please execute the following steps:

1. While database is mounted, drop the tempfiles:
SQL> alter database tempfile 1 drop;

NOTE: this workaround may give error ora-19951 "cannot modify control file until DBNEWID is completed"

2. Recreate the controlfile:
- SQL> alter database backup controlfile to trace resetlogs;

NOTE: Depending on how much NID had accomplished before failing, it may be required to recreate the controlfile with "set database" to new database name.

- SQL> startup nomount;
- SQL> create controlfile.....
- SQL> alter database open resetlogs;

OR

You can also request the backport patch for this bug to prevent this error again.


nid TARGET=SYS/oracle REVERT=YES LOGFILE=$HOME/nid.log

Monday, May 17, 2010

AGENT RECONFIGURE AFTER DB UPGRADE.

EM Grid Agent targets.xml Modification After Database Upgrade

http://gavinsoorma.com/category/em-grid-control/

If you upgrade an Oracle database or start the database from a new Oracle Home, the following needs to be done for GC to recognise the change and for the database to be monitored via the GRID.

Shutdown the agent on the box -

cd AGENT_HOME/bin

./emctl stop agent.

Next make a copy and then edit the following file -

Location – AGENT_HOME/sysman/emd/targets.xml

Targets AGENT_TOKEN=”1982fef1f5fe5b9c64f3d1e2edd8c4e875c889df”
Target TYPE=”oracle_emd” NAME=”frmbwds06:3872″/
Target TYPE=”host” NAME=”frmbwds06″/
Target TYPE=”oracle_listener” NAME=”LISTENER_frmbwds06″
Property NAME=”ListenerOraDir” VALUE=”/u01/oracle/10.2/network/admin”/
Property NAME=”LsnrName” VALUE=”LISTENER”/
Property NAME=”Machine” VALUE=”10.254.200.234″/
Property NAME=”OracleHome” VALUE=”/u01/oracle/10.2″/
Property NAME=”Port” VALUE=”1521″/
/Target
Target TYPE=”oracle_database” NAME=”adra.bankwest.com”
Property NAME=”OracleHome” VALUE=”/u01/oracle/10.2“/
Property NAME=”UserName” VALUE=”dbsnmp”/
Property NAME=”MachineName” VALUE=”10.254.200.234″/
Property NAME=”Port” VALUE=”1521″/
Property NAME=”SID” VALUE=”adra”/
Property NAME=”ServiceName” VALUE=”adra.bankwest.com”/
Property NAME=”password” VALUE=”8b139f39a2544353″ ENCRYPTED=”TRUE”/
Property NAME=”Role” VALUE=”NORMAL”/
/Target
/Targets

Make changes to ORACLE_HOME – Notice above I have changed the OH for adra to point to the 10g HOME.

Restart the agent -

cd $AGENT_HOME/bin

./emctl start agent -

After a minute or so -

./emctl status agent – to make sure uploads are happening to the repository on prdu010.

oracle@frmbwds06 bin ./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 1593568
Parent Process ID : 2007106
Agent URL : https://frmbwds06:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-03-14 13:20:18
Started by user : unknown
Last Reload : 2009-03-14 13:20:18
Last successful upload : 2009-03-17 09:33:06
Last attempted upload : 2009-03-17 09:37:39
Total Megabytes of XML files uploaded so far : 102.26
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 30.16%
Last attempted heartbeat to OMS : 2009-03-17 10:43:55
Last successful heartbeat to OMS : 2009-03-17 10:12:39
—————————————————————
Agent is Running and Ready
oracle@frmbwds06 bin

EM Grid Control Agent Target Discovery

Sometimes , you may install the 10g OEM GRID agent before you actually create a new database or you may add a new database to an existing box.

This new database will have to be discovered by GRID CONTROL.

I created a database called test on a box which already had a Grid agent running on it.

This database has to be discovered by the agent and it’s details uploaded to the GRID.

Make sure the ORAINVENTORY location in /etc/oraInst.loc matches the location when you installed the AGENT.

Go to the new AGENT_HOME/bin and issue command agentca- d

test:/u01/ofsap/agent10g/bin> ./agentca -d
Stopping the agent using /u01/ofsap/agent10g/bin/emctl stop agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Stopping agent … stopped.
Running agentca using /u01/ofsap/agent10g/oui/bin/runConfig.sh
ORACLE_HOME=/u01/ofsap/agent10g ACTION=Configure MODE=Perform
RESPONSE_FILE=/u01/ofsap/agent10g/response_file RERUN=TRUE
INV_PTR_LOC=/etc/oraInst.loc
COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform – mode finished for action: Configure
Perform – mode finished for action: Configure

You can see the log file:
/u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log
test:/u01/ofsap/agent10g/bin>

Output of log -
/u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log

The action configuration is performing
——————————————————
The plug-in Agent Configuration Assistant is running

Performing free port detection on host=prdu024.bankwest.com
Performing targets discovery and agent configuration
Starting the agent
AgentPlugIn:agent configuration finished with status = true

The plug-in Agent Configuration Assistant has successfully been performed
——————————————————
The action configuration has successfully completed
###################################################

Now login to the database as sys, unlock the dbsnmp user and also change
the password of dbsnmp user – for example to temp.

Login to the GRID , and click on targets – choose the new box and then the
new database and configure the dbsmp user .

Once this is complete, the GRID screen will show both the agent and
database on the targets page.


Troubleshooting Grid Control Agent issues with startup

Sometimes the GRID CONTROL agent will not start because an old HTTP process is still running on the host – this usually happens if the agent has crashed for any reason.

Example -

Check status of agent.

oracle(DATABASE)@hostname:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent is Not Running

START AGENT -

oracle(DATABASE)@fhostname:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent …… failed.
Failed to start HTTP listener.
Consult the log files in: /u01/oracle/agent10g/sysman/log

To solve the problem.

CHECK IF PORT 3872 – PORT USED BY AGENT IS IN USE.

oracle(DATABASE)@hostname:netstat -an | grep 3872
tcp4 0 0 *.3872 *.* LISTEN

oracle(DATABASE)@hostname:ps -ef | grep emagent
oracle 864486 1 0 Jan 27 – 5:36 /u01/oracle/agent10g/perl/bin/perl /u01/oracle/agent10g/bin/emwd.pl agent /u01/oracle/agent10g/sysman/log/emagent.nohup
oracle 1593344 864486 0 Jan 27 – 97:38 /u01/oracle/agent10g/bin/emagent

KILL ALL EMAGENT PROCESSES STILL RUNNING -

oracle(DATABASE)@hostname:ps -ef | grep emagent | awk ‘ {print $2}’ | xargs kill -9

oracle(DATABASE)@hostname:ps -ef | grep emagent| grep -v grep

No emagent process running now.

cd $AGENT_HOME/bin

oracle(DATABASE)@hostname:pwd
/u01/oracle/agent10g/bin

START AGENT

oracle(DATABASE)@hostname:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

AGENT WILL START.


Troubleshooting Agent Startup and Upload

If the agent in GRID control in not uploading to the OMS -

cd $AGENT_HOME/bin

oracle(DATABASE)@prdu030:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ………. started.

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 856310
Parent Process ID : 897132
Agent URL : https://prdu030.bankwest.com:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-08-07 12:27:29
Started by user : oracle
Last Reload : 2009-08-07 12:27:29
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 5004
Size of XML files pending upload(MB) : 20.53
Available disk space on upload filesystem : 69.35%
Collection Status : Disabled by Upload Manager
Last successful heartbeat to OMS : 2009-08-07 12:27:39
—————————————————————
Agent is Running and Ready

Steps to fix this issue -

cd $AGENT_HOME/sysman/emd

rm lastupld.xml agntstmp.txt

cd $AGENT_HOME/sysman/emd/upload

rm *.*

cd $AGENT_HOME/sysman/emd/state

rm *.*

No need to remove storage directory under $AGENT_HOME/sysman/emd/state

cd $AGENT_HOME/bin

./emctl start agent

./emctl status agent

If the Last successful heartbeat to OMS is not shown as successful, then secure the agent .

cd $AGENT_HOME/bin

./emctl secure agent

Make sure the password prompted for matches the OMS password – usually the sysman password for the OMS URL -

Once agent is secured -

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent is Not Running

oracle(DATABASE)@prdu030:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ……… started.

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 217220
Parent Process ID : 864280
Agent URL : https://prdu030.bankwest.com:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-08-07 12:55:43
Started by user : oracle
Last Reload : 2009-08-07 12:55:43
Last successful upload : 2009-08-07 12:56:00
Total Megabytes of XML files uploaded so far : 2.00
Number of XML files pending upload : 6
Size of XML files pending upload(MB) : 3.96
Available disk space on upload filesystem : 69.31%
Last successful heartbeat to OMS : 2009-08-07 12:55:52—————————————————————
Agent is Running and Ready