Agent Configuration Assistant Failed

Agent Configuration Assistant Failed

Today I hit the following error while trying to perform a standalone Grid Control Agent install:




There could be a number of reasons as to why the agent configuration assistant would fail while you are trying to install the Grid Control Agent. The commands which failed are listed in the $AGENT_HOME/cfgtoollogs/configToolsFailedCommands file.


[oracle@myserver agent10g] cat cfgtoollogs/configToolFailedCommands
# Copyright © 1999, 2009, Oracle. All rights reserved.
oracle.sysman.emcp.agent.AgentPlugIn


If you search metalink with “oracle.sysman.emcp.agent.AgentPlugIn” you will get a number of hits but the 2 most useful ones I have looked at are:

Subject:
Troubleshooting the error oracle.sysman.emcp.agent.AgentPlugIn raised by the Agent Configuration Assistant (AgentCA)
Doc ID:
740628.1


Subject:
Troubleshooting the 'oracle.sysman.emcp.agent.AgentPlugIn has failed' error
Doc ID:
734981.1


In my particular case, the agent failed because I entered an incorrect agent password during the install. Unfortunately there is no way to re-enter the password within OUI at this point so login to your server and follow the steps below:

1. cd $AGENT_HOME/bin
2. ./emctl stop agent
3. ./emctl unsecure agent
4. ./emctl secure agent
   
   • At this point you will be prompted for the agent registration password. If you input the incorrect password it will fail. Run this again until you enter the correct password.
   
   
5. ./emctl start agent

Now you should be able to login to your Grid Control console and see this target.

Setting EM Blackouts from GUI and Command Line

Setting EM Blackouts from the GUI and Command Line

Oracle Enterprise Manager provides you with the ability to monitor your environments and alert you once specified thresholds have been reached. Blackouts allow you to suspend monitoring so you do not get notified. This is useful for scheduled maintenance windows, such as cold backups, where the application and/or database may not be available.

As well, blackouts also suspends data collection for the given targets. This means that certain metrics such as availability will not be affected.

To create a blackout from the GUI, login to Enterprise Manager, navigate to the target you would like to blackout and at the bottom of the page under Related Links you will see a Blackouts link. You will be brought to the following page:





On the screen above you can view any blackouts that may currently be in effect as well as create new ones. To create a new blackout click on the create button.



On this page you can create a name for the blackout, with the default being “Blackout-”. You can also select the targets you wish to set the blackout for. As you can see from the screenshot I am going to set a blackout for an infrastructure application server (infra10g).

You can also provide a reason for the blackout by click on the Reason drop down list. Quite a few are available, from Server Bounce to Security Patch. Jobs can be disabled by deselecting the Run jobs during the blackout checkbox. If your applying a security patch then you may not want a scheduled backup to run as it will either error or cause problems.



The next screen allows you to select which components within the target will be blacked out. I selected a full blackout but you can select certain members if the outage will only affect specific components.



This screen allows you to schedule the blackout. It can either start immediately or you can choose a date along with a duration. Blackouts can be repeating as well, so you only have to create one for that monthly maintenance window for example.



The last screen provides a summary and once you have finished reviewing click on the Finish button.

You can also set blackouts from the command line, which is useful if you have some maintenance scripts which are not executed from Enterprise Managers job system. I’ve only tested this on linux but it should be the same for windows.

To set a blackout for a list of targets:

emctl start blackout [[:]]…. [-d Duration]

To set a blackout for all targets on a host:

emctl start blackout [-nodeLevel] [-d ]

-nodeLevel tells the agent to stop monitoring all targets on the server.

-d Duration allows you to set a duration in the format of [days] hh:mm. ex. 1 02:05 means the blackout will last for 1 day, 2 hours and 5 minutes.

For example, to use this in a script in which all targets will be unavailable you would start a blackout at the beginning of the script and stop it at the end:

cd $AGENT_HOME/bin
./emctl start blackout alltargets-myserver –nodeLevel

cd $AGENT_HOME/bin
./emctl stop blackout alltargets-myserver

Troubleshooting

In case you hit issues with blackouts take a look at the following notes:

Subject: Agent Blackout Initiated By Emctl Command Not Ending Doc ID: 559577.1

Subject: EMDiagkit Download and Master Index Doc ID: 421053.1

Subject: How to Troubleshoot the EM 10gR1 Blackout Sub-system Doc ID: 284024.1

Subject: Troubleshooting Blackouts in EM 10g Grid Control using EMDiag Kit Doc ID: 300671.1

They provide alot of information and solutions to different scenarios. I hit an issue over the weekend in which the blackout didn’t end properly. When I tried to stop it from the command line:

[oracle@myserver ~]$ /u01/app/oracle/product/agent10g/bin/emctl stop blackout alltargets-myserver
Oracle Enterprise Manager 10g Release 3 Grid Control 10.2.0.3.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Blackout stop Error : Blackout name alltargets-myserver is invalid

When trying to end the blackout via Enterprise Manager:

Error stopping the blackout on "infra10g": ORA-20710: Agent-side blackouts cannot be edited or stopped ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE", line 501 ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE", line 3262 ORA-06512: at "SYSMAN.MGMT_BLACKOUT", line 74 ORA-06512: at "SYSMAN.MGMT_BLACKOUT_UI", line 1167 ORA-06512: at line 1 .

To fix this problem I performed the following:
1. Shutdown the agent on the target server myserver
2. Removed the blackouts.xml file under $AGENT_HOME/sysman/emd
3. Used note 421053.1 to install the EMDiag kit
4. Logged in as sysman on the Enterprise Managers repository database and executed the following query:

select blackout_guid, blackout_name
from mgmt_blackouts;

BLACKOUT_GUID BLACKOUT_NAME
-------------------------------- ----------------------
30E2956CA329F0E59FBDF50951F2578E alltargets-myserver

5. Then executed:

exec mgmt_diag.KillBlackout(HEXTORAW(‘30E2956CA329F0E59FBDF50951F2578E’));

6. Restarted the agent on myserver and when I looked in Enterprise manager the blackout had cleared.

I have seen the command used above for some other scenarios but not this one specifically. Before executing any commands in your environment please test first.

10g Enterprise Manager Grid Control and its advisors

Life after 10g Enterprise Manager Grid Control and its advisors

Filed under: Oracle 10g New Features,Oracle 11g New Features,Oracle Best Practices,Oracle How To,Oracle Performance — H.Tonguç Yılmaz @ 7:35 am

In my opinion history will show that, life of an Oracle DBA is divided into two stages; before and after 10g Enterprise Manager Grid Control :) I still remember some old days after paying to Oracle every possible extra licensing you may pay, everybody were paying lots of extra money to some bunch of other companies’ tools just to manage Oracle effectively. But somehow, someday Oracle decided to end its customers’ pain; Grid Control, Automatic Storage Management(ASM) and SQL Developer are examples of this strategy, so those were now the “good” old days just for some companies I guess.

Everybody even presidents needs, loves advisors. It is the same with Oracle, after 10g and with 11g Oracle is continuing to invest on its advisories. Memory, SQL Access and Tuning, Segment advisors have important parts in our lives now. But with this post I planned to discuss on some less popular advisors(I believe) like Undo, Redo and Automatic Checkpoint Tuning Advisors with 10g and also some 11g delights like Partitioning and Data Recovery advisors.

The Undo Advisor assists in correctly sizing the undo tablespace. The Undo Advisor can also be used to set the low threshold value of the undo retention period for any Oracle Flashback requirements. And to access the Undo Advisor in the Database Control you may follow; Administration > Undo Management > Undo Advisor

Also after 10g Rollback Monitoring improved; when a transaction rolls back, the event is recorded in the view V$SESSION_LONGOPS, if the process takes more than six seconds. This view enables you to estimate when the monitored rollback process will finish.

SELECT TIME_REMAINING, SOFAR/TOTALWORK*100 PCT
FROM V$SESSION_LONGOPS
WHERE SID = 9
AND OPNAME =’Transaction Rollback’

Redo Log Tuning Advisory and Automatic Checkpoint Tuning are also new features introduced with 10G. The size of the redo log files can influence performance, because the behavior of the database writer and archiver processes depend on the redo log sizes. Generally, larger redo log files provide better performance, however it must balanced out with the expected recovery time, as rule of thumb switching logs at most once every fifteen-twenty minutes. Undersized log files increase checkpoint activity and increase CPU usage.

Checkpoint frequency is affected by several factors, including log file size and the setting of the FAST_START_MTTR_TARGET initialization parameter. If the FAST_START_MTTR_TARGET parameter is set to limit the instance recovery time, Oracle automatically tries to checkpoint as frequently as necessary. Under this condition, the size of the log files should be large enough to avoid additional checkpointing due to under sized log files.

The redo logfile sizing advisory is specified by column optimal_logfile_size of v$instance_recovery. This feature require setting the parameter “fast_start_mttr_target” for the advisory to take effect and populate the column optimal_logfile_size.

You can try this with different settings of “FAST_START_MTTR_TARGET” but -
* If FAST_START_MTTR_TARGET is set to such a small value that it is impossible to do a recovery within its time frame then target_mttr field of v$instance_recovery contains the effective MTTR target which is larger than FAST_START_MTTR_TARGET.
* If FAST_START_MTTR_TARGET is set to such a high value that even in worst case entire buffer cache is dirty) recovery would not take that long, then target_mrrt field contains the estimated mttr in worst-case scnerios.

Code Listing 117 – Redo Log Tuning Advisory Demo

And to access the Redo Logfile Size Advisor you may follow; Administration > Storage > Redo Log Groups > Sizing Advice

10g also supports automatic checkpoint tuning. But by default, this feature is not enabled, because FAST_START_MTTR_TARGET has a default value of 0. It is an advancement over the MTTR related parameter introduced in earlier versions. The idea is to use the periods of low I/O usage to advance checkpoints and therefore improve availability.

To enable automatic checkpoint tuning, unset FAST_START_MTTR_TARGET or set it to a nonzero value(This is measured in seconds). If you set this parameter to zero this feature will be disabled. When you enable fast-start checkpointing, remove or disable(set to 0) the following initialization parameters:
- LOG_CHECKPOINT_INTERVAL
- LOG_CHECKPOINT_TIMEOUT
- FAST_START_IO_TARGET
Enabling fast-start checkpointing can be done statically using the initialization files or dynamically using -

SQL> alter system set FAST_START_MTTR_TARGET=10;

Enabling fast-start checkpointing increases the average number of writes per transaction that DBWn issues for a given workload. However, if the system is not already near or at its maximum I/O capacity, then fast-start checkpointing has a negligible impact on performance. View V$MTTR_TARGET_ADVICE will give information on the additional I/O’s on various values of FAST_START_MTTR_TARGET. But if -
- FAST_START_MTTR_TARGET is set to a low value – Fast-start checkpointing is more aggressive. The average number of writes per transaction that DBWn issues is higher in order to keep the thread checkpoint sufficiently advanced to meet the requested MTTR.
- FAST_START_MTTR_TARGET is set to a high value – Fast-start checkpointing in less aggressive, and the average number of writes per transaction that DBWn issues is lower.
- FAST_START_MTTR_TARGET is unset – automatic checkpoint tuning is in effect. Average number of writes per transaction is reduced but at the same time MTTR is highest.

So what about future releases and Oracle’s advisory strategy, I advise :) you to check 11g Manageability presentation’s fifth slide for some hints. After 11g a new advisor called Partition Advisor which analyzes the data access patterns and suggests the decision to choose the partitioning scheme and the partitioning column(s) will be assisting us especially during development and testing stages. By the way before 10g if you drop a partitioned table Oracle removed all the partitions at once, so time and resource consuming process. After 10g Release 2 when you drop a partitioned table, partitions are dropped one by one, maybe you already felt the change :)

Data Recovery Advisor will be another friend which automatically diagnose data failures and recommend repairs. You can repair failures manually or request that they be repaired automatically. Of course Enterprise Manager includes interfaces for these new advisors with 11g.

Testing Information : the scripts mentioned are tested on Oracle Database 10g Enterprise Edition Release 10.2.0.3.0

Refences Used :
Oracle® Database Performance Tuning Guide 10g Release 2 (10.2)
Note:265831.1 – Automatic Checkpoint Tuning in 10g
Note:274264.1 – REDO LOGS SIZING ADVISORY
Note 180894.1 – “V$INSTANCE_RECOVERY”
Note 151062.1 – Init.ora Parameter “FAST_START_MTTR_TARGET” Reference Note
Note 30754.1 – Init.ora Parameter “LOG_CHECKPOINT_INTERVAL” Reference Note
Note 30755.1 – Init.ora Parameter “LOG_CHECKPOINT_TIMEOUT” Reference Note
Note 68931.1 – Init.ora Parameter “FAST_START_IO_TARGET” Reference Note
http://psoug.org/reference/dbms_advisor.html

Comments (2)

August 16, 2007

Using Automatic Database Diagnostic Monitor Manually

Filed under: Oracle 10g New Features,Oracle How To,Oracle Performance — H.Tonguç Yılmaz @ 1:36 pm

For Oracle after 10g the statistical data needed for accurate diagnosis of a problem is saved in the Automatic Workload Repository(AWR). The Automatic Database Diagnostic Monitor(ADDM) analyzes the AWR data on a regular basis, then locates the root causes of performance problems, provides recommendations for correcting any problems and identifies non-problem areas of the system. Because AWR is a repository of historical performance data, ADDM can be used to analyze performance issues after the event.

This decision tree Oracle developed in years under ADDM improves root-cause analysis and after this amount of research and development you pay additional cost for this feature even you are using Enterprise Edition.

The goal of the analysis is to reduce a single throughput metric called DB time. DB time is the cumulative time spent by the database server in processing user requests. It includes wait time and CPU time of all non-idle user sessions. DB time is displayed in the V$SESS_TIME_MODEL and V$SYS_TIME_MODEL views.

The background process Manageability Monitor Process(MMON) schedules the automatic running of the ADDM. You only need to make sure that the initialization parameters STATISTICS_LEVEL is set to TYPICAL or ALL, in order for the AWR to gather its cache of performance statistics. MMON schedules the ADDM to run every time the AWR collects its most recent snapshot. To view the ADDM’s findings:
* Use the OEM Database Control, the primary interface for diagnostic monitoring is the Oracle Enterprise Manager Database Control.
* Run the Oracle-provided script addmrpt.sql, very similar to statspack or awr reports.
* Run DBMS_ADVISOR APIs
Note: The DBMS_ADVISOR package requires the ADVISOR privilege.

Code Listing 112 – Manual ADDM Demo

ADDM analysis finding consists of the following four components:
* The definition of the problem itself
* The root cause of the performance problem
* Recommendation(s) to fix the problem
* The rationale for the proposed recommendations

Example ADDM Report

FINDING 1: 31% impact (7798 seconds)
————————————
SQL statements were not shared due to the usage of literals. This resulted in additional hard parses which were consuming significant database time.

RECOMMENDATION 1: Application Analysis, 31% benefit (7798 seconds)
ACTION: Investigate application logic for possible use of bind variables
instead of literals. Alternatively, you may set the parameter “cursor_sharing” to “force”.
RATIONALE: SQL statements with PLAN_HASH_VALUE 3106087033 were found to be using literals. Look in V$SQL for examples of such SQL statements.

In order to determining optimal I/O performance, ADDM uses the default value of the parameter DBIO_EXPECTED, which is 10 milliseconds.

SELECT PARAMETER_VALUE
FROM DBA_ADVISOR_DEF_PARAMETERS
WHERE ADVISOR_NAME=’ADDM’
AND PARAMETER_NAME=’DBIO_EXPECTED’

Today’s hardwares can be significantly different, so you may set the parameter value one time for all subsequent ADDM executions of course for better advices. Since this is not an init.ora parameter, the parameter can be changed again with DBMS_ADVISOR supplied package;

DBMS_ADVISOR.SET_DEFAULT_TASK_PARAMETER(‘ADDM’,'DBIO_EXPECTED’, 8000);

There are two important views you will be checking for the results of the ADDM analysis:

1. DBA_ADVISOR_RECOMMENDATIONS: This view displays the results of completed diagnostic tasks with recommendations for the problems identified in each run. The recommendations should be looked at in the order of the RANK column, as this relays the magnitude of the problem for the recommendation. The BENEFIT column gives the benefit to the system you can expect after the recommendation is carried out.

2. DBA_ADVISOR_FINDINGS: This view displays all the findings and symptoms that the diagnostic monitor encountered along with the specific recommendation.

Testing Information : the scripts mentioned are tested on Oracle Database 10g Enterprise Edition Release 10.2.0.3.0

Refences Used :
Automatic Database Diagnostic Monitor (ADDM) in Oracle Database 10g at oracle-base.com
Oracle® Database Performance Tuning Guide 10g Release 2 (10.2) – Chapter 6 Automatic Performance Diagnostics

AGENT RECONFIGURE AFTER DB UPGRADE.

EM Grid Agent targets.xml Modification After Database Upgrade

http://gavinsoorma.com/category/em-grid-control/

If you upgrade an Oracle database or start the database from a new Oracle Home, the following needs to be done for GC to recognise the change and for the database to be monitored via the GRID.

Shutdown the agent on the box -

cd AGENT_HOME/bin

./emctl stop agent.

Next make a copy and then edit the following file -

Location – AGENT_HOME/sysman/emd/targets.xml

Targets AGENT_TOKEN=”1982fef1f5fe5b9c64f3d1e2edd8c4e875c889df”
Target TYPE=”oracle_emd” NAME=”frmbwds06:3872″/
Target TYPE=”host” NAME=”frmbwds06″/
Target TYPE=”oracle_listener” NAME=”LISTENER_frmbwds06″
Property NAME=”ListenerOraDir” VALUE=”/u01/oracle/10.2/network/admin”/
Property NAME=”LsnrName” VALUE=”LISTENER”/
Property NAME=”Machine” VALUE=”10.254.200.234″/
Property NAME=”OracleHome” VALUE=”/u01/oracle/10.2″/
Property NAME=”Port” VALUE=”1521″/
/Target
Target TYPE=”oracle_database” NAME=”adra.bankwest.com”
Property NAME=”OracleHome” VALUE=”/u01/oracle/10.2“/
Property NAME=”UserName” VALUE=”dbsnmp”/
Property NAME=”MachineName” VALUE=”10.254.200.234″/
Property NAME=”Port” VALUE=”1521″/
Property NAME=”SID” VALUE=”adra”/
Property NAME=”ServiceName” VALUE=”adra.bankwest.com”/
Property NAME=”password” VALUE=”8b139f39a2544353″ ENCRYPTED=”TRUE”/
Property NAME=”Role” VALUE=”NORMAL”/
/Target
/Targets

Make changes to ORACLE_HOME – Notice above I have changed the OH for adra to point to the 10g HOME.

Restart the agent -

cd $AGENT_HOME/bin

./emctl start agent -

After a minute or so -

./emctl status agent – to make sure uploads are happening to the repository on prdu010.

oracle@frmbwds06 bin ./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 1593568
Parent Process ID : 2007106
Agent URL : https://frmbwds06:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-03-14 13:20:18
Started by user : unknown
Last Reload : 2009-03-14 13:20:18
Last successful upload : 2009-03-17 09:33:06
Last attempted upload : 2009-03-17 09:37:39
Total Megabytes of XML files uploaded so far : 102.26
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 30.16%
Last attempted heartbeat to OMS : 2009-03-17 10:43:55
Last successful heartbeat to OMS : 2009-03-17 10:12:39
—————————————————————
Agent is Running and Ready
oracle@frmbwds06 bin

EM Grid Control Agent Target Discovery

Sometimes , you may install the 10g OEM GRID agent before you actually create a new database or you may add a new database to an existing box.

This new database will have to be discovered by GRID CONTROL.

I created a database called test on a box which already had a Grid agent running on it.

This database has to be discovered by the agent and it’s details uploaded to the GRID.

Make sure the ORAINVENTORY location in /etc/oraInst.loc matches the location when you installed the AGENT.

Go to the new AGENT_HOME/bin and issue command agentca- d

test:/u01/ofsap/agent10g/bin> ./agentca -d
Stopping the agent using /u01/ofsap/agent10g/bin/emctl stop agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Stopping agent … stopped.
Running agentca using /u01/ofsap/agent10g/oui/bin/runConfig.sh
ORACLE_HOME=/u01/ofsap/agent10g ACTION=Configure MODE=Perform
RESPONSE_FILE=/u01/ofsap/agent10g/response_file RERUN=TRUE
INV_PTR_LOC=/etc/oraInst.loc
COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform – mode finished for action: Configure
Perform – mode finished for action: Configure

You can see the log file:
/u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log
test:/u01/ofsap/agent10g/bin>

Output of log -
/u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log

The action configuration is performing
——————————————————
The plug-in Agent Configuration Assistant is running

Performing free port detection on host=prdu024.bankwest.com
Performing targets discovery and agent configuration
Starting the agent
AgentPlugIn:agent configuration finished with status = true

The plug-in Agent Configuration Assistant has successfully been performed
——————————————————
The action configuration has successfully completed
###################################################

Now login to the database as sys, unlock the dbsnmp user and also change
the password of dbsnmp user – for example to temp.

Login to the GRID , and click on targets – choose the new box and then the
new database and configure the dbsmp user .

Once this is complete, the GRID screen will show both the agent and
database on the targets page.

Troubleshooting Grid Control Agent issues with startup

Sometimes the GRID CONTROL agent will not start because an old HTTP process is still running on the host – this usually happens if the agent has crashed for any reason.

Example -

Check status of agent.

oracle(DATABASE)@hostname:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent is Not Running

START AGENT -

oracle(DATABASE)@fhostname:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent …… failed.
Failed to start HTTP listener.
Consult the log files in: /u01/oracle/agent10g/sysman/log

To solve the problem.

CHECK IF PORT 3872 – PORT USED BY AGENT IS IN USE.

oracle(DATABASE)@hostname:netstat -an | grep 3872
tcp4 0 0 *.3872 *.* LISTEN

oracle(DATABASE)@hostname:ps -ef | grep emagent
oracle 864486 1 0 Jan 27 – 5:36 /u01/oracle/agent10g/perl/bin/perl /u01/oracle/agent10g/bin/emwd.pl agent /u01/oracle/agent10g/sysman/log/emagent.nohup
oracle 1593344 864486 0 Jan 27 – 97:38 /u01/oracle/agent10g/bin/emagent

KILL ALL EMAGENT PROCESSES STILL RUNNING -

oracle(DATABASE)@hostname:ps -ef | grep emagent | awk ‘ {print $2}’ | xargs kill -9

oracle(DATABASE)@hostname:ps -ef | grep emagent| grep -v grep

No emagent process running now.

cd $AGENT_HOME/bin

oracle(DATABASE)@hostname:pwd
/u01/oracle/agent10g/bin

START AGENT

oracle(DATABASE)@hostname:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

AGENT WILL START.

Troubleshooting Agent Startup and Upload

If the agent in GRID control in not uploading to the OMS -

cd $AGENT_HOME/bin

oracle(DATABASE)@prdu030:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ………. started.

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 856310
Parent Process ID : 897132
Agent URL : https://prdu030.bankwest.com:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-08-07 12:27:29
Started by user : oracle
Last Reload : 2009-08-07 12:27:29
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 5004
Size of XML files pending upload(MB) : 20.53
Available disk space on upload filesystem : 69.35%
Collection Status : Disabled by Upload Manager
Last successful heartbeat to OMS : 2009-08-07 12:27:39
—————————————————————
Agent is Running and Ready

Steps to fix this issue -

cd $AGENT_HOME/sysman/emd

rm lastupld.xml agntstmp.txt

cd $AGENT_HOME/sysman/emd/upload

rm *.*

cd $AGENT_HOME/sysman/emd/state

rm *.*

No need to remove storage directory under $AGENT_HOME/sysman/emd/state

cd $AGENT_HOME/bin

./emctl start agent

./emctl status agent

If the Last successful heartbeat to OMS is not shown as successful, then secure the agent .

cd $AGENT_HOME/bin

./emctl secure agent

Make sure the password prompted for matches the OMS password – usually the sysman password for the OMS URL -

Once agent is secured -

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent is Not Running

oracle(DATABASE)@prdu030:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ……… started.

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 217220
Parent Process ID : 864280
Agent URL : https://prdu030.bankwest.com:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-08-07 12:55:43
Started by user : oracle
Last Reload : 2009-08-07 12:55:43
Last successful upload : 2009-08-07 12:56:00
Total Megabytes of XML files uploaded so far : 2.00
Number of XML files pending upload : 6
Size of XML files pending upload(MB) : 3.96
Available disk space on upload filesystem : 69.31%
Last successful heartbeat to OMS : 2009-08-07 12:55:52—————————————————————
Agent is Running and Ready

Troubleshooting Grid Control/ Grid Log files

Troubleshooting Grid Control/ Grid Log files
http://oracledbasupport.co.uk

When troubleshooting problems in the Grid Control framework, it is important to remember that Grid Control is a J2EE application deployed to the Oracle Application Server 10g J2EE and Web Cache installation using an Oracle database as a repository.

When faced with a problem, first localize the problem to the affected component

1. Troubleshooting the OMR
2. Troubleshooting the OMS
3. Troubleshooting the OMA










1. Troubleshooting the OMR ( Repository/Database )

Connectivity: Access to the OMR is through the database listener. Please locate listener log file at ORACLE_HOME/network/log. The log file contains record of every connection and connection request received by the listener. Listener errors are recorded in the log file with a TNS-xxxxx format. If necessary, enable tracing for listener to record more diagnostic information. Trace files for the listener can be found in ORACLE_HOME/network/trace

Availability: If OMR database is unavailable for some other reason, check database alert log and trace files to determine the root cause.

Space usage: Space problems may occur in the OMR database if the OMR tablespaces are unable to accept new information due to lack of free space in the data files.

Performance: OMR database performance problems will normally trigger Grid Control alerts as metric threshold values are crossed.

2. Troubleshooting the OMS ( Management Service )

OMS log files are produced for – 1> Oracle Web Cache, 2> Oracle HTTP Server, 3> Oracle Application Server Containers for J2EE, and 4> the Oracle Process Monitor and Notification subcomponents.

1> Oracle Web Cache Log files may be found in ORACLE_HOME/webcache/logs. There are two key log files for Web Cache: the access_log (which records every connection to the Web Cache) and the event_log (which contains data about Web Cache availability and errors).

2> The Oracle HTTP Server provides the access_log and error_log in ORACLE_HOME/Apache/Apache/logs. As with Web Cache, the access_log can grow very large and is therefore “rotated” every 12 hours. Older logs are preserved in ORACLE_HOME/Apache/Apache/logs.

3> Oracle Application Server Containers for J2EE for generated log files for Grid Control may be found in ORACLE_HOME/j2ee/OC4J_EM/log/OC4J_EM_default_island_1. OC4J generates several log files that provide diagnostic information, including:

default-web-access.log: Contains information about each request received by the component. Information includes the IP address of the requester, date and time of the request, the URL that was specified in the request, and the result code. All requests should come from the OHS; all result codes should indicate success (200). This information is valuable when troubleshooting connection difficulties between the OHS and OC4J.

em-application.log: Contains information about all events, errors, and exceptions associated with the EM application. This is excellent information for troubleshooting Java errors.

global-application.log: Contains information about events, errors, and exceptions relating to the OC4J JVM that are not specific to the EM application. This is also a good source of information for troubleshooting Java errors.

server.log: Includes availability information for the OC4J_EM component, including start and stop times.

4> The Oracle Process Monitor and Notification (OPMN) system provides logs for each of the OMS components in ORACLE_HOME/opmn/logs. Key log files include:

HTTP_Server: In cases where the OHS will not start, this log file often contains pertinent error messages that can help diagnose the problem.

OC4J~OC4J_EM~default_island~1: Contains any errors received while starting the OC4J_EM component. This can be helpful in troubleshooting Java errors and global configuration problems.

3. Troubleshooting the OMA ( Management Agent )

Connectivity between the OMA and OMS: When a single management agent is unable to connect to the OMS, the problem will normally be found on the OMA’s server. If multiple agents are unable to connect, the problem may lie with the OMS or underlying network. Check AGENT_HOME/sysman/config/emd.properties and verify that the repository URL is correct.

Ensure you can ping the host identified as the repository URL. attempt to telnet to the OMS host.

Upload throughput as the OMA reports information on metric targets through the OMS to the OMR: OMA logs may be found in AGENT_HOME/sysman/log. OMA upload errors are recorded in AGENT_HOME/sysman/log/emdctl.trc.

Target discovery as new targets are added to a server: Use emctl config listtargets or check AGENT_HOME/sysman/emd/targets.xml to determine which targets are monitored by the agent. Remember to make a backup copy of the targets.xml file prior to any modifications. Errors with target discovery will be reported in AGENT_HOME/sysman/log/agentca.log.

Oracle Enterprise Manager 10g Grid Control comprises three major components:
Oracle Management Repository (OMR)
Oracle Management Service (OMS)
Oracle Management Agent (OMA)

•

Oracle Enterprise Manager 10g Grid Control comprises three major components:

• Oracle Management Repository (OMR)
• Oracle Management Service (OMS)
• Oracle Management Agent (OMA)

Key configuration files for the OMA include:

–./sysman/emd/targets.xml

–./sysman/config/emd.properties

–./sysman/config/emagentlogging.properties •

To start the Grid Control framework, do the following:
1. Start the OMR database listener
2. Start the OMR database
3. Start all OMSs
4. Start the OMA on the OMS/OMR server
5. Start the OMA on managed servers

To stop the Grid Control framework:
1. Stop the OMA on managed servers (optional).
2. Stop the OMA on the OMS/OMR server.
3. Stop all OMSs.
4. Stop the OMR database.
5. Stop the OMR database listener.

Oracle Management Agent : Must be installed on each managed host, Must be in its own ORACLE_HOME , Communicates with the OMS via HTTP or HTTPS

Oracle Management Service : Includes member components:, Oracle HTTP Server (OHS), Oracle Application Server Containers for J2EE (OC4J), OracleAS Web Cache, Is a J2EE application deployed on Oracle Application Server 10g, Connects to the OMR by using Java Database Connectivity

Oracle Management Repository : Resides in an Oracle database, Includes schema objects belonging to SYSMAN, Can be installed in a preexisting database, Can be configured to contain other management data, including:, Oracle Application Server infrastructure database, Oracle Recovery Manager catalog

Oracle Process Monitor and Notification Control utility $ opmnctl startall $ opmnctl stopall $ opmnctl startproc ias-component=OC4J $ opmnctl stopproc process-type=OC4J_EM $ opmnctl status -l

Enterprise Manager Control Utility (emctl) $ emctl start oms $ emctl stop oms $ emctl status oms $ emctl start iasconsole $ emctl stop iasconsole

Distributed Configuration Manager Control (dcmctl) $ dcmctl start $ dcmctl start –ct WebCache $ dcmctl stop $ dcmctl getstate $ dcmctl listcomponents

Example Output of Commands

$ /opt/oracle/product/oms10g/opmn/bin/opmnctl status -l

Processes in Instance: EnterpriseManager0.test
-------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component      | process-type       |     pid | status   |        uid |  memused |    uptime | ports
-------------------+--------------------+---------+----------+------------+----------+-----------+------
DSA                | DSA                |     N/A | Down     |        N/A |      N/A |       N/A | N/A
LogLoader          | logloaderd         |     N/A | Down     |        N/A |      N/A |       N/A | N/A
HTTP_Server        | HTTP_Server        |     749 | Alive    | 1325924656 |   194208 | 123:54:07 | http1:7778,http2:7200,https1:4444,https2:1159,http3:4889
dcm-daemon         | dcm-daemon         |     629 | Alive    | 1325924655 |    24956 | 123:54:39 | N/A
OC4J               | home               |   23906 | Alive    | 1325924658 |    33148 | 121:24:21 | ajp:12502,rmi:12402,jms:12602
OC4J               | OC4J_EMPROV        |   23907 | Alive    | 1325924659 |    57724 | 121:24:21 | ajp:12503,rmi:12403,jms:12603
OC4J               | OC4J_EM            |   12150 | Alive    | 1325924667 |   242044 | 1193:01:~ | ajp:12501,rmi:12401,jms:12601
WebCache           | WebCache           |   23908 | Alive    | 1325924660 |   106924 | 121:24:21 | http:7777,invalidation:9401,statistics:9402
WebCache           | WebCacheAdmin      |   23909 | Alive    | 1325924661 |    15652 | 121:24:21 | administration:9400

$ /opt/oracle/product/oms10g/opmn/bin/opmnctl status

Processes in Instance: EnterpriseManager0.test
-------------------+--------------------+---------+---------
ias-component      | process-type       |     pid | status
-------------------+--------------------+---------+---------
DSA                | DSA                |     N/A | Down
LogLoader          | logloaderd         |     N/A | Down
HTTP_Server        | HTTP_Server        |     749 | Alive
dcm-daemon         | dcm-daemon         |     629 | Alive
OC4J               | home               |   23906 | Alive
OC4J               | OC4J_EMPROV        |   23907 | Alive
OC4J               | OC4J_EM            |   12150 | Alive
WebCache           | WebCache           |   23908 | Alive
WebCache           | WebCacheAdmin      |   23909 | Alive

To Start and stop individual ias components use : opmnctl startproc ias-component=OC4J (See IAS-Component Above)

To Start and stop individual processes use : opmnctl startproc process-type=OC4J_EMPROV (See Process-Type Above)

$ /opt/oracle/product/oms10g/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...
================================================================================
opmn id=test:6201
5 of 6 processes stopped. ias-instance id=EnterpriseManager0.test
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------------------
ias-component/process-type/process-set:
OC4J/OC4J_EM/default_island

Error
--> Process (pid=12150)
time out while waiting for a managed process to stop
Log:
/opt/oracle/product/oms10g/opmn/logs/OC4J~OC4J_EM~default_island~1

opmnctl: graceful stop of processes failed, trying forceful shutdown...

$ /opt/oracle/product/oms10g/opmn/bin/opmnctl status
Unable to connect to opmn.
Opmn may not be up.

EM Agent Deploy Log Files

Posted By Sagar Patil

Connectivity Log File	Location
prereq.log	$OMS_HOME/sysman/prov/agentpush//prereqs/local
prereq.out	$OMS_HOME/sysman/prov/agentpush//prereqs/local
prereq.err	$OMS_HOME/sysman/prov/agentpush//prereqs/local

Prerequisite Log File	Location
prereq.log	$OMS_HOME/sysman/prov/agentpush//prereqs/
prereq.out	$OMS_HOME/sysman/prov/agentpush//prereqs/
prereq.err	$OMS_HOME/sysman/prov/agentpush//prereqs/

Logs	Location
EMAgentPush.log	/sysman/prov/agentpush/logs/ : :Agent Deploy application logs.
remoteInterfaces.log	/sysman/prov/agentpush/logs/ :Logs of the remote interfaces layer
install.log/.err	/sysman/prov/agentpush//logs// :Log or error of the new agent installation or new cluster agent installation
upgrade.log/.err	/sysman/prov/agentpush//logs// : Log or error of the upgrade operation using Agent Deploy
nfsinstall.log/err	/sysman/prov/agentpush//logs//: Log or error of the agent installation using the Shared Agent Home option in Agent Deploy.
clusterUpgrade.log/err	/sysman/prov/agentpush//logs//:Log or error of the cluster upgrade operation using Agent Deploy
sharedClusterUpgradeConfig.log/err	/sysman/prov/agentpush//logs//:Log or error of the config operation in case of upgrade on a shared cluster
config.log/err	/sysman/prov/agentpush//logs//:Log or error of the configuration of shared cluster in case of an agent installation on a shared cluster.
preinstallscript.log/.err	/sysman/prov/agentpush//logs//:Log/error of the running of preinstallation script, if specified Clone Agent Configuration If you have a working EM grid console configuration, you can TAR a client configuration and clone it at other nodes. Tar the working agent at node $ cd /opt/oracle/product $ ls -l total 8 drwxr-xr-x 65 oracle9 oinstall 2048 Oct 10 2006 9.2.0 drwxrwx--- 3 oracle9 oinstall 96 Nov 19 13:29 agent_10g $ tar –cvf agent.tar agent_10g Set the ORACLE_HOME to /opt/oracle/product/agent10g< ?xml:namespace prefix = o /> $export ORACLE_HOME=/opt/oracle/product/agent10g 4. Run the installer as shown below - $cd $ORACLE_HOME/oui/bin Or $cd /opt/oracle/product/agent10g/oui/bin $./runInstaller -clone -forceClone ORACLE_HOME=$ORACLE_HOME ORACLE_HOME_NAME=agent10g -noconfig –silent 5. Run the root.sh as root user /opt/oracle/product/agent10g/agent10g/root.sh 6. Amend the REPOSITORY_URL and emdWalletSrcUrl parameters in the /sysman/config/emp.properties configuration file to make the agent point to right grid server 7. Run the agent configuration assistant $cd $ORACLE_HOME/bin or $cd /opt/oracle/product/agent10g/bin $./agentca -f 8. Secure the agent $./emctl secure agent 9. Start the agent, if not already $./emctl start agent