How to change the number of OACORE JVMs in R12 with minimal disruption

Goal: To change the number of OACORE JVMs from 1 to 5 in R12 with almost no outage.

Highlights:

- No need to run autoconfig
- Minimal outage required (probaly < 1 min)
- Preserve the changes even if autoconfig is run at a later time.

Steps
1. On the Apache node, cd $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml
2. Backup the opmn.xml
3. Edit the following section of opmn.xml


< /process-type>
< process-type id="oacore" module-id="OC4J" status="enabled" working-dir="$ORACLE_HOME/j2ee/home">
< module-data>
< category id="start-parameters">
< data id="java-options" value="-server -verbose:gc –Xmx1024M –Xms1024M -XX:MaxPermSize=160M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/apps/local/CEPRD/inst/apps/CEPRD_c7-ipnt-53/ora/10.1.3/j2ee/oacore/config/jazn.xml">
< data id="java-bin" value="/apps/local/CEPRD/inst/apps/CEPRD_c7-ipnt-53/admin/scripts/java.sh">
< data id="oc4j-options" value="-out /apps/local/CEPRD/inst/apps/CEPRD_c7-ipnt-53/logs/ora/10.1.3/opmn/oacorestd.out -err /apps/local/CEPRD/inst/apps/CEPRD_c7-ipnt-53/logs/ora/10.1.3/opmn/oacorestd.err">
< /category>
< category id="stop-parameters">
< data id="java-options" value="-server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false">
< /category>
< category id="security-parameters">
< data id="wallet-file" value="file:/apps/local/CEPRD/inst/apps/CEPRD_c7-ipnt-53/certs/Apache">
< /category>
< /module-data>
< start retry="2" timeout="600">
< stop timeout="120">
< restart retry="2" timeout="720">
< port id="default-web-site" protocol="ajp" range="21585-21589">
< port id="rmi" range="20085-20089">
< port id="jms" range="23085-23089">
< process-set id="default_group" numprocs="1">
< /process-type>


In the above block, change the tag from < process-set id="default_group" numprocs="1">to < process-set id="default_group" numprocs="5">.

4. In the context file, change the following block from < oacore_nprocs oa_var="s_oacore_nprocs">1 to < oacore_nprocs oa_var="s_oacore_nprocs">5

5. Do NOT run autoconfig
6. adopmnctl.sh reload
7. Validate that 5 JVMs have been indeed started by adopmnctl.sh status

Risks and things to watch out for:

1. Make sure the port range for rmi and jms is 5.
2. Make sure those ports are not in use by using netstat command.
3. If the port range is not specifying 5 ports, assign 5 ports as per the port pool value.

For eg, if your port pool is 17, then rmi range will be 20085 - 20089 and jms range will be 23085- 23089.

4. Make similar change in context file so that your changes are not
5. You have a choice of customizing your template rather than directly changing your xml file as well.

R12 - How to enable Apache, OC4J and OPMN logging

Whenever raise SR with Oracle support for R12 application logging related issue, they will ask you to produce the logging details by enabling the Apache, OC4J and OPMN components and its services. Compared to 11i, Oracle Applications R12 introduces new easy way in order to enable logging for any of the components that are being used within it. The details about how to enable logging of these component you can also follow as explained in this post.

Oracle Configuration Home for Apache, OC4J & OPMN is
$ORA_CONFIG_HOME/10.1.3

Apache server is still being used and its configuration path is as following:
$ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf

but OC4J services, by default Oracle Applications R12 creates 3 OC4J instances:
1. OACore: runs OA Framework-based applications
2. Forms: runs Forms-base applications
3. OAFM (Oracle Apps Fusion Middleware): runs web services, mapviewer, ascontrol

Configuration files path for the above 3 OC4J instances is:
$ORA_CONFIG_HOME/10.1.3/j2ee/[oacore, forms, oafm]/config

Configuration file used to adjust log file path for the above 3 OC4J instances is:
$ORA_CONFIG_HOME/10.1.3/j2ee/[oacore, forms, oafm]/application-deployments/[oacore, forms, oafm]/orion-application.xml

Log files types:
Oracle Applications / E-Business Suite R12 can be configured to generate one of two types of logs files as following:

1. Plain text (default):
- No log rotation available
- Log files need to be manually removed
2. Oracle Diagnostics Logging (ODL) XML:
- Logs rotation based on filesize, directory size
3. R12 logs by default are configured to generate plain text

Log files locations:
All log files are written in one home ( $LOG_HOME ) which is by default the (logs) directory under ($INST_TOP ).

Please note that after enbaling the logging for any of the following services you have to bounce the services in order the changes to take effect.

Steps to enable Apache logging:
Apache Logging ( Plain Text )

1) Edit the file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf and set
LogLevel warn (s_apache_loglevel in context file)
2) Bounce the apache
3) Try to access the home URL or reproduce the issue.
4) Collect the following logfiles from $LOG_HOME/ora/10.1.3/Apache
access_log.
error_log.

Values that can be set to LogLevel variable in httpd.conf file
LogLevel = emerg,alert,crit,error,warn,notice,info,debug.

Apache Logging (ODL Logging)
1) Set the following parameters in file $ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
OraLogMode [oracle|odl|apache]
OraLogSeverity [message type]:[message level]

Message type: INTERNAL_ERROR, ERROR, WARNING, NOTIFICATION & TRACE
Message level: 1-32 (1 most severe, 32 least)
2) Bounce the apache
3) Try to access the home URL or reproduce the issue.
4) Collect the logfiles from $LOG_HOME/ora/10.1.3/Apache/oracle

Steps to enable OC4J logging:

Log file path is specified in the file
$ORA_CONFIG_HOME/10.1.3/j2ee/[oacore, forms, oafm]/application-deployments/[oacore, forms, oafm]/orion-application.xml

1. Open the file to set the following log level $ORA_CONFIG_HOME/10.1.3/j2ee/[oacore, forms, oafm]/config/j2ee-logging.xml
2. Come to the location located as
logger name=’oracle’ level=’NOTIFICATION:1‘ …..
3. Set the desired logging using following values
message type:message level
Message type: INTERNAL_ERROR, ERROR, WARNING, NOTIFICATION & TRACE
Message level: 1-32 (1 most severe, 32 least)
4. Locate the log file path from the file
$ORA_CONFIG_HOME/10.1.3/j2ee/[oacore, forms, oafm]/application-deployments/[oacore,forms,oafm]/orion-application.xml
(Will be identified with tag : log file path=… /log)
5. Bounce the OC4J instance and reproduce the issue
6. Collect the log files from the following locations .

Plain text
$LOG_HOME/ora/10.1.3/j2ee/[oacore, forms, oafm]/[oacore,forms,oafm]_[default_group_1]/application.log

ODL Log
$LOG_HOME/ora/10.1.3/j2ee/[oacore, forms, oafm]/[oacore,forms,oafm]_[default_group_1]/log.xml

Steps to enable OPMN logging:
There are 3 log files generated:
- (opmn.log): contains commands, exit status
- (opmn.dbg): contains debug log messages
- (opmn.out): contains stdout & stderr messages

1) Open the file $ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml to set the logging parameter
2) Logging is enabled per component (internal, ons or pm)
3) Levels that can be set are (component codes) as following:
none, fatal, error, warn, notify (written to .log)
debug1, debug2, debug3, debug4 (written to .dbg)
Ex :
opmnctl set target=log comp=warn
opmnctl set target=debug comp=debug1
4) Bounce opmn services and reproduce the issue
5) Collect the opmn log files generated in $LOG_HOME/ora/10.1.3/opmn
opmn.log , opmn.dbg and opmn.out

NOTE : Logfiles can be enabled for rotation using parameter s_opmn_log_rotation_size, s_opmn_log_rotation_time in opmn.xml

Components (Services) Status & Logs:
1. You can check the status of the services/components by passing 'status' parameter to any AD script in $ADMIN_SCRIPTS_HOME.

adapcctl.sh status

2. The various logs written while starting and stopping the services are created in ($LOG_HOME/appl/admin/log/), the logs files are:

TNS Listener Start/Stop log - adalnctl.txt
Fulfillment Server Start/Stop log - jtffmctl.txt
Oracle HTTP Server start/stop log - adapcctl.txt
Concurrent Managers and ICM start/stop log - adcmctl.txt
Forms OC4J start/stop log - adformsctl.txt
OACore OC4J start/stop log - adoacorectl.txtq
OAFM OC4J start/stop log - adoafmctl.txt
OPMN start/stop log - adopmnctl.txt

(adstpall.log) and (adstrtal.log) will record the messages while starting or stopping all services.