Thursday, March 6, 2008

ISSUE: 115102 upgrade Workflow Mailer Service

ISSUE:

After 115102 upgrade Workflow Mailer Service | Workflow Agent Listener Service | Workflow Document Web Services Service DO NOT STARTUP

Steps to reproduce:

1. configure WF mailer as per Note:268085.1
2. Activate WF Concurrent processing

Concurrent : Manager -> Administer
Workflow Agent Listener Service
Workflow Mailer Service
Workflow Document Web Services Service
Activate all of them


##
## Verify that workflow mailer and agent listener services running successfully
## without exceptions in the log files. [they are not]
##

dbtier.applmgr->ls -l $APPLCSF/$APPLLOG/FNDCPGSC*.txt
-rw-r--r-- 1 applmgr oaa 359 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199414.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199415.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199416.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199417.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199418.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199419.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:08 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199420.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:09 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199421.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:09 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199422.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:11 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199423.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:11 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199424.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:13 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199425.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:13 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199426.txt
-rw-r--r-- 1 applmgr oaa 1270 Mar 8 17:15 /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199427.txt

##
## Here's the error that was reported in one of the log files
## after I Activated WF% cm queues
##
dbtier.applmgr->more /u02/app/applmgr/common_cm/log/BHUP_dbtier/FNDCPGSC199415.txt
LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.Logger.Logger(String, int) : Logging to System.out until necessary parameters are retrieved for Logger to be properly started.
LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.initializeStateMachine() : BEGIN
LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN
LOG_ID_UNKNOWN : oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext() : BEGIN
Could not start Service Component Container because an unexpected RuntimeException or other Throwable occurred -> java.lang.No
ClassDefFoundError: oracle/apps/pass/profiles/Profiles
java.lang.NoClassDefFoundError: oracle/apps/pass/profiles/Profiles
at oracle.apps.fnd.profiles.ExtendedProfileStore.clear(Compiled Code)
at oracle.apps.fnd.security.SessionManager.setUser(Compiled Code)
at oracle.apps.fnd.security.SessionManager.setUser(Compiled Code)
at oracle.apps.fnd.wf.apps.ContextFactory.makeContext(Compiled Code)
at oracle.apps.fnd.wf.common.WorkflowContext.(Compiled Code)
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.getNewWorkflowContext(Compiled Code)
at oracle.apps.fnd.cp.gsc.SvcComponentContainer.start(Compiled Code)
at oracle.apps.fnd.cp.gsm.GSMSvcComponentContainer.main(Compiled Code)

dbtier.applmgr->echo $AF_CLASSPATH
/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/java/appsborg.zip:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/product/8.0.6_115102/forms60/java:/u02/app/applmgr/common/java


dbtier.applmgr->grep jar $APPL_TOP/admin/adjborg2.txt
/u02/app/applmgr/product/iAS_115102/jlib/jssl-1_1.jar
/u02/app/applmgr/product/iAS_115102/jlib/javax-ssl-1_1.jar
/u02/app/applmgr/product/iAS_115102/rdbms/jlib/aqapi.jar
/u02/app/applmgr/product/iAS_115102/rdbms/jlib/jmscommon.jar
/u02/app/applmgr/product/iAS_115102/Apache/xsu/xsu12.jar
/u02/app/applmgr/product/iAS_115102/Apache/xsu/oraclexmlsql.jar
/u02/app/applmgr/product/iAS_115102/rdbms/jlib/xsu12.jar
/u02/app/applmgr/product/iAS_115102/lib/http_client.jar
/u02/app/applmgr/product/iAS_115102/dm/lib/odmapi.jar
/u02/app/applmgr/product/iAS_115102/soap/webapps/soap/WEB-INF/lib/soap.jar


dbtier.applmgr->unzip -l $JAVA_TOP/appsborg2.zip | grep -i jms | wc -l
128

dbtier.applmgr->grep jar $APPL_TOP/admin/adjborg.txt
/u02/app/applmgr/product/8.0.6_115102/owm/jlib/jssl-1_1.jar
/u02/app/applmgr/product/8.0.6_115102/owm/jlib/javax-ssl-1_1.jar
/u02/app/applmgr/product/iAS_115102/dm/lib/odmapi.jar
dbtier.applmgr->

## this setting is coming from:
$APPL_TOP/admin/adovars.env

AF_CLASSPATH="/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/java/appsborg.zip:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/product/8.0.6_115102/forms60/java:/u02/app/applmgr/common/java"
export AF_CLASSPATH

show be:

Example of AF_CLASSPATH setting:
/local/java/jdk1.3.1/lib/dt.jar:/local/java/jdk1.3.1/lib/tools.jar:/slot05/appmgr/atgwfqacomn/java/appsborg2.zip:/slot05/appmgr/atgwfqaora/8.0.6/forms60/java:/slot05/appmgr/atgwfqacomn/java


dbtier.applmgr->grep classpath $APPL_TOP/admin/BHUP_dbtier.xml
# wrapper.classpath =
# wrapper.classpath =


/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/tools.jar:/u02/app/applmgr/common/java/appsborg.zip:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/product/8.0.6_115102/forms60/java:/u02/app/applmgr/common/java
/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/java/appsborg.zip:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/product/8.0.6_115102/forms60/java:/u02/app/applmgr/common/java

.:/u02/app/applmgr/common/java/jdbc111.zip:/u02/app/applmgr/common/java/xmlparserv2.zip:/u02/app/applmgr/common/java:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/common/util/jre/1.1.8/classes:/u02/app/applmgr/common/util/jre/1.1.8/lib:/u02/app/applmgr/common/util/jre/1.1.8/lib/classes.zip:/u02/app/applmgr/common/util/jre/1.1.8/lib/classes.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/java/3rdparty/RFJavaInt.zip:
/u02/app/applmgr/common/util/jre/1.1.8/bin/jre -mx128m -classpath .:/u02/app/applmgr/common/java/jdbc111.zip:/u02/app/applmgr/common/java/xmlparserv2.zip:/u02/app/applmgr/common/java:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/common/util/jre/1.1.8/classes:/u02/app/applmgr/common/util/jre/1.1.8/lib:/u02/app/applmgr/common/util/jre/1.1.8/lib/classes.zip:/u02/app/applmgr/common/util/jre/1.1.8/lib/classes.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/java/3rdparty/RFJavaInt.zip: -Dengine.LogPath=/u02/app/applmgr/common/admin/log/BHUP_dbtier -Dengine.TempDir=/u02/app/applmgr/common_cm/temp -Dengine.CommandPort=9300 -Dengine.AOLJ.config=/u02/app/applmgr/11.5/fnd/11.5.0/secure/dbtier_xupg.dbc -Dengine.ServerID=5000 -Ddebug=full -Dengine.LogLevel=9 -Dlog.ShowWarnings=false -Dengine.FaxEnabler=oracle.apps.jtf.fm.engine.rightfax.RfFaxEnablerImpl -Dengine.PrintEnabler=oracle.apps.jtf.fm.engine.rightfax.RfPrintEnablerImpl -Dfax.TempDir=/u02/app/applmgr/common/admin/log/BHUP_dbtier -Dprint.TempDir=/u02/app/applmgr/common/admin/log/BHUP_dbtier oracle.apps.jtf.fm.FulfillmentServer >> /u02/app/applmgr/common/admin/log/BHUP_dbtier/jtffmctl.txt

things that need to be changed:



SOLUTION
----------

##
## FIX AF_CLASSPATH
##

## EXISTING SETTING [WRONG] ##
dbtier.applmgr->grep s_adovar_afclasspath $APPL_TOP/admin/BHUP_dbtier.xml
/u02/app/applmgr/common/util/jre/1.1.8/lib/rt.jar:/u02/app/applmgr/common/util/jre/1.1.8/lib/i18n.jar:/u02/app/applmgr/common/java/appsborg.zip:/u02/app/applmgr/common/java/apps.zip:/u02/app/applmgr/product/8.0.6_115102/forms60/java:/u02/app/applmgr/common/java

## SHOULD BE ##
Example of AF_CLASSPATH setting:
/local/java/jdk1.3.1/lib/dt.jar:
/local/java/jdk1.3.1/lib/tools.jar:
/slot05/appmgr/atgwfqacomn/java/appsborg2.zip:
/slot05/appmgr/atgwfqaora/8.0.6/forms60/java:
/slot05/appmgr/atgwfqacomn/java

## OUR MAPPING BE ##
/usr/j2se/lib/dt.jar:
/usr/j2se/lib/tools.jar:
/u02/app/applmgr/common/java/appsborg2.zip:
/u02/app/applmgr/product/8.0.6_115102/forms60/java:
/u02/app/applmgr/common/java

## OUR MAPPING MT ##
/usr/j2se/lib/dt.jar:
/usr/j2se/lib/tools.jar:
/u01/app/applmgr/common/java/appsborg2.zip:
/u01/app/applmgr/product/8.0.6_115102/forms60/java:
/u01/app/applmgr/common/java


## FIXED SETTING ##
dbtier.applmgr->grep s_adovar_afclasspath $APPL_TOP/admin/BHUP_dbtier.xml
/usr/j2se/lib/dt.jar:/usr/j2se/lib/tools.jar:/u02/app/applmgr/common/java/appsborg2.zip:/u02/app/applmgr/product/8.0.6_115102/forms60/java:/u02/app/applmgr/common/java

midtier.applmgr->grep s_adovar_afclasspath $APPL_TOP/admin/BHUP_midtier.xml
/usr/j2se/lib/dt.jar:/usr/j2se/lib/tools.jar:/u01/app/applmgr/common/java/appsborg2.zip:/u01/app/applmgr/product/8.0.6_115102/forms60/java:/u01/app/applmgr/common/java


##
## FIX java exe definition
##

## EXISTING SETTING [WRONG] ##
dbtier.applmgr->grep -i AF_JRE_TOP $APPL_TOP/admin/BHUP_dbtier.xml
/u02/app/applmgr/common/util/jre/1.1.8

dbtier.applmgr->grep -i AFJVAPRG $APPL_TOP/admin/BHUP_dbtier.xml
/u02/app/applmgr/common/util/jre/1.1.8/bin/jre


## SHOULD BE ##

AF_JRE_TOP=/usr/j2se
export AF_JRE_TOP

AFJVAPRG=/usr/j2se/bin/java
export AFJVAPRG

## FIXED SETTING ##
dbtier.applmgr->grep -i AF_JRE_TOP $APPL_TOP/admin/BHUP_dbtier.xml
/usr/j2se

dbtier.applmgr->grep -i AFJVAPRG $APPL_TOP/admin/BHUP_dbtier.xml
/usr/j2se/bin/java

midtier.applmgr->grep -i AF_JRE_TOP $APPL_TOP/admin/BHUP_midtier.xml
/usr/j2se

midtier.applmgr->grep -i AFJVAPRG $APPL_TOP/admin/BHUP_midtier.xml
/usr/j2se/bin/java


##
## Re-run AutoConfig
##

## BE tier
$AD_TOP/bin/adconfig.sh contextfile=$APPL_TOP/admin/BHUP_dbtier.xml appspass=pass


## MT tier
$AD_TOP/bin/adconfig.sh contextfile=$APPL_TOP/admin/BHUP_midtier.xml appspass=pass



## MT tier
##
## comment out the following lines from jserv.properties and jserv.conf:
## refer to SR:780 11.5.10.2 AppsLocalLogin.jsp Internal Server Error
##

vi /u01/app/applmgr/product/iAS_115102/Apache/Jserv/etc/jserv.properties
#zones=mobile
#mobile.properties=/u01/app/applmgr/product/iAS_115102/Apache/Jserv/etc/mobile.properties

vi /u01/app/applmgr/product/iAS_115102/Apache/Jserv/etc/jserv.conf
#ApJServGroupMount /mobile balance://OACoreGroup/mobile



##
## Shutdown Services
##

## BE tier
/u02/app/applmgr/common/admin/scripts/BHUP_dbtier/adstpall.sh apps/pass

## MT tier
/u01/app/applmgr/common/admin/scripts/BHUP_midtier/adstpall.sh apps/pass



##
## Start Services
##

## BE tier
exit
su - applmgr
/u02/app/applmgr/common/admin/scripts/BHUP_dbtier/adstrtal.sh apps/pass


## MT tier
exit
su - applmgr
/u01/app/applmgr/common/admin/scripts/BHUP_midtier/adstrtal.sh apps/pass



##
## TEST call to external SMTP server
##
$AFJVAPRG -classpath $AF_CLASSPATH -Dprotocol=smtp -Dserver=mailhub.company.com \
-Dport=25 -Daccount=oracle.dba@company.com -Dconnect_timeout=120 \
oracle.apps.fnd.wf.mailer.Mailer

dbtier.applmgr->$AFJVAPRG -classpath $AF_CLASSPATH -Dprotocol=smtp -Dserver=mailhub.company.com \
-Dport=25 -Daccount=oracle.dba@company.com -Dconnect_timeout=120 \
oracle.apps.fnd.wf.mailer.Mailer
> -Dport=25 -Daccount=oracle.dba@company.com -Dconnect_timeout=120 \
> oracle.apps.fnd.wf.mailer.Mailer
Server mailhub.company.com at port 25 is reachable
Successfully connected to the SMTP account



REFERENCE:
Note:242941.1 How To Troubleshoot Java-based Workflow Notification Mailer In 11.5.9 and OWF.G
Note:278127.1 Workflow Notification Will Not Start After Cloning
Note:278163.1 Workflow Notification Mailer Will Not Start

No comments: