Showing posts with label Oracle EBS. Show all posts
Showing posts with label Oracle EBS. Show all posts

Thursday, April 2, 2015

EBS 12.2 adop apply_mode=downtime: Patching Downtime Is Back



EBS 12.2 -- adop apply-mode=downtime -- Patching downtime is back!

We are happy to find out that downtime for the EBS patches have come back :)
Actually it is optional to apply a patch with downtime, but it is more fast and it requires less system resources  as it seems.. On the other hand; with downtime mode; you 'll have an increased system downtime..
As you know, in EBS 12.2; we don’t have a maintenance mode anymore .. That's why; to apply a patch with downtime , we only need to stop our application tier services..

What we need to do is ;
1)Source our run environment.
2)Stop our application services
3)adop phase=apply apply_mode=downtime patches=PATCH_NUMBER

With apply_mode=downtime ; adop directly applies the patch .. No patching cycles...

But we need to face the facts;
As Oracle states;
Release 12.2 patches are not normally tested in downtime mode.
Downtime mode is only supported for production use where explicitly documented, or when directed by Oracle Support or Development.
So unless Oracle supports say that "This patch can be applied using apply_mode=downtime";  you need to solve the problems you may face during the adop downtime patching , by yourself..

There are some examples for these kind of situations;
Is it supported to apply 17050005 R12.HR_PF.C.delta.4 in downtime mode ? (Doc ID 1916385.1)   ->  If customer is using AD-TXK Delta 5 or above, then it is supported to apply the patch R12.HR_PF.C.delta.4 (17050005) in downtime mode.
Is It Possible To Apply ALL Patches During An Upgrade From 11i To 12.2.4 With Apply_mode=downtime (Doc ID 1918842.1) -> Can all post-12.2.4 patches can be applied in downtime mode? Yes, all post-12.2.4 patches can be applied in downtime mode as long as the Applications Tier processes have not been started (first time after upgrade).  Can downtime mode be used to apply patches once the upgrade is complete?  No.  Once the system is open for users, all subsequent patches must be applied on-line unless otherwise stated in the Readme or corresponding Note
Here another important doc: Oracle E-Business Suite Release 12.2.4 Readme (Doc ID 1617458.1) --> adop phase=apply apply_mode=downtime patches=17919161
Okay.. In my opinion;  Oracle suggests and supports this downtime apply mode for the fresh installations, upgrades and  for some specific product upgrades at the moment;
When the system is started to be used by the users ; in other words when we put data into it/ when we start the business processes,  we need to apply our patches online unless otherwise started in the Readme or corresponding note.

To be able to use downtime mode; you need to upgrade AD-TXK Delta 5, or you need to upgrade to 12.2.4.. Note that you need  to upgrade your AD-TXK DELTA 5 prior to 12.2.4 upgrade anyways :) When you apply AD TXK Delta 5 as a prereq for 12.2.4, you may even use downtime mode during the application of your 12.2.4  upgrade patch.

For  AD-TXK Delta 5 upgrade >
Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 (Doc ID 1617461.1)

For 12.2.4 Upgrade >
Oracle E-Business Suite Release 12.2.4 Readme (Doc ID 1617458.1)




EBS 12.2 Essential Bundle Fixes for AD Delta 5 and TXK Delta 5 (Doc ID 1934471.1)

EBS 12.2 Essential Bundle Fixes for AD Delta 5 and TXK Delta 5 (Doc ID 1934471.1)

if any of the below features are interesting to your deployment then please review the doc and apply the essential bundle patches on 12.2.5 environment. Hope this helps to stabilize your environment.

Section 4: Features and Fixes in the Current Code level
The bundle fixes include implementation of the following AD and TXK features and fixes.

4.1: AD Features and Fixes

  • The database connection module has been enhanced such that the former multiple connections during adop execution have been reduced to only two connections for all embedded SQL actions.
  • Concurrency issues during multi-node configuration have been fixed.
  • Redundancy issues have been addressed:
    • When calling validation on all nodes.
    • Unnecessary calls to the TXK API, have been removed from the cleanup phase.
    • Time-consuming database actions have been centralized, instead of being performed on all nodes.
  • Multinode logic has been changed to depend on a new table, adop_valid_nodes, instead of fnd_nodes.
  • An issue where AD Admin and AD Splice actions were not synchronized on shared slave nodes has been fixed.
  • Reporting capabilities have been improved for:
    • Abandon nodes and failed nodes.
    • Uncovered objects not being displayed after actualize_all in adopreports.
    • Out of sync nodes during fs_clone and abort.
  • Cutover improvements:
    • Restartability of cutover.
    • An obsoleted materialized view has been removed from processing during cutover.
  • xdfgen.pl has been enhanced to support execution against Oracle RAC databases where ipscan is enabled.
  • Support for valid comma-separated adop phases has been provided.
  • Several database-related performance issues have been fixed.
  • Improvements have been made in supporting hybrid, DMZ, non-shared, and shared configurations.
  • The adop utility has been enhanced to support host name containing the domain name.

4.2: TXK New Features and Fixes

  • Enhancements have been made to the provisioning tools used in multi-tier environments to perform operations such as adding or deleting nodes and adding or deleting managed servers.
  • An enhancement has been made to allow customization of the s_webport and s_http_listen_parameter context variables when adding a new node.
  • Performance improvements have been made for cloning application tier nodes, particularly in the pre-clone and post-clone phases.
  • Fixes related to cloning support for Oracle 12c Database have been provided.
  • Performance improvements have been made for managing application tier services, including implementation of the Managed Server Independence Mode feature (-msimode parameter to adstrtal.sh) to allow application tier services to be started or stopped without the WebLogic Administration Server running.
  • On a multi-node application tier system configuration, remote connectivity is no longer required for packaging the Oracle E-Business Suite WebLogic Server domain.
  • JVM heap size (-Xms and -Xmx) has been increased to 1 GB for the WebLogic Administration Server and all managed servers.


Cutover Phase Scenario - R12.2 Online Patching Cycle

Scenario

You have been asked to apply a patch and you completed the prepare phase and apply phase. Now customer has given a downtime for the cutover. You have run the finalize phase and the instance is ready for cutover.

  • How do we do it?
  • Pre-Health Check
  • Blackout
  • Cutover
  •  Unset Blackout
  • Post-Health Check
$ adop -status

Enter the APPS username: apps
Enter the APPS password:


Current Patching Session ID: 9

Node Name     Node Type     Phase      Status        Started                   Finished                  Elapsed
--------------- --------------- ----------- --------------- ------------------------------ ------------------------------ ------------
testserver01  master        APPLY      ACTIVE        17-NOV-13 02:49:24 -06:00        17-NOV-13 02:52:46 -06:00    0:03:22
                           PREPARE     COMPLETED          17-NOV-13 12:46:19 -06:00        17-NOV-13 01:00:28 -06:00        0:14:09
                           CUTOVER     NOT STARTED
                           CLEANUP     NOT STARTED




File System Synchronization Used in this Patching Cycle: None

For more information, run ADOP Status Report by using -detail option
Generating ADOP Status Report at location: /inst122i/applmgr/fs_ne/EBSapps/log/status_20131117_031159/adzdshowstatus.out
Please wait...
Done...!

adop exiting with status = 0 (Success)

$ adop phase=cutover

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

 Please wait. Validating credentials...


RUN file system context file: /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml
PATCH file system context file: /inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml

************* Start of  session *************
 version: 12.2.0
 started at: Sun Nov 17 2013 03:12:38

APPL_TOP is set to /inst122i/applmgr/fs2/EBSapps/appl
  Using ADOP Session ID from currently incomplete patching cycle
[START 2013/11/17 03:13:12] adzdoptl.pl run
  ADOP Session ID: 9
  Phase: cutover
  Log file: /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/adop_20131117_031215.log
  [EVENT]     Executing pre-cutover call-out script.
  [START 2013/11/17 03:13:33] cutover phase
    [EVENT]     [START 2013/11/17 03:13:43] Performing Cutover Phase steps
      [EVENT]     [START 2013/11/17 03:13:59] ICM Shutdown

You are running adcmctl.sh version 120.19.12020000.3

Shutting down concurrent managers for INST122I ...
ORACLE Password:
Submitted request 476567 for CONCURRENT FND SHUTDOWN


adcmctl.sh: exiting with status 0


adcmctl.sh: check the logfile /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/logs/appl/admin/log/adcmctl.txt for more information ... 

      [EVENT]     [END   2013/11/17 03:14:04] ICM Shutdown
      [EVENT]     [START 2013/11/17 03:14:10] Waiting for ICM to go down
      [EVENT]     [END   2013/11/17 03:17:45] Waiting for ICM to go down
      [EVENT]     [START 2013/11/17 03:17:50] Bringing down Middle Tier Services
        [EVENT]     Calling: /inst122i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl

Program :  started @ Sun Nov 17 03:17:53 2013

*** Log File = /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/TXK_CTRL_forceshutdown_Sun_Nov_17_03_17_53_2013/txkADOPCutOverPhaseCtrlScript_Sun_Nov_17_03_17_53_2013.log


Program :  completed @ Sun Nov 17 03:19:28 2013

      [EVENT]     [END   2013/11/17 03:19:28] Bringing down Middle Tier Services
        Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=4 restart=no abandon=yes defaultsfile=/inst122i/applmgr/fs1/EBSapps/appl/admin/INST122I_patch/adalldefaults.txt patchtop=/inst122i/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cutover.log driver=ucutover.drv
        ADPATCH Log directory: /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/log
      [EVENT]     [START 2013/11/17 03:21:16] Atomic File System Cutover

Program :  started @ Sun Nov 17 03:21:17 2013

*** Log File = /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/TXK_CTRL_ctxupdate_Sun_Nov_17_03_21_17_2013/txkADOPCutOverPhaseCtrlScript_Sun_Nov_17_03_21_17_2013.log


Program :  completed @ Sun Nov 17 03:22:53 2013

      [EVENT]     [END   2013/11/17 03:22:53] Atomic File System Cutover
      [EVENT]     [START 2013/11/17 03:23:00] Starting Middle Tier Services

Program :  started @ Sun Nov 17 03:23:00 2013

*** Log File = /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/TXK_CTRL_forcestartup_Sun_Nov_17_03_23_00_2013/txkADOPCutOverPhaseCtrlScript_Sun_Nov_17_03_23_00_2013.log


Program :  completed @ Sun Nov 17 03:32:21 2013

      [EVENT]     [END   2013/11/17 03:32:21] Starting Middle Tier Services
    [EVENT]     [END   2013/11/17 03:32:30] Performing Cutover Phase steps
  [END   2013/11/17 03:32:37] cutover phase
  [START 2013/11/17 03:32:38] Generating Post Cutover Reports
  [END   2013/11/17 03:32:39] Generating Post Cutover Reports
  [START 2013/11/17 03:32:43] Generating AD_ZD_LOGS Report
      [EVENT]     Report: /inst122i/applmgr/fs1/EBSapps/appl/ad/12.0.0/sql/ADZDSHOWLOG.sql

      [EVENT]     Output: /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/adzdshowlog.out

  [END   2013/11/17 03:32:48] Generating AD_ZD_LOGS Report
  [EVENT]     Executing post-cutover call-out script.
[END   2013/11/17 03:32:50] adzdoptl.pl run
adop phase=cutover - Completed Successfully

Log file: /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/adop_20131117_031215.log

adop exiting with status = 0 (Success)
you have mail in /var/spool/mail/apinst122i
$



Internal Logs for ICM Shutdown

      [EVENT]     [START 2013/11/17 03:13:59] ICM Shutdown
        [STATEMENT] Log file: UNDEF
      [EVENT]     [END   2013/11/17 03:14:04] ICM Shutdown
      [PROCEDURE] [START 2013/11/17 03:14:05] Cancelling ADZDPATCH concurrent program
        [STATEMENT] Concurrent Request Number: 476555
      [PROCEDURE] [END   2013/11/17 03:14:10] Cancelling ADZDPATCH concurrent program
      [EVENT]     [START 2013/11/17 03:14:10] Waiting for ICM to go down
        [STATEMENT] [START 2013/11/17 03:14:11] Check if ICM is running
          [PROCEDURE] [START 2013/11/17 03:14:14] ICM Status
          [PROCEDURE] [END   2013/11/17 03:14:16] ICM Status
          [STATEMENT]  ICM is active
        [STATEMENT] [END   2013/11/17 03:14:17] Check if ICM is running
        [PROCEDURE] ICM is still not down . Will wait for another minute.

        [STATEMENT] [START 2013/11/17 03:15:21] Check if ICM is running
          [PROCEDURE] [START 2013/11/17 03:15:24] ICM Status
          [PROCEDURE] [END   2013/11/17 03:15:25] ICM Status
          [STATEMENT]  ICM is active
        [STATEMENT] [END   2013/11/17 03:15:27] Check if ICM is running
        [PROCEDURE] ICM is still not down . Will wait for another minute.

        [STATEMENT] [START 2013/11/17 03:16:31] Check if ICM is running
          [PROCEDURE] [START 2013/11/17 03:16:34] ICM Status
          [PROCEDURE] [END   2013/11/17 03:16:36] ICM Status
          [STATEMENT]  ICM is active
        [STATEMENT] [END   2013/11/17 03:16:37] Check if ICM is running
        [PROCEDURE] ICM is still not down . Will wait for another minute.

        [STATEMENT] [START 2013/11/17 03:17:41] Check if ICM is running
          [STATEMENT]  ICM is not active
        [STATEMENT] [END   2013/11/17 03:17:44] Check if ICM is running
      [EVENT]     [END   2013/11/17 03:17:45] Waiting for ICM to go down



Internal Logs for Shutdown Middle Tier Services

[EVENT]     [START 2013/11/17 03:17:50] Bringing down Middle Tier Services
        [EVENT]     Calling: /inst122i/applmgr/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl
        [STATEMENT] Output directory: /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01
      [EVENT]     [END   2013/11/17 03:19:28] Bringing down Middle Tier Services



-----------
Values used
-----------
Context File        : /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml
PATCH Context File  : /inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml
OUT Directory       : /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01
Action              : forceshutdown
Console Logging     : off
Multi-node instance : NO



Executing SYSTEM command: /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/admin/scripts/adstpall.sh -nopromptmsg -abortCM -console=off
EXIT STATUS: 0
Copying the file
----------------
SOURCE : /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/logs/appl/admin/log/adstpall.log
TARGET : /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/TXK_CTRL_forceshutdown_Sun_Nov_17_03_17_53_2013/adstpall.log

/inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/admin/scripts/adstpall.sh went through successfully.



Internal Logs for Database Cutover

      [STATEMENT] Sourcing the Patch Edition environment: /inst122i/applmgr/fs1/EBSapps/appl/APPSINST122I_testserver01.env
      [PROCEDURE] [START 2013/11/17 03:19:35] Performing Automatic DB cutover
        [PROCEDURE] [Start 2013/11/17 03:19:36] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
          [PROCEDURE] Reading defaults file /inst122i/applmgr/fs1/EBSapps/appl/admin/INST122I_patch/adalldefaults.txt
          [PROCEDURE] Substituting tokens ...
          [PROCEDURE] Regenerating defaults file ...
        [PROCEDURE] [End   2013/11/17 03:19:37] ADOP::DefaultsFileUtil::substituteDefaultsFileTokensForPatchApplTop
        [STATEMENT] Log file: adop_20131117_031215.log
        Calling: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=4 restart=no abandon=yes defaultsfile=/inst122i/applmgr/fs1/EBSapps/appl/admin/INST122I_patch/adalldefaults.txt patchtop=/inst122i/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cutover.log driver=ucutover.drv
        ADPATCH Log directory: /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/log
        [PROCEDURE] Running: adpatch options=hotpatch,nocompiledb interactive=no console=no workers=4 restart=no abandon=yes defaultsfile=/inst122i/applmgr/fs1/EBSapps/appl/admin/INST122I_patch/adalldefaults.txt patchtop=/inst122i/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/driver logfile=cutover.log driver=ucutover.drv stdin=yes
      [PROCEDURE] [END   2013/11/17 03:20:52] Performing Automatic DB cutover


Additional Tasks

[PROCEDURE] [START 2013/11/17 03:20:53] Loading Jar files to DB
        [STATEMENT] No jars found to load
[PROCEDURE] [END   2013/11/17 03:20:54] Loading Jar files to DB



Internal Logs for Setting Patch Edition as Default Edition

[PROCEDURE] [START 2013/11/17 03:20:55] Setting Patch Edition as Default Edition
        [STATEMENT] sql stmt : <
                        declare
                          result varchar2(10);
                        begin
                          ad_zd_adop.cutover;
                        exception when others then
                          update ad_adop_sessions set status='F'
                          where adop_session_id=9;
                          commit;
                    raise_application_error(-20001,'Error while calling ad_zd_adop.cutover.' || sqlerrm);
                        end;>
        [STATEMENT] Sourcing the Run Edition environment: /inst122i/applmgr/fs2/EBSapps/appl/APPSINST122I_testserver01.env
      [PROCEDURE] [END   2013/11/17 03:21:00] Setting Patch Edition as Default Edition



Internal Logs for Flipping Snapshots

[PROCEDURE] [START 2013/11/17 03:21:07] Flipping Snapshots
        [STATEMENT] sql stmt : <
                     declare
                        result varchar2(10);
                     begin
                        ad_zd_adop.flip_snapshots;
                     exception when others then
                        update ad_adop_sessions set status='F'
                        where adop_session_id=9;
                        commit;
               raise_application_error(-20001,'Error while calling ad_zd_adop.flip_snapshots.' || sqlerrm);
                     end;>

      [PROCEDURE] [END   2013/11/17 03:21:10] Flipping Snapshots



Internal Logs for Atomic File System Cutover

Script Name : txkADOPCutOverPhaseCtrlScript.pl

-----------
Values used
-----------
Context File        : /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml
PATCH Context File  : /inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml
OUT Directory       : /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01
Action              : ctxupdate
Console Logging     : off
Multi-node instance : NO


What it does ?

·         Backup the run and patch edition context files to the FS_NE filesystem.
·         Backup the run and patch edition environment variables + EBSapps.env to the FS_NE filesystem.
·         Backup the run and patch edition fsclone_config.txt files to the FS_NE filesystem.
·         Reads the RUN edition context file version from the database and compares it with the filesystem.
Context File Version = 120.401.12020000.21
FS: serial_no_fs = 0
DB: serial_no_db = 0
FS: ctx_version_fs = 120.401.12020000.21
DB: ctx_version_db = 120.401.12020000.21
Context File: /inst122i/applmgr/fs2/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml is in SYNC with DB, no need to download.
·         Reads the PATCH edition context file version from the database and compares it with the filesystem.
Context File Version = 120.401.12020000.21
FS: serial_no_fs = 0
DB: serial_no_db = 0
FS: ctx_version_fs = 120.401.12020000.21
DB: ctx_version_db = 120.401.12020000.21
Context File: /inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/appl/admin/INST122I_testserver01.xml is in SYNC with DB, no need to download.


·         Updating Context Variable
NAME  : s_file_edition_type
VALUE : patch

Update successful
Updating Context Variable
-------------------------
NAME  : s_file_edition_type
VALUE : run

Update successful
·         Executes Autoconfig and creates the following files using its corresponding template files
o    INST122I_testserver01.env – environment variable for patch and run edition.
o    fsclone_config.txt – used by fsclone tool internally
o    EBSapps.env
·         Uploads the updated context file back to the database.
·         Copies the new RUN and PATCH configuration files to the cutover folder
o    INST122I_testserver01.xml
·         Copies the new RUN and PATCH deployment plans to the cutover folder
o    FMW_Home/Oracle_EBS-app1/deployment_plans
·         Copies the new RUN and PATCH OHS webtire configuration files to the cutover folder
o    FMW_Home/webtier/instances/EBS_web_INST122I_OHS1/config/OHS
·         Finally does cleanup activity
o    cleanupCaboStylesCache() - FMW_Home/Oracle_EBS-app1/applications/oacore/html/cabo/styles/cache





Internal Logs for Startup Services

      [EVENT]     [START 2013/11/17 03:23:00] Starting Middle Tier Services
      [EVENT]     [END   2013/11/17 03:32:21] Starting Middle Tier Services

================================
Inside forceStartupServices()...
================================

Executing SYSTEM command: /inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/admin/scripts/adstrtal.sh -nopromptmsg -console=off
EXIT STATUS: 0
Copying the file
----------------
SOURCE : /inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/logs/appl/admin/log/adstrtal.log
TARGET : /inst122i/applmgr/fs_ne/EBSapps/log/adop/9/cutover_20131117_031215/INST122I_testserver01/TXK_CTRL_forcestartup_Sun_Nov_17_03_23_00_2013/adstrtal.log


/inst122i/applmgr/fs1/inst/apps/INST122I_testserver01/admin/scripts/adstrtal.sh went through successfully.

Sunday, September 28, 2014

Oracle EBS R12 Blank AppsLogin / AppsLocalLogin Page

Oracle EBS-Blank Page Appslogin/AppsLocal Login Page in R12


There were quite a few times that I have encountered this blank front-end issue. The root cause was not the same every time it was investigated. There are many reasons for this issue to occur which I have compiled to make it easier for Apps DBA’s. Listed below are some of them.
Here is the error page while logging into Oracle E-Business Suite front end.
ebsblog2_1ss
These are some of the main reasons for the APPS blank login page in R12:
  • JSP pages not compiled properly after the clone
  • Data block corruption on Cloned environment with bulk synchronization issues
  • Database Listeners are down
  • Archive log space is full
  • DB or APPS file system is Full including /tmp location
  • Java Cache issues between the multiple middle-tiers
Here are different scenarios and work around/Fixes:
Scenario 1:  After the clone if any of the users experience user login issues other than super user with the below error message then please follow the steps in scenario 1.
The login in page may show Http-500 Internal server error (or) Http-404
1.       http 500 internal server error:
500 Internal Server Error
java.lang.NoClassDefFoundError
at oracle.apps.fnd.sso.AppsLoginRedirect.doGet(AppsLoginRedirect.java:623)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.evermind[Oracle Containers for J2EE 10g (10.1.3.0.0) ].server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(ReleaseResFilter.java:26)
Solution: (from Application Tier)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$cd $ADMIN_SCRIPTS_HOME
 
$adopmnctl.sh stopall — to shutdown opmn managed services viz., opmn, forms, oacore & oc4j
 
 $ cd $COMMON_TOP/_pages
 
$ mv _pages _pages_old
 
$mkdir _pages
 
 $ cd $FND_TOP/patch/115/bin or export PATH=$FND_TOP/patch/115/bin:$PATH
 
Compile all jsps.
$perl ojspCompile.pl –compile –flush -p 2 (or) ojspCompile.pl --compile --fast --quiet
(-p represents parallel threads, if you have enough horse power on your server, you can increase it to 8 or 10)
 
$cd $ADMIN_SCRIPTS_HOME
 
adopmnctl.sh startall
OR:
ORA-01578: ORACLE data block corrupted (file # 23, block # 221085)
ORA-01110: data file 23: ‘/RDBMS ORACLE_HOME/app/a_txn_data02.dbf’
ORA-26040: Data block was loaded using the NOLOGGING option
Solution: Login as apps user
Sqlplus apps/apps_password
1
2
3
4
5
6
7
8
9
10
11
12
13
set serveroutput on size 100000;
 declare
 begin
 WF_LOCAL_SYNCH.BULKSYNCHRONIZATION(
 P_ORIG_SYSTEM => 'ALL',
 P_PARALLEL_PROCESSES => null,
 P_LOGGING => null,
 P_RAISEERRORS => TRUE);
 exception
 when others then
 dbms_output.put_line('sqlerrm = ' || sqlerrm);
 end;
 /
Scenario 2: Database Listener is down
ebsblog2_2ss
Solution:  Start the database listener.
Scenario 3: Database Archive Locaion is 100% full or any mount point or tmp location is full(including concurrent log and output storage mount point)
Solution: Either temporarily move the files/archivelogs to another location or delete older files/archives (provided you had taken a cold backup sometime).
Scenario 4: IAS Cache initialization failed
This issue occurs only if you have more than one middle-tier and you had enabled Distributed Java Caching.
Error Message in log file(logfile resides in $LOG_HOME)
Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException:
IAS Cache initialization failed.
The Distributed Caching System failed to initialize on port: 12357.
The list of hosts in the distributed caching system is: [node 1] [node 2].
The port [port no] should be free on each host running the JVMs.
at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:214)
at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451)
In this scenario, s_java_object_cache_port value was listening on node 2 but not on node 1. So whichever connection went to node 1 it returned a blank page and occasionally it displayed 500 Internal Server error.
Solution:
Autoconfig
1)      Bring down opmn services on both the boxes.
1
2
3
$cd $ADMIN_SCRIPTS_HOME
 
$adopmnctl.sh stopall — to shutdown opmn managed services viz., opmn, forms, oacore & oc4j
2)       Verify no process is listening for s_java_object_cache_port value.
(To check the s_java_object_cache_port processes:
1
netstat -an | grep | grep LISTEN
Command to identify the OS process associated with a port
lsof -i tcp: )
3)       Change s_java_object_cache_port on all the nodes in the context file
4)      Run autoconfig
Non-Autoconfig
s_java_object_cache_port value is referrenced in two places
1) $INST_TOP/ora/10.1.3/javacache/admin/javacache.xml
2) Profile option name JTF_DIST_CACHE_PORT
Change the port value to a new number and bounce opmnctl services. Dont forget to change it in $CONTEXT_FILE. (To take effect during next autoconfig)
Verification:
Use commands listed in step 2 of autoconfig option.
If the new port mentioned still doesnt listen on the boxes you can use below workaround. This workaround will resolve the “IAS Cache initialization failed” even if the port is not listening.
Workaround
Set LONG_RUNNING_JVM=false in $INST_TOP/ora/10.1.3/j2ee/oacore/config/oc4j.properties
Bounce opmn services.
In 11i, if the database listener is down or if the archive location is full, atleast we get the internal server page while trying to access the login page, but in R12 nothing happens except for the blank page and apache error log will have an entry “oc4j_socket_recvfull timed out” with additional error message mod_oc4j: request to OC4J [mid-tier servername:ajp port] failed: Connect failed