Keep downtime short on 11i migration: Reusing a prepared software stack

Lots of production downtime can be eliminated by using a prepared software stack for upgrading. This tip explains how.

This tip is an excerpt from the paper "Keep downtime short on 11i migration" and is brought to you by the International Oracle Users Group (IOUG). Become a member of the IOUG to access the paper referenced here and a repository of technical content created for Oracle users by Oracle users.

This tip continues from Evaluating which tasks can done before and after, Applications database migration using the exp/imp method, and Patching improvements.

Reusing a prepared software stack

Lots of production downtime can be eliminated by using prepared software stack for upgrading. That means, we take the software stack of the latest successfully completed test upgrade and will carry out production upgrade with the same software as well. That means we don't have to apply any copy or generate drivers of patches, thus significantly reducing downtime. We will apply only database drivers of all patches.

There are two ways of doing upgrade with prepared software stack. First, when doing the installation for test upgrade, you have to create it exactly as you want it to be in production, because this is the environment which you'll clone or reuse for production later on.

The usually recommended way is to clone a successful test environment to another server or location and execute the production upgrade there. This approach is not the best, because if we really want to be sure of our production upgrade success and not have any surprises, we should use the software stack of last successful test upgrade in its original location. That means, when we have completed a successful test upgrade, we will just recreate the database and keep everything else as it is. This way we can do production upgrade with exactly the same file system and server environment as last successful test run. Also, we don't have to spend time on cloning efforts. Of course, it is always good and necessary to have a test environment available during and even after production upgrade. But normally, system testing and customization development is on separate, one of the first test-upgraded systems anyway, thus this environment isn't affected anyway. However, both software stack reuse methods are described here, in case you can't go with original software stack from last successful upgrade.

Reusing "origival" prepared software stack

Successfully upgraded and patched software stack is a prerequisite for this method.

  1. Shut down all services (of last test upgrade).
  2. Make sure there's no old applptch.txt file left in $APPL_TOP/admin/<SID> directory. If there is, remove or rename it, otherwise AutoPatch will try to load it to patch history tables the first time you execute it in normal mode.
  3. Prepare source database. There are several ways of doing it as described in other sections. (This tasks requires downtime, but in some cases, as upgrading database with export/import, some of the first tasks can be done before production downtime. See section about database migration). There are few issues when upgrading/moving database to a different host or instance name, see section "Modifying Applications Context to reflect database location" below. At the end of third phase we should have fully prepared file system (from last test run) and certified version of source Applications database online. Preparatory tasks such are initial statistics gathering and data purging etc. should have been done in source environment.
  4. Perform upgrade as required in Upgrade Manual and apply patches. Post-upgrade tasks should be done after all patches have been applied. There is one, but important difference compared to standard upgrade:
    • No need for applying any C or G patch drivers, not even AD and F-CUP patches which are normally applied in preinstall mode. We apply only D-drivers. Starting from AD.E the D drivers are run with "checkfile" option. This allows us to greatly reduce database patching time.
    • To enable patch prerequisite checking you have to run "Maintain snapshot information" under "Maintain Applications Files", to gather missing information about files copied to file system during previous test upgrade (which generated our file system as it is now). The information is uploaded to patch history tables, for later use of patch prerequisite checking. Normally it should be done after the whole upgrade is completed, since it takes significant amount of time to scan through the file system and it can be executed online afterwards. Also, we already have manually resolved all patch requirements and applied all of them during last test upgrade. However, if adpatch is unable to apply some patch because it fails the prerequisite checking, you can always use parameter "options=noprereq" when running adpatch.
    • It would be good idea to remove old log and request files from last test upgrade, that they wouldn't confuse you when monitoring progress of upgrade or solving problems.
  5. Do other post-upgrade tasks.
Reusing original prepared software stack is the recommended method, since it allows us to simulate production upgrade in exactly the same environment and with almost identical configuration to real production upgrade.

Modifying Applications Context to reflect database location

This section only applies if we are reusing our software stack and our new database is going to be in different host or port than set during software stack installation and last test upgrade.

There is a central repository in Applications, where all configuration information about physical servers, host names, ports, file locations are stored. This is called Applications Context. Applications Context lies in .xml file, under $APPL_TOP/admin directory. If source database is already at Applications 11i certified version and no upgrade is needed, just modify the tnsnames.ora entry specified in TWO_TASK variable to point to correct database instance. The supported and easiest way to do it is using Context Editor and AutoConfig (both come with 11.5.7)

  • Use Context Editor to change your data server host information in Apps context file <TWO_TASK>.xml. Several values like, host name, domain name, tns listener port, etc. can be changed if needed. Use editcontext command to modify the context file:


    Set DISPLAY variable to point to your host:

    export DISPLAY=xhost:0 (replace xhost with your host name)
    Run editcontext executable under COMMON_TOP:

    Run EditContext.cmd executable under COMMON_TOP:

    When Context Editor opens, it asks where your context file is located. This file is usually named <TWO_TASK>.xml under $APPL_TOP/admin. When the context file is opened, several configuration parameters can be modified, Data Server Host for example as seen from picture below, if we start using database on another host.
  • Since Context Editor only modifies the context xml file, we will use AutoConfig for configuring Applications to apply our changes in context. AutoConfig has two mandatory parameters, one for contextfile location, other for APPS user password.


    $AD_TOP/bin/ contextfile=$APPL_TOP/admin/PROD.xml appspass=apps
    %AD_TOP%/bin/adconfig.cmd contextfile=%APPL_TOP%/admin/PROD.xml appspass=apps
    After AutoConfig has been run, new changes should be in effect. If the database referenced by TWO_TASK is shut down or unavailable at the time, AutoConfig doesn't fail, it just writes error messages into a logfile $APPL_TOP/admin/log directory. Logfile's format is .log, meaning it contains information about day, month, hour and minute it was run.


    Even though this task doesn't actually require downtime, you should note that if you run AutoConfig in your new environment and TWO_TASK or LOCAL variable is pointing to source (currently running) production environment, AutoConfig can change profile information in the source database without asking. This won't be a problem if source database already has been shut down for migration, but may cause unability to log on to source system if it is sill in use (Changing APPS_WEB_AGENT or ICX_FORMS_LAUNCHER for example). As a precaution, this task should be performed only when database and Apps have already been taken down for migration.

  • Note that <TWO_TASK>.xml under $APPL_TOP/admin can also be modified without Context Editor, just open it up with text editor and search for "s_dbhost" variable, if you want to change database hostname. However, to apply the changes, AutoConfig must still be run.
  • Alternatively, you could copy your source database in place of the last test upgrade database. This requires database versions to be exactly the same and platforms to be binary compatible. Also, database controlfile might have to be recreated, if source database's name is different from what is specified in target instance's db_name initialization parameter. If file locations change, it is easy to apply those modifications also when recreating controlfile.
  • If there is a need to upgrade the database, choose appropriate method from section "Database Migration to Applications' Certified Version".

Reusing a "cloned" prepared software stack

Successfully upgraded and patched software stack is a prerequisite for this method.

There are several notes and recommendations about reusing prepared software stack by cloning. The main difference with reusing original prepared software stack is that Applications file system is mainly cloned to different location and we don't have to use AutoConfig to change any configuration, because correct configuration is set during software stack installation.

  1. Run Rapid Install in category 2 to install a new instance. Use the Create Upgrade file system option. This installs the technology stack, an APPL_TOP (that you will replace later), and creates configuration files.
  2. Create an "admin" directory under <COMMON_TOP>.
  3. Use the AD Cloning utility in pre-clone mode to preserve the environment (See metalink note 135792.1).
  4. Copy the Applications file system ($APPL_TOP, $OA_HTML, and $JAVA_TOP) from the successfully upgraded test system to the production system.
  5. Manually re-apply the configuration files saved by the AD Cloning utility. The configuration files are saved in <COMMON_TOP>/admin/clone.
  6. Set the TWO_TASK environment variable to use the source production database or copy of it. Apply all technology stack patches you previously applied to the test system. (Developer, 9iAS patches). Why we aren't just copying those over, is that copying itself doesn't leave any trace in Oracle Inventory, thus making Technology Stack patching harder later on.
  7. Perform Upgrade as required in Upgrade Manual and apply patches. Here we have exactly the same conditions as in "Reusing a original prepared software stack" section, that we have to apply only D drivers, etc..
  8. Do other post-upgrade tasks.

Simple cloning to different server using identical directory structures

There is actually one very simple and easy cloning mechanism - just copying all Oracle files to different server to identical directory structure. It would be good if both servers had exactly the same operating system version and patch level, but it is enough that both of the server configurations are certified by Oracle and more important, binary compatible. For example, we can clone from SunOS 2.6 operating system to Solaris 8 if both are on SPARC architecture. Other issue is testing, if we have tested our upgrade on Solaris 8, then doing the production upgrade on SunOS 2.6 could introduce us several more bugs, problems and might need additional patches for compatibility.

A simple cloning mechanism, when you can retain the same OS configuration, is following:

  1. Instead of only copying some directories like in previous example, copy entire Applications file system over to target.
  2. See your operating system specific installation documents to see where are the inventory pointer file oraInst.loc and oratab files stored, those should be copied over as well (/var/opt/oracle in Solaris, /var/opt/oracle and /etc for Tru64). And remember, that you might have to modify target system's kernel parameters as with installing Oracle software.
  3. When just copying executable files over, you also should relink those executables (relink command for Oracle 8i,9i,9iAS, for Developer6i)
  4. Use Context Editor and AutoConfig to change appropriate configuration options like host name, ports, database name if needed.
  5. Perform Upgrade as required in Upgrade Manual and apply patches. Again, only D-drivers are needed for patches, if we are upgrading with a filesystem cloned from a successful test upgrade.
  6. Do other post-upgrade tasks.
The only restriction with this method is, that you should have (fairly) identical servers for having same directory structure and OS groups and users. Note that this mechanism is meant for cloning in Unix, although can be in Windows, it is trickier because registry and service installation.

Conclusion on software stack reusing

Software stack reusing can significantly reduce downtime, especially when there's lots of different languages and modules involved. Forms and reports need to be generated for every language, this can be really time consuming in international installations.

As we see from number of required steps, reusing a software stack using cloning is significantly more complex, but its advantage is that you can retain the test environment while using a copy of this software stack for production upgrade. But, since normal Applications Upgrade project includes more than one test upgrade anyway, it usually is no problem to use last test software stack for production upgrade. Thus the first, reusing original software stack, is the recommended way.

This was last published in February 2005

Dig Deeper on Oracle applications implementation and upgrades

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.