Check 18c Upgrade Results – The cluster upgrade state is [UPGRADE FINAL]
18c Upgrade Getting to Results – The cluster upgrade state is [NORMAL]
This is a multi-part series for 18c Upgrade and suggest read the above 2 posts first.
Commands for gridSetup.sh
+ /u01/18.3.0.0/grid/gridSetup.sh -silent -skipPrereqs -applyRU /media/patch/Jan2019/28828717 -responseFile /sf_OracleSoftware/18cLinux/gridsetup_upgrade.rsp -J-Doracle.install.mgmtDB=false -J-Doracle.install.mgmtDB.CDB=false -J Doracle.install.crs.enableRemoteGIMR=false
Preparing the home to patch...
Applying the patch /media/patch/Jan2019/28828717...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2019-04-16_06-19-12AM/installerPatchActions_2019-04-16_06-19-12AM.log
Launching Oracle Grid Infrastructure Setup Wizard...
The response file for this session can be found at:
/u01/18.3.0.0/grid/install/response/grid_2019-04-16_06-19-12AM.rsp
You can find the log of this install session at:
/u01/app/oraInventory/logs/GridSetupActions2019-04-16_06-19-12AM/gridSetupActions2019-04-16_06-19-12AM.log
As a root user, execute the following script(s):
1. /u01/18.3.0.0/grid/rootupgrade.sh
Execute /u01/18.3.0.0/grid/rootupgrade.sh on the following nodes:
[racnode-dc1-1, racnode-dc1-2]
Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.
Successfully Setup Software.
As install user, execute the following command to complete the configuration.
/u01/18.3.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /sf_OracleSoftware/18cLinux/gridsetup_upgrade.rsp [-silent]
+ exit
oracle@racnode-dc1-1::/home/oracle
$
Basically, the error provided is utterly useless.
oracle@racnode-dc1-1::/home/oracle $ /u01/18.3.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /sf_OracleSoftware/18cLinux/gridsetup_upgrade.rsp -silent Launching Oracle Grid Infrastructure Setup Wizard... You can find the logs of this session at: /u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM [WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped. ACTION: Refer to the logs or contact Oracle Support Services. oracle@racnode-dc1-1::/home/oracle
Check logs from directory /u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM
oracle@racnode-dc1-1:+ASM1:/home/oracle $ cd /u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM $ ls -alrt total 1072 -rw-r----- 1 oracle oinstall 130 Apr 16 12:59 installerPatchActions_2019-04-16_12-59-56PM.log -rw-r----- 1 oracle oinstall 0 Apr 16 12:59 gridSetupActions2019-04-16_12-59-56PM.err drwxrwx--- 8 oracle oinstall 4096 Apr 16 13:01 .. -rw-r----- 1 oracle oinstall 1004378 Apr 16 13:01 gridSetupActions2019-04-16_12-59-56PM.out -rw-r----- 1 oracle oinstall 2172 Apr 16 13:01 time2019-04-16_12-59-56PM.log *** -rw-r----- 1 oracle oinstall 73047 Apr 16 13:01 gridSetupActions2019-04-16_12-59-56PM.log *** drwxrwx--- 2 oracle oinstall 4096 Apr 16 13:01 .
Check time2019-04-16_12-59-56PM.log
oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM $ cat time2019-04-16_12-59-56PM.log # Message # ElapsedTime # Current Time ( ms ) # Starting step:INITIALIZE_ACTION of state:init # 0 # 1555412405106 # Finished step:INITIALIZE_ACTION of state:init # 1 # 1555412405106 # Starting step:EXECUTE of state:init # 0 # 1555412405108 # Finished step:EXECUTE of state:init # 3 # 1555412405111 # Starting step:VALIDATE of state:init # 0 # 1555412405113 # Finished step:VALIDATE of state:init # 2 # 1555412405115 # Starting step:TRANSITION of state:init # 0 # 1555412405115 # Finished step:TRANSITION of state:init # 2 # 1555412405117 # Starting step:EXECUTE of state:CRSConfigTools # 0 # 1555412405117 # Finished step:EXECUTE of state:CRSConfigTools # 813 # 1555412405930 # Starting step:VALIDATE of state:CRSConfigTools # 0 # 1555412405930 # Finished step:VALIDATE of state:CRSConfigTools # 0 # 1555412405930 # Starting step:TRANSITION of state:CRSConfigTools # 0 # 1555412405930 # Finished step:TRANSITION of state:CRSConfigTools # 26591 # 1555412432521 # Starting step:INITIALIZE_ACTION of state:setup # 0 # 1555412432521 # Finished step:INITIALIZE_ACTION of state:setup # 0 # 1555412432521 # Starting step:EXECUTE of state:setup # 0 # 1555412432522 # Finished step:EXECUTE of state:setup # 6 # 1555412432528 # Configuration in progress. # 0 # 1555412436788 # Update Inventory in progress. # 0 # 1555412437768 # Update Inventory successful. # 52612 # 1555412490380 # Upgrading RHP Repository in progress. # 0 # 1555412490445 ================================================================================ # Upgrading RHP Repository failed. # 12668 # 1555412503112 ================================================================================ # Starting step:VALIDATE of state:setup # 0 # 1555412503215 # Finished step:VALIDATE of state:setup # 15 # 1555412503230 # Starting step:TRANSITION of state:setup # 0 # 1555412503230 # Finished step:TRANSITION of state:setup # 0 # 1555412503230 # Starting step:EXECUTE of state:finish # 0 # 1555412503230 # Finished step:EXECUTE of state:finish # 6 # 1555412503236 # Starting step:VALIDATE of state:finish # 0 # 1555412503237 # Finished step:VALIDATE of state:finish # 1 # 1555412503238 # Starting step:TRANSITION of state:finish # 0 # 1555412503238 # Finished step:TRANSITION of state:finish # 0 # 1555412503238 oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM
Check gridSetupActions2019-04-16_12-59-56PM.log
oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM $ grep -B2 -A100 'Executing RHPUPGRADE' gridSetupActions2019-04-16_12-59-56PM.log INFO: [Apr 16, 2019 1:01:30 PM] Starting 'Upgrading RHP Repository' INFO: [Apr 16, 2019 1:01:30 PM] Starting 'Upgrading RHP Repository' INFO: [Apr 16, 2019 1:01:30 PM] Executing RHPUPGRADE INFO: [Apr 16, 2019 1:01:30 PM] Command /u01/18.3.0.0/grid/bin/rhprepos upgradeSchema -fromversion 12.1.0.2.0 INFO: [Apr 16, 2019 1:01:30 PM] ... GenericInternalPlugIn.handleProcess() entered. INFO: [Apr 16, 2019 1:01:30 PM] ... GenericInternalPlugIn: getting configAssistantParmas. INFO: [Apr 16, 2019 1:01:30 PM] ... GenericInternalPlugIn: checking secretArguments. INFO: [Apr 16, 2019 1:01:30 PM] No arguments to pass to stdin INFO: [Apr 16, 2019 1:01:30 PM] ... GenericInternalPlugIn: starting read loop. INFO: [Apr 16, 2019 1:01:43 PM] Completed Plugin named: rhpupgrade INFO: [Apr 16, 2019 1:01:43 PM] ConfigClient.saveSession method called INFO: [Apr 16, 2019 1:01:43 PM] Upgrading RHP Repository failed. INFO: [Apr 16, 2019 1:01:43 PM] Upgrading RHP Repository failed. INFO: [Apr 16, 2019 1:01:43 PM] ConfigClient.executeSelectedToolsInAggregate action performed INFO: [Apr 16, 2019 1:01:43 PM] Exiting ConfigClient.executeSelectedToolsInAggregate method INFO: [Apr 16, 2019 1:01:43 PM] Adding ExitStatus SUCCESS_MINUS_RECTOOL to the exit status set INFO: [Apr 16, 2019 1:01:43 PM] ConfigClient.saveSession method called INFO: [Apr 16, 2019 1:01:43 PM] Calling event ConfigSessionEnding INFO: [Apr 16, 2019 1:01:43 PM] ConfigClient.endSession method called INFO: [Apr 16, 2019 1:01:43 PM] Completed Configuration INFO: [Apr 16, 2019 1:01:43 PM] Adding ExitStatus FAILURE to the exit status set INFO: [Apr 16, 2019 1:01:43 PM] All forked task are completed at state setup INFO: [Apr 16, 2019 1:01:43 PM] Completed background operations INFO: [Apr 16, 2019 1:01:43 PM] Validating state <setup> ================================================================================ WARNING: [Apr 16, 2019 1:01:43 PM] [WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped. ACTION: Refer to the logs or contact Oracle Support Services. ================================================================================ INFO: [Apr 16, 2019 1:01:43 PM] Advice is CONTINUE INFO: [Apr 16, 2019 1:01:43 PM] Completed validating state <setup> INFO: [Apr 16, 2019 1:01:43 PM] Verifying route success INFO: [Apr 16, 2019 1:01:43 PM] Waiting for completion of background operations INFO: [Apr 16, 2019 1:01:43 PM] Completed background operations INFO: [Apr 16, 2019 1:01:43 PM] Waiting for completion of background operations INFO: [Apr 16, 2019 1:01:43 PM] Completed background operations INFO: [Apr 16, 2019 1:01:43 PM] Executing action at state finish INFO: [Apr 16, 2019 1:01:43 PM] FinishAction Actions.execute called INFO: [Apr 16, 2019 1:01:43 PM] Finding the most appropriate exit status for the current application INFO: [Apr 16, 2019 1:01:43 PM] Completed executing action at state <finish> INFO: [Apr 16, 2019 1:01:43 PM] Waiting for completion of background operations INFO: [Apr 16, 2019 1:01:43 PM] Completed background operations INFO: [Apr 16, 2019 1:01:43 PM] Waiting for completion of background operations INFO: [Apr 16, 2019 1:01:43 PM] Completed background operations INFO: [Apr 16, 2019 1:01:43 PM] Moved to state <finish> INFO: [Apr 16, 2019 1:01:43 PM] Waiting for completion of background operations INFO: [Apr 16, 2019 1:01:43 PM] Completed background operations INFO: [Apr 16, 2019 1:01:43 PM] Validating state <finish> WARNING: [Apr 16, 2019 1:01:43 PM] Validation disabled for the state finish INFO: [Apr 16, 2019 1:01:43 PM] Completed validating state <finish> INFO: [Apr 16, 2019 1:01:43 PM] Terminating all background operations INFO: [Apr 16, 2019 1:01:43 PM] Terminated all background operations INFO: [Apr 16, 2019 1:01:43 PM] Successfully executed the flow in SILENT mode INFO: [Apr 16, 2019 1:01:43 PM] Finding the most appropriate exit status for the current application INFO: [Apr 16, 2019 1:01:43 PM] inventory location is/u01/app/oraInventory INFO: [Apr 16, 2019 1:01:43 PM] Finding the most appropriate exit status for the current application ================================================================================ INFO: [Apr 16, 2019 1:01:43 PM] Exit Status is -1 INFO: [Apr 16, 2019 1:01:43 PM] Shutdown Oracle Grid Infrastructure 18c Installer INFO: [Apr 16, 2019 1:01:43 PM] Unloading Setup Driver ================================================================================ oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM $
Due to Exit Status is -1 is probably why – The cluster upgrade state is [UPGRADE FINAL]
Why Upgrading RHP Repository when oracle_install_crs_ConfigureRHPS=false?
oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM $ grep -i rhp * gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:00:04 PM] Setting value for the property:oracle_install_crs_ConfigureRHPS in the bean:CRSInstallSettings gridSetupActions2019-04-16_12-59-56PM.log: oracle_install_crs_ConfigureRHPS false gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:00:37 PM] Created config job for rhpupgrade gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:00:37 PM] Selecting job named 'Upgrading RHP Repository' for retry gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:30 PM] Started Plugin named: rhpupgrade gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:30 PM] Starting 'Upgrading RHP Repository' gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:30 PM] Starting 'Upgrading RHP Repository' gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:30 PM] Executing RHPUPGRADE gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:30 PM] Command /u01/18.3.0.0/grid/bin/rhprepos upgradeSchema -fromversion 12.1.0.2.0 gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:43 PM] Completed Plugin named: rhpupgrade gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:43 PM] Upgrading RHP Repository failed. gridSetupActions2019-04-16_12-59-56PM.log:INFO: [Apr 16, 2019 1:01:43 PM] Upgrading RHP Repository failed. time2019-04-16_12-59-56PM.log: # Upgrading RHP Repository in progress. # 0 # 1555412490445 time2019-04-16_12-59-56PM.log: # Upgrading RHP Repository failed. # 12668 # 1555412503112 oracle@racnode-dc1-1:+ASM1:/u01/app/oraInventory/logs/GridSetupActions2019-04-16_12-59-56PM $
gridsetup_upgrade.rsp is used for upgrade and pertinent info shown.
## To upgrade clusterware and/or Automatic storage management of earlier ## ## releases ## ## - Fill out sections A,B,C,D and H ## #------------------------------------------------------------------------------- # Specify the required cluster configuration # Allowed values: STANDALONE, DOMAIN, MEMBERDB, MEMBERAPP #------------------------------------------------------------------------------- oracle.install.crs.config.ClusterConfiguration=STANDALONE #------------------------------------------------------------------------------- # Configure RHPS - Rapid Home Provisioning Service # Applicable only for DOMAIN cluster configuration # Specify 'true' if you want to configure RHP service, else specify 'false' #------------------------------------------------------------------------------- oracle.install.crs.configureRHPS=false oracle@racnode-dc1-1::/sf_OracleSoftware/18cLinux $ sdiff -iEZbWBst -w 150 gridsetup.rsp gridsetup_upgrade.rsp INVENTORY_LOCATION= | INVENTORY_LOCATION=/u01/app/oraInventory oracle.install.option= | oracle.install.option=UPGRADE ORACLE_BASE= | ORACLE_BASE=/u01/app/oracle oracle.install.crs.config.scanType= | oracle.install.crs.config.scanType=LOCAL_SCAN oracle.install.crs.config.ClusterConfiguration= | oracle.install.crs.config.ClusterConfiguration=STANDALONE oracle.install.crs.config.configureAsExtendedCluster= | oracle.install.crs.config.configureAsExtendedCluster=false oracle.install.crs.config.gpnp.configureGNS= | oracle.install.crs.config.gpnp.configureGNS=false oracle.install.crs.config.autoConfigureClusterNodeVIP= | oracle.install.crs.config.autoConfigureClusterNodeVIP=false oracle.install.asm.configureGIMRDataDG= | oracle.install.asm.configureGIMRDataDG=false oracle.install.asm.configureAFD= | oracle.install.asm.configureAFD=false oracle.install.crs.configureRHPS= | oracle.install.crs.configureRHPS=false oracle.install.crs.config.ignoreDownNodes= | oracle.install.crs.config.ignoreDownNodes=false oracle.install.config.managementOption= | oracle.install.config.managementOption=NONE oracle.install.crs.rootconfig.executeRootScript= | oracle.install.crs.rootconfig.executeRootScript=false
ora.cvu does not report any errors.
oracle@racnode-dc1-1:+ASM1:/home/oracle
$ crsctl stat res -w "TYPE = ora.cvu.type" -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cvu
1 ONLINE ONLINE racnode-dc1-2 STABLE
--------------------------------------------------------------------------------
oracle@racnode-dc1-1:+ASM1:/home/oracle
$ crsctl stat res -w "TYPE = ora.cvu.type" -p|grep RESULTS | sed 's/,/\n/g'
CHECK_RESULTS=
oracle@racnode-dc1-1:+ASM1:/home/oracle
$
Run rhprepos upgradeSchema -fromversion 12.1.0.2.0 – FAILED.
oracle@racnode-dc1-1::/home/oracle $ /u01/18.3.0.0/grid/bin/rhprepos upgradeSchema -fromversion 12.1.0.2.0 PRCT-1474 : failed to run 'mgmtca' on node racnode-dc1-2. oracle@racnode-dc1-1::/home/oracle $ ps -ef|grep pmon oracle 9722 4804 0 19:37 pts/0 00:00:00 grep --color=auto pmon oracle 10380 1 0 13:46 ? 00:00:01 asm_pmon_+ASM1 oracle 10974 1 0 13:46 ? 00:00:01 apx_pmon_+APX1 oracle 11218 1 0 13:47 ? 00:00:02 ora_pmon_hawk1 oracle@racnode-dc1-1::/home/oracle $ ssh racnode-dc1-2 Last login: Tue Apr 16 18:44:30 2019 ---------------------------------------- Welcome to racnode-dc1-2 OracleLinux 7.3 x86_64 FQDN: racnode-dc1-2.internal.lab IP: 10.0.2.15 Processor: Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz #CPU's: 2 Memory: 5709 MB Kernel: 4.1.12-61.1.18.el7uek.x86_64 ---------------------------------------- oracle@racnode-dc1-2::/home/oracle $ ps -ef|grep pmon oracle 9219 1 0 13:44 ? 00:00:01 asm_pmon_+ASM2 oracle 10113 1 0 13:45 ? 00:00:01 apx_pmon_+APX2 oracle 10619 1 0 13:45 ? 00:00:01 ora_pmon_hawk2 oracle 13200 13178 0 19:37 pts/0 00:00:00 grep --color=auto pmon oracle@racnode-dc1-2::/home/oracle $
In conclusion, the silent upgrade process is poorly documented at best.
Starting to wondering if the following parameters contributed to the issue:
-J-Doracle.install.mgmtDB=false -J-Doracle.install.mgmtDB.CDB=false -J Doracle.install.crs.enableRemoteGIMR=false