I was tasked to find supplemental logging details for Oracle database used with GoldenGate.

Note: this is not a pluggable database.

With ADD TRANDATA, use dba_log_groups and dba_log_group_columns.

With ADD SCHEMATRANDATA, use select * from table(logmnr$always_suplog_columns( SCHEMA, TABLE ));

Basically, one would need to run the query with logmnr pipeline function for all the tables in the schema.

Here is one process I used.

Create info_schematrandata.prm

$ cat info_schematrandata.prm
dblogin USERID ggs, PASSWORD *
info schematrandata *

Run ggsci using info_schematrandata.prm (full path is required)

$ ggsci paramfile /home/oracle/working/dinh/info_schematrandata.prm > info_schematrandata.log

Here is example for results (actual contains 12 schemas)

$ grep -i enable info_schematrandata.log
2020-06-01 05:19:35  INFO    OGG-06480  Schema level supplemental logging, excluding non-validated keys, is enabled on schema "SCOTT".
2020-06-01 05:19:35  INFO    OGG-01981  Schema level supplemental logging is enabled on schema "SCOTT" for all columns.

After finding the schemas, use logmnr pipeline function to find all the details.

select * from table(logmnr$always_suplog_columns('SCOTT','EMP')) order by intcol;
select * from table(logmnr$always_suplog_columns('SCOTT','BONUS')) order by intcol;
select * from table(logmnr$always_suplog_columns('SCOTT','DEPT')) order by intcol;

You can find demo with logmnr$always_suplog_columns at: GoldenGate 12c Features Found in 11.2.1.0.21 ???

References:

OGG: How To Log All Columns With Add Schematrandata To Get NOCOMPRESSUPDATES (Doc ID 1413142.1)

ADD SCHEMATRANDATA does not allow columns to be specified.
This enables logging of Primary Key columns only.
By default, updates are compressed.
In order to log all columns ADD TRANDATA would have to be used.
The ADD TRANDATA can be used in conjunction with ADD SCHEMATRANDATA to specify the non-primary key columns.

How to Check Supplemental Logging When ADD SCHEMATRANDATA is Enabled (Doc ID 1537837.1)

It is not listed in dba_log_groups or dba_log_group_columns.
select * from table(logmnr$always_suplog_columns( SCHEMA, TABLE ));

Effects of ADD TRANDATA and ADD SCHEMATRANDATA on an Oracle databases’ Supplemental Logging (Doc ID 2070331.1)

Some useful commands from ggsci:

INFO TRANDATA [container.]owner.table (info trandata *) did not work
INFO SCHEMATRANDATA schema            (info schematrandata *)
LIST TABLES table                     (list tables SCOTT.*)

Note to self:

$ cat list_table.prm
dblogin USERID ggs, PASSWORD *
list tables SCOTT.*

$ ggsci paramfile /home/oracle/working/dinh/list_table.prm > list_table.log

$ grep '\.' list_table.log | egrep -iv 'found|ggsci'| grep -A 10000 "Successfully logged into database."|grep -v database > table.log

$ cat table.log
SCOTT.EMP
SCOTT.BONUS
SCOTT.DEPT

$ cat read.sh
#!/bin/bash
IFS="."
while read f1 f3
do
echo "select * from table(logmnr\$always_suplog_columns('$f1','$f3')) order by intcol;"
done < /home/oracle/working/dinh/table.log
exit

$ ./read.sh > /tmp/suplog.sql

$ head /tmp/suplog.sql
select * from table(logmnr$always_suplog_columns('SCOTT','EMP')) order by intcol;
select * from table(logmnr$always_suplog_columns('SCOTT','BONUS')) order by intcol;
select * from table(logmnr$always_suplog_columns('SCOTT','DEPT')) order by intcol;

$ cat suplog.sql
set numw 8 lines 200 timing off echo off pages 10000 trimsp on tab off
column NAME_COL_PLUS_SHOW_PARAM format a30
column VALUE_COL_PLUS_SHOW_PARAM format a65 wrap
col owner for a20
col table_name for a20
col column_name for a30
col log_group_type for a20
col column_list for a80
col log_group_name for a30
col table_name for a30
spool Database_Supplemental_Logging_Details.log
pro ******** Database ********
SELECT
name,db_unique_name,open_mode,database_role,remote_archive,switchover_status,dataguard_broker,primary_db_unique_name
FROM v$database
;
pro ******** Database Supplemental Logging ********
SELECT
supplemental_log_data_min MIN,
supplemental_log_data_pk PK,
supplemental_log_data_ui UI,
supplemental_log_data_fk FK,
supplemental_log_data_all "ALL"
FROM v$database
;
pro ******** Table Supplemental Logging ********
pro
pro ******** GoldenGate: ADD TRANDATA ********
SELECT
g.owner, g.table_name, g.log_group_name, g.log_group_type,
DECODE(always,'ALWAYS','Unconditional',NULL,'Conditional') always,
LISTAGG(c.column_name, ', ') WITHIN GROUP (ORDER BY c.POSITION) column_list
FROM dba_log_groups g, dba_log_group_columns c
WHERE g.owner = c.owner(+)
AND g.log_group_name = c.log_group_name(+)
AND g.table_name = c.table_name(+)
GROUP BY g.owner, g.log_group_name, g.table_name, g.log_group_type, DECODE(always,'ALWAYS','Unconditional',NULL,'Conditional')
ORDER BY g.owner, g.log_group_name, g.table_name, g.log_group_type
;
pro ******** Schema Supplemental Logging ********
pro
pro ******** GoldenGate: ADD SCHEMATRANDATA ********
@/tmp/suplog.sql
exit

IMPORTANT: NO DB has been created !!!

Why is rename ASM DG?
1. Test RAC RMAN duplicate for standby similar to client environment.
2. Just because.

Rename Diskgroup name by renamedg command doesn't update the OCR (Doc ID 2392499.1)	

Bug 8740909  [11GR2-LNX-090726]OLD DISK GROUP RESOURCE NOT DELETED AFTER RENAMEDG

renamedg does not update resources, nor does renamedg update any file references within the database. 
Because of this, the original disk group resource is not automatically deleted. 
This resource can be manually deleted with Server Control Utility (SRVCTL).

==============================

[oracle@ol7-19-phx2 ~]$ asmcmd -V
asmcmd version 19.6.0.0.0
[oracle@ol7-19-phx2 ~]$

==============================

[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$


==============================

[oracle@ol7-19-phx1 ~]$ asmcmd umount RECO
[oracle@ol7-19-phx2 ~]$ asmcmd umount RECO

==============================

[oracle@ol7-19-phx1 ~]$ asmcmd lsdg -g
Inst_ID  State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  NORMAL  N         512             512   4096  1048576     98301    47769            32767            7501              0             Y  CRS/
      2  MOUNTED  NORMAL  N         512             512   4096  1048576     98301    47769            32767            7501              0             Y  CRS/
      1  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  DATA_PHX/
      2  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  DATA_PHX/

[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        OFFLINE OFFLINE                               STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$

==============================

[oracle@ol7-19-phx1 ~]$ renamedg dgname=RECO newdgname=FRA_PHX verbose=true

Parameters in effect:

         Old DG name       : RECO
         New DG name          : FRA_PHX
         Phases               :
                 Phase 1
                 Phase 2
         Discovery str        : (null)
         Clean              : TRUE
         Raw only           : TRUE
renamedg operation: dgname=RECO newdgname=FRA_PHX verbose=true
Executing phase 1
Discovering the group
Performing discovery with string:
ERROR: submiting READ on /dev/sdg failed due to kgfknm error
ERROR: -20(27041 /dev/sdg)ERROR: submiting READ on /dev/sda failed due to kgfknm error
ERROR: -20(27041 /dev/sda)ERROR: submiting READ on /dev/sdc failed due to kgfknm error
ERROR: -20(27041 /dev/sdc)ERROR: submiting READ on /dev/sdf failed due to kgfknm error
ERROR: -20(27041 /dev/sdf)ERROR: submiting READ on /dev/sde failed due to kgfknm error
ERROR: -20(27041 /dev/sde)ERROR: submiting READ on /dev/sdd failed due to kgfknm error
ERROR: -20(27041 /dev/sdd)ERROR: submiting READ on /dev/sdb failed due to kgfknm error
ERROR: -20(27041 /dev/sdb)ERROR: submiting READ on /dev/sda1 failed due to kgfknm error
ERROR: -20(27041 /dev/sda1)ERROR: submiting READ on /dev/sda2 failed due to kgfknm error
ERROR: -20(27041 /dev/sda2)ERROR: submiting READ on /dev/sdb1 failed due to kgfknm error
ERROR: -20(27041 /dev/sdb1)Identified disk UFS:/dev/sdg1 with disk number:0 and timestamp (33103213 1600008192)
Checking for hearbeat...
Re-discovering the group
Performing discovery with string:
ERROR: submiting READ on /dev/sdg failed due to kgfknm error
ERROR: -20(27041 /dev/sdg)ERROR: submiting READ on /dev/sda failed due to kgfknm error
ERROR: -20(27041 /dev/sda)ERROR: submiting READ on /dev/sdc failed due to kgfknm error
ERROR: -20(27041 /dev/sdc)ERROR: submiting READ on /dev/sdf failed due to kgfknm error
ERROR: -20(27041 /dev/sdf)ERROR: submiting READ on /dev/sde failed due to kgfknm error
ERROR: -20(27041 /dev/sde)ERROR: submiting READ on /dev/sdd failed due to kgfknm error
ERROR: -20(27041 /dev/sdd)ERROR: submiting READ on /dev/sdb failed due to kgfknm error
ERROR: -20(27041 /dev/sdb)ERROR: submiting READ on /dev/sda1 failed due to kgfknm error
ERROR: -20(27041 /dev/sda1)ERROR: submiting READ on /dev/sda2 failed due to kgfknm error
ERROR: -20(27041 /dev/sda2)ERROR: submiting READ on /dev/sdb1 failed due to kgfknm error
ERROR: -20(27041 /dev/sdb1)Identified disk UFS:/dev/sdg1 with disk number:0 and timestamp (33103213 1600008192)
Checking if the diskgroup is mounted or used by CSS
Checking disk number:0
Generating configuration file..
Completed phase 1
Executing phase 2
Looking for /dev/sdg1
Modifying the header
Completed phase 2
[oracle@ol7-19-phx1 ~]$

==============================

[oracle@ol7-19-phx1 ~]$ asmcmd mount FRA_PHX
[oracle@ol7-19-phx2 ~]$ asmcmd mount FRA_PHX

==============================

[oracle@ol7-19-phx1 ~]$ asmcmd lsdg FRA_PHX -g
Inst_ID  State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
      1  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  FRA_PHX/
      2  MOUNTED  EXTERN  N         512             512   4096  1048576     32767    32663                0           32663              0             N  FRA_PHX/

[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.FRA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        OFFLINE OFFLINE                               STABLE
      2        OFFLINE OFFLINE                               STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$

### For when DG is not online:
srvctl start diskgroup -diskgroup FRA_PHX -node "node_list"

==============================

[oracle@ol7-19-phx1 ~]$ srvctl remove diskgroup -g RECO
[oracle@ol7-19-phx1 ~]$ crsctl stat res -t -w "TYPE = ora.diskgroup.type"
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.DATA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.FRA_PHX.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol7-19-phx1              STABLE
      2        ONLINE  ONLINE       ol7-19-phx2              STABLE
      3        OFFLINE OFFLINE                               STABLE
--------------------------------------------------------------------------------
[oracle@ol7-19-phx1 ~]$

Q.E.D.

See RAC_19c_rman_duplicate_standby_same_sid.log

Confirmed!

*** Oracle Data Guard Broker and Static Service Registration (Doc ID 1387859.1)
Note: Static “_DGMGRL” entries are no longer needed as of Oracle Database 12.1.0.2 in Oracle Data Guard Broker configurations
that are managed by Oracle Restart, RAC One Node or RAC as the Broker will use the clusterware to restart an instance.

Upgrading 11.2.4 database with GoldenGate installed (don’t step on booby trap) to 19.3.

Does anyone know what the meaning of “Preupgrade Check Name” and wouldn’t be great of Oracle does not make you dig for it?

Please comment if you know the meaning of “Preupgrade Check Name”.

I will update post a week later with the info.

Run preupgrade_fixups.sql

oracle@db-fs-1:hawk:/home/oracle
$ sqlplus / as sysdba @/u01/app/oracle/cfgtoollogs/hawk/preupgrade/preupgrade_fixups.sql

SQL*Plus: Release 11.2.0.4.0 Production on Wed Jul 29 16:55:45 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 7
Generated on:            2020-07-29 16:54:11

For Source Database:     HAWK
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    1.  parameter_min_val         NO          Manual fixup recommended.
    2.  em_present                NO          Manual fixup recommended.
    3.  amd_exists                NO          Manual fixup recommended.
    4.  apex_manual_upgrade       NO          Manual fixup recommended.
    5.  streams_setup             NO          Manual fixup recommended.
    6.  trgowner_no_admndbtrg     YES         None.
    7.  pre_fixed_objects         YES         None.
    8.  tablespaces_info          NO          Informational only.
                                              Further action is optional.
    9.  exf_rul_exists            NO          Informational only.
                                              Further action is optional.
   10.  rman_recovery_version     NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade.  To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.

16:56:18 SYS @ hawk:>exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@db-fs-1:hawk:/home/oracle
$

Run postupgrade_fixups.sql

oracle@db-fs-1:hawk:/home/oracle
$ sqlplus / as sysdba @ /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-29_07-50-43PM/hawk/postupgrade_fixups.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Jul 29 21:16:52 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


Session altered.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


PL/SQL procedure successfully completed.


Package created.

No errors.

Package body created.


PL/SQL procedure successfully completed.

No errors.





Package created.

No errors.

Package body created.

No errors.
Executing Oracle POST-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 7
Generated on:            2020-07-29 19:50:53

For Source Database:     HAWK
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    9.  old_time_zones_exist      YES         None.
   10.  dir_symlinks              YES         None.
   11.  post_dictionary           YES         None.
   12.  post_fixed_objects        NO          Informational only.
                                              Further action is optional.
   13.  upg_by_std_upgrd          NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database upgrade is not
fully complete.  To resolve the outstanding issues, start by reviewing
the postupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.

PL/SQL procedure successfully completed.


Session altered.

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
oracle@db-fs-1:hawk:/home/oracle
$

Let’s search for trgowner_no_admndbtrg – no luck!

oracle@db-fs-1:hawk:/u01/app/oracle/cfgtoollogs/hawk/preupgrade
$ grep trgowner_no_admndbtrg *.*
preupgrade_fixups.sql:    --    CHECK/FIXUP name: trgowner_no_admndbtrg
preupgrade_fixups.sql:    fixup_result := dbms_preup.run_fixup('trgowner_no_admndbtrg',6) AND fixup_result;
preupgrade_messages.properties:#    bspeckha    04/05/18 - update trgowner_no_admndbtrg_check action
preupgrade_package.sql:Rem    hvieyra     01/11/18 - Bug 27344799: Add fixup for check trgowner_no_admndbtrg_check.
preupgrade_package.sql:    FUNCTION trgowner_no_admndbtrg_check    (result_txt OUT CLOB) RETURN NUMBER;
preupgrade_package.sql:    FUNCTION trgowner_no_admndbtrg_fixup    (result_txt IN OUT VARCHAR2, pSqlcode IN OUT NUMBER) RETURN number;
preupgrade_package.sql:FUNCTION trgowner_no_admndbtrg_check (result_txt OUT CLOB) RETURN NUMBER
preupgrade_package.sql:        result_txt := get_failed_check_xml('trgowner_no_admndbtrg',
preupgrade_package.sql:END trgowner_no_admndbtrg_check;
preupgrade_package.sql:FUNCTION trgowner_no_admndbtrg_fixup (
preupgrade_package.sql:END trgowner_no_admndbtrg_fixup;
oracle@db-fs-1:hawk:/u01/app/oracle/cfgtoollogs/hawk/preupgrade
$

Currently upgrading 11.2 to 19.3 database using dbua silent.

Run preupgrade.jar and no issues detected.

Run preupgrade_fixups.sql

Run dbua -silent error resulted with [FATAL] [DBT-20061]

Per Doc ID 2246770.1, ignore the error which may be hazardous.

Actually, I have already figured our the root cause; however, the method I used may not always be available for all environments.

Will post solution in a week.

HINT: probable cause is in the output which did not occur for first run.

$ env|grep HOME
OLD_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1
NEW_HOME=/u01/app/oracle/product/19.3.0.0/dbhome_1
HOME=/home/oracle
ORACLE_HOME=/u01/app/oracle/product/11.2.0.4/dbhome_1

$ $OLD_HOME/jdk/bin/java -jar $NEW_HOME/rdbms/admin/preupgrade.jar TERMINAL TEXT FILE
==================
PREUPGRADE SUMMARY
==================
  /u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade.log
  /u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade_fixups.sql
  /u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/postupgrade_fixups.sql

Execute fixup scripts as indicated below:

Before upgrade:

Log into the database and execute the preupgrade fixups
@/u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade_fixups.sql

After the upgrade:

Log into the database and execute the postupgrade fixups
@/u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/postupgrade_fixups.sql

Preupgrade complete: 2020-07-31T18:50:38
oracle@ol7-112-dg2:hawk:/home/oracle
$
	
*******************************************************

$ sqlplus / as sysdba @/u01/app/oracle/cfgtoollogs/hawk_stby/preupgrade/preupgrade_fixups.sql

SQL*Plus: Release 11.2.0.4.0 Production on Fri Jul 31 18:52:21 2020

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Executing Oracle PRE-Upgrade Fixup Script

Auto-Generated by:       Oracle Preupgrade Script
                         Version: 19.0.0.0.0 Build: 7
Generated on:            2020-07-31 18:50:36

For Source Database:     HAWK
Source Database Version: 11.2.0.4.0
For Upgrade to Version:  19.0.0.0.0

Preup                             Preupgrade
Action                            Issue Is
Number  Preupgrade Check Name     Remedied    Further DBA Action
------  ------------------------  ----------  --------------------------------
    1.  min_recovery_area_size    NO          Manual fixup required.
    2.  parameter_min_val         NO          Manual fixup recommended.
    3.  em_present                NO          Manual fixup recommended.
    4.  amd_exists                NO          Manual fixup recommended.
    5.  apex_manual_upgrade       NO          Manual fixup recommended.
    6.  tablespaces_info          NO          Informational only.
                                              Further action is optional.
    7.  exf_rul_exists            NO          Informational only.
                                              Further action is optional.
    8.  rman_recovery_version     NO          Informational only.
                                              Further action is optional.

The fixup scripts have been run and resolved what they can. However,
there are still issues originally identified by the preupgrade that
have not been remedied and are still present in the database.
Depending on the severity of the specific issue, and the nature of
the issue itself, that could mean that your database is not ready
for upgrade.  To resolve the outstanding issues, start by reviewing
the preupgrade_fixups.sql and searching it for the name of
the failed CHECK NAME or Preupgrade Action Number listed above.
There you will find the original corresponding diagnostic message
from the preupgrade which explains in more detail what still needs
to be done.
OL7-112-DG2:(SYS@hawk_stby:LOGICAL STANDBY>
	
*******************************************************
	
$ $NEW_HOME/bin/dbua -silent \
> -sid $ORACLE_SID \
> -oracleHome $ORACLE_HOME \
> -recompile_invalid_objects TRUE \
> -upgradeTimezone TRUE \
> -emConfiguration NONE \
> -upgrade_parallelism 2
Logs directory:  /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM
Performing Pre-Upgrade Checks...
============================
PRE- and POST- FIXUP ACTIONS
=============================
/u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk/upgrade.xml
/u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk/preupgrade_fixups.sql
/u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk/postupgrade_fixups.sql
[FATAL] [DBT-20061] One or more of the pre-upgrade checks on the database have resulted into error conditions that must be addressed before proceeding.
   ACTION: Refer to the pre-upgrade results location for details: /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk

*******************************************************

$ ls -l /u01/app/oracle/cfgtoollogs/dbua/upgrade2020-07-31_06-53-41PM/hawk
total 692
-rw-r-----. 1 oracle oinstall      1 Jul 31 18:53 checksBuffer.tmp
-rw-r-----. 1 oracle oinstall  41134 Jul 31 18:53 components.properties
-rw-r-----. 1 oracle oinstall  15085 Jul 31 18:53 dbms_registry_extended.sql
drwxr-x---. 3 oracle oinstall     21 Jul 31 18:53 oracle
-rw-r-----. 1 oracle oinstall  14051 Jul 31 18:53 parameters.properties
-rw-r-----. 1 oracle oinstall   8584 Jul 31 18:53 postupgrade_fixups.sql
-rw-r-----. 1 oracle oinstall   7884 Jul 31 18:53 preupgrade_driver.sql
-rw-r-----. 1 oracle oinstall  13082 Jul 31 18:53 preupgrade_fixups.sql
-rw-r-----. 1 oracle oinstall 101702 Jul 31 18:53 preupgrade_messages.properties
-rw-r-----. 1 oracle oinstall 461660 Jul 31 18:53 preupgrade_package.sql
-rw-r-----. 1 oracle oinstall   5215 Jul 31 18:53 sqls.log
drwxr-x---. 3 oracle oinstall     24 Jul 31 18:53 upgrade
-rw-r-----. 1 oracle oinstall  15082 Jul 31 18:53 upgrade.xml

*******************************************************

DBUA Silent : Director Service Registration Error DBT-20060 DBT-08010 (Doc ID 2246770.1)	

Case 2 :
SEVERE: [FATAL] [DBT-20061] One or more of the pre-upgrade checks on the database have resulted into error conditions that must be addressed before proceeding.

./dbua -silent -sid <SID> -oracleHome $ORACLE_HOME -J-Doracle.install.dbua.validate.all=false

I am not going to reiterate what is already out there; however, what is out there covers 12.1.0.2.0 vs 19.3.0.0.0.

Here is the syntax and what seems to be ambiguous is target version.

$./physru.sh <sysdba user> <primary TNS alias> <physical standby TNS alias> <primary db unique name> <physical standby db unique name> <target version> 

NOTE: This is NOT all the steps for upgrade but only applicable steps when running physru.sh.

First physru execution completed successfully.

### First physru execution:
Verifies that Data Guard Broker is disabled and FRA is configured.
Creates a guaranteed restore point
Converts the existing Physical Standby to a Logical Standby

oracle@ol7-112-dg2:hawk:/sf_working/upgrade19c
$ ./physru_v3.sh SYS hawk hawk_stby hawk hawk_stby 19.3.0.0
Please enter the sysdba password:

### Initialize script to either start over or resume execution
Jul 30 20:06:30 2020 [0-1] Identifying rdbms software version
Jul 30 20:06:31 2020 [0-1] database hawk is at version 11.2.0.4.0
Jul 30 20:06:31 2020 [0-1] database hawk_stby is at version 11.2.0.4.0
Jul 30 20:06:31 2020 [0-1] verifying fast recovery area is enabled at hawk and hawk_stby
Jul 30 20:06:31 2020 [0-1] verifying backup location at hawk and hawk_stby
Jul 30 20:06:31 2020 [0-1] verifying available flashback restore points
Jul 30 20:06:31 2020 [0-1] verifying DG Broker is disabled
Jul 30 20:06:31 2020 [0-1] looking up prior execution history
Jul 30 20:06:31 2020 [0-1] purging script execution state from database hawk
Jul 30 20:06:31 2020 [0-1] purging script execution state from database hawk_stby
Jul 30 20:06:31 2020 [0-1] starting new execution of script

### Stage 1: Backup user environment in case rolling upgrade is aborted
Jul 30 20:06:31 2020 [1-1] database hawk location for backup controlfile is /u01/app/oracle/fast_recovery_area
Jul 30 20:06:32 2020 [1-1] database hawk_stby location for backup controlfile is /u01/app/oracle/fast_recovery_area
Jul 30 20:06:32 2020 [1-1] creating restore point PRU_0000_0003 on database hawk_stby
Jul 30 20:06:32 2020 [1-1] backing up current control file on hawk_stby
Jul 30 20:06:32 2020 [1-1] created backup control file /u01/app/oracle/fast_recovery_area/PRU_0003_hawk_stby_f.f
Jul 30 20:06:32 2020 [1-1] creating restore point PRU_0000_0003 on database hawk
Jul 30 20:06:32 2020 [1-1] backing up current control file on hawk
Jul 30 20:06:32 2020 [1-1] created backup control file /u01/app/oracle/fast_recovery_area/PRU_0003_hawk_f.f

NOTE: Restore point PRU_0000_0001 and backup control file PRU_0003_hawk_stby_f.f
      can be used to restore hawk_stby back to its original state as a
      physical standby, in case the rolling upgrade operation needs to be aborted
      prior to the first switchover done in Stage 4.

### Stage 2: Create transient logical standby from existing physical standby
Jul 30 20:06:32 2020 [2-1] verifying RAC is disabled at hawk_stby
Jul 30 20:06:32 2020 [2-1] verifying database roles
Jul 30 20:06:32 2020 [2-1] verifying physical standby is mounted
Jul 30 20:06:32 2020 [2-1] verifying database protection mode
Jul 30 20:06:32 2020 [2-1] verifying transient logical standby datatype support
Jul 30 20:06:33 2020 [2-2] starting media recovery on hawk_stby
Jul 30 20:06:39 2020 [2-2] confirming media recovery is running
Jul 30 20:06:39 2020 [2-2] waiting for apply lag to fall under 30 seconds
Jul 30 20:06:52 2020 [2-2] apply lag measured at 13 seconds
Jul 30 20:06:52 2020 [2-2] stopping media recovery on hawk_stby
Jul 30 20:06:53 2020 [2-2] executing dbms_logstdby.build on database hawk
Jul 30 20:06:59 2020 [2-2] converting physical standby into transient logical standby
Jul 30 20:07:03 2020 [2-3] opening database hawk_stby
Jul 30 20:07:05 2020 [2-4] configuring transient logical standby parameters for rolling upgrade
Jul 30 20:07:05 2020 [2-4] starting logical standby on database hawk_stby
Jul 30 20:07:10 2020 [2-4] enabling log archive destination to database hawk_stby
Jul 30 20:07:11 2020 [2-4] waiting until logminer dictionary has fully loaded
Jul 30 20:07:51 2020 [2-4] dictionary load 03% complete
Jul 30 20:08:01 2020 [2-4] dictionary load 62% complete
Jul 30 20:08:11 2020 [2-4] dictionary load is complete
Jul 30 20:08:11 2020 [2-4] waiting for apply lag to fall under 30 seconds
Jul 30 20:08:15 2020 [2-4] apply lag measured at 3 seconds

NOTE: Database hawk_stby is now ready to be upgraded.  This script has left the
      database open in case you want to perform any further tasks before
      upgrading the database.  Once the upgrade is complete, the database must
      opened in READ WRITE mode before this script can be called to resume the
      rolling upgrade.

NOTE: If hawk_stby was previously a RAC database that was disabled, it may be
      reverted back to a RAC database upon completion of the rdbms upgrade.
      This can be accomplished by performing the following steps:

          1) On instance hawk, set the cluster_database parameter to TRUE.
          eg: SQL> alter system set cluster_database=true scope=spfile;

          2) Shutdown instance hawk.
          eg: SQL> shutdown abort;

          3) Startup and open all instances for database hawk_stby.
          eg: srvctl start database -d hawk_stby

oracle@ol7-112-dg2:hawk:/sf_working/upgrade19c

Second physru execution FAILED.

### Second physru execution to switchover (APPLICATION BROWNOUT):
Executes a switchover making the upgraded standby database the primary database.
Executes a flashback of the original primary database to the guaranteed restore point from step 1 and shuts it down.

[oracle@ol7-112-dg2 upgrade19c]$ ./physru_v3.sh SYS hawk hawk_stby hawk hawk_stby 19.3.0.0.0
Please enter the sysdba password:

### Initialize script to either start over or resume execution
Aug 01 01:55:56 2020 [0-1] Identifying rdbms software version
Aug 01 01:55:56 2020 [0-1] database hawk is at version 11.2.0.4.0
Aug 01 01:55:57 2020 [0-1] database hawk_stby is at version 19.0.0.0.0
Aug 01 01:56:00 2020 [0-1] verifying fast recovery area is enabled at hawk and hawk_stby
Aug 01 01:56:02 2020 [0-1] verifying backup location at hawk and hawk_stby
Aug 01 01:56:03 2020 [0-1] verifying available flashback restore points
Aug 01 01:56:04 2020 [0-1] verifying DG Broker is disabled
Aug 01 01:56:05 2020 [0-1] looking up prior execution history
Aug 01 01:56:08 2020 [0-1] last completed stage [2-4] using script version 0003
Aug 01 01:56:08 2020 [0-1] resuming execution of script

### Stage 3: Validate upgraded transient logical standby
Aug 01 01:56:09 2020 [3-1] database hawk_stby is no longer in OPEN MIGRATE mode
Aug 01 01:56:09 2020 [3-1] ERROR: hawk_stby is not at version 19.3.0.0.0
[oracle@ol7-112-dg2 upgrade19c]$

Second physru execution SUCCEEDED as 19.0.0.0.0 (base release) is used vs 19.3.0.0.0 (actual release).

[oracle@ol7-112-dg2 upgrade19c]$ ./physru_v3.sh SYS hawk hawk_stby hawk hawk_stby 19.0.0.0.0
Please enter the sysdba password:

### Initialize script to either start over or resume execution
Aug 01 02:48:40 2020 [0-1] Identifying rdbms software version
Aug 01 02:48:40 2020 [0-1] database hawk is at version 11.2.0.4.0
Aug 01 02:48:41 2020 [0-1] database hawk_stby is at version 19.0.0.0.0
Aug 01 02:48:45 2020 [0-1] verifying fast recovery area is enabled at hawk and hawk_stby
Aug 01 02:48:47 2020 [0-1] verifying backup location at hawk and hawk_stby
Aug 01 02:48:48 2020 [0-1] verifying available flashback restore points
Aug 01 02:48:49 2020 [0-1] verifying DG Broker is disabled
Aug 01 02:48:50 2020 [0-1] looking up prior execution history
Aug 01 02:48:53 2020 [0-1] last completed stage [2-4] using script version 0003
Aug 01 02:48:53 2020 [0-1] resuming execution of script

### Stage 3: Validate upgraded transient logical standby
Aug 01 02:48:54 2020 [3-1] database hawk_stby is no longer in OPEN MIGRATE mode
Aug 01 02:48:54 2020 [3-1] database hawk_stby is at version 19.0.0.0.0

### Stage 4: Switch the transient logical standby to be the new primary
Aug 01 02:48:59 2020 [4-1] waiting for hawk_stby to catch up (this could take a while)
Aug 01 02:49:00 2020 [4-1] waiting for apply lag to fall under 30 seconds
Aug 01 02:49:12 2020 [4-1] apply lag measured at 9 seconds
Aug 01 02:49:16 2020 [4-2] using fast switchover optimizations

NOTE: A switchover is about to be performed which will incur a brief outage
      of the primary database.  If you answer 'y' to the question below,
      database hawk_stby will become the new primary database, and database hawk
      will be converted into a standby in preparation for upgrade.  If you answer
      'n' to the question below, the script will exit, leaving the databases in
      their current roles.
Are you ready to proceed with the switchover? (y/n): y

Aug 01 02:49:31 2020 [4-2] continuing
Aug 01 02:49:31 2020 [4-2] switching hawk to become a logical standby
Aug 01 02:49:39 2020 [4-2] hawk is now a logical standby
Aug 01 02:49:39 2020 [4-2] waiting for standby hawk_stby to process end-of-redo from primary
Aug 01 02:49:44 2020 [4-2] switching hawk_stby to become the new primary
Aug 01 02:49:45 2020 [4-2] hawk_stby is now the new primary

### Stage 5: Flashback former primary to pre-upgrade restore point and convert to physical
Aug 01 02:49:49 2020 [5-1] shutting down database hawk
Aug 01 02:50:03 2020 [5-1] mounting database hawk
Aug 01 02:50:12 2020 [5-2] flashing back database hawk to restore point PRU_0000_0003
Aug 01 02:50:15 2020 [5-3] converting hawk into physical standby
Aug 01 02:50:17 2020 [5-4] shutting down database hawk

NOTE: Database hawk has been shutdown, and is now ready to be started
      using the newer version Oracle binary.  This script requires the
      database to be mounted (on all active instances, if RAC) before calling
      this script to resume the rolling upgrade.

[oracle@ol7-112-dg2 upgrade19c]$

Third and Final physru execution completed successfully.

### Execute physru for the third and final time. 
Start redo apply
Prompt whether to switch back to original configuration
Remove guaranteed restore points

[oracle@ol7-112-dg2 upgrade19c]$ ./physru_v3.sh SYS hawk hawk_stby hawk hawk_stby 19.0.0.0.0
Please enter the sysdba password:

### Initialize script to either start over or resume execution
Aug 01 02:48:40 2020 [0-1] Identifying rdbms software version
Aug 01 02:48:40 2020 [0-1] database hawk is at version 11.2.0.4.0
Aug 01 02:48:41 2020 [0-1] database hawk_stby is at version 19.0.0.0.0
Aug 01 02:48:45 2020 [0-1] verifying fast recovery area is enabled at hawk and hawk_stby
Aug 01 02:48:47 2020 [0-1] verifying backup location at hawk and hawk_stby
Aug 01 02:48:48 2020 [0-1] verifying available flashback restore points
Aug 01 02:48:49 2020 [0-1] verifying DG Broker is disabled
Aug 01 02:48:50 2020 [0-1] looking up prior execution history
Aug 01 02:48:53 2020 [0-1] last completed stage [2-4] using script version 0003
Aug 01 02:48:53 2020 [0-1] resuming execution of script

### Stage 3: Validate upgraded transient logical standby
Aug 01 02:48:54 2020 [3-1] database hawk_stby is no longer in OPEN MIGRATE mode
Aug 01 02:48:54 2020 [3-1] database hawk_stby is at version 19.0.0.0.0

### Stage 4: Switch the transient logical standby to be the new primary
Aug 01 02:48:59 2020 [4-1] waiting for hawk_stby to catch up (this could take a while)
Aug 01 02:49:00 2020 [4-1] waiting for apply lag to fall under 30 seconds
Aug 01 02:49:12 2020 [4-1] apply lag measured at 9 seconds
Aug 01 02:49:16 2020 [4-2] using fast switchover optimizations

NOTE: A switchover is about to be performed which will incur a brief outage
      of the primary database.  If you answer 'y' to the question below,
      database hawk_stby will become the new primary database, and database hawk
      will be converted into a standby in preparation for upgrade.  If you answer
      'n' to the question below, the script will exit, leaving the databases in
      their current roles.
Are you ready to proceed with the switchover? (y/n): y

Aug 01 02:49:31 2020 [4-2] continuing
Aug 01 02:49:31 2020 [4-2] switching hawk to become a logical standby
Aug 01 02:49:39 2020 [4-2] hawk is now a logical standby
Aug 01 02:49:39 2020 [4-2] waiting for standby hawk_stby to process end-of-redo from primary
Aug 01 02:49:44 2020 [4-2] switching hawk_stby to become the new primary
Aug 01 02:49:45 2020 [4-2] hawk_stby is now the new primary

### Stage 5: Flashback former primary to pre-upgrade restore point and convert to physical
Aug 01 02:49:49 2020 [5-1] shutting down database hawk
Aug 01 02:50:03 2020 [5-1] mounting database hawk
Aug 01 02:50:12 2020 [5-2] flashing back database hawk to restore point PRU_0000_0003
Aug 01 02:50:15 2020 [5-3] converting hawk into physical standby
Aug 01 02:50:17 2020 [5-4] shutting down database hawk

NOTE: Database hawk has been shutdown, and is now ready to be started
      using the newer version Oracle binary.  This script requires the
      database to be mounted (on all active instances, if RAC) before calling
      this script to resume the rolling upgrade.

[oracle@ol7-112-dg2 upgrade19c]$

[oracle@ol7-112-dg2 upgrade19c]$ ./physru_v3.sh SYS hawk hawk_stby hawk hawk_stby 19.0.0.0.0
Please enter the sysdba password:

### Initialize script to either start over or resume execution
Aug 01 03:26:16 2020 [0-1] Identifying rdbms software version
Aug 01 03:26:17 2020 [0-1] database hawk is at version 19.0.0.0.0
Aug 01 03:26:18 2020 [0-1] database hawk_stby is at version 19.0.0.0.0
Aug 01 03:26:26 2020 [0-1] verifying fast recovery area is enabled at hawk and hawk_stby
Aug 01 03:26:29 2020 [0-1] verifying backup location at hawk and hawk_stby
Aug 01 03:26:31 2020 [0-1] verifying available flashback restore points
Aug 01 03:26:34 2020 [0-1] verifying DG Broker is disabled
Aug 01 03:26:36 2020 [0-1] looking up prior execution history
Aug 01 03:26:39 2020 [0-1] last completed stage [5-4] using script version 0003
Aug 01 03:26:39 2020 [0-1] resuming execution of script

### Stage 6: Run media recovery through upgrade redo
Aug 01 03:26:47 2020 [6-1] upgrade redo region identified as scn range [995261, 2453907]
Aug 01 03:26:47 2020 [6-1] enabling log archive destination to database hawk
Aug 01 03:26:51 2020 [6-1] starting media recovery on hawk
Aug 01 03:26:57 2020 [6-1] confirming media recovery is running
Aug 01 03:26:59 2020 [6-1] waiting for media recovery to initialize v$recovery_progress
Aug 01 03:27:20 2020 [6-1] monitoring media recovery's progress
Aug 01 03:27:32 2020 [6-3] recovery of upgrade redo at 07% - estimated complete at Aug 01 03:31:24
Aug 01 03:27:57 2020 [6-3] recovery of upgrade redo at 26% - estimated complete at Aug 01 03:30:06
Aug 01 03:28:21 2020 [6-3] recovery of upgrade redo at 42% - estimated complete at Aug 01 03:30:00
Aug 01 03:28:45 2020 [6-3] recovery of upgrade redo at 52% - estimated complete at Aug 01 03:30:10
Aug 01 03:29:10 2020 [6-3] recovery of upgrade redo at 61% - estimated complete at Aug 01 03:30:25
Aug 01 03:29:36 2020 [6-3] recovery of upgrade redo at 73% - estimated complete at Aug 01 03:30:27
Aug 01 03:30:00 2020 [6-3] recovery of upgrade redo at 82% - estimated complete at Aug 01 03:30:35
Aug 01 03:30:24 2020 [6-3] recovery of upgrade redo at 90% - estimated complete at Aug 01 03:30:42
Aug 01 03:30:51 2020 [6-3] recovery of upgrade redo at 96% - estimated complete at Aug 01 03:30:55
Aug 01 03:31:12 2020 [6-4] media recovery has finished recovering through upgrade

### Stage 7: Switch back to the original roles prior to the rolling upgrade

NOTE: At this point, you have the option to perform a switchover
     which will restore hawk back to a primary database and
     hawk_stby back to a physical standby database.  If you answer 'n'
     to the question below, hawk will remain a physical standby
     database and hawk_stby will remain a primary database.

Do you want to perform a switchover? (y/n): y

Aug 01 03:31:26 2020 [7-1] continuing
Aug 01 03:31:36 2020 [7-2] waiting for apply lag to fall under 30 seconds
Aug 01 03:31:44 2020 [7-2] apply lag measured at 5 seconds
Aug 01 03:31:48 2020 [7-3] switching hawk_stby to become a physical standby
Aug 01 03:31:55 2020 [7-3] hawk_stby is now a physical standby
Aug 01 03:31:55 2020 [7-3] shutting down database hawk_stby
Aug 01 03:31:57 2020 [7-3] mounting database hawk_stby
Aug 01 03:32:08 2020 [7-3] starting media recovery on hawk_stby
Aug 01 03:32:15 2020 [7-3] confirming media recovery is running
Aug 01 03:32:16 2020 [7-3] waiting for standby hawk to process end-of-redo from primary
Aug 01 03:32:21 2020 [7-3] switching hawk to become the new primary
Aug 01 03:32:23 2020 [7-3] hawk is now the new primary
Aug 01 03:32:23 2020 [7-3] opening database hawk

### Stage 8: Statistics
script start time:                                           31-Jul-20 23:54:44
script finish time:                                          01-Aug-20 03:32:36
total script execution time:                                       +00 03:37:52
wait time for user upgrade:                                        +00 02:52:39
active script execution time:                                      +00 00:45:13
transient logical creation start time:                       31-Jul-20 23:54:46
transient logical creation finish time:                      31-Jul-20 23:55:14
primary to logical switchover start time:                    01-Aug-20 02:49:14
logical to primary switchover finish time:                   01-Aug-20 02:49:47
primary services offline for:                                      +00 00:00:33
total time former primary in physical role:                        +00 00:40:57
time to reach upgrade redo:
time to recover upgrade redo:                                      +00 00:03:44
primary to physical switchover start time:                   01-Aug-20 03:31:25
physical to primary switchover finish time:                  01-Aug-20 03:32:34
primary services offline for:                                      +00 00:01:09

SUCCESS: The physical rolling upgrade is complete

[oracle@ol7-112-dg2 upgrade19c]$

References:

Oracle11g Data Guard: Database Rolling Upgrade Shell Script (Doc ID 949322.1)

Data Guard physru_v3.sh Script Errors ORA-01403 ORA-06512 (Doc ID 2570572.1)

Oracle Database Rolling Upgrades

High_Level_Action_Plan

================================================================================
1. Convert physical standby to transient logical standby by running PHYSRU_v3.sh
================================================================================
Step100-Disable_DG_Broker_GoldenGate_Configuration.txt
Step110-Check_Primary_Standby_Configuration.txt
Step120-Convert_PhysicalStandby_To_TransientLogicalStandby_using_physru_v3.sh.txt
Verifies that Data Guard Broker is disabled and FRA is configured.
Creates a guaranteed restore point
Converts the existing Physical Standby to a Logical Standby

================================================================================
2. Upgrade the transient logical standby using DBUA
================================================================================
Step200-Upgrade_TransientLogicalStandby_using_dbua.txt

================================================================================
3. Convert transient logical standby to Primary by running PHYSRU_v3.sh
================================================================================
Step300-Convert_TransientLogicalStandby_To_Primary_using_physru_v3.sh.txt
Executes a switchover making the upgraded standby database the primary database.
Executes a flashback of the original primary database to the guaranteed restore point and shuts it down.

ORIGINAL_PRIMARY is down and ORIGINAL_STANDBY is now NEW_PRIMARY.

================================================================================
4. Configure 11.2 ORIGINAL_PRIMARY for 19c
================================================================================
Step400-Configure_11g_ORIGINAL_PRIMARY_for_19c.txt

================================================================================
5. Upgrade 11.2 ORIGINAL_PRIMARY to 19c by running PHYSRU_v3.sh
================================================================================
Step500-Upgrade19c_ORIGINAL_PRIMARY_using_physru_v3.sh.txt
Start redo apply
Prompt whether to switch back to original configuration
Remove guaranteed restore points

Data Guard is like a spare tire. Don’t wait to find there is no air when you need it.

[oracle@ol7-112-dg1 ~]$ dgmgrl
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sun Aug 2 16:57:28 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@hawk
Password:
Connected to "hawk"
Connected as SYSDBA.
DGMGRL> show configuration

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 52 seconds ago)

DGMGRL> switchover to hawk_stby
Performing switchover NOW, please wait...
Operation requires a connection to database "hawk_stby"
Connecting ...
Connected to "hawk_stby"
Connected as SYSDBA.
New primary database "hawk_stby" is opening...
Operation requires start up of instance "hawk" on database "hawk"
Starting instance "hawk"...
Connected to an idle instance.
ORACLE instance started.
Connected to "hawk"
Database mounted.
Connected to "hawk"
Switchover succeeded, new primary is "hawk_stby"
DGMGRL> show configuration

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk_stby - Primary database
    hawk      - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 57 seconds ago)

DGMGRL> exit
[oracle@ol7-112-dg1 ~]$

==================================================

[oracle@ol7-112-dg1 upgrade19c]$ sqlplus / as sysdba @ status

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 2 16:59:45 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


Session altered.

SQL> select NAME,DB_UNIQUE_NAME,OPEN_MODE,DATABASE_ROLE,PROTECTION_MODE,FLASHBACK_ON from v$database
  2  ;

NAME      DB_UNIQUE_NAME                 OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE      FLASHBACK_ON
--------- ------------------------------ -------------------- ---------------- -------------------- ------------------
HAWK      hawk                           MOUNTED              PHYSICAL STANDBY MAXIMUM PERFORMANCE  YES

SQL> ;
  1  select NAME,DB_UNIQUE_NAME,OPEN_MODE,DATABASE_ROLE,PROTECTION_MODE,FLASHBACK_ON from v$database
  2*
SQL> select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6  ;

PID                           INST   THREAD# CLIENT_P PROCESS   STATUS       SEQUENCE#    BLOCK# DELAY_MINS
------------------------ --------- --------- -------- --------- ------------ --------- --------- ----------
3118                             1         1 LGWR     RFS       RECEIVING          169      3237          0
3151                             1         1 N/A      MRP0      APPLYING_LOG       169      3237          0

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@ol7-112-dg1 upgrade19c]$

==================================================

[oracle@ol7-112-dg1 ~]$ dgmgrl
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sun Aug 2 17:00:03 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@hawk_stby
Password:
Connected to "hawk_stby"
Connected as SYSDBA.
DGMGRL> switchover to hawk
Performing switchover NOW, please wait...
Operation requires a connection to database "hawk"
Connecting ...
Connected to "hawk"
Connected as SYSDBA.
New primary database "hawk" is opening...
Operation requires start up of instance "hawk" on database "hawk_stby"
Starting instance "hawk"...
Connected to an idle instance.
ORACLE instance started.
Connected to "hawk_stby"
Database mounted.
Connected to "hawk_stby"
Switchover succeeded, new primary is "hawk"
DGMGRL> show configuration

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 59 seconds ago)

DGMGRL> exit
[oracle@ol7-112-dg1 ~]$

==================================================

[oracle@ol7-112-dg1 upgrade19c]$ sqlplus / as sysdba @ status

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 2 17:02:06 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0


Session altered.

SQL> select NAME,DB_UNIQUE_NAME,OPEN_MODE,DATABASE_ROLE,PROTECTION_MODE,FLASHBACK_ON from v$database
  2  ;

NAME      DB_UNIQUE_NAME                 OPEN_MODE            DATABASE_ROLE    PROTECTION_MODE      FLASHBACK_ON
--------- ------------------------------ -------------------- ---------------- -------------------- ------------------
HAWK      hawk                           READ WRITE           PRIMARY          MAXIMUM PERFORMANCE  YES

SQL> ;
  1  select NAME,DB_UNIQUE_NAME,OPEN_MODE,DATABASE_ROLE,PROTECTION_MODE,FLASHBACK_ON from v$database
  2*
SQL> select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6  ;

PID                           INST   THREAD# CLIENT_P PROCESS   STATUS       SEQUENCE#    BLOCK# DELAY_MINS
------------------------ --------- --------- -------- --------- ------------ --------- --------- ----------
3328                             1         1 LNS      LNS       WRITING            172      3252          0

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
[oracle@ol7-112-dg1 upgrade19c]$

This response file is generated from GUI.

The values can be 1,2,4,8,16,32, or 64 MB, depending on the specific disk group compatibility level.
Larger AU sizes typically provide performance advantages for data warehouse applications that use large sequential reads.

Which Is The Grow Up Factor When We Pass From AUsize Of 1 To AUsize Of 4? (Doc ID 1961116.1)

gridSetup.sh -silent -applyRU $PATCH_DIR/31305339 -responseFile ~/grid19c_upgrade_has.rsp -ignorePrereqFailure

--- grid19c_upgrade_has.rsp ---
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0
*** INVENTORY_LOCATION=<FILL IN PATH LOCATION>
*** ORACLE_BASE=<FILL IN PATH LOCATION>
oracle.install.option=UPGRADE
oracle.install.crs.config.scanType=LOCAL_SCAN
oracle.install.crs.config.ClusterConfiguration=STANDALONE
oracle.install.crs.config.configureAsExtendedCluster=false
oracle.install.crs.config.gpnp.configureGNS=false
oracle.install.crs.config.autoConfigureClusterNodeVIP=false
oracle.install.crs.config.gpnp.gnsOption=CREATE_NEW_GNS
oracle.install.crs.configureGIMR=false
oracle.install.asm.configureGIMRDataDG=false
oracle.install.crs.config.useIPMI=false
oracle.install.asm.diskGroup.AUSize=1
oracle.install.asm.gimrDG.AUSize=1
oracle.install.asm.configureAFD=false
oracle.install.crs.configureRHPS=false
oracle.install.crs.config.ignoreDownNodes=false
oracle.install.config.managementOption=NONE
oracle.install.config.omsPort=0
oracle.install.crs.rootconfig.executeRootScript=false

If you came here looking for solution, my apologies as I don’t have one.

If you came here looking for comfort, then I am here as you are not alone.

Honestly, it might just be easier to applyRU vs creategoldimage.
gridSetup.sh -applyRU $PATCH_DIR/31305339


Applied Patch 31305339 - GI Release Update 19.8.0.0.200714 for RAC environment.

==================================================

[FATAL] [INS-32700] The gold image creation failed for Grid 19.8

[root@ol7-19-lax1 ~]# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [761455134].
[root@ol7-19-lax1 ~]#

[root@ol7-19-lax2 ~]# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [19.0.0.0.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [761455134].
[root@ol7-19-lax2 ~]#

==================================================

[oracle@ol7-19-lax1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.
[oracle@ol7-19-lax1 ~]$

[oracle@ol7-19-lax2 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.
[oracle@ol7-19-lax2 ~]$

==================================================

[oracle@ol7-19-lax1 ~]$ . oraenv <<< +ASM1
ORACLE_SID = [+ASM1] ? The Oracle base remains unchanged with value /u01/app/oracle

[oracle@ol7-19-lax1 ~]$ $ORACLE_HOME/gridSetup.sh -creategoldimage -exclFiles $ORACLE_HOME/log,$ORACLE_HOME/.patch_storage -destinationlocation /u01/app/oracle/goldimage -silent
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-32700] The gold image creation failed. Check the install log /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-30-53PM for more details.
Setup failed.
[oracle@ol7-19-lax1 ~]$

==================================================

[oracle@ol7-19-lax1 ~]$ cd /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-30-53PM
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$ ls -l
total 1116
-rw-r--r--. 1 oracle oinstall 0 Aug 15 17:30 gridSetupActions2020-08-15_05-30-53PM.err
-rw-r--r--. 1 oracle oinstall 1052145 Aug 15 17:31 gridSetupActions2020-08-15_05-30-53PM.log
-rw-r--r--. 1 oracle oinstall 77842 Aug 15 17:31 gridSetupActions2020-08-15_05-30-53PM.out
-rw-r--r--. 1 oracle oinstall 129 Aug 15 17:30 installerPatchActions_2020-08-15_05-30-53PM.log
-rw-r--r--. 1 oracle oinstall 2158 Aug 15 17:31 time2020-08-15_05-30-53PM.log
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$

[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$ grep INS-32700 *
gridSetupActions2020-08-15_05-30-53PM.log:SEVERE: [Aug 15, 2020 5:31:09 PM] [FATAL] [INS-32700] The gold image creation failed. Check the install log /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-30-53PM for more details.
gridSetupActions2020-08-15_05-30-53PM.out:[FATAL] [INS-32700] The gold image creation failed. Check the install log /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-30-53PM for more details.
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$

[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$ grep -C20 INS-32700 gridSetupActions2020-08-15_05-30-53PM.log
INFO: [Aug 15, 2020 5:31:01 PM] File lib/clntsh.map marked to be zero out.
WARNING: [Aug 15, 2020 5:31:02 PM] Could not create symlink: /tmp/GridSetupActions2020-08-15_05-30-53PM/tempHome_1597512661914/log/procwatcher/prw.sh.
Refer associated stacktrace #oracle.install.ivw.common.driver.job.CreateGoldImageJob:7142
INFO: [Aug 15, 2020 5:31:02 PM] Executing [/u01/app/19.0.0/grid/OPatch/opatch, lspatches]
INFO: [Aug 15, 2020 5:31:02 PM] Starting Output Reader Threads for process /u01/app/19.0.0/grid/OPatch/opatch
INFO: [Aug 15, 2020 5:31:09 PM] 31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
INFO: [Aug 15, 2020 5:31:09 PM] 31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
INFO: [Aug 15, 2020 5:31:09 PM] 31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
INFO: [Aug 15, 2020 5:31:09 PM] 31281355;Database Release Update : 19.8.0.0.200714 (31281355)
INFO: [Aug 15, 2020 5:31:09 PM] The process /u01/app/19.0.0/grid/OPatch/opatch exited with code 0
INFO: [Aug 15, 2020 5:31:09 PM] Waiting for output processor threads to exit.
INFO: [Aug 15, 2020 5:31:09 PM]
INFO: [Aug 15, 2020 5:31:09 PM] OPatch succeeded.
INFO: [Aug 15, 2020 5:31:09 PM] Output processor threads exited.
INFO: [Aug 15, 2020 5:31:09 PM] Executing [/u01/app/19.0.0/grid/bin/zip, -q, -r, /u01/app/oracle/goldimage/grid_home_2020-08-15_05-30-53PM.zip, .]
INFO: [Aug 15, 2020 5:31:09 PM] Starting Output Reader Threads for process /u01/app/19.0.0/grid/bin/zip
INFO: [Aug 15, 2020 5:31:09 PM] The process /u01/app/19.0.0/grid/bin/zip exited with code 0
INFO: [Aug 15, 2020 5:31:09 PM] Waiting for output processor threads to exit.
INFO: [Aug 15, 2020 5:31:09 PM] Output processor threads exited.
INFO: [Aug 15, 2020 5:31:09 PM] Removing the goldimage file, as there was a failure.
SEVERE: [Aug 15, 2020 5:31:09 PM] [FATAL] [INS-32700] The gold image creation failed. Check the install log /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-30-53PM for more details.
INFO: [Aug 15, 2020 5:31:09 PM] Advice is ABORT
INFO: [Aug 15, 2020 5:31:09 PM] Adding ExitStatus FAILURE to the exit status set
INFO: [Aug 15, 2020 5:31:09 PM] Adding ExitStatus FAILURE to the exit status set
INFO: [Aug 15, 2020 5:31:09 PM] All forked task are completed at state setup
INFO: [Aug 15, 2020 5:31:09 PM] Completed background operations
INFO: [Aug 15, 2020 5:31:09 PM] Validating state
INFO: [Aug 15, 2020 5:31:09 PM] Completed validating state
INFO: [Aug 15, 2020 5:31:09 PM] Verifying route success
INFO: [Aug 15, 2020 5:31:09 PM] Waiting for completion of background operations
INFO: [Aug 15, 2020 5:31:09 PM] Completed background operations
INFO: [Aug 15, 2020 5:31:09 PM] Waiting for completion of background operations
INFO: [Aug 15, 2020 5:31:09 PM] Completed background operations
INFO: [Aug 15, 2020 5:31:09 PM] Executing action at state finish
INFO: [Aug 15, 2020 5:31:09 PM] FinishAction Actions.execute called
INFO: [Aug 15, 2020 5:31:09 PM] Completed executing action at state
INFO: [Aug 15, 2020 5:31:09 PM] Waiting for completion of background operations
INFO: [Aug 15, 2020 5:31:09 PM] Completed background operations
INFO: [Aug 15, 2020 5:31:09 PM] Waiting for completion of background operations
INFO: [Aug 15, 2020 5:31:09 PM] Completed background operations
INFO: [Aug 15, 2020 5:31:09 PM] Moved to state
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$

==================================================

DOES NOT APPLY

Bug 29220079 - Error INS-32700 Creating a GI Gold Image (Doc ID 29220079.8)

The fix for 29220079 is first included in
20.1.0
19.3.0.0.190416 (Apr 2019) Database Release Update (DB RU)

[INS-32700] The gold image creation failed. Check the install log

The trace file shows an error creating a directory (or a file):

WARNING: [Jan 16, 2019 12:23:13 AM] Could not create directory:
/tmp/GridSetupActions2019-01-16_00-22-41AM/tempHome_1547626992976/lib.

==================================================

WARNING: [Aug 15, 2020 5:30:57 PM] Command to get the files from '/u01/app/19.0.0/grid' not owned by 'oracle' failed.
WARNING: [Aug 15, 2020 5:30:59 PM] Following files are not readable: [/u01/app/19.0.0/grid/log/procwatcher/prw.sh, /u01/app/19.0.0/grid/log/procwatcher/PRW_SYS_ol7-19-lax1, /u01/app/19.0.0/grid/log/procwatcher/prwinit.ora, /u01/app/19.0.0/grid/crf/admin/run/crfmond, /u01/app/19.0.0/grid/crf/admin/run/crflogd]
WARNING: [Aug 15, 2020 5:31:02 PM] Could not create symlink: /tmp/GridSetupActions2020-08-15_05-30-53PM/tempHome_1597512661914/log/procwatcher/prw.sh.
WARNING: [Aug 15, 2020 5:31:09 PM] Validation disabled for the state finish

==================================================

[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$ ls -l /u01/app/19.0.0/grid
total 164
drwxrwxr-x. 3 root oinstall 22 Mar 4 01:03 acfs
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 acfsccm
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 acfsccreg
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 acfscm
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 acfsiob
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 acfsrd
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 acfsrm
drwxr-xr-x. 2 oracle oinstall 102 Jul 11 03:49 addnode
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:50 advmccb
drwxr-xr-x. 10 oracle oinstall 4096 Apr 17 2019 assistants
drwxrwxrwt. 6 root oinstall 52 Jul 11 03:52 auth
drwxr-xr-x. 2 root oinstall 12288 Aug 15 15:20 bin
drwxrwxr-x. 4 oracle oinstall 42 Jul 11 03:52 cdata
drwxr-x---. 3 oracle oinstall 18 Mar 4 00:50 cdp
drwxrwxr-x. 8 oracle oinstall 4096 Aug 15 14:47 cfgtoollogs
drwxr-x---. 4 oracle oinstall 31 Mar 4 00:50 cha
drwxr-xr-x. 3 oracle oinstall 19 Mar 4 00:30 client
drwxr-xr-x. 4 oracle oinstall 87 Jul 11 03:49 clone
drwxr-x---. 3 root oinstall 19 Jul 11 03:52 crf
drwxr-xr-x. 14 root oinstall 4096 Jul 11 03:52 crs
drwx--x--x. 5 oracle oinstall 41 Mar 4 00:50 css
drwxr-xr-x. 3 root oinstall 18 Jul 11 03:52 ctss
drwxr-xr-x. 3 oracle oinstall 19 Aug 15 15:08 ctx
drwxrwxr-x. 7 oracle oinstall 71 Apr 17 2019 cv
drwxr-xr-x. 3 oracle oinstall 19 Apr 17 2019 dbjava
drwxr-xr-x. 2 oracle oinstall 79 Aug 15 15:17 dbs
drwxr-xr-x. 5 oracle oinstall 4096 Jul 11 03:49 deinstall
drwxr-xr-x. 3 oracle oinstall 20 Apr 17 2019 demo
drwxr-xr-x. 3 oracle oinstall 20 Apr 17 2019 diagnostics
drwxr-xr-x. 13 oracle oinstall 4096 Apr 17 2019 dmu
-rw-r--r--. 1 oracle oinstall 852 Aug 18 2015 env.ora
drwxr-x---. 7 oracle oinstall 65 Jul 11 03:52 evm
drwxr-x---. 3 oracle oinstall 18 Jul 11 03:52 gipc
drwxr-x---. 3 oracle oinstall 18 Jul 11 03:52 gnsd
drwxr-x---. 8 oracle oinstall 4096 Jul 11 04:02 gpnp
-rwxr-x---. 1 oracle oinstall 3294 Mar 8 2017 gridSetup.sh
drwxr-xr-x. 4 oracle oinstall 32 Apr 17 2019 has
drwxr-xr-x. 3 oracle oinstall 19 Apr 17 2019 hs
drwxrwx---. 10 oracle oinstall 4096 Jul 11 03:52 install
drwxr-xr-x. 2 oracle oinstall 29 Apr 17 2019 instantclient
drwxr-x---. 14 oracle oinstall 4096 Jul 11 03:49 inventory
drwxr-xr-x. 8 oracle oinstall 82 Mar 4 02:18 javavm
drwxr-xr-x. 3 oracle oinstall 35 Apr 17 2019 jdbc
drwxr-xr-x. 6 root oinstall 4096 Aug 15 15:10 jdk
drwxr-xr-x. 2 oracle oinstall 8192 Aug 15 15:14 jlib
drwxr-xr-x. 10 oracle oinstall 4096 Apr 17 2019 ldap
drwxr-xr-x. 4 root oinstall 12288 Aug 15 15:15 lib
drwxrwxr-x. 6 oracle oinstall 67 Jul 11 04:07 log
drwxr-xr-x. 5 oracle oinstall 42 Apr 17 2019 md
drwxr-x---. 3 oracle oinstall 18 Jul 11 03:52 mdns
drwxr-xr-x. 10 oracle oinstall 4096 Mar 4 02:18 network
drwxr-xr-x. 5 oracle oinstall 46 Apr 17 2019 nls
drwxr-x---. 3 oracle oinstall 18 Jul 11 03:52 ohasd
drwxr-xr-x. 3 oracle oinstall 19 Aug 15 15:08 olap
drwxr-xr-x. 3 root oinstall 18 Jul 11 03:52 ologgerd
drwxr-x---. 14 oracle oinstall 4096 Aug 15 13:50 OPatch
drwxrwxr-x. 3 oracle oinstall 16 Aug 15 14:39 opatchautocfg
drwxr-xr-x. 8 oracle oinstall 77 Apr 17 2019 opmn
drwxr-xr-x. 4 oracle oinstall 34 Apr 17 2019 oracore
drwxr-xr-x. 3 oracle oinstall 18 Mar 4 00:26 oradiag_oracle
-rw-r-----. 1 oracle oinstall 56 Jul 11 03:52 oraInst.loc
drwxr-xr-x. 6 oracle oinstall 52 Apr 17 2019 ord
drwxr-xr-x. 4 oracle oinstall 66 Apr 17 2019 ords
drwxr-xr-x. 3 oracle oinstall 19 Apr 17 2019 oss
drwxr-xr-x. 3 root oinstall 18 Jul 11 03:52 osysmond
drwxr-xr-x. 8 oracle oinstall 4096 Jul 11 03:49 oui
drwxr-xr-x. 4 oracle oinstall 33 Apr 17 2019 owm
drwxr-xr-x. 5 root oinstall 39 Apr 17 2019 perl
drwxr-xr-x. 6 oracle oinstall 78 Apr 17 2019 plsql
drwxr-xr-x. 5 oracle oinstall 42 Apr 17 2019 precomp
drwxr-xr-x. 2 oracle oinstall 26 Apr 17 2019 QOpatch
drwxr-xr-x. 5 oracle oinstall 42 Apr 17 2019 qos
drwxr-xr-x. 6 oracle oinstall 68 Jul 11 03:52 racg
drwxr-xr-x. 15 oracle oinstall 4096 Mar 4 02:18 rdbms
drwxr-xr-x. 3 oracle oinstall 21 Apr 17 2019 relnotes
drwxr-xr-x. 7 oracle oinstall 102 Apr 17 2019 rhp
-rwxr-xr-x. 1 root oinstall 405 Jul 11 03:49 root.sh
-rwx------. 1 oracle oinstall 490 Apr 17 2019 root.sh.old
-rw-r-----. 1 oracle oinstall 10 Apr 17 2019 root.sh.old.1
-rwx------. 1 oracle oinstall 405 Apr 18 2019 root.sh.old.2
-rw-r-----. 1 oracle oinstall 10 Apr 17 2019 root.sh.old.3
-rwxr-xr-x. 1 oracle oinstall 415 Mar 4 00:40 root.sh.old.4
-rw-r-----. 1 oracle oinstall 10 Apr 17 2019 root.sh.old.5
-rwxr-xr-x. 1 root oinstall 414 Jul 11 03:49 rootupgrade.sh
-rwxr-x---. 1 oracle oinstall 628 Sep 3 2015 runcluvfy.sh
drwxr-xr-x. 5 oracle oinstall 4096 Apr 17 2019 sdk
drwxr-xr-x. 3 oracle oinstall 18 Apr 17 2019 slax
drwxr-xr-x. 6 oracle oinstall 4096 Aug 15 15:08 sqlpatch
drwxr-xr-x. 6 oracle oinstall 53 Jul 11 03:48 sqlplus
drwxr-xr-x. 8 oracle oinstall 77 Jul 11 03:52 srvm
drwxr-x---. 5 root oinstall 63 Mar 4 00:30 suptools
drwxr-xr-x. 4 oracle oinstall 29 Apr 17 2019 tomcat
drwxr-xr-x. 3 oracle oinstall 35 Apr 17 2019 ucp
drwxr-xr-x. 7 oracle oinstall 71 Apr 17 2019 usm
drwxr-xr-x. 2 oracle oinstall 33 Apr 17 2019 utl
-rw-r-----. 1 oracle oinstall 500 Feb 6 2013 welcome.html
drwxr-xr-x. 3 oracle oinstall 18 Apr 17 2019 wlm
drwxr-xr-x. 3 oracle oinstall 19 Apr 17 2019 wwg
drwxr-xr-x. 5 oracle oinstall 4096 Aug 15 15:19 xag
drwxr-x---. 6 oracle oinstall 58 Apr 17 2019 xdk
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-30-53PM]$

==================================================

[oracle@ol7-19-lax2 ~]$ . oraenv <<< +ASM2
ORACLE_SID = [hawk2] ? The Oracle base remains unchanged with value /u01/app/oracle

[oracle@ol7-19-lax2 ~]$ $ORACLE_HOME/gridSetup.sh -creategoldimage -exclFiles $ORACLE_HOME/log,$ORACLE_HOME/.patch_storage -destinationlocation /u01/app/oracle/goldimage -silent
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-32700] The gold image creation failed. Check the install log /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-49-37PM for more details.
Setup failed.
[oracle@ol7-19-lax2 ~]$

==================================================

[oracle@ol7-19-lax1 logs]$ $ORACLE_HOME/gridSetup.sh -creategoldimage -exclFiles $ORACLE_HOME/log,$ORACLE_HOME/.patch_storage -destinationlocation /u01/app/oracle/goldimage -silent -debug
Launching Oracle Grid Infrastructure Setup Wizard...

[FATAL] [INS-32700] The gold image creation failed. Check the install log /u01/app/oraInventory/logs/GridSetupActions2020-08-15_05-58-28PM for more details.
Setup failed.

[oracle@ol7-19-lax1 logs]$ ls -alrt
total 204
drwxrwx---. 2 oracle oinstall 4096 Jul 11 03:52 GridSetupActions2020-07-11_03-46-06AM
-rw-r-----. 1 oracle oinstall 0 Jul 11 04:09 oraInstall2020-07-11_04-08-44AM.err
-rw-r-----. 1 oracle oinstall 117 Jul 11 04:09 oraInstall2020-07-11_04-08-44AM.out.ol7-19-lax2
-rw-r-----. 1 oracle oinstall 0 Jul 11 04:09 oraInstall2020-07-11_04-08-44AM.err.ol7-19-lax2
-rw-r-----. 1 oracle oinstall 41420 Jul 11 04:09 UpdateNodeList2020-07-11_04-08-44AM.log
-rw-r-----. 1 oracle oinstall 32660 Jul 11 04:09 installActions2020-07-11_04-08-44AM.log
-rw-r-----. 1 oracle oinstall 151 Jul 11 04:11 oraInstall2020-07-11_04-08-44AM.out
drwxrwx---. 3 oracle oinstall 4096 Jul 11 04:11 GridSetupActions2020-07-11_04-08-44AM
-rw-r-----. 1 oracle oinstall 94 Jul 11 04:18 time2020-07-11_04-16-02AM.log
-rw-r-----. 1 oracle oinstall 0 Jul 11 04:18 oraInstall2020-07-11_04-16-02AM.out
-rw-r-----. 1 oracle oinstall 0 Jul 11 04:18 oraInstall2020-07-11_04-16-02AM.err
-rw-r-----. 1 oracle oinstall 17442 Jul 11 04:18 installActions2020-07-11_04-16-02AM.log
drwxrwx---. 2 oracle oinstall 4096 Jul 11 04:24 InstallActions2020-07-11_04-16-02AM
-rw-r-----. 1 oracle oinstall 24443 Aug 15 15:03 OPatch2020-08-15_02-57-03PM.log
-rw-r-----. 1 oracle oinstall 35327 Aug 15 15:15 OPatch2020-08-15_03-03-52PM.log
drwxrwx---. 6 oracle oinstall 4096 Aug 15 17:58 .
-rw-r-----. 1 oracle oinstall 14673 Aug 15 17:58 installActions2020-08-15_05-58-28PM.log
drwxrwx---. 5 oracle oinstall 92 Aug 15 17:58 ..
drwxr-xr-x. 2 oracle oinstall 4096 Aug 15 17:58 GridSetupActions2020-08-15_05-58-28PM
[oracle@ol7-19-lax1 logs]$

[oracle@ol7-19-lax1 logs]$ cd GridSetupActions2020-08-15_05-58-28PM
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-58-28PM]$ ls -l
total 1256
-rw-r--r--. 1 oracle oinstall 1853 Aug 15 17:58 gridSetupActions2020-08-15_05-58-28PM.err
-rw-r--r--. 1 oracle oinstall 1179847 Aug 15 17:58 gridSetupActions2020-08-15_05-58-28PM.log
-rw-r--r--. 1 oracle oinstall 87723 Aug 15 17:58 gridSetupActions2020-08-15_05-58-28PM.out
-rw-r--r--. 1 oracle oinstall 129 Aug 15 17:58 installerPatchActions_2020-08-15_05-58-28PM.log
-rw-r--r--. 1 oracle oinstall 2158 Aug 15 17:58 time2020-08-15_05-58-28PM.log
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-58-28PM]$ cat gridSetupActions2020-08-15_05-58-28PM.err
---# Begin Stacktrace #---------------------------
ID: oracle.install.ivw.common.driver.job.CreateGoldImageJob:7962
java.nio.file.NoSuchFileException: /tmp/GridSetupActions2020-08-15_05-58-28PM/tempHome_1597514317225/log/procwatcher/prw.sh
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.createSymbolicLink(UnixFileSystemProvider.java:457)
at java.nio.file.Files.createSymbolicLink(Files.java:1043)
at oracle.install.ivw.common.driver.job.CreateGoldImageJob.createTemporaryHome(CreateGoldImageJob.java:844)
at oracle.install.ivw.common.driver.job.CreateGoldImageJob.createGoldImage(CreateGoldImageJob.java:1016)
at oracle.install.ivw.common.driver.job.CreateGoldImageJob.call(CreateGoldImageJob.java:327)
at oracle.install.ivw.common.driver.job.CreateGoldImageJob.call(CreateGoldImageJob.java:2463)
at oracle.install.ivw.common.driver.job.CreateGoldImageJob.call(CreateGoldImageJob.java:121)
at oracle.install.ivw.crs.driver.CRSImageSetupDriver.setup(CRSImageSetupDriver.java:403)
at oracle.install.commons.base.interview.common.action.SetupAction$1.call(SetupAction.java:62)
at oracle.install.commons.base.interview.common.action.SetupAction$1.call(SetupAction.java:58)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

---# End Stacktrace #-----------------------------
[oracle@ol7-19-lax1 GridSetupActions2020-08-15_05-58-28PM]$

Typically, creategoldimage is used to create image for install or upgrade; however, creategoldimage is too BUGGY.


$GRID_HOME/gridSetup.sh -creategoldimage -exclFiles $ORACLE_HOME/log,$ORACLE_HOME/.patch_storage -destinationlocation /u01/app/oracle/goldimage -silent


In the discussion with LDC, he was thinking of using tar and I did not know if tar will work.

Here’s the proof of concept that it works.

Upgrade_Oracle_Restart_from_12.2_to_19.8.pdf

Looks like OOP is back again; however, I have not had the opportunity to test.

When you test, it’s important to test rollback as well.

GRID Out Of Place (OOP) Rollback Disaster

Steps for Minimal Downtime Grid Infrastructure Out of Place ( OOP ) Patching using gridSetup.sh (Doc ID 2662762.1)
May 13, 2020
Oracle Database – Enterprise Edition – Version 19.6.0.0.0 and later
This is applicable only for cluster environment,not the SIHA(Oracle Restart)

In general, the following steps are involved:

1) Installing and Patching the Grid infrastructure (software only)

/u01/app/19.7.0.0/grid/gridSetup.sh -ApplyRU 3089972
Chose the option “Install Software only” and select all the nodes.

2) Switching the Grid Infrastructure Home

Run the gridSetup.sh from the target home

/u01/app/19.7.0.0/grid/gridSetup.sh -SwitchGridhome

It will launch the GUI (you could run it in silent as well with a response file)

During this phase, you can select the automated root.sh option if you wanted. Otherwise it will prompt the root.sh.

I have been working on tasks for weekly tablespace segment advisor to shrink all segments residing in tablespace.

There are many blogs out there with the same info; however, it was not to the requirements and this is a combinations after research.

Here is a demo for 19c; however, the preparations have been tested in 11.2.

=======================================================
### SQL Scripts
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ ls -l
total 11
-rwxrwxrwx 1 vagrant vagrant 1048 Aug 23 04:07 10-advise.sql
-rwxrwxrwx 1 vagrant vagrant  257 Aug 23 03:31 20-benefit.sql
-rwxrwxrwx 1 vagrant vagrant  475 Aug 23 04:05 30-space_save.sql
-rwxrwxrwx 1 vagrant vagrant  722 Aug 23 03:36 40-recommendations.sql
-rwxrwxrwx 1 vagrant vagrant  431 Aug 23 04:09 99-delete.sql
-rwxrwxrwx 1 vagrant vagrant  141 Aug 23 04:03 set_global_var.sql
-rwxrwxrwx 1 vagrant vagrant  259 Aug 23 04:00 test.sql

=======================================================
### Create test case.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ sqlplus / as sysdba @ test.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:10:55 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> drop table big_table purge;
drop table big_table purge
           *
ERROR at line 1:
ORA-00942: table or view does not exist


Elapsed: 00:00:00.01
SQL> create table big_table (id number, name char(200)) tablespace USERS;

Table created.

Elapsed: 00:00:00.02
SQL> insert into big_table select rownum,'a' from dual connect by rownum<900000; 899999 rows created. Elapsed: 00:00:12.65 SQL> commit;

Commit complete.

Elapsed: 00:00:01.86
SQL> delete from big_table where mod(id,10)<>0;

810000 rows deleted.

Elapsed: 00:00:37.72
SQL> commit;

Commit complete.

Elapsed: 00:00:00.08
SQL> exit

=======================================================
### Run advise.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ sqlplus / as sysdba @ 10-advise.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:13:40 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> DECLARE
  2    l_object_id   NUMBER;
  3    l_object_type VARCHAR2(32767) := 'TABLESPACE';
  4    l_attr1       VARCHAR2(32767) := '&v_tablespace';
  5    l_task_name   VARCHAR2(32767) := '&v_task_name';
  6  BEGIN
  7  DBMS_ADVISOR.create_task (
  8    advisor_name => 'Segment Advisor',
  9    task_name    => l_task_name
 10  );
 11
 12  DBMS_ADVISOR.create_object (
 13    task_name   => l_task_name,
 14    object_type => l_object_type,
 15    attr1       => l_attr1,
 16    attr2       => NULL,
 17    attr3       => NULL,
 18    attr4       => 'null',
 19    attr5       => NULL,
 20    object_id   => l_object_id
 21  );
 22
 23  DBMS_ADVISOR.set_task_parameter (
 24    task_name => l_task_name,
 25    parameter => 'RECOMMEND_ALL',
 26    value     => 'TRUE');
 27
 28  DBMS_ADVISOR.execute_task (
 29    task_name => l_task_name
 30  );
 31
 32  END;
 33  /
old   4:   l_attr1       VARCHAR2(32767) := '&v_tablespace';
new   4:   l_attr1       VARCHAR2(32767) := 'USERS';
old   5:   l_task_name   VARCHAR2(32767) := '&v_task_name';
new   5:   l_task_name   VARCHAR2(32767) := 'SEGMENT_ADVISOR_TBS_USERS';
SQL> set feedback on echo on head on
SQL> select task_name, advisor_name
  2  from DBA_ADVISOR_TASKS
  3  where advisor_name='Segment Advisor'
  4  ;

TASK_NAME                      ADVISOR_NAME
------------------------------ ---------------------------------------
SEGMENT_ADVISOR_TBS_USERS      Segment Advisor

1 row selected.

SQL> exit

=======================================================
### Review benefit.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ sqlplus / as sysdba @ 20-benefit.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:14:49 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> SELECT benefit_type FROM DBA_ADVISOR_RECOMMENDATIONS WHERE task_name='&v_task_name' order by 1;
old   1: SELECT benefit_type FROM DBA_ADVISOR_RECOMMENDATIONS WHERE task_name='&v_task_name' order by 1
new   1: SELECT benefit_type FROM DBA_ADVISOR_RECOMMENDATIONS WHERE task_name='SEGMENT_ADVISOR_TBS_USERS' order by 1

BENEFIT_TYPE
------------------------------------------------------------------------------------------------------------------------------------------------------
Enable row movement of the table SYS.BIG_TABLE and perform shrink, estimated savings is 180447064 bytes.
SQL> exit

=======================================================
### Review space saving.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ sqlplus / as sysdba @ 30-space_save.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:15:48 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> col segment_name for a30
SQL> SELECT
  2  segment_name,
  3  round(allocated_space/1024/1024,1) alloc_mb,
  4  round(used_space/1024/1024,1) used_mb,
  5  round(reclaimable_space/1024/1024) reclaim_mb,
  6  round(reclaimable_space/allocated_space*100,0) pctsave
  7  FROM TABLE(dbms_space.asa_recommendations())
  8  where tablespace_name='&v_tablespace'
  9  order by pctsave desc
 10  ;

SEGMENT_NAME                     ALLOC_MB    USED_MB RECLAIM_MB    PCTSAVE
------------------------------ ---------- ---------- ---------- ----------
BIG_TABLE                             216       43.9        172         80
SQL> exit

=======================================================
### Create recommendations.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ sqlplus / as sysdba @ 40-recommendations.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:16:24 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

***********************************************************************
***   nohup sqlplus "/ as sysdba" @run.sql > run.out 2>&1 &         ***
***********************************************************************
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
oracle@db-fs-1:hawk:/sf_working/segment_advisor

=======================================================
### Review SQL script.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ cat run.sql
set echo on timing on

alter table "SYS"."BIG_TABLE" enable row movement;
alter table "SYS"."BIG_TABLE" shrink space COMPACT;
alter table "SYS"."BIG_TABLE" shrink space;

exit

=======================================================
### Run SQL script using nohup.
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ nohup sqlplus "/ as sysdba" @run.sql > run.out 2>&1 &
[1] 27417
oracle@db-fs-1:hawk:/sf_working/segment_advisor
$
[1]+  Done                    nohup sqlplus "/ as sysdba" @run.sql > run.out 2>&1
oracle@db-fs-1:hawk:/sf_working/segment_advisor
$

=======================================================
### Review results.
=======================================================

Live Run for comparison.
/home/oracle/segment_advisor$ grep ORA- run.out | sort | uniq -c
   6 ORA-00054: resource busy and acquire with NOWAIT specified or timeout expired
   2 ORA-10631: SHRINK clause should not be specified for this object
  12 ORA-10636: ROW MOVEMENT is not enabled

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ grep ORA- run.out | sort | uniq -c

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ grep Elapsed run.out | sort | uniq -c
      1 Elapsed: 00:00:00.02
      1 Elapsed: 00:00:02.91
      1 Elapsed: 00:00:06.84

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ cat run.out
nohup: ignoring input

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:18:07 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL>
SQL> alter table "SYS"."BIG_TABLE" enable row movement;

Table altered.

Elapsed: 00:00:00.02
SQL> alter table "SYS"."BIG_TABLE" shrink space COMPACT;

Table altered.

Elapsed: 00:00:06.84
SQL> alter table "SYS"."BIG_TABLE" shrink space;

Table altered.

Elapsed: 00:00:02.91
SQL>
SQL> exit

=======================================================
### Delete Advisor Task
=======================================================

oracle@db-fs-1:hawk:/sf_working/segment_advisor
$ sqlplus / as sysdba @ 99-delete.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Sun Aug 23 04:20:04 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> DECLARE
  2    l_object_id      NUMBER;
  3    l_object_type VARCHAR2(32767) := 'TABLESPACE';
  4    l_attr1       VARCHAR2(32767) := 'v_tablespace';
  5    l_task_name      VARCHAR2(32767) := '&v_task_name';
  6  BEGIN
  7  DBMS_ADVISOR.delete_task (
  8    task_name => l_task_name
  9  );
 10  END;
 11  /
old   5:   l_task_name   VARCHAR2(32767) := '&v_task_name';
new   5:   l_task_name   VARCHAR2(32767) := 'SEGMENT_ADVISOR_TBS_USERS';

PL/SQL procedure successfully completed.

SQL> select task_name, advisor_name
  2  from DBA_ADVISOR_TASKS
  3  where advisor_name='Segment Advisor'
  4  ;

no rows selected

SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
oracle@db-fs-1:hawk:/sf_working/segment_advisor
$

SQL Scripts for tablespace_segment_advisor

On November 2, 2017, I had blogged about Monitoring Standby – SQLPlus or DGMGRL

Since the post, I do not recall using dgmgrl much for monitoring lag.

Almost 3 years later and 19c, let’s revisit the topic.

Here is what monitoring lag looks like from SQLPlus.
Notice BLOCK# increased which mean transfer is working.

SQL> r
  1  select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6*

                                        CLIENT                                                  DELAY
PID                       INST  THREAD# PROCESS      PROCESS   STATUS       SEQUENCE#   BLOCK#   MINS
------------------------ ----- -------- ------------ --------- ------------ --------- -------- ------
9589                         1        1 LGWR         RFS       RECEIVING          175     8540      0
9059                         1        1 N/A          MRP0      APPLYING_LOG       175     8540      0

SQL> r
  1  select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6*

                                        CLIENT                                                  DELAY
PID                       INST  THREAD# PROCESS      PROCESS   STATUS       SEQUENCE#   BLOCK#   MINS
------------------------ ----- -------- ------------ --------- ------------ --------- -------- ------
9589                         1        1 LGWR         RFS       RECEIVING          175     8554      0
9059                         1        1 N/A          MRP0      APPLYING_LOG       175     8554      0

SQL> 

For 19c, show configuration lag will provide info on lag and is knowing Lag is 0 seconds good enough?

[oracle@ol7-112-dg2 sql]$ dgmgrl /
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sat Sep 19 12:50:58 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "hawk_stby"
Connected as SYSDG.
DGMGRL> show configuration lag

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
                Transport Lag:      0 seconds (computed 9 seconds ago)
                Apply Lag:          0 seconds (computed 9 seconds ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 20 seconds ago)

DGMGRL> show configuration lag verbose

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
                Transport Lag:      0 seconds (computed 12 seconds ago)
                Apply Lag:          0 seconds (computed 12 seconds ago)

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    TraceLevel                      = 'USER'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'
    ConfigurationWideServiceName    = ''

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS
DGMGRL>

Using SendQEntries shows LOG_SEQ but RecvQEntries does not.

DGMGRL> show database hawk_stby RecvQEntries
STANDBY_RECEIVE_QUEUE
              STATUS     RESETLOGS_ID           THREAD              LOG_SEQ       TIME_GENERATED       TIME_COMPLETED        FIRST_CHANGE#         NEXT_CHANGE#       SIZE (KBs)

DGMGRL> show database hawk SendQEntries
PRIMARY_SEND_QUEUE
        STANDBY_NAME       STATUS     RESETLOGS_ID           THREAD              LOG_SEQ       TIME_GENERATED       TIME_COMPLETED        FIRST_CHANGE#         NEXT_CHANGE#       SIZE (KBs)
                          CURRENT       1047346434                1                  175  09/19/2020 12:32:29                                   2984164                                 23586

DGMGRL> show database hawk_stby RecvQEntries
STANDBY_RECEIVE_QUEUE
              STATUS     RESETLOGS_ID           THREAD              LOG_SEQ       TIME_GENERATED       TIME_COMPLETED        FIRST_CHANGE#         NEXT_CHANGE#       SIZE (KBs)

DGMGRL>

Disable apply and compare differences between SQLPlus and DGMGRL.

DGMGRL> edit database hawk_stby set state=APPLY-OFF
> ;
Succeeded.
DGMGRL> show database hawk_stby

Database - hawk_stby

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-OFF
  Transport Lag:      0 seconds (computed 8 seconds ago)
  Apply Lag:          0 seconds (computed 8 seconds ago)
  Average Apply Rate: (unknown)
  Real Time Query:    OFF
  Instance(s):
    hawk

Database Status:
SUCCESS

DGMGRL>

### From Primary:

*** gv$managed_standby ***

                                        CLIENT                                                  DELAY
PID                       INST  THREAD# PROCESS      PROCESS   STATUS       SEQUENCE#   BLOCK#   MINS
------------------------ ----- -------- ------------ --------- ------------ --------- -------- ------
9030                         1        1 LNS          LNS       WRITING            180     1311      0

SQL> r
  1  select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6*

                                        CLIENT                                                  DELAY
PID                       INST  THREAD# PROCESS      PROCESS   STATUS       SEQUENCE#   BLOCK#   MINS
------------------------ ----- -------- ------------ --------- ------------ --------- -------- ------
9030                         1        1 LNS          LNS       WRITING            180     1314      0

SQL>

### From Standby:

*** gv$archived_log ***

 DEST_ID  THREAD# APPLIED    MAX_SEQ MAX_TIME             DELTA_SEQ DETA_MIN
-------- -------- --------- -------- -------------------- --------- --------
       1        1 NO             179 19-SEP-2020 13:12:25         5 39.93333
       1        1 YES            174 19-SEP-2020 12:32:29

SQL> r
  1  select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6*

                                        CLIENT                                                  DELAY
PID                       INST  THREAD# PROCESS      PROCESS   STATUS       SEQUENCE#   BLOCK#   MINS
------------------------ ----- -------- ------------ --------- ------------ --------- -------- ------
9589                         1        1 LGWR         RFS       RECEIVING          180      284      0

SQL> r
  1  select PID,inst_id inst,thread#,client_process,process,status,sequence#,block#,DELAY_MINS
  2  from gv$managed_standby
  3  where BLOCK#>1
  4  and status not in ('CLOSING','IDLE')
  5  order by status desc, thread#, sequence#
  6*

                                        CLIENT                                                  DELAY
PID                       INST  THREAD# PROCESS      PROCESS   STATUS       SEQUENCE#   BLOCK#   MINS
------------------------ ----- -------- ------------ --------- ------------ --------- -------- ------
9589                         1        1 LGWR         RFS       RECEIVING          180      298      0

SQL>

From DGMGRL:

DGMGRL> show configuration lag

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
                Transport Lag:      0 seconds (computed 6 seconds ago)
                Apply Lag:          6 minutes 48 seconds (computed 6 seconds ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 31 seconds ago)

DGMGRL> show database hawk SendQEntries
PRIMARY_SEND_QUEUE
        STANDBY_NAME       STATUS     RESETLOGS_ID           THREAD              LOG_SEQ       TIME_GENERATED       TIME_COMPLETED        FIRST_CHANGE#         NEXT_CHANGE#       SIZE (KBs)
                          CURRENT       1047346434                1                  180  09/19/2020 13:12:25                                   2992416                                   270

DGMGRL> show database hawk_stby RecvQEntries
STANDBY_RECEIVE_QUEUE
              STATUS     RESETLOGS_ID           THREAD              LOG_SEQ       TIME_GENERATED       TIME_COMPLETED        FIRST_CHANGE#         NEXT_CHANGE#       SIZE (KBs)
   PARTIALLY_APPLIED       1047346434                1                  175  09/19/2020 12:32:29  09/19/2020 13:12:16              2984164              2992388            24709
         NOT_APPLIED       1047346434                1                  176  09/19/2020 13:12:16  09/19/2020 13:12:17              2992388              2992393                1
         NOT_APPLIED       1047346434                1                  177  09/19/2020 13:12:17  09/19/2020 13:12:20              2992393              2992400                2
         NOT_APPLIED       1047346434                1                  178  09/19/2020 13:12:20  09/19/2020 13:12:20              2992400              2992403                1
         NOT_APPLIED       1047346434                1                  179  09/19/2020 13:12:20  09/19/2020 13:12:25              2992403              2992416                3

DGMGRL>

APPLY-ON

DGMGRL> edit database hawk_stby set state=APPLY-ON;
Succeeded.
DGMGRL> show configuration lag

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
                Transport Lag:      0 seconds (computed 3 seconds ago)
                Apply Lag:          0 seconds (computed 3 seconds ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 41 seconds ago)

DGMGRL> /

Configuration - my_dg_config

  Protection Mode: MaxPerformance
  Members:
  hawk      - Primary database
    hawk_stby - Physical standby database
                Transport Lag:      0 seconds (computed 4 seconds ago)
                Apply Lag:          0 seconds (computed 4 seconds ago)

Fast-Start Failover:  Disabled

Configuration Status:
SUCCESS   (status updated 56 seconds ago)

DGMGRL> show database hawk_stby

Database - hawk_stby

  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      0 seconds (computed 6 seconds ago)
  Apply Lag:          0 seconds (computed 6 seconds ago)
  Average Apply Rate: 1.00 KByte/s
  Real Time Query:    OFF
  Instance(s):
    hawk

Database Status:
SUCCESS

DGMGRL>

It would be nice if show configuration lag is able to provide some high level info frequently asked by management.

What is lag time, how many sequence is the standby behind, what is the apply rate, what is LOG_SEQ at primary and standby?

Not too long ago, I had blogged about When To Use dgmgrl / vs dgmgrl sys@tns

I believe this is New Feature for 19c (but not 100% certain) may resolved the question above?.

DEMO:
Connect using OS authentication from standby host.

ERROR:
ORA-01017: invalid username/password; logon denied

[oracle@ol7-112-dg2 ~]$ dgmgrl /
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Sun Sep 20 14:22:13 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "hawk_stby"
Connected as SYSDG.
DGMGRL> validate database hawk;

  Database Role:    Primary database

  Ready for Switchover:  Yes

  Managed by Clusterware:
    hawk:  NO
    Validating static connect identifier for the primary database hawk...

ORA-01017: invalid username/password; logon denied

    Warning: Ensure primary database's StaticConnectIdentifier property
    is configured properly so that the primary database can be restarted
    by DGMGRL after switchover

DGMGRL> validate database hawk_stby;

  Database Role:     Physical standby database
  Primary Database:  hawk

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    hawk     :  NO
    hawk_stby:  NO
    Validating static connect identifier for the primary database hawk...

ORA-01017: invalid username/password; logon denied

    Warning: Ensure primary database's StaticConnectIdentifier property
    is configured properly so that the primary database can be restarted
    by DGMGRL after switchover

  Log Files Cleared:
    hawk Standby Redo Log Files:       Cleared
    hawk_stby Online Redo Log Files:   Not Cleared
    hawk_stby Standby Redo Log Files:  Available

DGMGRL>

DEMO:
Connect to primary using tns from standby host.

DGMGRL> connect sys/oracle@hawk
Connected to "hawk"
Connected as SYSDBA.
DGMGRL> validate database hawk;

  Database Role:    Primary database

  Ready for Switchover:  Yes

  Managed by Clusterware:
    hawk:  NO
    Validating static connect identifier for the primary database hawk...
    The static connect identifier allows for a connection to database "hawk".

DGMGRL> validate database hawk_stby;

  Database Role:     Physical standby database
  Primary Database:  hawk

  Ready for Switchover:  Yes
  Ready for Failover:    Yes (Primary Running)

  Managed by Clusterware:
    hawk     :  NO
    hawk_stby:  NO
    Validating static connect identifier for the primary database hawk...
    The static connect identifier allows for a connection to database "hawk".

  Log Files Cleared:
    hawk Standby Redo Log Files:       Cleared
    hawk_stby Online Redo Log Files:   Not Cleared
    hawk_stby Standby Redo Log Files:  Available

DGMGRL>

This will at least address one example for when to use TNS vs OS authentication for DGMGRL.

IMPORTANT: This is not a recommendation to rebuild indexes.

The post will outline SQL used to determine index to rebuild.

PL/SQL will be used to check table lock for the underlying index and if there is no lock, then rebuild index else skip rebuild for index.

1.Download Index Sizing and create copy index_est_proc_2.sql.org

2. Create table index_rebuild.

SQL> desc index_rebuild
 Name                                      Null?    Type
 ----------------------------------------- -------- ----------------------------
 TABLE_OWNER                               NOT NULL VARCHAR2(30)
 TABLE_NAME                                NOT NULL VARCHAR2(30)
 INDEX_NAME                                NOT NULL VARCHAR2(20)
 LEAF_BLOCKS                                        NUMBER
 TARGET_SIZE                                        NUMBER

SQL>

3. Update index_est_proc_2.sql and include the following insert into table index rebuild.

if m_leaf_estimate < &m_scale_factor * r.leaf_blocks then
  dbms_output.put_line(
    to_char(sysdate,'hh24_mi_ss') || '|table|' ||
    trim(r.table_name) || '|index|' ||
    trim(r.index_name) || '|' || 'Current Leaf blocks|' || trim(to_char(r.leaf_blocks,'999,999,999')) || '|Target size|' || 
    trim(to_char(m_leaf_estimate,'999,999,999'))
  );

  -- Insert data into table index_rebuild as well as output to terminal.
  insert into index_rebuild(table_owner,table_name,index_name,leaf_blocks,target_size)
  values
  (UPPER('&m_owner'),trim(r.table_name),trim(r.index_name),r.leaf_blocks,m_leaf_estimate);
  dbms_output.new_line;
end if;

4. Create plsql_rebuild_idx.sql

set timing on time on serveroutput on size unlimited trimsp on tab off lines 200
col TABLE_OWNER for a30
col TABLE_NAME for a30
col INDEX_NAME for a35
col USERNAME for a10
col MACHINE for a10
col MODULE for a30
-- Display current user session info.
select s.username as Username,
       s.machine as Machine,
       s.module as Module,
       s.sid as SessionID,
       p.pid as ProcessID,
       p.spid as "UNIX ProcessID"
from
v$session s, v$process p
where s.sid = sys_context ('userenv','sid')
and s.PADDR = p.ADDR
;
set echo on
-- Rebuild indexes with LEAF_BLOCKS < 16000000 and edit as required.
select * from index_rebuild where LEAF_BLOCKS < 16000000;
exit
lock table index_rebuild in EXCLUSIVE mode WAIT 120;
DECLARE
  l_sql varchar2(1000);
  l_ct  number;
BEGIN
FOR d in (
  select TABLE_OWNER, TABLE_NAME, INDEX_NAME, LEAF_BLOCKS from index_rebuild order by leaf_blocks asc
)
LOOP
  select count(*) into l_ct
  from v$locked_object a, v$session b, dba_objects c
  where b.sid = a.session_id
  and a.object_id = c.object_id
  and c.object_type='TABLE'
  and c.owner=d.TABLE_OWNER
  and c.object_name=d.TABLE_NAME
  and d.LEAF_BLOCKS < 16000000;
  IF l_ct = 0 THEN
    dbms_output.put_line( '-- Check lock for owner|table|index : ' ||d.TABLE_OWNER||'.'||d.TABLE_NAME||'.'||d.INDEX_NAME||'='||l_ct );
    l_sql := 'alter index '||d.TABLE_OWNER||'.'||d.INDEX_NAME||' rebuild online parallel 4';
    dbms_output.put_line (l_sql);
    execute immediate l_sql;
    delete from index_rebuild where TABLE_OWNER=d.TABLE_OWNER and TABLE_NAME=d.TABLE_NAME and INDEX_NAME=d.INDEX_NAME;
  END IF;
END LOOP;
END;
/
delete from index_rebuild;
commit;
exit

5. Run plsql_rebuild_idx.sql using nohup

nohup sqlplus "/ as sysdba" @ plsql_rebuild_idx.sql > plsql_rebuild_idx.log 2>&1 &

6. Review

$ cat plsql_rebuild_idx.log
nohup: ignoring input

SQL*Plus: Release 12.1.0.2.0 Production on Thu Sep 24 14:13:00 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Advanced Analytics
and Real Application Testing options

14:13:00 SQL> select * from index_rebuild;

TABLE_OWNER                    TABLE_NAME                     INDEX_NAME           LEAF_BLOCKS TARGET_SIZE
------------------------------ ------------------------------ -------------------- ----------- -----------
XXXX                           YYYYYYYYYY1                    ZZZZZZZZZZZ_M            9721430     4328586
XXXX                           YYYYYYYYYY2                    ZZZZZZZZZZZ_MP          15865953     5848673

Elapsed: 00:00:00.00
14:13:00 SQL> lock table index_rebuild in EXCLUSIVE mode WAIT 120;

Table(s) Locked.

Elapsed: 00:00:00.00
14:13:00 SQL> DECLARE
14:13:00   2    l_sql varchar2(1000);
14:13:00   3    l_ct  number;
14:13:00   4  BEGIN
14:13:00   5  FOR d in (
14:13:00   6    select TABLE_OWNER, TABLE_NAME, INDEX_NAME, LEAF_BLOCKS from index_rebuild order by leaf_blocks asc
14:13:00   7  )
14:13:00   8  LOOP
14:13:00   9    select count(*) into l_ct
14:13:00  10    from v$locked_object a, v$session b, dba_objects c
14:13:00  11    where b.sid = a.session_id
14:13:00  12    and a.object_id = c.object_id
14:13:00  13    and c.object_type='TABLE'
14:13:00  14    and c.owner=d.TABLE_OWNER
14:13:00  15    and c.object_name=d.TABLE_NAME;
14:13:00  16    IF l_ct = 0 THEN
14:13:00  17      dbms_output.put_line( '-- Check lock for owner|table|index : ' ||d.TABLE_OWNER||'.'||d.TABLE_NAME||'.'||d.INDEX_NAME||'='||l_ct );
14:13:00  18      l_sql := 'alter index '||d.TABLE_OWNER||'.'||d.INDEX_NAME||' rebuild online parallel 4';
14:13:00  19      dbms_output.put_line (l_sql);
14:13:00  20      execute immediate l_sql;
14:13:00  21      delete from index_rebuild where TABLE_OWNER=d.TABLE_OWNER and TABLE_NAME=d.TABLE_NAME and INDEX_NAME=d.INDEX_NAME;
14:13:00  22    END IF;
14:13:00  23  END LOOP;
14:13:00  24  END;
14:13:00  25  /
-- Check lock for owner|table|index : XXXX.YYYYYYYYYY1.ZZZZZZZZZZZ_M=0
alter index XXXX.ZZZZZZZZZZZ_M rebuild online parallel 4
-- Check lock for owner|table|index : XXXX.YYYYYYYYYY2.ZZZZZZZZZZZ_MP=0
alter index XXXX.ZZZZZZZZZZZ_MP rebuild online parallel 4

PL/SQL procedure successfully completed.

Elapsed: 04:00:23.08
18:13:23 SQL> commit;

Commit complete.

Elapsed: 00:00:00.01
18:13:23 SQL> exit

7. Run index_est_proc_2.sql.org (screen output only) or index_est_proc_2.sql (screen output and insert into index_rebuild table) to determine if any more indexes are listed for rebuild.

Note: The first rebuild contained a few dozen of indexes for rebuild but was not automated.

Later, there were only 2 indexes for rebuild as shown above from real production environment before minor improvements, e.g. — Display current user session info.

Q.E.D.

Here are the contents of for a tweet I started.

Hard to imagine setting SQLPATH ahd glogin.sql would affect upgrade.

That’s the only changes I made.

“ORA-04023: Object SYS.STANDARD Could Not Be Validated or Authorized” during database upgrade (Doc ID 984511.1)

Documentation does not mention having customized glogin.sql and/or SQLPATH will cause upgrade to fail. Documentation has now been updated.

You can read more about Upgrade and profile scripts

AutoUpgrade validates glogin for you before starting. It looks like any improvements are only added to AutoUgrade and not traditional methods. Not only do you need to read the documentation, you also need to read again because of changes. 

Lastly, here are Useful List of Examples which I find helpful.

I still remembered the cliche a manager used to tell me, “Slow and steady win the race.”

Looks like it is true with Oracle software as you never know what’s going to get.

I am trying to install 19.3 database software and apply Patch 31305339 – GI Release Update 19.8.0.0.200714 which failed misserably.

There is little to no information for the failure (at least from what I was able to ascertain).

Resolving the issue requires starting over from the beginning.

Here are the steps taken:

[oracle@ol7-112-dg1 ~]$ cat /etc/system-release
Oracle Linux Server release 7.7

[oracle@ol7-112-dg1 ~]$ ll /etc/ora*
-rw-r--r--. 1 root   root      32 Aug  8  2019 /etc/oracle-release
-rw-rw-r--. 1 oracle oinstall 790 Oct 14 19:13 /etc/oratab

[oracle@ol7-112-dg1 ~]$ echo $new_db_home
/u01/app/oracle/product/19.3.0.0/db_1

[oracle@ol7-112-dg1 ~]$ rm -rf $new_db_home

[oracle@ol7-112-dg1 ~]$ ls $new_db_home
ls: cannot access /u01/app/oracle/product/19.3.0.0/db_1: No such file or directory

[oracle@ol7-112-dg1 ~]$ echo $zip_loc
/vagrant_software

[oracle@ol7-112-dg1 ~]$ ls -l $zip_loc/LINUX.X64_193000_db_home.zip
-rwxrwxrwx. 1 vagrant vagrant 3059705302 Sep  5  2019 /vagrant_software/LINUX.X64_193000_db_home.zip

[oracle@ol7-112-dg1 ~]$ unzip -qo $zip_loc/LINUX.X64_193000_db_home.zip -d $new_db_home; echo $?; ls $new_db_home
0
addnode     crs   dbjava       dmu      hs             jdbc  md       olap     ords  plsql    rdbms          runInstaller   sqlj      ucp
apex        css   dbs          drdaas   install        jdk   mgw      OPatch   oss   precomp  relnotes       schagent.conf  sqlpatch  usm
assistants  ctx   deinstall    dv       instantclient  jlib  network  opmn     oui   QOpatch  root.sh        sdk            sqlplus   utl
bin         cv    demo         env.ora  inventory      ldap  nls      oracore  owm   R        root.sh.old    slax           srvm      wwg
clone       data  diagnostics  has      javavm         lib   odbc     ord      perl  racg     root.sh.old.1  sqldeveloper   suptools  xdk

[oracle@ol7-112-dg1 ~]$ $new_db_home/OPatch/opatch version
OPatch Version: 12.2.0.1.17

OPatch succeeded.

[oracle@ol7-112-dg1 ~]$ unzip -qo $zip_loc/p6880880_190000_LINUX.zip -d $new_db_home; echo $?
0

[oracle@ol7-112-dg1 ~]$ $new_db_home/OPatch/opatch version
/u01/app/oracle/product/19.3.0.0/db_1/OPatch/opatch: line 839: [: too many arguments
/u01/app/oracle/product/19.3.0.0/db_1/OPatch/opatch: line 839: [: too many arguments
Java (1.7) could not be located. OPatch cannot proceed!
OPatch returns with error code = 1

[oracle@ol7-112-dg1 ~]$ $new_db_home/OPatch/opatch version -jdk $new_db_home/jdk
OPatch Version: 12.2.0.1.21

OPatch succeeded.

[oracle@ol7-112-dg1 ~]$ cd $new_db_home
[oracle@ol7-112-dg1 db_1]$ ls -l /home/oracle/patch/31305339
total 132
drwxr-x---. 5 oracle oinstall     81 Jul 10 05:20 31281355
drwxr-x---. 5 oracle oinstall     62 Jul 10 05:17 31304218
drwxr-x---. 5 oracle oinstall     62 Jul 10 05:18 31305087
drwxr-x---. 4 oracle oinstall     48 Jul 10 05:20 31335188
drwxr-x---. 2 oracle oinstall   4096 Jul 10 05:18 automation
-rw-rw-r--. 1 oracle oinstall   5054 Jul 10 05:46 bundle.xml
-rw-rw-r--. 1 oracle oinstall 120878 Jul 20 19:04 README.html
-rw-r--r--. 1 oracle oinstall      0 Jul 10 05:17 README.txt

[oracle@ol7-112-dg1 db_1]$ pwd
/u01/app/oracle/product/19.3.0.0/db_1

[oracle@ol7-112-dg1 db_1]$ ./runInstaller -debug -applyRU /home/oracle/patch/31305339
Preparing the home to patch…
Applying the patch /home/oracle/patch/31305339…
OPatch command failed while applying the patch. For details look at the logs from /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto/.
[oracle@ol7-112-dg1 db_1]$ cd /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto
-bash: cd: /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/opatchauto: No such file or directory
[oracle@ol7-112-dg1 db_1]$ cd /u01/app/oracle/product/19.3.0.0/db_1/cfgtoollogs/
[oracle@ol7-112-dg1 cfgtoollogs]$ ll
total 4
drwxrwx---. 2 oracle oinstall 4096 Oct 17 16:44 oui
[oracle@ol7-112-dg1 cfgtoollogs]$

[oracle@ol7-112-dg1 db_1]$ ./runInstaller -debug -applyRU /home/oracle/patch/31305339

ERROR: The home is not clean. This home cannot be used since there was a failed OPatch execution in this home. Use a different home to proceed.

[oracle@ol7-112-dg1 db_1]$


Executing "gridSetup.sh" Fails with "ERROR: The home is not clean"(Doc ID 2279633.1)
Fails with "ERROR: The home is not clean" .

[root]# rm-Rf 
[oracle]$ unzip linuxx64_12201_grid_home.zip -d 

This is a followup Database runInstaller applyRU Failed Me

I finally figured out my error which I should have seen from the beginning and better error reporting would have helped.

Can you guess what’s wrong?

unzip -qod $ORACLE_HOME /vagrant_software/p6880880_190000_LINUX.zip; echo $?
versus
unzip -qod $ORACLE_HOME /vagrant_software/p6880880_190000_Linux-x86-64.zip; echo $?

The wrong platform for opatch was used.
Here is what should have been deployed.

[oracle@ol7-112-dg1 ~]$ unzip -qod $ORACLE_HOME /vagrant_software/p6880880_190000_Linux-x86-64.zip; echo $?
0
[oracle@ol7-112-dg1 ~]$
[oracle@ol7-112-dg1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
29585399;OCW RELEASE UPDATE 19.3.0.0.0 (29585399)
29517242;Database Release Update : 19.3.0.0.190416 (29517242)
OPatch succeeded.
[oracle@ol7-112-dg1 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.21
OPatch succeeded.
[oracle@ol7-112-dg1 ~]$

[oracle@ol7-112-dg1 ~]$ echo $ORACLE_HOME
/u01/app/oracle/product/19.3.0.0/db_1
[oracle@ol7-112-dg1 ~]$

### This failed:
[oracle@ol7-112-dg1 ~]$ $ORACLE_HOME/OPatch/opatch lspatches
/u01/app/oracle/product/19.3.0.0/db_1/OPatch/opatch: line 839: [: too many arguments
/u01/app/oracle/product/19.3.0.0/db_1/OPatch/opatch: line 839: [: too many arguments
Java (1.7) could not be located. OPatch cannot proceed!
OPatch returns with error code = 1
[oracle@ol7-112-dg1 ~]$

### This works but why?
[oracle@ol7-112-dg1 ~]$ $ORACLE_HOME/OPatch/opatch version -jdk $ORACLE_HOME/jdk
OPatch Version: 12.2.0.1.21
OPatch succeeded.
[oracle@ol7-112-dg1 ~]$

### Here is java version and noticed it's 64-Bit
[oracle@ol7-112-dg1 bin]$ $ORACLE_HOME/jdk/bin/java -version
java version "1.8.0_201"
Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
[oracle@ol7-112-dg1 bin]$

$ORACLE_HOME/runInstaller -applyRU /home/oracle/patch/31305339 should now work.

What’s up Doc!

So there I was, reading documentation and planning upgrade but still not perfect.

DBUA Command-Line Syntax for Active and Silent Mode

changeUserTablespacesReadOnly does not show from help but exists in documenation.

$ which dbua
/app/product/19.3.0.0/bin/dbua

$ dbua -help
Usage: dbua [<flag>] [<option>]
Following are the possible flags:
-createPartialBackup – Flag to create a new offline partial RMAN backup by setting the user tablespaces in R/O mode.
-backupLocation
-disableParallelUpgrade – Flag to disable the parallel execution of database upgrade.
-executePreReqs – Flag to execute the pre-upgrade checks alone for the specified database.
-sid | -dbName
-sid
-dbName
-help – Shows this usage help.
-ignorePreReqs – Ignore error conditions in pre-upgrade checks.
-keepEvents – Flag to keep the configured database events during upgrade.
-silent – This flag allows you to carry on configuration in silent mode.
-sid | -dbName
-sid
-dbName
-skipListenersMigration – Flag to skip the listener migration process as part of the database upgrade.

Following are the possible options:
[-asmsnmpPassword – <Specify ASMSNMP user password>]
[-backupLocation – <Specify directory to backup your database before starting the upgrade>]
[-createGRP – <true | false> To create a guaranteed restore point when database is in archive log and flashback mode.]
[-createListener – <true | false> To create a listener in newer release Oracle home specify listenrName:lsnrPort.]
[-dbName – <Specify Database Name>]
[-oracleHome – <Specify the Oracle home path of the database>]
[-sysDBAUserName – <User name with SYSDBA privileges>]
[-sysDBAPassword – <Password for sysDBAUserName user name>]
[-dbsnmpPassword – <Specify DBSNMP user password>]
[-disableUpgradeScriptLogging – <true | false> This command disables the detailed log generation for running SQL scripts during the upgrade process. By default this is enabled. To enable the log generation, don’t specify this command.]
[-emConfiguration – <DBEXPRESS | CENTRAL | BOTH | NONE>]
[-dbsnmpPassword – <Specify DBSNMP user password>]
[-emPassword – <Specify EM admin user password>]
[-emUser – <Specify EM admin username to add or modify targets>]
[-emExpressPort – <Specify the port where EM Express will be configured>]
[-omsHost – <Specify EM management server host name>]
[-omsPort – <Specify EM management server port number>]
[-asmsnmpPassword – <Specify ASMSNMP user password>]
[-ignoreScriptErrors – <true | false> Specify this flag for ignoring ORA errors during custom scripts.]
[-initParam – <Specify a comma separated list of initialization parameter values of the format name=value,name=value>]
[-initParamsEscapeChar – <Specify escape character for comma when a specific initParam has multiple values. If the escape character is not specified backslash is the default escape character>]
[-excludeInitParams – <Specify a comma separated list of initialization parameters to be excluded.>]
[-keepDeprecatedParams – <true | false> To retain deprecated parameters during database upgrade.]
[-localListenerWithoutAlias – To set LOCAL_LISTENER without TNS Alias.]
[-listeners – <To register the database with existing listeners, specify listeners by comma separated listenerName:Oracle Home. Listeners from lower release home are migrated to newer release home. Specifying -listeners lsnrName1,lsnrName2 or -listeners lsnrName1:<Oracle home path>,-listeners lsnrName2:<Oracle home path>, DBUA searches specified listeners from GI home (if configured), target home and source home>]
[-localRacSid – <Specify the local System Identifier of the cluster database if the cluster database is not registered in OCR>]
[-logDir – <Specify the path to a custom log directory>]
[-newGlobalDbName – <Specify New Global Database Name. This option can only be used for Oracle Express Edition upgrade>]
[-newSid – <Specify New System Identifier. This option can only be used for Oracle Express Edition upgrades>]
[-newInitParam – <Specify a comma separated list of initialization parameter values of the format name=value,name=value. Use this option to specify parameters that are allowed only on the target Oracle home>]
[-initParamsEscapeChar – <Specify escape character for comma when a specific initParam has multiple values. If the escape character is not specified backslash is the default escape character>]
[-oracleHomeUserPassword – <Specify Oracle Home user password>]
[-pdbs – <Specify a comma separated list with the names of the pluggable databases (PDB) that will be upgraded. Specify ALL to select all or NONE to select none of the pluggable databases for upgrade>]
-sid | -dbName
-sid – <Specify System Identifier>
[-oracleHome – <Specify the Oracle home path of the database>]
[-sysDBAUserName – <User name with SYSDBA privileges>]
[-sysDBAPassword – <Password for sysDBAUserName user name>]
-dbName – <Specify Database Name>
[-oracleHome – <Specify the Oracle home path of the database>]
[-sysDBAUserName – <User name with SYSDBA privileges>]
[-sysDBAPassword – <Password for sysDBAUserName user name>]
[-pdbsWithPriority – <Specify a comma separated list of pluggable databases (PDB) to be upgraded along with its corresponding priorities (being 1 the top priority) of the format <pdb name>:<upgrade priority>,<pdb name>:<upgrade priority> >]
-sid | -dbName
-sid – <Specify System Identifier>
[-oracleHome – <Specify the Oracle home path of the database>]
[-sysDBAUserName – <User name with SYSDBA privileges>]
[-sysDBAPassword – <Password for sysDBAUserName user name>]
-dbName – <Specify Database Name>
[-oracleHome – <Specify the Oracle home path of the database>]
[-sysDBAUserName – <User name with SYSDBA privileges>]
[-sysDBAPassword – <Password for sysDBAUserName user name>]
[-performFixUp – <true | false> Enable or disable fix ups for the silent upgrade mode.]
[-postUpgradeScripts – <Specify a comma separated list of SQL scripts with their complete pathnames. These scripts will be executed at the end of the upgrade>]
[-preUpgradeScripts – <Specify a comma separated list of SQL scripts with their complete pathnames. These scripts will be executed before the upgrade>]
[-recompile_invalid_objects – <true | false> Recompile invalid objects as part of the upgrade.]
[-upgrade_parallelism – <Specify number of CPU’s to be used for parallel upgrade>]
[-upgradeTimezone – <true | false> Upgrade the timezone files of the database.]
[-upgradeXML – <Specify the path to the existing pre-upgrade XML file> This option only applies to in-place database upgrades.]
[-useExistingBackup – <true | false> To restore database using existing RMAN backup.]
[-useGRP – <Specify the name of the existing guaranteed restore point> To restore the database using a specified guaranteed restore point.]

Even when -createListener show as valid syntax, using -createListener is not recognized.
$ cat run_dbua.sh

 
date
/app/product/19.3.0.0/bin/dbua -silent \
-sid db01 \
-oracleHome /app/product/11.2.0.4 \
-useGRP upgrade19c \
-recompile_invalid_objects TRUE \
-upgradeTimezone TRUE \
-emConfiguration NONE \
-skipListenersMigration \
-createListener false \  --- failed
-upgrade_parallelism 8
date
exit

./run_dbua.sh: line 10: -createListener: command not found
This works.
/app/product/19.3.0.0/bin/dbua -silent -sid db01 -skipListenersMigration -oracleHome /app/product/11.2.0.4 -recompile_invalid_objects true -upgradeTimezone true -emConfiguration NONE -upgrade_parallelism 4 -createListener false

What am i missing?

Hopefully, you will have better luck than I did.

Update: there as a space from line above thanks to https://twitter.com/VincePoore

$ grep -r '[[:blank:]]$' run_dbua.sh
-skipListenersMigration \

$ grep -r '[[:blank:]]$' run_dbua.sh | wc -l
1

Please click on link to open document:

Validate And Perform 19c Data Guard Switchover