Troubleshooting ORA-01017 from DGMGRL validate

I was reviewing new 19c Data Guard implementation.

DGMGRL validate is my favorite command to detect any issues.

Using validate, shows issue for one of the databases in Data Guard Configuration.

Apologizes for the funky format, since wordpress changed layout, it’s becoming more difficult to use, but I digress.

--- Here are the errors:

DGMGRL> validate network configuration for all;
ORA-01017: invalid username/password; logon denied

DGMGRL> validate static connect identifier for all;
ORA-01017: invalid username/password; logon denied

--- Check TNS from $DB_HOME:

$ cat tnsnames.ora

FALCON =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 10.1.71.242)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = FALCON)
)
)

--- Connect using TNS from above SUCCEED.

$ sqlplus sys@FALCON as sysdba

--- Connect using static connect identifier FAILED.

$ sqlplus sys@'(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.1.71.242)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=FALCON_DGMGRL)(INSTANCE_NAME=FALCON)(SERVER=DEDICATED)(STATIC_SERVICE=TRUE)))' as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Nov 10 14:00:29 2020
Version 19.8.0.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.

ORA-01017: invalid username/password; logon denied

--- Check password file from DB_HOME to find file size is different comparing to other host.
--- Create copy for password file and copy from other host to this host.
--- What's strange is if password file was an issue, then why did sqlplus sys@FALCON as sysdba succeeded?
--- Regardless, I like to keep them the same.

$ ls -l orapw*
-rw-r-----. 1 oracle oinstall 4608 Nov 10 14:11 orapwFALCON
-rw-r-----. 1 oracle oinstall 1536 Nov 9 16:45 orapwFALCON.bak

--- Check listener.ora from $GI_HOME and there's there's the problem.
ORACLE_HOME is set to GI vs DB home.

$ diff listener.ora listener.ora.bak
16c16
< (ORACLE_HOME = /u01/product/19c)
---
> (ORACLE_HOME = /u01/grid/19c)

Thinking out loud here.

Oracle Data Guard Broker and Static Service Registration (Doc ID 1387859.1)

Note: Static “_DGMGRL” entries are no longer needed as of Oracle Database 12.1.0.2 in Oracle Data Guard Broker configurations.

Since database was upgraded from 11.2 to 19c, existing static connect identifier was carried forward.

It might be better to remove existing configuration and create new configuration without _DGMGL; otherwise, may encounter the same issue for next upgrade to 20c.