[oracle@racnode-dc1-2 dbs]$ srvctl start database -d hawk PRCR-1079 : Failed to start resource ora.hawk.db CRS-5017: The resource action "ora.hawk.db start" encountered the following error: ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+DATA/HAWK/spfilehawk.ora' ORA-17503: ksfdopn:10 Failed to open file +DATA/HAWK/spfilehawk.ora ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Operation not permitted ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba) . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/racnode-dc1-2/crs/trace/crsd_oraagent_oracle.trc". CRS-2674: Start of 'ora.hawk.db' on 'racnode-dc1-2' failed CRS-5017: The resource action "ora.hawk.db start" encountered the following error: ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+DATA/HAWK/spfilehawk.ora' ORA-17503: ksfdopn:10 Failed to open file +DATA/HAWK/spfilehawk.ora ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Operation not permitted ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba) . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/racnode-dc1-1/crs/trace/crsd_oraagent_oracle.trc". CRS-2674: Start of 'ora.hawk.db' on 'racnode-dc1-1' failed CRS-2632: There are no more servers to try to place resource 'ora.hawk.db' on that would satisfy its placement policy
Starting from sqlplus did not help either and why does it even matter?
[oracle@racnode-dc1-2 dbs]$ sqlplus / as sysdba SQL*Plus: Release 12.1.0.2.0 Production on Thu Feb 28 18:11:58 2019 Copyright (c) 1982, 2014, Oracle. All rights reserved. Connected to an idle instance. 18:11:59 SYS @ hawk2:>startup; ORA-01078: failure in processing system parameters ORA-01565: error in identifying file '+DATA/HAWK/spfilehawk.ora' ORA-17503: ksfdopn:10 Failed to open file +DATA/HAWK/spfilehawk.ora ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Operation not permitted ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 54321 (oinstall), current egid = 54322 (dba) 18:12:01 SYS @ hawk2:>exit Disconnected [oracle@racnode-dc1-2 dbs]$
SRVCTL Fails to Start Instance with ORA-17503 ORA-27303 But sqlplus Startup is Fine [1322959.1]
Even though the situation did not match support note, the solution provided did work
[oracle@racnode-dc1-2 dbs]$ id oracle uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54318(asmdba),54322(dba),54323(backupdba),54324(oper),54325(dgdba),54326(kmdba) [oracle@racnode-dc1-2 dbs]$ ls -l $GRID_HOME/bin/oracle -rwxrwxr-x 1 oracle oinstall 292020952 Feb 27 22:57 /u01/app/12.1.0.1/grid/bin/oracle [oracle@racnode-dc1-2 dbs]$ chmod 6751 $GRID_HOME/bin/oracle [oracle@racnode-dc1-2 dbs]$ ls -l $GRID_HOME/bin/oracle -rwsr-s--x 1 oracle oinstall 292020952 Feb 27 22:57 /u01/app/12.1.0.1/grid/bin/oracle [oracle@racnode-dc1-2 dbs]$ ls -l $ORACLE_HOME/bin/oracle -rwxrwsr-x 1 oracle dba 324409192 Feb 27 22:51 /u01/app/oracle/12.1.0.1/db1/bin/oracle [oracle@racnode-dc1-2 dbs]$ ========================================================================================== [oracle@racnode-dc1-1 dbs]$ ls -l $GRID_HOME/bin/oracle -rwxrwxr-x 1 oracle oinstall 292020952 Feb 27 21:41 /u01/app/12.1.0.1/grid/bin/oracle [oracle@racnode-dc1-1 dbs]$ chmod 6751 $GRID_HOME/bin/oracle [oracle@racnode-dc1-1 dbs]$ ls -l $GRID_HOME/bin/oracle -rwsr-s--x 1 oracle oinstall 292020952 Feb 27 21:41 /u01/app/12.1.0.1/grid/bin/oracle [oracle@racnode-dc1-1 dbs]$ ls -l $ORACLE_HOME/bin/oracle -rwxrwsr-x 1 oracle dba 324409192 Feb 27 21:35 /u01/app/oracle/12.1.0.1/db1/bin/oracle [oracle@racnode-dc1-1 dbs]$ [oracle@racnode-dc1-1 dbs]$ srvctl start database -d hawk [oracle@racnode-dc1-1 dbs]$ srvctl status database -d hawk -v Instance hawk1 is running on node racnode-dc1-1. Instance status: Open. Instance hawk2 is running on node racnode-dc1-2. Instance status: Open. [oracle@racnode-dc1-1 dbs]$ [oracle@racnode-dc1-1 dbs]$ crsctl stat res -t -------------------------------------------------------------------------------- Name Target State Server State details -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.CRS.dg ONLINE ONLINE racnode-dc1-1 STABLE ONLINE ONLINE racnode-dc1-2 STABLE ora.DATA.dg ONLINE ONLINE racnode-dc1-1 STABLE ONLINE ONLINE racnode-dc1-2 STABLE ora.FRA.dg ONLINE ONLINE racnode-dc1-1 STABLE ONLINE ONLINE racnode-dc1-2 STABLE ora.LISTENER.lsnr ONLINE ONLINE racnode-dc1-1 STABLE ONLINE ONLINE racnode-dc1-2 STABLE ora.asm ONLINE ONLINE racnode-dc1-1 Started,STABLE ONLINE ONLINE racnode-dc1-2 Started,STABLE ora.net1.network ONLINE ONLINE racnode-dc1-1 STABLE ONLINE ONLINE racnode-dc1-2 STABLE ora.ons ONLINE ONLINE racnode-dc1-1 STABLE ONLINE ONLINE racnode-dc1-2 STABLE -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.MGMTLSNR 1 ONLINE ONLINE racnode-dc1-1 169.254.203.248 172. 16.9.10,STABLE ora.cvu 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.hawk.db 1 ONLINE ONLINE racnode-dc1-1 Open,STABLE 2 ONLINE ONLINE racnode-dc1-2 Open,STABLE ora.mgmtdb 1 ONLINE ONLINE racnode-dc1-1 Open,STABLE ora.oc4j 1 OFFLINE OFFLINE STABLE ora.racnode-dc1-1.vip 1 ONLINE ONLINE racnode-dc1-1 STABLE ora.racnode-dc1-2.vip 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.scan1.vip 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.scan2.vip 1 ONLINE ONLINE racnode-dc1-2 STABLE ora.scan3.vip 1 ONLINE ONLINE racnode-dc1-2 STABLE -------------------------------------------------------------------------------- [oracle@racnode-dc1-1 dbs]$