Quantcast
Channel: Thinking Out Loud
Viewing all articles
Browse latest Browse all 668

ORA-12514: TNS:listener during switchover using DGMGRL

$
0
0

Not sure what I am doing wrong because DEFAULT StaticConnectIdentifier does not seem to work.

Google and MOS were not much help.

Using StaticConnectIdentifier as created by dmgrl would fail with the following error during switchover.
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

The error can be found in the listener.log

When using tnsnames for StaticConnectIdentifier, switchover works like a charm.

I keep thinking, there must be something wrong with my configuration but as you can see from the test case, the only changes were to StaticConnectIdentifier.

Both databases are on the same host with each listening on different ports.

Updated: Just found this note
DGMGRL>switchover to Fails with ORA-12514 (Doc ID 1582927.1)
Oracle Database – Enterprise Edition – Version 12.1.0.1 to 12.1.0.1 [Release 12.1]

Interesting to see if this will work.

oracle@arrow:hawklas:/home/oracle
$ dgmgrl /
DGMGRL for Linux: Version 11.2.0.4.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> connect sys/oracle@dc_las
Connected.
DGMGRL> connect sys/oracle@dc_san
Connected.
DGMGRL> show configuration verbose

Configuration - dg_hawk

  Protection Mode: MaxPerformance
  Databases:
    hawklas - Primary database
    hawksan - Physical standby database

  Properties:
    FastStartFailoverThreshold      = '30'
    OperationTimeout                = '30'
    FastStartFailoverLagLimit       = '30'
    CommunicationTimeout            = '180'
    ObserverReconnect               = '0'
    FastStartFailoverAutoReinstate  = 'TRUE'
    FastStartFailoverPmyShutdown    = 'TRUE'
    BystandersFollowRoleChange      = 'ALL'
    ObserverOverride                = 'FALSE'
    ExternalDestination1            = ''
    ExternalDestination2            = ''
    PrimaryLostWriteAction          = 'CONTINUE'

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database verbose hawklas

Database - hawklas

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    hawklas

  Properties:
    DGConnectIdentifier             = 'dc_las'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '1800'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = ''
    LogFileNameConvert              = ''
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold  = '30'
    SidName                         = 'hawklas'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=hawklas_DGMGRL)(INSTANCE_NAME=hawklas)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '8191'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database verbose hawksan

Database - hawksan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds (computed 0 seconds ago)
  Apply Lag:       0 seconds (computed 0 seconds ago)
  Apply Rate:      32.00 KByte/s
  Real Time Query: ON
  Instance(s):
    hawksan

  Properties:
    DGConnectIdentifier             = 'dc_san'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '30'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '1800'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = ''
    LogFileNameConvert              = ''
    FastStartFailoverTarget         = ''
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    ApplyLagThreshold               = '0'
    TransportLagThreshold           = '0'
    TransportDisconnectedThreshold  = '30'
    SidName                         = 'hawksan'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=hawksan_DGMGRL)(INSTANCE_NAME=hawksan)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '8191'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database hawksan

Database - hawksan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds (computed 1 second ago)
  Apply Lag:       0 seconds (computed 1 second ago)
  Apply Rate:      32.00 KByte/s
  Real Time Query: ON
  Instance(s):
    hawksan

Database Status:
SUCCESS
DGMGRL> connect sys/oracle@dc_las
Connected.
DGMGRL> switchover to hawksan
Performing switchover NOW, please wait...
Operation requires a connection to instance "hawksan" on database "hawksan"
Connecting to instance "hawksan"...
Connected.
New primary database "hawksan" is opening...
Operation requires startup of instance "hawklas" on database "hawklas"
Starting instance "hawklas"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up instance "hawklas" of database "hawklas"

DGMGRL> exit
oracle@arrow:hawklas:/home/oracle
$ sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Jan 10 17:36:20 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SYS@hawklas> startup
ORACLE instance started.

Total System Global Area 1068937216 bytes
Fixed Size                  2260088 bytes
Variable Size             385876872 bytes
Database Buffers          675282944 bytes
Redo Buffers                5517312 bytes
Database mounted.
Database opened.
SYS@hawklas> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@arrow:hawklas:/home/oracle
$ dgmgrl /
DGMGRL for Linux: Version 11.2.0.4.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration

Configuration - dg_hawk

  Protection Mode: MaxPerformance
  Databases:
    hawksan - Primary database
    hawklas - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
ORA-16534: switchover, failover or convert operation in progress
DGM-17017: unable to determine configuration status

DGMGRL> show configuration

Configuration - dg_hawk

  Protection Mode: MaxPerformance
  Databases:
    hawksan - Primary database
    hawklas - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> edit database hawksan set property StaticConnectIdentifier='dc_san';
Property "staticconnectidentifier" updated
DGMGRL> edit database hawklas set property StaticConnectIdentifier='dc_las';
Property "staticconnectidentifier" updated'
DGMGRL> connect sys/oracle@dc_san
Connected.
DGMGRL> switchover to hawklas
Performing switchover NOW, please wait...
Operation requires a connection to instance "hawklas" on database "hawklas"
Connecting to instance "hawklas"...
Connected.
New primary database "hawklas" is opening...
Operation requires startup of instance "hawksan" on database "hawksan"
Starting instance "hawksan"...
ORACLE instance started.
Database mounted.
Database opened.
Switchover succeeded, new primary is "hawklas"
DGMGRL> show configuration

Configuration - dg_hawk

  Protection Mode: MaxPerformance
  Databases:
    hawklas - Primary database
    hawksan - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database hawksan

Database - hawksan

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds (computed 1 second ago)
  Apply Lag:       0 seconds (computed 1 second ago)
  Apply Rate:      0 Byte/s
  Real Time Query: ON
  Instance(s):
    hawksan

Database Status:
SUCCESS

DGMGRL> exit
oracle@arrow:hawklas:/home/oracle
$
oracle@arrow:hawklas:/home/oracle
$ tns
oracle@arrow:hawklas:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$ tnsping dc_las

TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 10-JAN-2016 17:50:08

Copyright (c) 1997, 2013, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (SDU=32767) (SEND_BUF_SIZE=32767) (RECV_BUF_SIZE=32767) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = arrow)(PORT = 1521))) (CONNECT_DATA = (SERVICE_NAME = hawklas)))
OK (0 msec)
oracle@arrow:hawklas:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$ tnsping dc_san

TNS Ping Utility for Linux: Version 11.2.0.4.0 - Production on 10-JAN-2016 17:50:14

Copyright (c) 1997, 2013, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/sqlnet.ora


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (SDU=32767) (SEND_BUF_SIZE=32767) (RECV_BUF_SIZE=32767) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = arrow)(PORT = 1531))) (CONNECT_DATA = (SERVICE_NAME = hawksan)))
OK (0 msec)
oracle@arrow:hawklas:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$ lsnrctl status listener_las

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-JAN-2016 17:50:26

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow)(PORT=1521)(SEND_BUF_SIZE=32767)(RECV_BUF_SIZE=32767)))
STATUS of the LISTENER
------------------------
Alias                     listener_las
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                10-JAN-2016 16:15:51
Uptime                    0 days 1 hr. 34 min. 34 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/dbhome_1/network/log/listener_las.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521)))
Services Summary...
Service "foo2" has 1 instance(s).
  Instance "foo2", status UNKNOWN, has 1 handler(s) for this service...
Service "hawk" has 1 instance(s).
  Instance "hawklas", status READY, has 1 handler(s) for this service...
Service "hawklas" has 2 instance(s).
  Instance "hawklas", status UNKNOWN, has 2 handler(s) for this service...
  Instance "hawklas", status READY, has 1 handler(s) for this service...
Service "hawklas_DGB" has 1 instance(s).
  Instance "hawklas", status READY, has 1 handler(s) for this service...
The command completed successfully
oracle@arrow:hawklas:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$ lsnrctl status listener_san

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-JAN-2016 17:50:29

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow)(PORT=1531)(SEND_BUF_SIZE=32767)(RECV_BUF_SIZE=32767)))
STATUS of the LISTENER
------------------------
Alias                     listener_san
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                10-JAN-2016 16:15:54
Uptime                    0 days 1 hr. 34 min. 35 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/dbhome_1/network/log/listener_san.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531)))
Services Summary...
Service "hawk" has 1 instance(s).
  Instance "hawksan", status READY, has 1 handler(s) for this service...
Service "hawksan" has 2 instance(s).
  Instance "hawksan", status UNKNOWN, has 2 handler(s) for this service...
  Instance "hawksan", status READY, has 1 handler(s) for this service...
Service "hawksan_DGB" has 1 instance(s).
  Instance "hawksan", status READY, has 1 handler(s) for this service...
The command completed successfully
oracle@arrow:hawklas:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$

It seems that static registration is not working for db_unique_name_DMGRL

oracle@arrow:hawksan:/home/oracle
$ dgmgrl /
DGMGRL for Linux: Version 11.2.0.4.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show database hawklas StaticConnectIdentifier
  StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=hawklas_DGMGRL)(INSTANCE_NAME=hawklas)(SERVER=DEDICATED)))'
DGMGRL> show database hawksan StaticConnectIdentifier
  StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=hawksan_DGMGRL)(INSTANCE_NAME=hawksan)(SERVER=DEDICATED)))'
DGMGRL> exit
oracle@arrow:hawksan:/home/oracle
$ lsnrctl status listener_las

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-JAN-2016 22:16:09

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521)(SEND_BUF_SIZE=32767)(RECV_BUF_SIZE=32767)))
STATUS of the LISTENER
------------------------
Alias                     listener_las
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                10-JAN-2016 22:02:05
Uptime                    0 days 0 hr. 14 min. 3 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/dbhome_1/network/log/listener_las.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521)))
Services Summary...
Service "foo2" has 1 instance(s).
  Instance "foo2", status UNKNOWN, has 1 handler(s) for this service...
Service "hawk" has 1 instance(s).
  Instance "hawklas", status READY, has 1 handler(s) for this service...
Service "hawklas" has 2 instance(s).
  Instance "hawklas", status UNKNOWN, has 2 handler(s) for this service...
  Instance "hawklas", status READY, has 1 handler(s) for this service...
Service "hawklas_DGB" has 1 instance(s).
  Instance "hawklas", status READY, has 1 handler(s) for this service...
The command completed successfully
oracle@arrow:hawksan:/home/oracle
$ lsnrctl status listener_san

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 10-JAN-2016 22:16:12

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531)(SEND_BUF_SIZE=32767)(RECV_BUF_SIZE=32767)))
STATUS of the LISTENER
------------------------
Alias                     listener_san
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                10-JAN-2016 22:02:03
Uptime                    0 days 0 hr. 14 min. 9 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/11.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/product/11.2.0/dbhome_1/network/log/listener_san.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531)))
Services Summary...
Service "hawk" has 1 instance(s).
  Instance "hawksan", status READY, has 1 handler(s) for this service...
Service "hawksan" has 2 instance(s).
  Instance "hawksan", status UNKNOWN, has 2 handler(s) for this service...
  Instance "hawksan", status READY, has 1 handler(s) for this service...
Service "hawksan_DGB" has 1 instance(s).
  Instance "hawksan", status READY, has 1 handler(s) for this service...
The command completed successfully
oracle@arrow:hawksan:/home/oracle
$ sqlplus sys@'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=hawklas_DGMGRL)(INSTANCE_NAME=hawklas)(SERVER=DEDICATED)))' as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Jan 10 22:17:09 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Enter password:
ERROR:
ORA-12514: TNS:listener does not currently know of service requested in connect
descriptor


Enter user-name: ^C
oracle@arrow:hawksan:/home/oracle
$ sqlplus sys@'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=hawksan_DGMGRL)(INSTANCE_NAME=hawksan)(SERVER=DEDICATED)))' as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Jan 10 22:17:20 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Enter password:
ERROR:
ORA-12514: TNS:listener does not currently know of service requested in connect
descriptor


Enter user-name: ^C
oracle@arrow:hawksan:/home/oracle
$ sqlplus sys@'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=hawklas_DGB)(INSTANCE_NAME=hawklas)(SERVER=DEDICATED)))' as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Jan 10 22:17:48 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

ARROW:(SYS@hawklas):PRIMARY> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@arrow:hawksan:/home/oracle
$ sqlplus sys@'(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=arrow.localdomain)(PORT=1531))(CONNECT_DATA=(SERVICE_NAME=hawksan_DGB)(INSTANCE_NAME=hawksan)(SERVER=DEDICATED)))' as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Sun Jan 10 22:17:55 2016

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Enter password:

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

ARROW:(SYS@hawksan):PHYSICAL STANDBY> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
oracle@arrow:hawksan:/home/oracle
$

$ tns
oracle@arrow:hawksan:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$ grep -i dg listener.ora
        (GLOBAL_NAME = hawklas_DGB)
        (GLOBAL_NAME = hawklas_DGMGRL)
        (GLOBAL_NAME = hawksan_DGB)
        (GLOBAL_NAME = hawksan_DGMGRL)
oracle@arrow:hawksan:/u01/app/oracle/product/11.2.0/dbhome_1/network/admin
$

UPDATED Jan 16, 2015.

Thanks to Shivananda Rao (http://shivanandarao-oracle.com/) for pointing out the obvious which I was blind to.

IT’S NOT GLOBAL_NAME = hawklas_DGMGRL !!!

It’s GLOBAL_DBNAME = hawklas_DGMGRL (Missing DB)

R.I.P.

 



Viewing all articles
Browse latest Browse all 668