The deployment of updated PSMA data to the PRODUCTION environment involves the 'switching' of database tables containing previous data to tables containing new data. This typically does not cause any perceivable outage to services for customers.
The tables should have associated indexes which maintain search performance.
Incident Description
The deployment of updated GNAF data on 18 March initially appeared to be successful but shortly after the deployment, address verification services began to degrade significantly to failure.
On investigation it was found that the tables containing the new data did not have their associated indexes which resulted in unacceptably poor performance.
An attempt was made to create the indexes on the tables in-situ, however, this was unsuccessful due to active sessions on the database effectively 'locking' the table and preventing the creation of indexes.
A decision was made to restart the database which dropped all active sessions and allowed the indexes to be created. Shortly after the restart, all address verification services returned to normal expected operation.
Lessons learnt and actions
A step has been added to the new data deployment process to confirm required indexes exist on tables before they are deployed to any environment.
The deployment of new data took place during business hours. This was not normal practice but was done as the deployment was way overdue. This incident highlights the importance of conducting deployments out-of-hours to minimise disruption to customers in the event of unexpected issues.
Posted Mar 20, 2019 - 12:31 AEDT
Resolved
Address verification services have now been restored. The post mortem details for this incident will be updated in the next 24hrs.
Posted Mar 18, 2019 - 17:36 AEDT
Investigating
PSMA Cloud and Addresses API are experiencing address verification issues (no response). We are currently investigating and hope to have all services back online ASAP.
Posted Mar 18, 2019 - 15:31 AEDT
This incident affected: APIs (Addresses API) and PSMA Cloud.