Resolved -
Snyk engineers have been monitoring the environment and confirmed that all systems are operating within normal baselines. The incident response has been stood down.
Jan 16, 21:25 UTC
Update -
All Recurring Tests from yesterday are now complete.
We have enabled today's batch; however, some tests may be approximately 1-2 hours behind their normal schedule.
Due to remediation actions taken during this incident, we are observing delays in the propagation of issue data to the Reporting and Rest API after import.
We will continue monitor and provide updates as we progress.
Jan 16, 14:17 UTC
Update -
Recurring tests are progressing well, and we are not observing any of the symptoms seen yesterday.
We will continue to monitor progress and keep you updated.
Jan 16, 08:06 UTC
Monitoring -
Mitigation efforts have been implemented across all environments. Recurring tests are being rerun in all environments, with SNYK-AU-01 set to resume at 4 AM ACT. Customers may experience delayed results over the next 48 hours. Additionally, there may be a brief fluctuation in risk scores and reachability, which will be addressed in the next test batch. Snyk engineers are closely monitoring all environments.
Jan 15, 19:59 UTC
Update -
Snyk engineers are continuing to deploy the mitigation to individual tenants. The mitigation has been applied to the Multi-Tenant US region. Customers may experience delays in recurring tests for the environment over the next 48 hours. Mitigation efforts are ongoing.
Jan 15, 18:53 UTC
Identified -
Our engineers have identified a fix, which is currently being deployed.
We will provide a further update once the implementation is complete.
Jan 15, 15:51 UTC
Update -
It has been identified that all our environments are affected.
The investigation remains ongoing, and all recurring tests have been suspended pending identification and mitigation of the root cause.
Jan 15, 14:22 UTC
Update -
The investigation remains ongoing, and all recurring tests have been suspended pending identification and mitigation of the root cause.
Jan 15, 14:16 UTC
Update -
Due to the impact across both app.snyk.io and app.au.snyk.io, customers may notice a temporary fluctuation in risk/priority scores for recurring test results.
Our investigations into the root cause of this incident remain ongoing.
Jan 15, 13:22 UTC
Update -
Our investigations are ongoing, and we have identified that customers in our app.au.snyk.io environment are also affected.
Therefore, customers in the specified environment may notice that the Daily Recurring Tests scheduled for execution on the 14th of January may be delayed. We have not cancelled any tests in this environment to date.
We do not expect any other flows across both affected environments to be affected.
Jan 15, 12:23 UTC
Update -
To avoid impacting our planned Recurring Tests for today (15th January), we will cancel any Daily Recurring Tests from yesterday's (14th January) schedule that haven't initiated or completed.
We will provide further updates as our investigation progresses.
Jan 15, 10:48 UTC
Investigating -
Our Engineers are investigating the degradation affecting our Daily Recurring Tests, across the app.snyk.io environment.
Some customers may notice that the Daily Recurring Tests that were scheduled to run on the 14th of January may be delayed. This may also affect Daily Recurring Tests scheduled for today (15th January).
The Snyk Incident Response has been invoked. We will keep you updated on our progress.
Jan 15, 09:38 UTC