Delays in updating reporting data
Incident Report for Snyk

Resolved
Please read our recent updates for more context on this incident. We’re pleased to report that the historical issue data in Snyk Reporting from November 30th to December 6th has now been backfilled.

Our method here has been to run an insert query that effectively rolls forward all issues from the affected dates. This approach will give you much better data for these dates, and where we can, we will be refining this further to improve that accuracy.

The total issue counts and list of issues in the issue tab are as up to date as our data freshness allows – at the moment this is up to 9 hours due to a rapid increase in the number of monitored projects as a result of the Log4Shell vulnerability. We are working hard to bring this data freshness back to a smaller, reliable window.

Further work is already in progress to overhaul the reporting platform allowing for better reliability and scalability, and an RCA on this incident will be published in the coming days. Please feel free to reach out to support with any questions or concerns.
Posted Dec 17, 2021 - 16:34 UTC
Identified
Note that this incident affects only the Summary and Issues tabs of the Reporting section; the Dependencies and Licenses tabs are unaffected.

High loads on our systems mean that our freshness is unfortunately between 8 and 9 hours at this time, we are continuing to work to reduce this lag.

Backfilling of the data for the time between 30 November and 5 December has began and is starting to be reflected in our reporting, we will confirm when this is completed, the current outlook is tomorrow (17Dec). Data for all other dates is correct.

We are moving this incident into identified status as data dips backfill is in progress and we understand and are working on mitigations for our systems being under high load at this current time and expect to be able to stabilise this going forward.
Posted Dec 16, 2021 - 18:38 UTC
Update
Note that this incident affects only the Summary and Issues tabs of the Reporting section; the Dependencies and Licenses tabs are unaffected.

We continue to work to bring our data freshness permanently back into the 3 hour freshness window, it remains between 1 and 5 hours at any point in time at the moment.

Work on backfilling missing data between 30 November and 5 December is still ongoing. We are pursuing multiple avenues to ensure that the data is complete and correct as soon as possible but it is proving challenging. Historical data between 30 November and 5 December remains incomplete, which also affects current total counts from 30 November onwards. Data for all other dates is correct.

We will continue to keep this incident in the “Investigating” state until we are able to confirm a full and permanent fix.
Posted Dec 15, 2021 - 10:27 UTC
Update
Note that this incident affects only the Summary and Issues tabs of the Reporting section; the Dependencies and Licenses tabs are unaffected.

As a result of high load resulting from the recent Log4J vulnerability announcement, we are having challenges keeping our reporting data within the 5 hours freshness window. We are working hard to stabilise this as soon as possible.

Work on backfilling missing data between 30 November and 5 December is ongoing. We are pursuing multiple avenues to ensure that the data is complete and correct as soon as possible. Historical data between 30 November and 5 December remains incomplete, which also affects current total counts from 30 November onwards. Data for all other dates is correct.

We will continue to keep this incident in the “Investigating” state until we are able to confirm a full and permanent fix.
Posted Dec 13, 2021 - 18:10 UTC
Update
Note that this incident affects only the Summary and Issues tabs of the Reporting section; the Dependencies and Licenses tabs are unaffected.

We have overcome the issue with the temporary fix, so reporting data is back to being correct with a 5 hour freshness window.

Work on backfilling missing data between 30 November and 5 December is ongoing. We are pursuing multiple avenues to ensure that the data is complete and correct as soon as possible. Historical data between 30 November and 5 December remains incomplete, which also affects current total counts from 30 November onwards. Data for all other dates is correct.

We will continue to keep this incident in the “Investigating” state until we are able to confirm a full and permanent fix.
Posted Dec 11, 2021 - 10:02 UTC
Update
Note that this incident affects only the Summary and Issues tabs of the Reporting section; the Dependencies and Licenses tabs are unaffected.

We have overcome the issue with the temporary fix. Reporting data is currently updating, and we expect to be back within a 5 hour freshness window by 02:00 UTC.

Work on backfilling missing data between 30 November and 5 December is ongoing. We are pursuing multiple avenues to ensure that the data is complete and correct as soon as possible. Historical data between 30 November and 5 December remains incomplete, which also affects current total counts from 30 November onwards. Data for all other dates is correct.

We will continue to keep this incident in the “Investigating” state until we are able to confirm a full and permanent fix.
Posted Dec 10, 2021 - 23:30 UTC
Update
Our temporary fix has had an issue meaning that reporting freshness has again been significantly impacted, we are working as our top priority to get this back up and running to ensure that current reporting is correct.
Posted Dec 10, 2021 - 19:30 UTC
Update
The temporary fix is still in place, so current reporting data continues to be correct with a 5 hour freshness window.

Work on backfilling missing data between 30 November and 5 December is ongoing. We are pursuing multiple avenues to ensure that the data is complete and correct as soon as possible. Historical data between 30 November and 5 December remains incomplete, which also affects current total counts from 30 November onwards. Data for all other dates is correct.

We will continue to keep this incident in the “Investigating” state until we are able to confirm a full and permanent fix.
Posted Dec 10, 2021 - 18:00 UTC
Update
We have made progress since our last update:

We are still using the temporary fix, which means that our reporting data is within 5 hours of current.

However, the data synchronization issue has been resolved, for past and future data coming in.

Only historical data between 30 November and 5 December, and current total counts from 30 November onwards, continue to be affected by the historical data dips that we continue to work on backfilling. We have caught up on data for all other dates.

We will keep this incident in the "Investigating" state until we are able to confirm a full and permanent fix.
Posted Dec 09, 2021 - 16:55 UTC
Update
The temporary fix means that our data is within 5 hours current at the moment for reporting. We have identified a fix for the data synchronization issue we saw on some projects, it has been rolled out and we are monitoring to ensure that further issues are not seen.

Historical (between 30Nov & 7Dec) data and current total counts remain impacted by the historical data dips that we continue to work on backfilling.

Only Historical data from after November 30 is being affected.
We will continue to leave this incident in investigating status until we are able to confirm a full and permanent fix.
Posted Dec 08, 2021 - 19:37 UTC
Update
With our temporary fix in place our reporting system freshness continue to improve, and our teams are making progress towards a full fix.

We have identified are working on filling in missing data gaps, while we continue to work on streamlining the data flow in a steady pace.

We are currently investigating a data synchronization issue with the current fix, that manifests in some projects having an inconsistent number of issues (that is changing through the day).

Only Historical data from after November 30 is being affected.
We will continue to leave this incident in investigating status until we are able to confirm a full and permanent fix.
Posted Dec 07, 2021 - 19:03 UTC
Update
Our mitigation for the current database replication problems are proving successful. At this time, the freshness of our reporting system is greatly improved, but we can not yet guarantee this to be within the usual 3 hour window. We aim to have an update for that tomorrow (Tuesday). Historical data from 30th November until 7th December will contain gaps which we are in the process of filling.

This mitigation enables the reporting data being returned to be correct and up to date as soon as possible. As well as this, we have teams focussed on a full fix. We will leave this incident in investigating status until we are able to confirm a full and permanent fix.
Posted Dec 06, 2021 - 19:43 UTC
Update
Our reporting services is unfortunately still running behind in updating data, During this time, the graphs and totals showing the latest number of issues are inaccurate, and not all issues for the past two days will be visible in the issues tab or in the reporting API.

We appreciate this has significant impact to our customers using reporting and are very focused on resolving is asap. We made some further changes to our systems late yesterday (Sunday) and initial indications are that the system is catching up (although this will take some hours). We will update later today (Monday UTC).

Once the system has caught up then we will will need to repair any gaps in the data to avoid ‘dips’ in the historic information and we are preparing to do that.
Posted Dec 06, 2021 - 08:04 UTC
Update
Our reporting service is still running behind on updating data. During this time, the graphs and totals showing the latest number of issues are inaccurate, and not all issues for the past two days will be visible in the issues tab or in the reporting API. We are actively working on resolving this incident and will continue to provide further updates here.
Posted Dec 05, 2021 - 13:20 UTC
Update
Our reporting service is still running behind on updating data. During this time, the graphs and totals showing the latest number of issues are inaccurate, and not all issues for the past two days will be visible in the issues tab or in the reporting API. We are actively working on resolving this incident and will provide further updates here.
Posted Dec 04, 2021 - 12:33 UTC
Update
We believe we have identified the cause of this issue and are working to resolve it.
Posted Dec 02, 2021 - 11:15 UTC
Investigating
Our reporting service is currently experiencing slowness in data freshness and may update data later than 3 hours after test or import. Our team is looking into it and will update the status as we investigate and resolve it.
Posted Nov 30, 2021 - 12:59 UTC