Affected
Degraded performance from 4:00 PM to 11:06 PM
Degraded performance from 4:00 PM to 11:06 PM
- ResolvedResolved
The Zooniverse team is excited to announce that we've launched our new backend stats service, ERAS, and project + organization stats pages have returned to speedy working order. Thank you for your patience while we built and deployed this new service!
- UpdateUpdate
Our dev team continues their work on a new stats service to replace the current, permanently degraded service. Work on the new service is nearing completion, with roll-out of the new service taking place by the end of October.
A reminder: project stats via the current degraded service will often take multiple minutes to show, and is only successful a fraction (<50%) of the time. That said, stats can still be obtained in many cases with a little perseverance and page reloads. We thank you for your continued patience and look forward to the new service in the coming weeks.
- MonitoringMonitoring
After working with our cloud computing vendor for multiple weeks, we were unable to restore the stats service to its expected performance. While we were able to bring a degraded version of the service back online, we expect long query times, request timeouts, and overall poor performance from this service for the near future.
In response, our dev team has started building a new stats service to replace the older, permanently degraded service. Work has begun, and we aim to bring the new service online by September 2023.
We thank you for your continued patience.
- UpdateUpdate
We are continuing to make slow progress in restoring the stats service. On Thur 6 April, we were able to revive the service, but it is operating in a degraded state (requests are running slow, plots on project stats page will take up to a minute each to populate, etc). The Zooniverse development team continues to work with our cloud computing vendor to improve the performance, and we are beginning to take longer term steps (over next few months) to replace the problematic backend service.
- UpdateUpdate
We continue to work with our cloud computing vendor to resolve the issue that continues to affect our stats service. Again, we thank you for your continued patience.
- IdentifiedIdentified
Investigations indicate that the cause of the ongoing stats service outage is due to an issue with a cloud-hosted backend service. We are working with our cloud computing vendor to pinpoint and resolve the issue; thank you for your patience as we work to bring the stats service back online
- InvestigatingInvestigating
Starting at 10am US Central time on Monday, February 27, the Zooniverse stats service became non-responsive. This service provides histograms of classification and comments on a project stats page, which are not functioning at this time. We are actively investigating the issue and will provide updates here as we learn more.
Please know that this issue does not affect data collection in any way: classification submitted to project workflows and comments submitted to Talk are all functioning normally. Furthermore, project level stats that appear on the project landing page, individual stats on personalized Zooniverse homepage, and individual stats on the new FEM version of the project classify page all remain functional.