Zooniverse - Notice history

All systems operational

Operational

Project Pages and Classifications - Operational

Project Builder - Operational

Talk - Operational

Caesar - Operational

Classrooms - Operational

Notice history

Feb 2023

Stats Service Down / Degraded
  • Resolved
    UTC
    Resolved

    The Zooniverse team is excited to announce that we've launched our new backend stats service, ERAS, and project + organization stats pages have returned to speedy working order. Thank you for your patience while we built and deployed this new service!

  • Update
    UTC
    Update

    Our dev team continues their work on a new stats service to replace the current, permanently degraded service. Work on the new service is nearing completion, with roll-out of the new service taking place by the end of October.

    A reminder: project stats via the current degraded service will often take multiple minutes to show, and is only successful a fraction (<50%) of the time. That said, stats can still be obtained in many cases with a little perseverance and page reloads. We thank you for your continued patience and look forward to the new service in the coming weeks.

  • Monitoring
    UTC
    Monitoring

    After working with our cloud computing vendor for multiple weeks, we were unable to restore the stats service to its expected performance. While we were able to bring a degraded version of the service back online, we expect long query times, request timeouts, and overall poor performance from this service for the near future.

    In response, our dev team has started building a new stats service to replace the older, permanently degraded service. Work has begun, and we aim to bring the new service online by September 2023.

    We thank you for your continued patience.

  • Update
    UTC
    Update

    We are continuing to make slow progress in restoring the stats service. On Thur 6 April, we were able to revive the service, but it is operating in a degraded state (requests are running slow, plots on project stats page will take up to a minute each to populate, etc). The Zooniverse development team continues to work with our cloud computing vendor to improve the performance, and we are beginning to take longer term steps (over next few months) to replace the problematic backend service.

  • Update
    UTC
    Update

    We continue to work with our cloud computing vendor to resolve the issue that continues to affect our stats service. Again, we thank you for your continued patience.

  • Identified
    UTC
    Identified

    Investigations indicate that the cause of the ongoing stats service outage is due to an issue with a cloud-hosted backend service. We are working with our cloud computing vendor to pinpoint and resolve the issue; thank you for your patience as we work to bring the stats service back online

  • Investigating
    UTC
    Investigating

    Starting at 10am US Central time on Monday, February 27, the Zooniverse stats service became non-responsive. This service provides histograms of classification and comments on a project stats page, which are not functioning at this time. We are actively investigating the issue and will provide updates here as we learn more.

    Please know that this issue does not affect data collection in any way: classification submitted to project workflows and comments submitted to Talk are all functioning normally. Furthermore, project level stats that appear on the project landing page, individual stats on personalized Zooniverse homepage, and individual stats on the new FEM version of the project classify page all remain functional.

Jan 2023

Slow & Failed Project Page Loading
  • Resolved
    UTC
    Resolved

    Working with our cloud computing vendor, we've fully restored services and are seeing normal page load activity. We will remain vigilant in case the issue arises again, but we expect no further issues at this time.

  • Identified
    UTC
    Identified

    The Zooniverse team has identified the primary cause of the FEM issue and is working with our cloud computing vendor to path the underlying problem. We have implemented a temporary work-around that restores FEM projects to a normal working state and hope to have a permanent fix in place in the near future.

  • Update
    UTC
    Update

    The Zooniverse team has been hard at work investigating the issues. Fortunately, it looks like the issue only affects projects on the new front end code base (FEM) such as PRINT, Planet Hunters TESS, and other transcription projects. The issue seems to be intermittent and depends on region (e.g., OK in US, failed connections in UK -- or vice versa). We are working with our cloud computing vendor to identify and fix the issue.

  • Investigating
    UTC
    Investigating

    We are investigating volunteer reports of inaccessible project pages that will not load, particularly the PRINT project (https://www.zooniverse.org/projects/printmigrationnetwork/print) and other projects on the new front end code base (FEM).

Dec 2022

No notices reported this month

Dec 2022 to Feb 2023

Next