Rambling Readers System Status

Live Status\Incident History

September 8, 2025

Failed Execution / R2 Replication

Outage

Opened: Sep 8, 2025, 1:05 PM UTC

Duration: 56min 33sec

  • Opened
    Sep 8, 2025, 1:05 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Sep 8, 2025, 2:02 PM UTC

    Issue resolved on monitor recovery.

September 1, 2025

Job Not Completed / R2 Replication

Outage

Opened: Sep 1, 2025, 2:02 PM UTC

Duration: 35sec

  • Opened
    Sep 1, 2025, 2:02 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Sep 1, 2025, 2:02 PM UTC

    Issue resolved on monitor recovery.

August 28, 2025

Failed Execution / R2 Replication

Outage

Opened: Aug 28, 2025, 9:06 PM UTC

Duration: 55min 48sec

  • Opened
    Aug 28, 2025, 9:06 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Aug 28, 2025, 10:01 PM UTC

    Issue resolved on monitor recovery.

August 17, 2025

Job Not Completed / R2 Replication

Outage

Opened: Aug 17, 2025, 4:02 PM UTC

Duration: 58min 38sec

  • Opened
    Aug 17, 2025, 4:02 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Aug 17, 2025, 5:01 PM UTC

    Issue resolved on monitor recovery.

August 16, 2025

Site Unreachable / Web Application

Outage

Opened: Aug 16, 2025, 7:25 PM UTC

Duration: 8min 48sec

  • Opened
    Aug 16, 2025, 7:25 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Aug 16, 2025, 7:34 PM UTC

    Issue resolved on monitor recovery.

August 15, 2025

Site Unreachable / Web Application

Outage

Opened: Aug 15, 2025, 5:18 PM UTC

Duration: 1min 31sec

  • Opened
    Aug 15, 2025, 5:18 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Aug 15, 2025, 5:19 PM UTC

    Issue resolved on monitor recovery.

August 13, 2025

Job Not Completed / R2 Replication

Outage

Opened: Aug 13, 2025, 8:02 PM UTC

Duration: 59min 14sec

  • Opened
    Aug 13, 2025, 8:02 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Aug 13, 2025, 9:01 PM UTC

    Issue resolved on monitor recovery.

July 31, 2025

Failed Execution / R2 Replication

Missing Data

Opened: Jul 31, 2025, 7:01 PM UTC

Duration: 50min 30sec

  • Opened
    Jul 31, 2025, 7:01 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 31, 2025, 7:51 PM UTC
  • Resolved
    Jul 31, 2025, 7:51 PM UTC

    This was just a failed worker and the message was re-queued

Site Unreachable / Web Application

Outage

Opened: Jul 31, 2025, 9:22 AM UTC

Duration: 14min 52sec

  • Opened
    Jul 31, 2025, 9:22 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 31, 2025, 9:36 AM UTC

    Issue resolved on monitor recovery.

  • Resolved
    Jul 31, 2025, 10:20 AM UTC

    Caused by issue at Webdock data centre: https://status.webdock.io/cmdr7b90g00qprrob85w64j0y

July 24, 2025

Site Unreachable / Celery Status (task queue)

Outage

Opened: Jul 24, 2025, 2:00 AM UTC

Duration: 1hr 59min 55sec

  • Opened
    Jul 24, 2025, 2:00 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 24, 2025, 4:00 AM UTC

    Issue resolved on monitor recovery.

  • Resolved
    Jul 24, 2025, 7:56 AM UTC

    Caused by power issues at Webdock where our server is hosted. See: https://status.webdock.io/cmdgo959n00406abcdu66ubw9

Site Unreachable / Web Application

Outage

Opened: Jul 24, 2025, 12:38 AM UTC

Duration: 1hr 26min 49sec

  • Opened
    Jul 24, 2025, 12:38 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 24, 2025, 2:05 AM UTC

    Issue resolved on monitor recovery.

  • Resolved
    Jul 24, 2025, 7:55 AM UTC

    Caused by power issues at Webdock where our server is hosted. See: https://status.webdock.io/cmdgo959n00406abcdu66ubw9

July 22, 2025

Site Unreachable / Web Application

Outage

Opened: Jul 22, 2025, 7:28 AM UTC

Duration: 9min 50sec

  • Opened
    Jul 22, 2025, 7:28 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 22, 2025, 7:38 AM UTC

    Issue resolved on monitor recovery.

July 21, 2025

Site Unreachable / Web Application

Outage

Opened: Jul 21, 2025, 1:08 PM UTC

Duration: 8min 30sec

  • Opened
    Jul 21, 2025, 1:08 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 21, 2025, 1:17 PM UTC

    Issue resolved on monitor recovery.

July 20, 2025

Job Not Completed / Database Backup

Maintenance

Posted: Jul 20, 2025, 4:06 PM UTC

  • Opened
    Jul 20, 2025, 4:06 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 20, 2025, 5:08 PM UTC
  • Resolved
    Jul 20, 2025, 5:08 PM UTC

    Database backup suspended due to long running resource intensive process running on the VPS.

July 19, 2025

Failed Execution / R2 Replication

Outage

Opened: Jul 19, 2025, 10:00 PM UTC

Duration: 1min 3sec

  • Opened
    Jul 19, 2025, 10:00 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 19, 2025, 10:01 PM UTC

    Issue resolved on monitor recovery.

Healthcheck Assertion Failed / Celery Status (task queue)

Outage

Opened: Jul 19, 2025, 8:01 AM UTC

Duration: 1hr 9min 16sec

  • Opened
    Jul 19, 2025, 8:01 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 19, 2025, 9:10 AM UTC
  • Resolved
    Jul 19, 2025, 9:11 AM UTC

    Celery queue was unresponsive. Fixed by restarting the bookwyrm-worker.service. Might have been caused by currently generate image preview cards which started with around 450,000 items in the queue and is now down to some 220,000.

July 13, 2025

Failed Execution / Database Backup

Outage

Opened: Jul 13, 2025, 7:46 PM UTC

Duration: 3hrs 45min 34sec

  • Opened
    Jul 13, 2025, 7:46 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 13, 2025, 11:31 PM UTC

    Issue resolved on monitor recovery.

July 8, 2025

Failed Execution / Database Backup

Maintenance

Posted: Jul 8, 2025, 7:30 PM UTC

  • Opened
    Jul 8, 2025, 7:30 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 9, 2025, 10:16 PM UTC

Failed Execution

Maintenance

Posted: Jul 8, 2025, 7:30 PM UTC

  • Opened
    Jul 8, 2025, 7:30 PM UTC
  • Resolved
    Jul 9, 2025, 10:16 PM UTC

Site Unreachable / Web Application

Minor Outage

Opened: Jul 8, 2025, 5:11 PM UTC

Duration: 1min 14sec

  • Opened
    Jul 8, 2025, 5:11 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 8, 2025, 5:13 PM UTC

    Issue resolved on monitor recovery.

  • Resolved
    Jul 8, 2025, 10:35 PM UTC

    Misconfiguration of S3 Storage custom domain

Healthcheck Assertion Failed / Celery Status (task queue)

Minor Outage

Opened: Jul 8, 2025, 5:10 PM UTC

Duration: 2hrs 26sec

  • Opened
    Jul 8, 2025, 5:10 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 8, 2025, 7:10 PM UTC

    Issue resolved on monitor recovery.

  • Resolved
    Jul 8, 2025, 10:34 PM UTC

    Misconfiguration of S3 Storage custom domain

July 3, 2025

Site Unreachable / Web Application

Outage

Opened: Jul 3, 2025, 4:40 PM UTC

Duration: 1min 36sec

  • Opened
    Jul 3, 2025, 4:40 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jul 3, 2025, 4:41 PM UTC

    Issue resolved on monitor recovery.

June 14, 2025

Site Unreachable / Web Application

Maintenance

Posted: Jun 14, 2025, 2:56 PM UTC

  • Opened
    Jun 14, 2025, 2:56 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 14, 2025, 3:06 PM UTC

    Issue resolved on monitor recovery.

  • Unresolved
    Jun 14, 2025, 3:08 PM UTC

    Issue re-opened on monitor failure.

  • Resolved
    Jun 14, 2025, 3:12 PM UTC

    Issue resolved on monitor recovery.

  • Unresolved
    Jun 14, 2025, 3:32 PM UTC

    Issue re-opened on monitor failure.

  • Resolved
    Jun 14, 2025, 3:50 PM UTC

    Issue resolved on monitor recovery.

June 12, 2025

Site Unreachable / Web Application

Outage

Opened: Jun 12, 2025, 9:40 AM UTC

Duration: 5min 2sec

  • Opened
    Jun 12, 2025, 9:40 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 12, 2025, 9:45 AM UTC

    Issue resolved on monitor recovery.

June 10, 2025

Site Unreachable / Web Application

Outage

Opened: Jun 10, 2025, 11:37 AM UTC

Duration: 1min 57sec

  • Opened
    Jun 10, 2025, 11:37 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 10, 2025, 11:39 AM UTC

    Issue resolved on monitor recovery.

June 1, 2025

Job Not Completed

Outage

Opened: Jun 1, 2025, 5:06 PM UTC

Duration: 2hrs 5min 36sec

  • Opened
    Jun 1, 2025, 5:06 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 1, 2025, 7:12 PM UTC

    Issue resolved on monitor recovery.

Job Not Completed

Maintenance

Posted: Jun 1, 2025, 2:25 PM UTC

  • Opened
    Jun 1, 2025, 2:25 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Jun 1, 2025, 2:30 PM UTC

April 18, 2025

Site Unreachable / Web Application

Outage

Opened: Apr 18, 2025, 12:01 PM UTC

Duration: 19min 57sec

  • Opened
    Apr 18, 2025, 12:01 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Apr 18, 2025, 12:20 PM UTC
  • Resolved
    Apr 18, 2025, 12:21 PM UTC

    System became unresponsive. Resolved by restarting VPS.

Failed Execution

Outage

Opened: Apr 18, 2025, 12:00 PM UTC

Duration: 20min 23sec

  • Opened
    Apr 18, 2025, 12:00 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Apr 18, 2025, 12:20 PM UTC

    Issue resolved on monitor recovery.

  • Resolved
    Apr 18, 2025, 12:22 PM UTC

    System became unresponsive. Resolved by restarting VPS.

April 4, 2025

Job Not Completed / Database Backup

Outage

Opened: Apr 4, 2025, 8:07 PM UTC

Duration: 2hrs 36min 17sec

  • Opened
    Apr 4, 2025, 8:07 PM UTC

    Issue opened on monitor failure.

  • Identified
    Apr 4, 2025, 10:43 PM UTC
  • Resolved
    Apr 4, 2025, 10:43 PM UTC

    Cron job for the database back-up was accidentally deleted when another cron job was being configured. This appears to be due to a bug with webmin that listed two different cron jobs with the same name in the GUI, leading me to think they were duplicates.

Job Not Completed

Maintenance

Posted: Apr 4, 2025, 1:59 AM UTC

  • Opened
    Apr 4, 2025, 1:59 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Apr 4, 2025, 10:49 AM UTC
  • Resolved
    Apr 4, 2025, 10:49 AM UTC

    This was due to maintenance activity associated with optimising images stored on the server to reduce disk space requirements.

April 3, 2025

Job Not Completed

Maintenance

Posted: Apr 3, 2025, 1:10 AM UTC

  • Opened
    Apr 3, 2025, 1:10 AM UTC

    Issue opened on monitor failure.

  • Resolved
    Apr 3, 2025, 7:58 PM UTC

    Issue resolved on monitor recovery.

April 1, 2025

Job Not Completed

Maintenance

Posted: Apr 1, 2025, 9:04 PM UTC

  • Opened
    Apr 1, 2025, 9:04 PM UTC

    Issue opened on monitor failure.

  • Resolved
    Apr 2, 2025, 8:18 AM UTC

    Issue resolved on monitor recovery.