A backup queue that retries without drama

Database Vault backs up PostgreSQL, MySQL, MongoDB, SQLite, Firebase and Supabase to Google Cloud Storage. Backups are the kind of feature nobody notices until the day one is missing, so the architecture is mostly about not failing silently.

Two apps, one queue

The product is a Next.js dashboard and an Express + TypeScript worker. The dashboard never performs a backup. It creates a job. The worker, which can run and scale separately, picks jobs up from a BullMQ queue backed by Redis. A slow dump can't slow down the UI, and a dashboard deploy can't interrupt a running backup.

Retries with backoff

Networks and databases fail transiently, so a failed backup is retried up to three times with exponential backoff:

await queue.add("backup", payload, {
  attempts: 3,
  backoff: { type: "exponential", delay: 5_000 },
});

Every job moves through explicit states, from pending through to completed or failed. "It will try again at 14:05" is a better message than an error.

Immediate and scheduled use the same path

"Back up now" and "every night at 02:00" differ only in when the job is added. Both end up in the same queue and the same worker code, so there is one path to test and one place to fix bugs.

Connections as singletons

Redis, Supabase and Cloud Storage clients are created once and reused. It's a small thing that avoids connection exhaustion when many jobs run together.

Telling people

I built the notification side: transactional emails for backup results and account events, plus in-app notifications, with templates in both the Next.js app and the worker. A backup that succeeded quietly is fine. A backup that failed needs to reach a human, and the email is how it does.

Takeaways

• Put long-running work in a worker with its own queue.
• Make retries a default, and make job state visible.
• Give "now" and "scheduled" the same code path.
• Spend as much care on the failure email as on the backup.