Introduction
There is a common saying that an untested backup is not a backup. But what if your backups don't even get created properly and you do not realize it? In this post I want to dive into why that's a bad thing and what methods we as admins have at our disposal to conter it.
Why Monitoring Backups Matters
Backup failures can often go unnoticed for extended periods, sometimes days or even weeks, which can lead to severe consequences. Silent failures - those that don't trigger obvious alerts - pose an even greater risk because they remain undetected until data loss occurs. Early detection of these issues is crucial to preventing potential data loss and ensuring the integrity of your backups. Effective monitoring not only safeguards your data but also provides peace of mind for infrastructure teams, knowing that backup processes are running smoothly.
The Problem: Common Monitoring Gaps
Even daily success emails are easy to miss when they do not come; you can easily not realize that the daily check-in is missing, which creates a blind spot in your backup monitoring.
Manual Checks Are Unreliable
Relying on manual checks for backups demands constant discipline and sharp memory, which can easily falter during busy periods. This approach often leaves gaps in visibility, especially when managing backup statuses across multiple systems, increasing the risk of unnoticed failures.
Delayed Notifications
Email-only alerts frequently get lost in overflowing inboxes, delaying awareness of critical backup failures. Without real-time notifications, team members might miss urgent alerts, hindering prompt response and resolution.
Lack of Integration
One significant challenge in backup monitoring is the lack of integration between alert systems and the communication tools that teams use daily. When backup alerts are isolated from platforms like Microsoft Teams or Discord, it becomes difficult for teams to coordinate timely responses. This disconnect can lead to delays in addressing critical issues, increasing the risk of data loss. Integrating backup notifications directly into team communication channels ensures that alerts are visible, actionable, and can prompt immediate collaboration among team members.
The Solution: remote-backups Notifications
remote-backups now supports flexible notification delivery through:
- Discord
- Microsoft Teams
- Telegram
- Webhook
- More soon
To not have duplication or potentially outdated documentation in a blog post, please refer to the documentation on how to configure each type: https://docs-next.bennetg.de/products/remote-backups/datastore/notifications
Following these steps will ensure your backup alerts are routed promptly and reliably to your preferred communication channels.
Case Study: Detecting Backup Failures Early
Recently, we migrated an on-premises Proxmox Backup Server (PBS) to remote-backups, which now backs up a local Proxmox VE (PVE) cluster. During the initial infrastructure review to document requirements, the engineer and I discovered that notification emails were incorrectly configured. As a result, backup failure alerts were not being correctly sent.
Two weeks before our review, the on-prem PBS host experienced a kernel panic following a system update, rendering it non-functional. This led to two weeks of missed backups - a critical risk in any daily business environment where data integrity is vital.
After recovering from this incident and completing the migration to remote-backups, we implemented Microsoft Teams alerts to notify the team immediately whenever backups or other jobs fail. To add an extra layer of reliability, we also set up a webhook integration with the customer’s internal developer portal. This ensures that if notifications through Teams fail for whatever reason, another alert mechanism is in place.
Thanks to these improvements, the customer now has confidence that any backup failure will be promptly detected and communicated, minimizing risk and improving overall data protection.
This case highlights the importance of proper alert configuration and the value of integrating multiple notification channels to safeguard critical backup processes.
Conclusion
Backup monitoring isn't optional - it's essential. With remote-backups' flexible notification system, you can ensure your team stays informed about backup health in real-time, using the tools they already use daily.