Insight · Operations

"We take backups" is not a recovery position

A backup you have never restored is a hypothesis. Four things separate a recovery point from a file that happens to be large: what is inside it, whether anything checks it, whether the retention label is true, and who is structurally able to trigger a restore.

1. A database dump is not a backup of an ERP

A construction ERP's most irreplaceable content is not in PostgreSQL. It is the drawings, the site photos, the signed permits, the approved PDFs — the filestore. A dump restored without them gives you a system that knows an attachment exists, references it on a record, and cannot produce it.

Each Majal recovery archive contains three things, on purpose:

dump.sql
A transaction-consistent PostgreSQL export — consistent, not merely recent.
filestore/
Every uploaded drawing, photo, PDF and attachment.
manifest.json
Platform, PostgreSQL and installed-module versions.

The manifest is the one people leave out, and it is the one that decides whether a restore is possible at all in eighteen months. A dump is only restorable into a compatible environment, and "compatible" is a fact about module versions, not a feeling. Recording it at capture time costs nothing; reconstructing it during an incident costs the incident.

2. A checksum you compute once is decoration

Every archive is hashed with SHA-256 and the hash is stored in Majal. The part that matters is that it is checked again — before download and before restore.

Silent corruption is not exotic. Disks, filesystems, copies and object stores all fail quietly, and a corrupt backup is indistinguishable from a good one until the moment you need it, which is by definition the moment you have no alternative. A hash computed at write time and never re-read tells you that the file was fine once.

Archives are written with mode 0600 below the protected application data volume. The application refuses a configured backup path outside that volume, and checks minimum free storage before exporting — because a truncated archive that stopped when the disk filled is exactly the corrupt-but-present file the checksum exists to catch.

3. Retention labels should not lie about age

Majal keeps seven curated slots: today, yesterday, the day before yesterday, a weekly checkpoint refreshed every Monday, a two-week checkpoint on every second ISO week, a monthly checkpoint on the first of each month, and a three-month checkpoint on 1 January, April, July and October.

These are described as retention tiers, not fabricated exact ages, and the screen always shows the actual timestamp and age. That distinction sounds pedantic until somebody plans a recovery around a slot labelled "monthly" that is in fact five weeks old — a five-week-old assumption during an incident is worse than no assumption.

For the same reason, a new installation shows empty scheduled slots and fills them over time, rather than back-filling labels it cannot honour.

4. A live web process must never be able to replace its own database

This is the structural point, and it is the one that most "one-click restore" features get wrong.

If the application can restore itself, then anything that compromises the application — a bad session, a mis-scoped administrator, a bug in a controller — can destroy the system and its history in one call. The convenience and the catastrophe are the same code path.

Majal's restore is therefore two steps by construction:

  1. A Platform Owner opens a verified recovery point and chooses Prepare Restore.
  2. They type the database name and a confirmation phrase.
  3. Majal issues a one-time recovery code that expires after two hours and writes a checksum-bound request into the protected data volume.
  4. A deployment operator — a different person, at the host — runs the restore script with that request ID and code.

Note what the split actually enforces: the application can only ever request. It cannot execute. Two roles, a short expiry and a checksum-bound request are three independent things an attacker or an accident would have to defeat.

The script itself is not a shortcut either. It stops the application, validates the code, expiry, archive path, checksum and ZIP contents, restores into a replacement database, restores the filestore, validates the installed-module table, and only then removes the temporary safety database. If validation fails, it rolls the original database and filestore back before restarting.

The line worth quoting Never test restore for the first time during an incident. Run a quarterly restore drill into an isolated non-production environment, and record the measured recovery time — because the number you have measured is the only recovery time objective you can promise a client.

What seven local recovery points do not protect against

They are fast operational rollback: somebody deleted a project, an import went wrong, an upgrade went badly. They are not a disaster-recovery strategy, and Majal's own documentation says so before anybody has to find out.

The gap is host or account loss. Seven archives on the same volume as the database share the volume's fate, and archives reachable with the application's own credentials share the application's fate. So the documented pre-sale requirements are:

  • copy encrypted backups to a separate account and region;
  • make at least one copy immutable against application credentials;
  • monitor the daily job, archive age, free space and off-site replication — a backup job that silently stopped is the classic failure, and it is only visible if somebody is watching the age of the newest archive;
  • define client-specific recovery-point and recovery-time objectives, in numbers;
  • keep the encryption key outside the application host;
  • test both database records and attachment downloads after a restore, not just that the login page loads;
  • take a protected point immediately before every application upgrade.

Why this belongs on a product page at all

Because "we take backups" is what everybody says, and it distinguishes nothing. The questions that distinguish something are: is the filestore in it, is the checksum re-verified before use, does the age shown match the age it has, can the web process restore itself, and when did you last measure a restore.

Those are answerable questions, and the answers are either yes or they are marketing.

In the product

Recovery points and the two-step restore live in majal_administration, alongside the client role model and the immutable administration audit. See the feature summary →