Kubernetes backup for stateful services: choosing tools and testing restores

Why back up Kubernetes for stateful services

A Kubernetes cluster tolerates small failures well: a pod dies, it’s recreated, the service responds again. That works for stateless workloads. When you have databases, queues, artifact stores or logging systems, the cost of mistakes rises sharply: you lose not just processes, but data and history.

In real incidents rarely only one thing is lost. Often PersistentVolumes (or on-disk data), manifests (Deployment/StatefulSet, Service, Ingress, RBAC), Secrets and ConfigMaps, and surrounding configuration suffer at the same time: StorageClass, access policies, network parameters. If you restore PVs but forget connection secrets or permissions, the app won’t start, even though the data is physically present.

Stateful services are also harder because of startup order and consistency. PostgreSQL with replication or Kafka require careful recovery: which nodes start first, how volumes are attached, and whether split‑brain might occur. So backup here is not just copying files, it’s a way to return to a predictable state.

It helps to separate two scenarios up front:

• A pod failure is often fixed by recreating it and users may not notice.
• Loss of on-disk data or cluster metadata without backups almost always becomes downtime, manual reconstruction and risk of irreversible loss.

Kubernetes backups address common failures: accidental namespace deletion, a bad helm upgrade, broken permissions, a node with state lost, volume corruption, or migrating to another cluster. But they won’t save you from everything. If malware encrypts application data and that encrypted data makes it into backups, or if backups aren’t verified by restores, reliability is an illusion. Backup only makes sense together with a clear restore plan and regular checks.

What to back up: data, configuration and dependencies

Backing up Kubernetes for stateful services is rarely just “copy the volume”. For proper recovery you need the data and the way it was connected, the permissions, and several non‑obvious dependencies.

First answer the key question: do you plan to restore the application in the same cluster after an outage, or bring it up in a new cluster from scratch? In the second case the backup usually needs to be broader and stricter.

A practical way is to think in three layers:

• Resources in a namespace (Deployments, StatefulSets, Services, Ingress, ConfigMaps).
• CRDs and related resources (for example, from database operators).
• Access and permissions (ServiceAccounts, Roles/RoleBindings, sometimes ClusterRole/ClusterRoleBinding).

Secrets need special attention. You can copy them as‑is, but consider how they’re protected. If Secrets are encrypted in the cluster, restoring without keys (KMS or API server keys) can yield corrupted values. It’s better if keys are stored separately from backups and have their own rotation and recovery procedures.

For PVs it depends on the storage. CSI snapshots provide fast point‑in‑time recovery but usually only within compatible storage and often within one data center. If snapshots aren’t available, you must rely on application‑level backups (dumps) or the storage system’s capabilities.

For databases it’s usually best to combine approaches. Volume snapshots help with quick RTO, and logical dumps (pg_dump, mysqldump) add portability and protection from logical errors (e.g. accidental table deletion). Practical example: take a snapshot before a release and run a logical dump at night, and verify that the dump actually restores.

Don’t forget external dependencies: an external S3 bucket, external DB, LDAP, licenses, DNS records. You can’t “back them up” with a Kubernetes tool, but you can record configuration and endpoints and set up backup and restore testing for those systems so the chain doesn’t break during an incident.

Strategy requirements: RPO, RTO and recovery model

A backup strategy starts not with a tool but with two questions: how much data are you willing to lose, and how quickly do you need to be back after a failure. For stateful services this is crucial because problems usually come from data, not manifests.

RPO (Recovery Point Objective) in plain terms is “how fresh the data should be after recovery”. If RPO is 15 minutes, you may lose at most the last 15 minutes of changes.

RTO (Recovery Time Objective) is “how long you can be unavailable”. If RTO is 1 hour, the service must be running again within an hour, even if limited.

Define RPO/RTO by talking to the business and ops: what hurts more — data loss or downtime? For internal Git and CI RTO often matters more, while payments or medical records prioritize RPO.

Backup frequency and type follow from RPO. Full backups give a clear restore point but are heavy in time and storage. Incremental backups save space and speed up regular snapshots, but complicate restore and increase testing requirements.

Document the minimum: target RPO/RTO for each critical service, snapshot frequency and retention, actual restore time from tests, and who triggers a recovery and on what signal.

Then choose a recovery model. “Kubernetes backup” at the cluster level (resources, manifests, Secrets, PVC metadata) helps quickly rebuild services, but doesn’t always guarantee consistency inside volumes. Storage‑level backups (volume snapshots, replication) often give a more reliable path for databases and queues but require understanding how they map to Kubernetes objects and startup order.

A DR plan isn’t only for “big” companies: one critical service can make a second cluster or site mandatory. As a rule: if downtime within one data center is unacceptable, plan recovery in another availability zone or separate cluster. Decide in advance where backups live and who has access.

Velero, Kasten, Portworx: how they differ conceptually

Velero, Kasten K10 and Portworx PX‑Backup solve similar problems (Kubernetes backup and restore) but differ in approach. Understanding each product’s core makes choosing easier.

Velero is often chosen as a basic starting point. Out of the box it collects Kubernetes resources (manifests, namespaces, CRDs with proper config), while handling on‑disk data depends on the environment. For snapshots and correct operation with specific storage you usually need plugins (CSI, object storage providers, etc.). It’s flexible and generally cheaper license‑wise but requires careful setup and maintenance.

Kasten K10 targets a more managed enterprise backup platform: policies, schedules, reports, application‑centric visibility, role model, and convenient restore scenarios. It’s chosen where audit, compliance, delegation and a clear operational model matter. Licensing is typically more expensive but needs less “assembly from parts”.

Portworx PX‑Backup is logical when backup is tightly tied to the storage layer and storage functions matter: consistent snapshots, volume mobility, cross‑cluster migration. It’s especially convenient if Portworx is already in use. If not, you add another platform to operate.

Evaluate not only features but total cost of ownership: licensing, operational costs (updates, plugins, troubleshooting restores), storage integrations and control requirements (policies, RBAC, audit).

If you run on‑prem (e.g. in your own DC) and need clear processes and support, factor in resources for regular restore tests and managing the whole chain, not just the tool.

How to pick a tool for your cluster: non‑marketing criteria

Start from how your data and storage are arranged, not the brand. For stateful services the deciding factor is how the tool handles volumes, snapshots and real recovery scenarios, not the UI.

First check storage compatibility. If you expect fast restores via snapshots, the tool must support CSI snapshots for your StorageClasses (and actually work in your cluster, not only in docs). If snapshots are unavailable, decide in advance what will replace them: file copying, sidecar agents, or storage integration. This directly affects RPO and downtime.

Next look at policies and restore workflows. It should allow rules by namespace and labels, exclude unnecessary objects, restore volumes consistently, and restore into a different namespace (for tests) or into another cluster (for DR/migrations). Also require clear diagnostics: what was backed up, what was skipped and why.

Security is separate. Organizations with strict requirements need backup encryption, role separation, operation audit and a history of actions: who ran backups, who restored, and what exactly changed.

Finally, check how the tool fits into on‑call processes. A good sign is a predictable restore procedure for the night shift: minimal manual steps, sensible alerts and the ability to run practice restores regularly.

If unsure, run a short pilot on one critical stateful service: the same backup and restore scenario and the same metrics. Often that’s enough to make the decision obvious.

Where to store backups: S3‑compatible, NFS, on‑prem and offsite

Storage location often matters more than the tool. The same backup can be reliable or useless depending on whether the storage survives a cluster outage, human error and site failures.

Object storage (S3‑compatible) usually wins for regular backups: it scales, supports versioning, and often offers immutability features. File storage (NFS) is convenient for local access and simple integration but is often a single point of failure, especially if it sits near the cluster.

The 3‑2‑1 rule for Kubernetes: three copies (e.g. daily, weekly, monthly), two media types (object and file or archive), and one copy offsite (another DC or provider).

Credentials deserve a separate mention. Create a backup account in the storage separate from production and give it minimal rights. A good practice is to separate accounts per environment (dev/stage/prod) so a test script can’t delete prod backups.

Decide retention according to investigations and rollback needs: how long to keep daily, weekly and yearly archives. Enable versioning and protections against deletion/overwrite (immutability or at least forbid delete for the backup account), otherwise a mistaken rm can render backups meaningless.

For on‑prem put backups so they survive a site outage: the repository should not live on the same racks and power zone as the cluster. If you have a second site, replicate there to avoid the worst case: “the cluster and backups died together”.

Step by step: build a working backup strategy

Start with an inventory. List critical namespaces and stateful services: databases, queues, file stores, and the PVs attached to them. Note which data can be rebuilt from external systems (e.g. replicas) and which exist only in the cluster.

Then define rules. Policy should answer: how often to back up, how long to keep copies, and what to exclude. Configurations (manifests, CRDs) can be saved more often; large volumes less often but kept longer. Exclusions matter: caches, temporary directories, test namespaces and data that’s easier to recreate.

A simple scheme often works: daily backup of PVs and resources in critical namespaces, more frequent backups of configurations only, clear retention (e.g. 7 daily, 4 weekly, 6 monthly), and an explicit list of exclusions (dev, staging or specific labels).

Next set up access. Create a separate service account for backups, give it only required roles and access to storage secrets. Ensure it has no excessive rights and secrets aren’t exposed in pipelines.

Don’t skip encryption. Enable backup encryption and verify restore access: can the team start the service after a restore without hunting for keys, tokens and certificates? A common issue is data restored but the app won’t start due to missing Secrets or wrong volume permissions.

Capture the setup in short documentation, otherwise the strategy dies. One page is usually enough: what is backed up, where copies live, how to run backup and restore, who is responsible. Add minimal commands (backup, list, restore, verify), on‑call contacts, windows for test restores and success criteria (what checks must pass after recovery).

How to test restores: regular scenarios and metrics

A backup is only as good as your ability to restore it. Restore tests should be regular, repeatable and produce clear numbers.

The safest start is to restore into a separate namespace. That checks PV/PVC creation, StatefulSet/Deployment startup, and Secrets/ConfigMaps being applied without touching production. It’s convenient to add a suffix to resource names and temporarily disable ingress so the test doesn’t become an accidental deployment.

Next level — test on another cluster. This DR scenario reveals issues with DNS, CSI driver versions, storage access and external dependencies (external DBs, queues, licenses). Run such tests at least quarterly or after major cluster changes.

What to validate after restore

Seeing pods in Running state isn’t enough. Check application‑level consistency: for Postgres run control queries and compare row counts, for Kafka read several latest messages, for a file service verify hashes of a couple of reference files. Good rule: validate what matters to the business, not what’s easiest for engineers.

Metrics and automation

To measure real RTO, record the time from the restore command to when the service responds to a simple health check (plus one “real” request) and has warmed caches.

Reporting usually needs: actual RTO (total time and time to app readiness), percentage of successful restores over a period, “drift” (what did not restore or changed), and actual data loss (real RPO measured by checkpoints).

Automate tests on a schedule: nightly restore into a test namespace, monthly restore to a standby cluster, and save logs and timings in a central journal.

Common mistakes and pitfalls when backing up stateful workloads in Kubernetes

What fails most often

The most painful case: archives exist but the service cannot be restored. The reason is usually not the tool but mismatches between how the cluster was configured and how you try to bring it up after failure.

A typical problem: PVs don’t come up because the target cluster has a different StorageClass or lacks the required CSI driver. Backups restore manifests, but storage cannot provision volumes with the same parameters (disk type, zones, encryption settings). Pods get stuck in Pending and time is wasted.

Second trap: lost secrets and keys. The app seems deployed but can’t decrypt data, connect to the DB or open TLS certificates. This is especially painful for stateful services: data is restored but keys differ and the service refuses to run.

Third mistake: snapshots taken without freezing writes. For PostgreSQL, MySQL, Elasticsearch and similar systems consistency matters. A hot snapshot without quiesce (or without the DB’s backup mechanism) can produce subtle corruption: restore may succeed but later reveal broken indexes or missing transactions.

Fourth problem: overly broad access to backups. If backups sit in one place and access is too wide, they can be deleted accidentally by an operator, a cleanup script or a retention policy mistake.

And the most common reason for failure: restores haven’t been tested for months. On the incident day you discover CRD version changes, expired tokens, the new cluster can’t reach the repository, or the restore requires manual steps nobody remembers.

How to reduce risk

A short set of regular checks helps:

• Map StorageClass and CSI for recovery: know what will be available in the new cluster and how volumes map.
• Store secrets so they can be restored with data, and separately verify encryption keys.
• For databases use a consistent mode: either application‑aware backups or snapshots with quiesce and integrity checks.
• Separate roles: who creates backups, who can delete them, who can run restores.
• Test restores on a schedule and record actual RTO and data loss points.

A simple routine: every two weeks restore a test namespace with Postgres and the app, then run one or two control operations (login, write, read). It’s cheaper than troubleshooting during a real outage.

Short checklist before enabling backups in production

Before enabling backups on a production cluster, record a minimum set of rules you can check in 10 minutes. This doesn't replace a full runbook but catches common failures before the first incident.

First agree what you consider data loss and how much downtime is tolerable. Different stateful services have different criticality: a DB, a queue and a file store are rarely equal.

Then confirm:

• A list of critical services exists, each with RPO and RTO and a chosen recovery model (same cluster or new cluster).
• Backup covers not only volumes (PV) but manifests, settings and dependencies: Namespace, CRD, ConfigMap, Secrets, and Ingress and access policies if needed for startup.
• Backup storage is isolated: separate credentials, protection from bulk deletion, versioning or immutability enabled, and a plan for credential compromise.
• A restore test exists and the date of the last successful run is known, plus measured RTOs: time to Pod start, time to application readiness and data checks.
• An owner is assigned: who monitors schedules, who reacts to errors, where the short restore instruction lives and how to access it quickly during an incident.

Make the checklist concrete with an example. For instance: "PostgreSQL for billing: RPO 15 minutes, RTO 1 hour, restore into a new namespace, checks: record counts and latest transactions." If it can’t be described in one line, the plan isn’t ready.

After going live, set a reminder to review settings whenever storage, access policies or Kubernetes versions change. Such changes most often break restores.

Next steps: pilot, runbook and backup infrastructure

After that, the tool name matters less than turning backup into a repeatable process. The most practical path is a short pilot where you observe real timings, sizes, load and restore quality.

Start with 1–2 applications that reflect your typical risk: a PostgreSQL with PVC and a small service with critical Secrets and ConfigMaps. Run the full cycle: backup, simulate data loss (in a test environment) and restore, then verify at the application level (login, queries, background jobs).

A 2–3 week plan can stay simple: choose pilot apps and RPO/RTO, set schedules and retention, run several restore scenarios (same cluster, separate namespace, another cluster if available), record metrics (time, size, success) and write a short 1–2 page runbook.

At the same time assess infrastructure. Bottlenecks are often disks and network: is nighttime bandwidth enough, is there IOPS headroom, how much space is needed with retention, is there offsite or storage isolation?

If on‑prem, plan server resources for the backup repository and redundancy: separate nodes, RAID or replication, power, monitoring and an offsite scheme. Sometimes a systems integrator like GSE.kz is involved to supply servers and help build infrastructure and backup processes with support and placement requirements in mind.