Backup and Recovery

POS owns its PostgreSQL database and must be recoverable without reading or restoring a platform database. Provider-specific adapters, including Kenyan mobile-money and fiscalization adapters, do not change that ownership rule.

Current rehearsal

The repository includes a non-destructive logical backup/restore rehearsal:

./scripts/remote-backup-restore-rehearsal.sh

The script defaults to root@damplabs.com, namespace emali2-test, and the emali2-pos-postgres StatefulSet. Override those values only when deliberately targeting another approved environment:

POS_REMOTE_HOST=user@example-host \
POS_K8S_NAMESPACE=example-namespace \
POS_POSTGRES_WORKLOAD=statefulset/example-pos-postgres \
./scripts/remote-backup-restore-rehearsal.sh

kubectl runs on the SSH host. The custom-format dump, temporary restore database, and all row material remain inside the PostgreSQL pod. A restrictive umask is used, and an exit trap removes the temporary database and dump after success, interruption, or failure.

The rehearsal:

creates a compressed custom-format pg_dump;
restores it into a uniquely named temporary database;
requires pg_restore --exit-on-error to complete;
compares the POS base-table count and latest successful Flyway version;
compares a deterministic checksum of critical merchant, store, terminal, shift, sale, line, tender, receipt, refund, stock, and audit rows;
compares column definitions and stable structural metadata for indexes, constraints, routines, and user triggers;
compares POS sequence state; and
drops the temporary database, deletes the dump, and verifies both are gone.

It reports only counts, versions, hashes, byte size, elapsed time, and status. It does not print business rows or credentials.

Verified test evidence

On 17 July 2026, the corrected rehearsal completed against the current remote test POS database:

backup_restore_ok schema=pos tables=50 flyway=v29 critical_checksum=match schema_objects=match sequences=match dump_bytes=212830 elapsed_seconds=3 cleanup=verified

A separate SSH/kubectl check found zero matching temporary databases and zero matching dump files after the final run.

On 24 July 2026, the same rehearsal passed again after the payment-resolution schema release:

backup_restore_ok schema=pos tables=52 flyway=v33 critical_checksum=match schema_objects=match sequences=match dump_bytes=225383 elapsed_seconds=3 cleanup=verified

A separate remote kubectl check again reported restore_databases=0 restore_dumps=0.

After the merchant integration-client registry release on 24 July 2026, the rehearsal passed at the new schema baseline:

backup_restore_ok schema=pos tables=54 flyway=v34 critical_checksum=match schema_objects=match sequences=match dump_bytes=231190 elapsed_seconds=3 cleanup=verified

The rehearsal now requires the workload's authoritative POS_DATABASE_NAME and refuses to substitute PostgreSQL's bootstrap POSTGRES_DB. It validates the namespace and workload reference before assembling the remote kubectl command from shell-quoted arguments. Six guards and ShellCheck cover the database selection, SSH-only kubectl path, cleanup trap and input validation.

This proves that the current PostgreSQL 16 test database can be logically dumped and restored into a fresh database in the same pod, that the selected critical data and schema structure survive that cycle, and that temporary rehearsal artifacts are removed.

Off-cluster backup foundation

The backup image and suspended Kubernetes CronJob provide a provider-neutral off-cluster path. They use PostgreSQL's custom dump format and the standard S3 API; no country, payment provider, fiscal authority, or cloud vendor is part of the backup contract.

The backup path fails closed unless all of these controls pass:

the S3 URI names a dedicated external bucket and any custom endpoint uses a fully qualified, verified HTTPS hostname outside the POS Kubernetes cluster;
bucket versioning is enabled;
object lock is enabled with a default GOVERNANCE or COMPLIANCE retention;
every dump and completion manifest is encrypted locally with an age X25519 public recipient before upload;
S3 server-side encryption is AES256 or aws:kms;
uploaded byte sizes and ciphertext SHA-256 metadata match, both objects report the required SSE mode, and both report an applied object-lock mode and retention expiry; and
the encrypted completion manifest is uploaded only after the encrypted dump.

The backup workload receives only the age public recipient. The private age identity must remain in a separately controlled recovery vault and must never be placed in emali2-pos-backup-secrets or the POS application Secret. S3 credentials must be dedicated to the backup bucket and limited to the required versioning, object-lock, upload, and read-after-write verification operations.

The committed schedule is 01:17 UTC and suspend: true. Deploying that suspended definition creates no backup pod and does not require credentials:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_BACKUP_IMAGE_TAG=<immutable-tag> \
POS_BACKUP_SUSPEND=true \
./k8s/deploy-backup-test.sh

Before enabling it, create emali2-pos-backup-secrets through the approved secret-management workflow with these keys:

POS_BACKUP_S3_URI
POS_BACKUP_AGE_RECIPIENT
POS_BACKUP_SSE_MODE
POS_BACKUP_KMS_KEY_ID when using aws:kms
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
AWS_ENDPOINT_URL only for an S3-compatible provider with a verified HTTPS endpoint

The guarded repository workflow for creating or rotating both backup Secrets is scripts/reconcile-pos-backup-secrets.sh. It accepts one regular, non-symlink JSON bundle owned by the current user with no group/other permissions. The bundle contains six exact objects:

storage: S3 URI with non-empty prefix, region, AES256 or aws:kms, the required KMS key id for aws:kms, and an optional public HTTPS endpoint;
archiveWriter and recoveryReader: distinct access-key identities;
age: the public recipient and separately vaulted recovery identity;
pitr: cluster id plus the dedicated replication username/password; and
recoveryDatabase: credentials used only by the isolated recovered database and application smoke.

Do not place this bundle in the repository. Create it under a protected operator path with umask 077. PLAN validates its schema, separation and permissions locally, performs no SSH/network call and prints only key counts:

POS_BACKUP_SECRET_BUNDLE_FILE=/secure/operator/path/pos-backup-bundle.json \
POS_BACKUP_SECRET_MODE=PLAN \
./scripts/reconcile-pos-backup-secrets.sh

CREATE additionally requires age-keygen; it derives the recipient from the private identity and refuses a mismatch before contacting the cluster. It then requires the exact remote context, both backup CronJobs suspended, both managed Secrets absent, and the explicit create confirmation:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_BACKUP_SECRET_BUNDLE_FILE=/secure/operator/path/pos-backup-bundle.json \
POS_BACKUP_SECRET_MODE=CREATE \
POS_BACKUP_SECRET_CONFIRM=CREATE_POS_BACKUP_SECRETS \
./scripts/reconcile-pos-backup-secrets.sh

The manifest is streamed over SSH standard input and applied with Kubernetes server-side apply. Secret values are not command arguments, Terraform values, temporary manifest files or a client-side last-applied annotation. The command verifies exact Secret key names and reports counts only; it never unsuspends a CronJob. Rotation requires both Secrets to exist, both schedules still suspended, POS_BACKUP_SECRET_MODE=ROTATE, and the separate ROTATE_POS_BACKUP_SECRETS confirmation. Nine guards cover no-network plan, file permissions, confirmations, schedule state, age key matching, split cloud and database credentials, provider-neutral SSE, partial Secret state and secret-free command output.

Read-only storage readiness gate

Before enabling either backup schedule, run the digest-pinned storage auditor. It reads only bucket control metadata and performs no object upload, copy, delete, retention change, policy change or bucket mutation. It proves bucket reachability, versioning, Object Lock with a positive default retention, default encryption matching POS_BACKUP_SSE_MODE, and absence of public ACL grants. AWS_PUBLIC_ACCESS_BLOCK additionally requires all four AWS public access controls; use S3_PRIVATE_ACL only for a compatible provider that does not implement the AWS public-access-block API and whose private-bucket setting has been verified through its provider control plane.

Validate the isolated Kubernetes Job and NetworkPolicy without reading the backup Secret or creating any resources:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_BACKUP_AUDIT_IMAGE=ghcr.io/mainamartin/pos-postgres-backup@sha256:9e4bd250cbad86c422f7089a3323482bd6f894ba92e80c6212ee212baa986a18 \
POS_BACKUP_STORAGE_AUDIT_CONFIRM=RUN_READ_ONLY_STORAGE_AUDIT \
POS_BACKUP_STORAGE_AUDIT_DRY_RUN_ONLY=true \
./scripts/run-pos-backup-storage-audit.sh

After the approved secret-management workflow creates the backup Secret, run the real AWS gate with the same digest and confirmation, omitting POS_BACKUP_STORAGE_AUDIT_DRY_RUN_ONLY. For a compatible non-AWS provider, also set POS_BACKUP_ACCESS_CONTROL_MODE=S3_PRIVATE_ACL. The runner reads only Secret key names during preflight, mounts no POS application or database credentials, denies ingress, permits only cluster DNS and external HTTPS, requires a read-only root filesystem, accepts only one exact sanitized READY line, and removes its Job and NetworkPolicy on success or failure. Its output never includes the bucket, endpoint, access identity or credential values.

The audit implementation has six fail-closed command tests and five guarded runner checks. A native-architecture integration passes the same auditor, age-encrypted backup, manifest-last commit, isolated restore and cleanup against a TLS S3-compatible server. A fresh native ARM64 run on 17 July reports storage READY, exact ciphertext and decrypted-dump checksums, matching Flyway/table state, and zero residual restore databases. The storage-readiness image was first qualified as pos-postgres-backup:test-20260716-storage-readiness1 at sha256:9e4bd250cbad86c422f7089a3323482bd6f894ba92e80c6212ee212baa986a18. Both remote backup CronJobs now reference the consolidated immutable release test-20260717-partial-refunds-v29-1 but remain suspended with zero Jobs and zero pods; changing the suspended templates did not enable WAL archiving.

Enabling requires the explicit operator confirmation:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_BACKUP_IMAGE_TAG=<immutable-tag> \
POS_BACKUP_SUSPEND=false \
POS_BACKUP_ENABLE_CONFIRM=ENABLE_OFF_CLUSTER_BACKUPS \
./k8s/deploy-backup-test.sh

Both deploy modes execute kubectl only through the configured SSH host, which defaults to root@damplabs.com.

The restore verifier requires POS_RESTORE_CONFIRM=ISOLATED_RESTORE, rejects the live emali2-pos-postgres service, accepts only a fresh database whose name starts with emali2_pos_restore_drill_, verifies encrypted-object and decrypted dump checksums, then compares the POS table count and Flyway version. It drops the drill database by default. The age private identity belongs only in the separate, temporary restore workload.

The automated integration drill uses two separate PostgreSQL containers and a TLS S3-compatible server with an ephemeral KMS key. It has verified age plus SSE encryption, versioning, object lock, manifest-last commit, encrypted and decrypted checksums, restore into the isolated server, Flyway/table parity, and post-drill cleanup. This is implementation evidence, not evidence that a real off-cluster test bucket has been provisioned or that its operators can meet the agreed RPO/RTO.

External storage selection checkpoint

The provider catalog was rechecked on 16 July 2026. The directly provisionable Railway bucket is not acceptable for POS backup or PITR: Railway documents object versioning, object locks, server-side encryption and bucket lifecycle configuration as unsupported. It can remain an application-asset option, but must not be used to satisfy immutable recovery evidence. See the Railway Storage Buckets documentation.

Two compatible choices remain pending account-owner, data-location and cost approval:

Use	Candidate	Evidence and trade-off
Africa-region pilot/production	Amazon S3 general-purpose bucket in Cape Town, `af-south-1`	AWS publishes a regional HTTPS endpoint and native S3 Versioning/Object Lock. This is the preferred regional-control option for an Africa deployment.
Low-volume test	Backblaze B2	Starts at USD 6.95/TB/month, pay-as-you-go with no minimum storage duration. Its S3-compatible API implements Get Bucket Versioning, the complete Object Lock call set, AES-256 server-side encryption and bucket access logs. Account regions are not Africa-local, so data-location approval is required.

See the AWS S3 endpoint table, S3 Object Lock documentation, Backblaze B2 pricing, and Backblaze S3 Object Lock support. For either provider, provisioning must enable versioning, Object Lock with an approved default retention, public-access blocking, server-side encryption and access logging before credentials are placed in Kubernetes. Use separate least-privilege archive-writer and recovery-reader identities. No external bucket was created during this checkpoint.

AWS provisioning package

The repository now includes infra/aws-pos-backup, a region-neutral Terraform root module validated with Terraform 1.5.7 and AWS provider 6.55.0. The operator must supply the approved region, globally unique bucket name, environment and retention policy; no country or AWS region is embedded in the implementation. The example uses Cape Town only as an example for an Africa-hosted deployment.

The module creates a general-purpose S3 bucket with Versioning, Object Lock at bucket creation, positive default GOVERNANCE or COMPLIANCE retention, bucket-owner-enforced ownership, all four public-access blocks, TLS-only access, and default SSE-KMS with an S3 bucket key. Both the bucket and rotating customer-managed KMS key have Terraform prevent_destroy guards. Its only lifecycle action aborts incomplete multipart uploads; it never expires a completed backup version.

Two separate least-privilege policies are emitted. The archive writer may put and verify encrypted objects under the configured prefix, but it cannot delete, change retention or bypass governance. The recovery reader may list and read current or versioned objects and decrypt SSE-KMS, but it cannot upload, delete, change retention or bypass governance. Optional IAM users have no console password and, critically, the module creates no access key. Reusable credentials must be created and rotated outside Terraform so that neither their identifier nor secret is stored in Terraform state.

The module does not create the organization's access-audit destination. Before pilot activation, route S3 data events or server-access evidence to an approved, separately protected audit destination and verify the retention/access policy. Do not use the Object-Lock backup bucket itself as a server-access-log destination.

The static security guard runs in CI and rejects access-key resources, credential fields, region literals, delete/retention-bypass permissions, missing Object Lock/KMS/public-access controls, or a missing writer/reader split. CI also runs terraform init -lockfile=readonly and terraform validate against the pinned provider lock file. See infra/aws-pos-backup/README.md for the plan/apply and credential-handoff sequence.

Live remote prerequisite audit — 17 July 2026

A metadata-only audit through ssh root@damplabs.com confirmed that the test namespace is active and both emali2-pos-postgres-backup and emali2-pos-postgres-pitr-base-backup remain suspended with no last run. Their manifests reference the dedicated future emali2-pos-backup-secrets Secret, but that Secret and emali2-pos-pitr-recovery-secrets are absent. No Secret values were decoded. The running SeaweedFS StatefulSet is an in-cluster application/test store and is not off-cluster immutable recovery evidence.

This is the expected fail-closed state until the account owner approves the provider, region, cost, RPO/RTO and retention policy. Do not unsuspend either CronJob or enable WAL archiving merely because the Terraform module validates. First apply the provider infrastructure, inject separately controlled credentials, pass the no-write storage audit, create encrypted logical and physical backups, and complete the isolated remote recovery drill.

WAL and point-in-time recovery foundation

The provider-neutral PITR layer uses PostgreSQL physical base backups and the same external S3-compatible, age-encrypted, versioned and object-locked storage contract. It does not depend on a cloud brand or country-specific service.

The pos-postgres-pitr image adds five fail-closed operations:

archive-wal.sh is called synchronously by PostgreSQL archive_command. It validates the WAL/history filename, encrypts the source with the age public recipient, uploads it under an environment and cluster-specific prefix, and returns success only after ciphertext size/hash, plaintext size/hash, SSE, object-lock mode and retention expiry are verified;
pitr-base-backup.sh runs an authenticated PostgreSQL 16 pg_basebackup, verifies it with pg_verifybackup, records the PostgreSQL system identifier and the exact start/end WAL LSN values from backup_manifest, then uploads the encrypted physical base and encrypted completion manifest;
restore-wal.sh accepts only valid PostgreSQL WAL/history names and writes only below /var/lib/postgresql/pitr-restore/... after ciphertext and plaintext verification; and
pitr-prepare-restore.sh requires POS_PITR_RESTORE_CONFIRM=ISOLATED_PITR_RESTORE, an empty isolated data directory and exactly one recovery target (time, LSN or named restore point). It rejects unsafe tar paths, runs pg_verifybackup, writes the recovery configuration with an absolute destination below the validated isolated root, creates recovery.signal without starting PostgreSQL, and adds a recovery-only local-socket bootstrap rule plus a higher-priority scram-sha-256 rule for every TCP connection made as the restored POS role; and
rotate-recovery-role-password.sh runs only against the confirmed isolated root after promotion. It uses the local socket bootstrap to replace the restored role's source credential with the dedicated recovery password, atomically replaces the socket trust rule with scram-sha-256, reloads HBA, rejects passwordless socket authentication, and proves the new password over both the socket and TCP before the POS smoke starts. Root execution is rejected so the atomic replacement cannot change HBA ownership.

The primary PostgreSQL initialization adds one authenticated host replication ... scram-sha-256 HBA rule for the explicitly configured replication user and does not add a trust rule. The narrow socket-only trust rule described above is written only into the isolated recovered data directory and is replaced with SCRAM immediately after rotation; the POS smoke container has no socket mount and never starts before that replacement. The replication credentials and the write-only archive credentials belong in the backup Secret; the age private identity still belongs only in the recovery vault.

The local integration drill has verified a real-size PostgreSQL WAL encrypt/upload/decrypt checksum, a PostgreSQL 16 physical base backup, pg_verifybackup, encrypted manifest-last commit, safe recovery preparation, and startup of the recovered server. Recovery stops at a named restore point, contains both the physical-base row and a committed post-base row, excludes a later row, preserves the PostgreSQL system identifier and all 52 Flyway v33 POS tables, and promotes out of recovery. The restored POS role is then rotated to a recovery-only password, the temporary socket trust is removed, and the original source password is rejected over both the socket and TCP. The real standalone POS image starts with the rotated credential, reports UP, and returns 401 from a protected route without an access token. A third, non-root sidecar then creates an ephemeral RSA key inside the isolated pod, serves only its JWKS over loopback, signs a five-minute pos-api token carrying the generic POS_SERVICE role, and validates an authenticated merchant bootstrap 200 against a recovered projection. It prints neither token nor response and removes its private key, authorization header and response file before recording success. No live OAuth client or POS application Secret is mounted in that sidecar. The latest measured local prepare-to-promotion sample was 17 seconds; that is local implementation evidence, not a remote RTO. Four backup, thirteen PITR and six remote-runner guard tests verify insecure configuration, invalid filenames, mutable/missing images, live-data directory/confirmation rejection, forced TCP SCRAM on both transports, non-root password rotation, loopback-only ephemeral recovery authentication, and absence of the live application Secret from the recovery pod. Three additional role-provisioning guards and a scratch-PostgreSQL integration test verify immutable Job rendering, creation, SCRAM authentication, physical-replication protocol access, password rotation, old-password rejection, absence of role memberships, and denied POS-table reads. The amd64 images are published as:

pos-postgres-backup:test-20260716-storage-readiness1 at sha256:9e4bd250cbad86c422f7089a3323482bd6f894ba92e80c6212ee212baa986a18;
pos-postgres-pitr:test-20260715-pitr-recovery-start1 at sha256:b4397541661b3098498766aa3bfee45749e65c75f5b71aa961f0c08623ce716a (historical, before recovery-password rotation); and
pos-postgres-pitr:test-20260717-recovery-password2 at sha256:8918294f9f86f23044c7e538ca8f40ebc46ae86bdd719499f4fadb116c2e4775 (historical hardened password-rotation image); and
pos-postgres-pitr:test-20260717-authenticated-smoke1 at sha256:4cdad5e814c035860b46266780aeac788893e26ece31bd9b9c19746730295d8f (current recovery image with ephemeral authenticated business smoke).

The current image is verified on a native ARM64 Docker engine, builds as a loadable AMD64 image, and was copied to GHCR through a separately authenticated registry client after Docker Desktop's push path stalled. Registry inspection proves the published image is Linux/AMD64 at the digest above. Do not substitute the historical image merely because an older manifest references it.

Install the physical-base schedule inertly without changing the primary database:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_BACKUP_IMAGE_TAG=test-20260716-storage-readiness1 \
POS_PITR_IMAGE_TAG=test-20260717-authenticated-smoke1 \
POS_PITR_ENABLE_ARCHIVING=false \
POS_PITR_BASE_BACKUP_SUSPEND=true \
./k8s/deploy-pitr-test.sh

The remote 02:43 UTC Sunday base-backup CronJob is installed in that state, is suspended, and has zero Jobs/pods. The backup credential Secret is absent; the primary remains on stock PostgreSQL with no PITR arguments or egress policy.

Dedicated physical-replication role

scripts/provision-pos-pitr-replication-role.sh is the only supported provisioning path. It renders a short-lived, digest-pinned PostgreSQL 16 Job, uses Secret references rather than literal credentials or command arguments, authenticates only with the separate PostgreSQL administrator Secret, never the POS application Secret, and creates or rotates the configured role as LOGIN REPLICATION with SCRAM, connection limit two, and no superuser, database-creation, role-creation, inheritance, bypass-RLS, role-membership, schema, table, sequence, or function privileges. The Job emits only boolean completion evidence and is deleted and verified absent together with its two temporary NetworkPolicies. Those policies permit only cluster DNS and PostgreSQL; the Job has no service-account token, added Linux capabilities, writable root filesystem, or external egress.

Server-side validation is safe before the backup Secret exists and does not create a Job or change PostgreSQL:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_PITR_ROLE_CONFIRM=PROVISION_POS_PITR_REPLICATION_ROLE \
POS_PITR_ROLE_DRY_RUN_ONLY=true \
./scripts/provision-pos-pitr-replication-role.sh

That validation passes against the test cluster with zero role Jobs. The backup Secret remains absent, and the primary retains its existing pod UID, stock image, Ready state, and zero restarts. After the guarded database-identity split has created the separate administrator identity and the approved secret manager has created both its Secret and emali2-pos-backup-secrets with a generated replication username and at least 32-character password, the authorized one-shot execution is:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_PITR_ROLE_CONFIRM=PROVISION_POS_PITR_REPLICATION_ROLE \
POS_PITR_ROLE_DRY_RUN_ONLY=false \
./scripts/provision-pos-pitr-replication-role.sh

The live identity split completed on 17 July after a validated pre-migration custom-format dump and a bounded test API outage. PostgreSQL now reads its bootstrap identity from emali2-pos-postgres-admin-secrets, while the POS service continues to receive only the separate least-privilege application identity. The application role owns the database, pos schema and all POS relations, has no administrative capabilities or inherited memberships, and the temporary transition role was removed. Both backup CronJobs reference test-20260717-partial-refunds-v29-1, remain suspended and have no completed run. emali2-pos-backup-secrets and the recovery Secret remain absent, so no external backup or WAL archiving was enabled.

The same checks are executable without reading Secret values or changing the cluster/database:

EMALI2_TEST_KUBE_CONTEXT=default \
POS_REMOTE_HOST=root@damplabs.com \
./scripts/audit-pos-recovery-readiness.sh

The auditor checks only Kubernetes metadata, Secret key names and a PostgreSQL session forced into read-only mode. It reports the role split, Secret readiness, least-privilege physical-replication role, archive mode/command, logical and base-backup schedules, pod health, Flyway version and POS table count. It never prints role names, Secret values, object-store locations or private recovery material. Set POS_RECOVERY_AUDIT_REQUIRE_READY=true in an actual change-window gate to make NOT_READY exit with status 2.

The current sanitized baseline after Flyway v34 is:

status=NOT_READY postgres_ready=true postgres_restarts=0 database_identity_split=true admin_secret_ready=true backup_secret_ready=false recovery_secret_ready=false replication_role_ready=false archive_mode_enabled=false archive_command_configured=false logical_backup_enabled=false pitr_base_backup_enabled=false schema_current=true flyway=34 tables=54

This is the intended fail-safe state, not a database-health failure. On 17 July, the digest-pinned storage-audit manifest and isolated PITR drill both passed Kubernetes server-side dry-run validation through root@damplabs.com; each reported resources_created=false, and a follow-up label query found no Pod, Job, PVC or NetworkPolicy residue.

Do not run the real command merely to test connectivity: it creates or rotates the database role. k8s/deploy-pitr-test.sh requires the same role confirmation and runs this provisioner successfully before it can apply the base schedule, change the primary image/arguments, add egress, or restart PostgreSQL. Enabling synchronous archiving additionally requires the external bucket, all Secret keys, POS_PITR_ENABLE_ARCHIVING=true, and the separate confirmation POS_PITR_ENABLE_CONFIRM=ENABLE_POS_PITR_ARCHIVING. That database restart must use an approved test change window and a rollback plan.

Guarded remote recovery drill

scripts/run-remote-pos-pitr-drill.sh is the fail-closed remote orchestration path. It performs all Kubernetes operations through SSH, creates a unique PVC and Pod rather than reusing the live database volume, denies all ingress, and allows only cluster DNS plus external HTTPS egress. An init container decrypts and verifies the selected physical base into the isolated root; the PostgreSQL container then replays archived WAL, reaches the configured time, LSN, or named target, and promotes. The runner verifies the expected PostgreSQL system identifier, Flyway version, and POS table count. A no-integration-profile POS sidecar uses only the recovery database credentials, has no live application Secret or external integration configuration, disables projection refresh and device enrollment, and must report UP while enforcing 401 on a protected route. The runner then deletes the Pod, NetworkPolicy, and PVC and proves that cleanup completed.

The recovery Secret is deliberately separate from the primary's write-only archive credentials. Provision it through the approved secret manager as emali2-pos-pitr-recovery-secrets with these keys; do not place their values in the repository or shell history:

POS_BACKUP_S3_URI;
POS_BACKUP_SSE_MODE;
optional POS_BACKUP_KMS_KEY_ID, required by the restore process when the configured mode is aws:kms;
POS_RESTORE_AGE_IDENTITY (the recovery-vault private identity);
read-only AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION;
optional AWS_ENDPOINT_URL for a public, verified-HTTPS S3-compatible endpoint; and
POS_DATABASE_USERNAME and POS_DATABASE_NAME, which must identify the POS role and database contained in the physical base; and
POS_DATABASE_PASSWORD, a generated recovery-only value that must differ from every live/source application, administrator, replication, archive and storage credential. The drill rotates the restored role to this value after promotion; do not copy the live POS database password into this Secret.

Both restore containers receive the authoritative SSE mode and optional KMS key id from this recovery Secret. This prevents an SSE-KMS bucket from silently falling back to the restore script's AES256 default. Six remote-runner guards assert both values reach both containers, and the corrected manifest passes the real test cluster's server-side dry run with zero resources created.

The server-side manifest validation can run before those credentials exist and does not create resources:

EMALI2_TEST_KUBE_CONTEXT=<exact-context> \
POS_PITR_IMAGE_TAG=test-20260717-authenticated-smoke1 \
POS_PITR_SMOKE_IMAGE_TAG=test-20260716-country-neutral-boundary1 \
POS_PITR_CLUSTER_ID=<selected-cluster-id> \
POS_PITR_BASE_MANIFEST_KEY=<selected-encrypted-manifest-key> \
POS_PITR_RECOVERY_TARGET_TIME=<approved-UTC-target> \
POS_PITR_EXPECTED_SYSTEM_IDENTIFIER=<system-id-from-base-evidence> \
POS_PITR_EXPECTED_FLYWAY_VERSION=34 \
POS_PITR_EXPECTED_TABLE_COUNT=54 \
POS_PITR_DRILL_CONFIRM=RUN_ISOLATED_POS_PITR_DRILL \
POS_PITR_DRILL_DRY_RUN_ONLY=true \
./scripts/run-remote-pos-pitr-drill.sh

The native ARM64 cryptographic integration was rerun on 24 July 2026 against the current POS service image. It verified age-encrypted WAL and base-backup round trips, manifest-last commit, the named recovery target, promotion, recovery-only password rotation, rejection of the source password, removal of temporary socket trust, Flyway v33/52-table parity, application health, protected-route 401, authenticated bootstrap 200, ephemeral drill identity cleanup and full container/network/volume cleanup.

That dry run passes against the test cluster, renders the ephemeral JWKS and authenticated bootstrap sidecar, and leaves zero drill resources. Remove only POS_PITR_DRILL_DRY_RUN_ONLY=true after the external artifacts, recovery Secret, target, expected values, capacity, and operator approval have all been verified. It is not permission to enable WAL archiving or to restart the live database.

The encryption integration must run on the Docker engine's native architecture. Do not use an emulated amd64 age binary on an ARM development host as recovery evidence; build the PITR test image for linux/arm64 locally and set POS_PITR_TEST_PLATFORM=linux/arm64. CI and the test cluster execute the amd64 image on native amd64 infrastructure.

The 17 July native ARM64 integration archived and restored encrypted WAL, committed and verified an encrypted physical base, honored a named recovery target, promoted in 15 seconds, rotated the recovered role password, rejected the source password, removed temporary socket trust, started the standalone POS service at UP, enforced 401 on a protected route, and completed an ephemeral-key authenticated merchant bootstrap with 200. All test containers, networks and volumes were covered by the cleanup trap.

What this does not prove

This is not a production disaster-recovery sign-off. It does not prove:

an enabled real off-cluster backup schedule or recovery after PVC, node, cluster, or region loss;
enabled and monitored WAL archiving or a point-in-time recovery on the remote test/production estate;
retention, immutability/object lock, key-management, or backup-access audit;
startup of a remote isolated replacement PostgreSQL service from real off-cluster artifacts, with infrastructure-loss and regional-failure scope;
a remote authenticated sale/report smoke against real off-cluster artifacts; the implemented application smoke proves startup, database health and the authentication boundary only; or
an agreed recovery point objective (RPO) or recovery time objective (RTO).

Required before pilot

Before a controlled pilot handles production data:

agree the RPO, RTO, retention period, and restore authority;
configure encrypted, access-logged, off-cluster backups with separate credentials and retention protection;
enable and monitor WAL archiving/PITR when required by the agreed RPO;
restore into an isolated replacement database and run migrations plus authenticated POS application smoke tests;
rehearse credential and encryption-key access without exposing secrets;
verify alerting for failed or stale backups; and
record evidence, timings, residual data checks, and named operator approval.

Never point a rehearsal at production without an approved change window, capacity review, and a recovery-specific runbook.