Skip to content

POS Observability

The standalone POS service exports platform and durable-work measurements for Prometheus without exposing merchant or customer data. This is source and deployment preparation only; it does not mean monitoring has been installed in either remote environment.

Network boundary

The business API remains on port 8814. Actuator health and Prometheus run on a separate management listener on port 9814 in the TEST and production manifests.

  • The public application Service exposes only port 8814.
  • Ingress therefore cannot route /actuator/prometheus.
  • Only pods selected through the emali2-observability namespace may reach port 9814.
  • Kubernetes probes use the management listener.
  • Browser and Android readiness checks use the limited /readyz path on the application listener; they do not receive component or metric details.

The Prometheus endpoint is unauthenticated because the network boundary is the scrape credential. Do not expose port 9814 through a Service, Ingress, NodePort, or load balancer.

Aggregate measurements

The service refreshes one aggregate database snapshot every 30 seconds. It publishes queue counts and oldest-work age for:

  • durable outbox and inbox delivery;
  • country-neutral customer payment requests;
  • country-neutral electronic refunds;
  • country-neutral fiscalization jobs;
  • exact-session terminal revocations;
  • stale terminals, recent settlement exceptions, and shifts open over 24 hours.

Refresh failure retains the last known values and changes a separate health gauge. Meter names and tags never contain organization, store, provider, terminal, customer, till, phone, idempotency, authorization, transaction, or receipt identifiers. The meters have no business tags, preventing both data leakage and unbounded Prometheus cardinality.

Prometheus Operator installation

The optional repository manifest k8s/observability/pos-monitoring.yaml contains a PodMonitor and privacy-safe PrometheusRule. It is deliberately not applied by the POS deployment scripts: the cluster monitoring owner must first confirm the Prometheus Operator CRDs, namespace, RBAC, and resource selectors.

Read-only preflight:

ssh root@damplabs.com \
  kubectl api-resources --api-group=monitoring.coreos.com -o name
ssh root@damplabs.com \
  kubectl get namespace emali2-observability

The result must contain both podmonitors.monitoring.coreos.com and prometheusrules.monitoring.coreos.com. The current TEST monitoring stack was checked read-only and selects resources with release=emali2-observability; the manifest carries that label. Prometheus must also retain permission to discover pods in emali2-test and emali2-pos.

During an approved monitoring change window, validate before applying:

ssh root@damplabs.com \
  kubectl apply --dry-run=server \
  -f - < k8s/observability/pos-monitoring.yaml
ssh root@damplabs.com \
  kubectl apply -f - < k8s/observability/pos-monitoring.yaml

Then verify target discovery, one successful refresh, and every rule in the Prometheus UI. Do not route alerts until a test alert has reached the intended on-call receiver and been acknowledged.

CI validates all expressions with the same digest-pinned Prometheus promtool version currently declared by the cluster monitoring resource.

Initial alert policy

Critical alerts cover an unavailable scrape target, stale metric refresh, outbox delay, delayed customer payment/refund work, and delayed terminal revocation. Warnings cover manual-review queues, inbox failures, fiscalization delay, stale terminals, recent settlement exceptions, long-running shifts, and database-pool contention.

These thresholds are conservative starting points for TEST. Review them after a pilot produces real traffic and latency baselines. Backup freshness remains governed by the separate backup/recovery runbook; the backup schedules are suspended by default, so this rule set intentionally does not claim backup monitoring is active.