Executive Summary

This document proposes the redesign and migration of Jajiga's production platform from a three-node Docker Swarm cluster to a purpose-built, on-premise Kubernetes platform — covering everything from physical server provisioning and switching to cluster architecture, stateful-service placement, observability, and a phased zero-downtime cutover.

Why migrate

The current Swarm stack — reviewed directly from the production docker-compose.yml — runs roughly 15 services with hardcoded database credentials inside the compose file, host ports exposed directly on the LAN interface, no ingress/TLS layer, and a mix of pinned and floating (:latest) image tags. These are workable trade-offs for a 3-node Swarm cluster, but they become liabilities as the platform grows: no native rolling-update safety net, no policy-based network segmentation, no secret encryption at rest in Git, and a single orchestrator with materially weaker self-healing and scheduling guarantees than Kubernetes.

What this proposal covers

• Physical & network layer — server roles, LAN/switching topology, out-of-band management.
• Virtualization layer — VMware ESXi/vCenter design carved out of two 72-core hosts.
• Kubernetes cluster — Kubespray-provisioned cluster, 3 control-plane nodes (1 off-host for quorum resilience), 4 worker nodes.
• Cluster networking & add-ons — Cilium (CNI), Traefik (ingress), cert-manager, Longhorn (storage), Sealed Secrets.
• External stateful services — MariaDB, MongoDB (new 3-node replica set), Redis, MinIO kept outside the cluster on 3 dedicated bare-metal/VM hosts.
• Delivery pipeline — a single generic Helm chart, packaged and versioned through a Nexus OCI registry, deployed from GitLab CI.
• Observability — Prometheus, Grafana, Loki, carried over and extended from the exporters already running today.
• Migration & cutover plan — parallel-cluster strategy, stateless-first rollout, stateful data migration, traffic cutover, and rollback plan.
• Project roadmap — a 13-phase execution plan mapped one-to-one to the Jira epic already opened for this initiative (TECH-1792 → TECH-1804).

Current State Assessment

The current staging/production stack was reviewed directly from its Docker Swarm compose definition. It is a monolithic services: block of roughly 15 containers sharing one Docker network, with data persistence handled by named local volumes and bind mounts on a single host's filesystem.

Service inventory

Fourteen services were verifiable directly from the reviewed docker-compose.yml. Four more — frontend, kootam, search, and minio — are known to exist in production from earlier discussion but were not present in the specific compose file reviewed for this document; they're listed separately below so the gap is explicit rather than guessed at.

Findings that directly shape this proposal

Traffic & availability profile

Target Architecture Overview

The target platform deliberately splits the world into two halves: a stateless Kubernetes cluster that only runs application workloads, and an external stateful tier of purpose-run database and object-storage hosts that the cluster talks to over the LAN. This is not a compromise forced by time constraints — for a single-operator team, it is the more reliable design: database failure domains stay outside the blast radius of cluster upgrades, CNI changes, or scheduler bugs, and existing backup/tuning tooling keeps working unmodified.

Figure 2.1 — Full request & data path: users enter through Traefik, land on worker VMs spread across two ESXi hosts, and reach the external data tier over the LAN; Observability scrapes both halves independently.

Layered view

Physical & Network Layer

Hardware inventory

LAN & switching topology

All hosts — the two ESXi servers, the external control-plane node, and the three data-tier hosts — sit on the same flat LAN segment, measured at ~1 Gbps between the ESXi hosts and the external control-plane node. That single fact removes the biggest usual risk for a 3-node etcd control plane (write latency), so no cross-site etcd tuning is required.

Out-of-band management (iLO / iDRAC / IPMI)

Whatever the physical servers' vendor is (HPE iLO, Dell iDRAC, or generic IPMI), out-of-band management should be treated as production infrastructure in its own right:

• Put the management interfaces on the isolated management VLAN above — never expose iLO/iDRAC directly on the same broadcast domain as application or pod traffic.
• Change default credentials immediately and store them as a Sealed Secret / offline password-manager entry — not in any Git repository.
• Enable virtual console + virtual media so ESXi/OS reinstalls and BIOS-level troubleshooting never depend on physical site access.
• Configure remote syslog / SNMP alerting from iLO/iDRAC for hardware health events (disk predictive failure, PSU, ECC memory errors) and route them into the same Loki/Alertmanager stack from Chapter 09 — hardware failures should surface in the same place as application alerts.
• Enable scheduled firmware/BIOS update windows as a recurring maintenance task, tracked alongside Kubernetes version upgrades.

Virtualization Layer — ESXi & vCenter

Each 72-core / 128 GB physical host is carved into three VMs: one Kubernetes control-plane node and two worker nodes. vCenter manages both hosts as a single cluster, which unlocks DRS affinity rules — the actual mechanism that protects the two-server design from correlated failures.

VM layout per host

Sizing & performance guidance

• NUMA alignment — with 72 physical cores typically split across 2 sockets (2 NUMA nodes), size each worker VM to fit inside a single socket's core count (e.g. ~34–36 vCPU on a 36-core-per-socket host) to avoid cross-NUMA memory access penalties. Confirm the exact per-socket core count in vCenter → Host → Hardware before finalizing VM sizes.
• Network adapter type — use VMXNET3, not the legacy E1000 emulated adapter, for every VM's network interface to get near-native throughput.
• DRS anti-affinity rules — create an anti-affinity rule keeping the two control-plane VMs that live on the same physical estate apart from each other where possible, and a second rule keeping worker VM pairs spread so that a single host outage removes exactly one worker from each "pair," never two workers that back the same replica.
• Dedicated Longhorn disk — attach a separate VMDK (its own datastore or at least its own virtual disk, not sharing the OS disk) to each worker VM for Longhorn's block storage, so storage I/O contention never competes with the OS/container runtime disk.
• Reserve, don't overcommit, control-plane resources — set CPU/memory reservations on the three control-plane VMs so etcd write latency never degrades under host memory pressure from worker VMs.

Kubernetes Cluster Design

The cluster is bootstrapped from scratch with Kubespray, chosen over a bare kubeadm install for its built-in HA control-plane playbooks and CNI selection, and over a managed offering because the platform is fully on-premise/bare-metal.

Control plane — 3 nodes, 1 off-host

• control-plane-1 (esxi-host-a), control-plane-2 (esxi-host-b), control-plane-3 (separate physical/VM host, same LAN, ~1 Gbps to the other two).
• This gives etcd real quorum resilience: losing either ESXi host still leaves 2 of 3 control-plane members alive, so the cluster keeps accepting writes.
• Control-plane nodes are tainted (node-role.kubernetes.io/control-plane:NoSchedule) so application pods never land there — with 8 GB reserved per CP VM this has ample headroom for etcd + API server + controller-manager + scheduler.

Worker pool — 4 nodes

• worker-1, worker-2 on esxi-host-a; worker-3, worker-4 on esxi-host-b.
• All application Deployments run here. Kubernetes' default scheduler, combined with pod anti-affinity rules on the Helm chart side, spreads replicas of the same service across both physical hosts automatically.
• Sizing envelope: total requests across all workloads must fit inside 2 worker VMs (the surviving half after a single host loss) — this is the binding constraint for every resource request/limit set in the Helm values.

Kubespray configuration highlights

Cluster Networking & Add-ons

Cilium — CNI

Cilium's eBPF datapath gives lower per-packet overhead than iptables-based CNIs, ships with native NetworkPolicy enforcement for segmenting namespaces (e.g. keeping production-data Secrets unreachable from pods that don't need them), and includes Hubble — a flow-level network observability UI that is disproportionately valuable for a solo operator troubleshooting connectivity without a networking team to lean on. Hubble UI is enabled from day one.

Traefik — Ingress & TLS

Traefik is deployed in the ingress-system namespace as the single entry point for all HTTP(S) traffic into the cluster, paired with cert-manager for automated Let's Encrypt certificate issuance and renewal. Compared to ingress-nginx, Traefik's CRD-based IngressRoute configuration and built-in dashboard are easier for a one-person team to reason about, and its native Kubernetes Gateway API support keeps the door open for a future migration path without another ingress-controller swap.

Longhorn — in-cluster block storage

With databases and MinIO living outside the cluster, Longhorn's remaining job is smaller but still necessary: persistent volumes for Prometheus (metrics history), Grafana (dashboards/config), and Loki (log chunks). Installed on a dedicated disk per worker VM (Chapter 04), with a replica count of 2–3 depending on final worker count, and snapshot/backup targets pointed at the external MinIO cluster.

Sealed Secrets

Every credential that would otherwise sit in plaintext in a Helm values.yaml — database connection strings, API keys, the Sentry DSN currently embedded in the Swarm compose file — is instead encrypted client-side with kubeseal against the cluster's public key before being committed to Git. Only the Sealed Secrets controller running inside the cluster can decrypt it. This was chosen over HashiCorp Vault deliberately: Vault's dynamic secrets, leasing, and multi-tenant ACL model solve problems this platform doesn't have yet, at the cost of another stateful service a one-person team would need to run, unseal, and back up.

External Stateful Services Architecture

All persistent data services run outside Kubernetes, on 3 dedicated bare-metal/VM hosts on the same LAN. The cluster reaches them through Kubernetes ExternalName Services, so applications always resolve a stable in-cluster DNS name and never hardcode an IP.

apiVersion: v1
kind: Service
metadata:
  name: mariadb
  namespace: production-data
spec:
  type: ExternalName
  externalName: mariadb-vm1.internal.jajiga.com

MariaDB — two independent instances

Both instances are distributed across the 3 data-tier hosts (e.g. primary + one replica on two hosts, WordPress instance on the third) so that no single host loss removes both a MariaDB primary and its replica at once.

MongoDB — 3-node Replica Set (built from scratch)

The current 3 Mongo nodes are reachable but their replication status is unconfirmed, so this is treated as a clean build on MongoDB 7.0.35 (the version already running today — pinned, not :latest):

1. Install MongoDB 7.0.35 independently on each of the 3 data-tier hosts.
2. Enable keyfile authentication for secure internal replica-set communication.
3. Run rs.initiate() to form the replica set (1 primary, 2 secondaries).
4. Migrate data: if the current instance is genuinely standalone, add the new nodes as secondaries for an online initial sync (lower downtime); otherwise use mongodump/mongorestore.
5. Update application connection strings to the multi-host replica-set form: mongodb://user:pass@mongo1,mongo2,mongo3/db?replicaSet=rs0

Redis

Deployed as a single primary today with no persistence configured; for the new platform, add Redis Sentinel across the 3 data-tier hosts for automatic failover, and set an authentication password (currently absent) via Sealed Secrets.

MinIO — distributed mode

Migrated from its current mode to MinIO distributed mode across the 3 data-tier hosts for erasure-coded redundancy. Data migration from the existing MinIO instance uses mc mirror — one of the lower-risk parts of this whole migration, since it can run incrementally ahead of cutover with a final delta sync at the end.

Placement matrix — surviving a single host loss

Network security for the data tier

Because Cilium NetworkPolicy has no effect on traffic to hosts outside the cluster, LAN-level firewall rules (iptables/nftables or VLAN isolation) must restrict inbound connections on the data-tier hosts to only the Kubernetes node IP range — closing the direct-internet-exposed ports seen in the current Swarm compose file.

Delivery Pipeline — Helm, Nexus & GitLab CI

One generic Helm chart, not fifteen

Rather than a dedicated chart per service, all stateless services are defined as entries in a single values.yaml map, rendered by one shared set of templates (Deployment, Service, optional Ingress). Adding a new service is a values-file change, not a new template — the right trade-off for a single maintainer supporting ten developers who don't yet read Kubernetes manifests themselves.

jajiga-platform/
  Chart.yaml
  values.yaml          # one block per service
  templates/
    _helpers.tpl
    deployment.yaml     # ranges over .Values.services
    service.yaml
    ingress.yaml
  sealed-secrets/        # kubeseal output, safe to commit

Nexus as an OCI registry

Helm ≥3.8 speaks the OCI registry protocol natively, so the same Nexus instance already used for Docker images can host chart packages without a separate Helm-specific repository type:

helm package .
helm push jajiga-platform-0.1.0.tgz oci://nexus.jajiga.com:PORT/helm-charts

GitLab CI pipeline stages

Monitoring & Logging

The current Swarm stack already runs node-exporter, mysql-exporter, and promtail — a real head start. Observability is designed in from the start of the migration, not bolted on afterward, and it is the same stack that watches both the cluster and the external data tier.

Metrics — kube-prometheus-stack

• Deployed via the official kube-prometheus-stack Helm chart into production-monitoring: Prometheus, Grafana, and Alertmanager together, with a large library of ready-made dashboards.
• In-cluster services are scraped automatically via ServiceMonitor CRDs.
• The existing node-exporter and mysql-exporter continue running on the 3 external data-tier hosts, scraped through Prometheus' additionalScrapeConfigs — no rewrite needed, just a new scrape target.
• iLO/iDRAC hardware alerts (Chapter 03) and MongoDB/Redis-specific exporters are added to the same Prometheus instance so hardware, database, and application signals live in one place.

Logs — Loki + Promtail

• Loki deployed in-cluster, backed by a Longhorn-provisioned volume for chunk storage.
• The existing Promtail configuration is extended to also ship logs from the 3 external data-tier hosts, alongside in-cluster pod logs collected via a Promtail DaemonSet.
• Grafana becomes the single pane of glass for both metrics (Prometheus datasource) and logs (Loki datasource).

Why this is built before anything migrates

Migration Strategy & Cutover Plan

Parallel-cluster strategy

The Kubernetes cluster is built and validated alongside the running Swarm cluster — never in place of it — until the very end. This is the single biggest lever for meeting the zero-downtime constraint: at every point before final cutover, the Swarm cluster remains the system of record and can absorb 100% of production traffic if anything in the new platform needs more time.

Phase order and why

Cutover sequence

1. DNS/load-balancer weighting — shift a small percentage of read-only or low-risk traffic to the Kubernetes ingress first, watching Grafana dashboards built in Chapter 09.
2. Full traffic switch — once error rates and latency match or beat the Swarm baseline, move 100% of traffic to Traefik.
3. Soak period — keep the Swarm cluster warm (not decommissioned) for an observation window before touching it, per the roadmap in Chapter 11.
4. Decommission — only after the soak period passes cleanly, the legacy Swarm infrastructure is powered down and its hardware reclaimed (potentially into the data tier or as future cluster capacity).

Rollback plan

Project Roadmap & Timeline

The 13 phases below map one-to-one to the Jira epic already opened for this initiative (TECH-1792 → TECH-1804), all owned by Milad Heidari. The original 8-month estimate has been re-planned to a tighter but still realistic ≈27 weeks (≈6.5 months), assuming this remains one engineer's primary but not sole responsibility. It includes explicit buffer around the highest-risk phases (stateful migration, cutover) rather than assuming best-case execution throughout.

Gantt Chart — 13-Phase Execution Plan

Phases are grouped into five stages that mirror the migration strategy from Chapter 10 — foundation work, cluster bootstrap, observability, migration & validation, and the highest-risk cutover work. Bar color encodes the group; exact dates and duration are printed for every phase so nothing depends on reading bar length alone.

Reference table

Risk Register

Post-Migration Operations & Maintenance

Recurring operational tasks

Day-2 principles

• Everything as code — cluster bootstrap (Ansible/Kubespray inventory), workloads (Helm values), and secrets (sealed, encrypted) all live in Git; there should be no manual kubectl edit that isn't reflected back into a commit.
• Alert before it's an incident — Grafana/Alertmanager thresholds are set from the real baselines captured during Chapter 09's rollout, not guessed defaults.
• Backups are only real if restored — the quarterly restore drills above are non-negotiable; an untested backup is a hope, not a plan.
• Capacity headroom review — as more of the 10-person dev team's services land on the cluster, periodically re-validate the "survive one host loss" math from Chapter 04 against actual usage, not the original estimate.

Namespace & RBAC Reference

Only the production environment moves to Kubernetes at this stage — running separate dev/staging clusters is not justified for a one-person DevOps team, and namespace separation by architectural layer (rather than by dev team, which isn't yet Kubernetes-literate) keeps day-to-day operation simple while still giving real RBAC boundaries where they matter most.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: developer-readonly
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "services"]
  verbs: ["get", "list", "watch"]

No equivalent Role is created for production-data — the absence of a binding is the access control. Database credentials never become visible to the development team through kubectl, regardless of what they can see in production.

Glossary