In this tutorial, you will learn how to clean up the MicroK8s cluster removing all the containers, deployments, configmaps, secrets, etc from the previous tutorials. MariaDB and EJBCA are deployed using a Helm chart on the cleaned-up cluster. The database is backed up and restored that way all the configuration that has been done from the previous tutorials is not lost.

Over the last year, all the EJBCA tutorials have been demonstrated on a MicroK8s virtual machine. EJBCA was deployed with YAML files and it would be nice to redeploy EJBCA using Helm. It’s time to clean up and clear out the cluster starting with a fresh deployment of EJBCA using a backup of the MariaDB database.

If you want to clean house, start fresh, and use Helm then this tutorial is for you.

The tutorial covers these steps:

  • Review what is deployed in the MicroK8s cluster

  • Backup the MariaDB database

  • Reset the MicroK8s cluster

  • Install required services on the MicroK8s cluster

  • Deploy MariaDB using Helm and restore the MariaDB backup

  • Deploy EJBCA using Helm

Prerequisites

For this tutorial, EJBCA Community Docker container version 8.2.0.1 was used.

Before you begin, you need:

Step 1 - Review what is deployed in the cluster

If you have been trying out the previous tutorials, there are a lot of pieces and parts deployed and configured in MicroK8s from following the tutorials. Follow the steps below to review what is currently deployed in the cluster:

  1. Review all of the deployments and items deployed:

    $ kubectl get all,ingress,secret,pv,pvc,svc,no --all-namespaces

    • The output is similar to the following:

      NAMESPACE NAME READY STATUS RESTARTS AGE
      cert-manager pod/cert-manager-55cf8685cb-tztzk 1/1 Running 15 (2d21h ago) 151d
      cert-manager pod/cert-manager-cainjector-fbd548cb8-nrr7l 1/1 Running 20 (2d21h ago) 151d
      cert-manager pod/cert-manager-webhook-655b4d58fb-46zzh 1/1 Running 12 (2d21h ago) 151d
      ejbca-cert-manager pod/ejbca-cert-manager-issuer-6c5549d575-qd52x 1/1 Running 30 (2d21h ago) 137d
      ejbca-k8s pod/ejbca-deployment-78fdbbb5c4-n5v2p 1/1 Running 12 (2d21h ago) 167d
      ejbca-k8s pod/ejbca-httpd-7d646c85f9-2n68j 1/1 Running 3 (2d21h ago) 48d
      ejbca-k8s pod/mariadb-deployment-5665d96759-sdg24 1/1 Running 3 (2d21h ago) 48d
      ingress pod/nginx-ingress-microk8s-controller-sqk8c 1/1 Running 2 (2d21h ago) 24d
      istio-system pod/istio-egressgateway-66fc798fb8-6cmc4 0/1 Running 22 (2d21h ago) 442d
      istio-system pod/istio-ingressgateway-668c8c965b-s7725 0/1 Running 22 (2d21h ago) 442d
      istio-system pod/istiod-7fc54f9bd-qhwxg 1/1 Running 27 (2d21h ago) 442d
      kube-system pod/calico-kube-controllers-7c8c9d85c9-htp24 1/1 Running 3 (2d21h ago) 48d
      kube-system pod/calico-node-c4ffk 1/1 Running 0 2d21h
      kube-system pod/coredns-6f5f9b5d74-wxbmz 1/1 Running 28 (2d21h ago) 449d
      kube-system pod/hostpath-provisioner-69cd9ff5b8-fx8ps 1/1 Running 45 (2d21h ago) 449d
      metallb-system pod/controller-9556c586f-gzbm6 1/1 Running 28 (2d21h ago) 449d
      metallb-system pod/speaker-58lm5 1/1 Running 39 (2d21h ago) 449d
      pkirules pod/ejbca-cm-issuer-helloworld-one-6c886556f9-xth78 1/1 Terminating 8 (2d21h ago) 136d
       
      NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      cert-manager service/cert-manager ClusterIP 10.152.183.223 <none> 9402/TCP 151d
      cert-manager service/cert-manager-webhook ClusterIP 10.152.183.154 <none> 443/TCP 151d
      default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 449d
      ejbca-k8s service/ejbca-database-service ClusterIP 10.152.183.44 <none> 3306/TCP 449d
      ejbca-k8s service/ejbca-internal ClusterIP 10.152.183.133 <none> 443/TCP,80/TCP 449d
      ejbca-k8s service/ejbca-service ClusterIP 10.152.183.46 <none> 8009/TCP 449d
      ingress service/ejbca-node1 LoadBalancer 10.152.183.171 172.16.170.240 80:31205/TCP,443:30622/TCP 449d
      istio-system service/istio-egressgateway ClusterIP 10.152.183.125 <none> 80/TCP,443/TCP 442d
      istio-system service/istio-ingressgateway LoadBalancer 10.152.183.92 172.16.170.241 15021:30350/TCP,80:31533/TCP,443:30430/TCP,31400:32547/TCP,15443:31322/TCP 442d
      istio-system service/istiod ClusterIP 10.152.183.76 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 442d
      kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 449d
      metallb-system service/webhook-service ClusterIP 10.152.183.52 <none> 443/TCP 449d
       
      NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
      ingress daemonset.apps/nginx-ingress-microk8s-controller 1 1 1 1 1 <none> 449d
      kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 449d
      metallb-system daemonset.apps/speaker 1 1 1 1 1 kubernetes.io/os=linux 449d
       
      NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
      cert-manager deployment.apps/cert-manager 1/1 1 1 151d
      cert-manager deployment.apps/cert-manager-cainjector 1/1 1 1 151d
      cert-manager deployment.apps/cert-manager-webhook 1/1 1 1 151d
      ejbca-cert-manager deployment.apps/ejbca-cert-manager-issuer 1/1 1 1 137d
      ejbca-k8s deployment.apps/ejbca-deployment 1/1 1 1 167d
      ejbca-k8s deployment.apps/ejbca-httpd 1/1 1 1 449d
      ejbca-k8s deployment.apps/mariadb-deployment 1/1 1 1 449d
      istio-system deployment.apps/istio-egressgateway 0/1 1 0 442d
      istio-system deployment.apps/istio-ingressgateway 0/1 1 0 442d
      istio-system deployment.apps/istiod 1/1 1 1 442d
      kube-system deployment.apps/calico-kube-controllers 1/1 1 1 449d
      kube-system deployment.apps/coredns 1/1 1 1 449d
      kube-system deployment.apps/hostpath-provisioner 1/1 1 1 449d
      metallb-system deployment.apps/controller 1/1 1 1 449d
       
      NAMESPACE NAME DESIRED CURRENT READY AGE
      cert-manager replicaset.apps/cert-manager-55cf8685cb 1 1 1 151d
      cert-manager replicaset.apps/cert-manager-cainjector-fbd548cb8 1 1 1 151d
      cert-manager replicaset.apps/cert-manager-webhook-655b4d58fb 1 1 1 151d
      ejbca-cert-manager replicaset.apps/ejbca-cert-manager-issuer-6c5549d575 1 1 1 137d
      ejbca-k8s replicaset.apps/ejbca-deployment-78fdbbb5c4 1 1 1 167d
      ejbca-k8s replicaset.apps/ejbca-httpd-7d646c85f9 1 1 1 449d
      ejbca-k8s replicaset.apps/mariadb-deployment-5665d96759 1 1 1 449d
      istio-system replicaset.apps/istio-egressgateway-66fc798fb8 1 1 0 442d
      istio-system replicaset.apps/istio-egressgateway-7bf6457867 0 0 0 442d
      istio-system replicaset.apps/istio-ingressgateway-668c8c965b 1 1 0 442d
      istio-system replicaset.apps/istio-ingressgateway-8576d6c55f 0 0 0 442d
      istio-system replicaset.apps/istiod-77f9c8bf68 0 0 0 442d
      istio-system replicaset.apps/istiod-7fc54f9bd 1 1 1 442d
      kube-system replicaset.apps/calico-kube-controllers-5cdbddfc9 0 0 0 449d
      kube-system replicaset.apps/calico-kube-controllers-75b6ddb7b8 0 0 0 408d
      kube-system replicaset.apps/calico-kube-controllers-79568db7f8 0 0 0 449d
      kube-system replicaset.apps/calico-kube-controllers-7c8c9d85c9 1 1 1 48d
      kube-system replicaset.apps/calico-kube-controllers-7c9cdf4bfc 0 0 0 396d
      kube-system replicaset.apps/calico-kube-controllers-7fcc7d8496 0 0 0 167d
      kube-system replicaset.apps/calico-kube-controllers-b5fcf779f 0 0 0 60d
      kube-system replicaset.apps/calico-kube-controllers-df49bb54 0 0 0 364d
      kube-system replicaset.apps/coredns-6f5f9b5d74 1 1 1 449d
      kube-system replicaset.apps/hostpath-provisioner-69cd9ff5b8 1 1 1 449d
      metallb-system replicaset.apps/controller-9556c586f 1 1 1 449d
       
      NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE
      ejbca-k8s ingress.networking.k8s.io/ejbca-node1 public ejbca-node1.ejbca-k8s 127.0.0.1 80, 443 24d
       
      NAMESPACE NAME TYPE DATA AGE
      cert-manager secret/cert-manager-webhook-ca Opaque 3 151d
      cert-manager secret/sh.helm.release.v1.cert-manager.v1 helm.sh/release.v1 1 151d
      ejbca-cert-manager secret/ejbca-ca-secret Opaque 1 151d
      ejbca-cert-manager secret/ejbca-secret kubernetes.io/tls 2 151d
      ejbca-cert-manager secret/sh.helm.release.v1.ejbca-cert-manager-issuer.v1 helm.sh/release.v1 1 137d
      ejbca-csr-signer secret/ejbca-client-cert kubernetes.io/tls 2 442d
      ejbca-csr-signer secret/ejbca-credentials Opaque 1 442d
      ejbca-k8s secret/ejbca-mariadb-secret Opaque 1 449d
      ejbca-k8s secret/mariadb-secret Opaque 1 449d
      ejbca-k8s secret/mtls-trust-chain Opaque 1 47d
      ejbca-k8s secret/tls-ejbca-node1 kubernetes.io/tls 2 60d
      istio-system secret/external-ca-cert Opaque 1 442d
      istio-system secret/istio-ca-secret istio.io/ca-root 5 442d
      metallb-system secret/memberlist Opaque 1 449d
      metallb-system secret/webhook-server-cert Opaque 4 449d
      vault secret/tls-api-vault kubernetes.io/tls 2 160d
      vault secret/vault-ha-tls Opaque 3 160d
       
      NAMESPACE NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
      persistentvolume/pvc-031c7c9d-63dd-42b7-8c20-83e9e57782d7 100M RWO Delete Bound ejbca-k8s/mariadb-pvc microk8s-hostpath <unset> 449d
      persistentvolume/pvc-230714e3-ed56-498f-8386-c84b941e9b00 1Gi RWO Delete Bound vault/data-vault-1 microk8s-hostpath <unset> 160d
      persistentvolume/pvc-3d0cc22e-3c9a-4e8a-a867-9f99b9f97424 1Gi RWO Delete Bound vault/audit-vault-2 microk8s-hostpath <unset> 160d
      persistentvolume/pvc-4cacc08e-2802-4b54-9af5-b386750fa715 1Gi RWO Delete Bound vault/audit-vault-0 microk8s-hostpath <unset> 160d
      persistentvolume/pvc-5be3658f-abda-4512-873a-38b4e9e46556 1Gi RWO Delete Bound vault/audit-vault-1 microk8s-hostpath <unset> 160d
      persistentvolume/pvc-6d665f7a-c89e-4536-8b3a-f8f5a79a4b00 1Gi RWO Delete Bound vault/data-vault-0 microk8s-hostpath <unset> 160d
      persistentvolume/pvc-903289f8-db05-44c2-8ded-fdb93a34fb21 1Gi RWO Delete Bound vault/data-vault-2 microk8s-hostpath <unset> 160d
      persistentvolume/pvc-b1d34ec9-10db-4b8d-ab3d-d8ba7810be14 30M RWO Delete Bound ejbca-k8s/ejbca-pvc microk8s-hostpath <unset> 449d
       
      NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
      ejbca-k8s persistentvolumeclaim/ejbca-pvc Bound pvc-b1d34ec9-10db-4b8d-ab3d-d8ba7810be14 30M RWO microk8s-hostpath <unset> 449d
      ejbca-k8s persistentvolumeclaim/mariadb-pvc Bound pvc-031c7c9d-63dd-42b7-8c20-83e9e57782d7 100M RWO microk8s-hostpath <unset> 449d
      vault persistentvolumeclaim/audit-vault-0 Bound pvc-4cacc08e-2802-4b54-9af5-b386750fa715 1Gi RWO microk8s-hostpath <unset> 160d
      vault persistentvolumeclaim/audit-vault-1 Bound pvc-5be3658f-abda-4512-873a-38b4e9e46556 1Gi RWO microk8s-hostpath <unset> 160d
      vault persistentvolumeclaim/audit-vault-2 Bound pvc-3d0cc22e-3c9a-4e8a-a867-9f99b9f97424 1Gi RWO microk8s-hostpath <unset> 160d
      vault persistentvolumeclaim/data-vault-0 Bound pvc-6d665f7a-c89e-4536-8b3a-f8f5a79a4b00 1Gi RWO microk8s-hostpath <unset> 160d
      vault persistentvolumeclaim/data-vault-1 Bound pvc-230714e3-ed56-498f-8386-c84b941e9b00 1Gi RWO microk8s-hostpath <unset> 160d
      vault persistentvolumeclaim/data-vault-2 Bound pvc-903289f8-db05-44c2-8ded-fdb93a34fb21 1Gi RWO microk8s-hostpath <unset> 160d
       
      NAMESPACE NAME STATUS ROLES AGE VERSION
      node/microk8-01 Ready <none> 449d v1.29.4 microk8s-hostpath <unset> 159d

You have now reviewed what has been deployed in MicroK8s, continue to the next step to backup the EJBCA database.

Step 2 - Backup the EJBCA database

Time was invested to configure EJBCA following the tutorials and it would be great to not lose the EJBCA configuration. Follow these steps to backup the EJBCA database before the MicroK8s cluster is reset:

  1. Review what directories are present in the VM host:

    $ ls -l

    • The output is similar to the following:

      total 20
      -rw-rw-r--. 1 user user 3612 Apr 24 07:56 ca.crt
      drwxrwxr-x. 3 user user 4096 Jan 26 06:30 cert-manager
      drwxrwxr-x. 3 user user 88 Mar 25 2023 csr-api
      drwxrwxr-x. 2 user user 4096 May 17 11:58 k81
      drwx------. 3 user microk8s 22 Mar 18 2023 snap
      drwxrwxr-x. 2 user user 4096 Jun 9 15:34 sql
      drwxrwxr-x. 2 user user 4096 Jan 2 12:01 vault

  2. Install the MariaDB client using the DNF package manager:

    $ sudo dnf install -y <https://dlm.mariadb.com/2690116/MariaDB/mariadb-10.10.2/yum/rhel8-amd64/rpms/MariaDB-client-10.10.2-1.el8.x86_64.rpm> \
    <https://dlm.mariadb.com/2690045/MariaDB/mariadb-10.10.2/yum/rhel8-amd64/rpms/MariaDB-backup-10.10.2-1.el8.x86_64.rpm> \
    <https://dlm.mariadb.com/2690100/MariaDB/mariadb-10.10.2/yum/rhel8-amd64/rpms/MariaDB-common-10.10.2-1.el8.x86_64.rpm> \
    <https://dlm.mariadb.com/2690079/MariaDB/mariadb-10.10.2/yum/rhel8-amd64/rpms/MariaDB-shared-10.10.2-1.el8.x86_64.rpm>

  3. Backup the database to the SQL directory:

    $ mariadb-dump -h 10.152.183.44 -u root -p --all-databases --quick --lock-tables=false --system=users --insert-ignore --force > sql/20240610_BackupToMigrate.sql

    • The output is similar to the following:

      # mariadb-dump: Couldn't execute 'SELECT CONCAT(QUOTE(u.user), '@', QUOTE(u.Host)) AS u FROM mysql.user u /*!80001 LEFT JOIN mysql.role_edges e ON u.user=e.from_user AND u.host=e.from_host WHERE e.from_user IS NULL */ /*M!100005 WHERE is_role='N' */': Illegal mix of collations (utf8mb4_general_ci,COERCIBLE) and (utf8mb4_uca1400_ai_ci,COERCIBLE) for operation '=' (1267)

You have now backed up the EJBCA MariaDB database using the mariadb-dump utility. Continue to the next step to reset the MicroK8s cluster.

Step 3 - Reset the MicroK8s cluster

When MicroK8s is reset, the cluster is cleaned up by removing the entire configuration and restoring the cluster to the default install from when MicroK8s was first installed. To reset the cluster follow these steps:

  1. Uninstall the ejbca cert-manager Issuer Helm deployment:

    $ helm uninstall -n ejbca-cert-manager ejbca-cert-manager-issuer

    • The output is similar to the following:

      release "ejbca-cert-manager-issuer" uninstalled

  2. Uninstall the cert-manager Helm deployment:

    $ helm uninstall -n cert-manager cert-manager

    • The output is similar to the following:

      release "cert-manager" uninstalled

  3. Uninstall istio using istioctl:

    $ csr-api/ejbca-csr-signer/istio-1.16.3/bin/istioctl uninstall --purge --force

    • The output is similar to the following:

      failed to get proxy infos: failure running port forward process: failure running port forward process: error upgrading connection: error dialing backend: dial tcp 172.16.170.187:10250: connect: connection refused
      All Istio resources will be pruned from the cluster
      Proceed? (y/N)

  4. Enter y to continue.

    • The output is similar to the following:

      Removed IstioOperator:istio-system:installed-state.
      Removed Deployment:istio-system:istio-egressgateway.
      Removed Deployment:istio-system:istio-ingressgateway.
      Removed Deployment:istio-system:istiod.
      Removed Service:istio-system:istio-egressgateway.
      Removed Service:istio-system:istio-ingressgateway.
      Removed Service:istio-system:istiod.
      Removed ConfigMap:istio-system:istio.
      Removed ConfigMap:istio-system:istio-sidecar-injector.
      Removed Pod:istio-system:istio-egressgateway-66fc798fb8-6cmc4.
      Removed Pod:istio-system:istio-ingressgateway-668c8c965b-s7725.
      Removed Pod:istio-system:istiod-7fc54f9bd-qhwxg.
      Removed ServiceAccount:istio-system:istio-egressgateway-service-account.
      Removed ServiceAccount:istio-system:istio-ingressgateway-service-account.
      Removed ServiceAccount:istio-system:istio-reader-service-account.
      Removed ServiceAccount:istio-system:istiod.
      Removed ServiceAccount:istio-system:istiod-service-account.
      Removed RoleBinding:istio-system:istio-egressgateway-sds.
      Removed RoleBinding:istio-system:istio-ingressgateway-sds.
      Removed RoleBinding:istio-system:istiod.
      Removed RoleBinding:istio-system:istiod-istio-system.
      Removed Role:istio-system:istio-egressgateway-sds.
      Removed Role:istio-system:istio-ingressgateway-sds.
      Removed Role:istio-system:istiod.
      Removed Role:istio-system:istiod-istio-system.
      Removed EnvoyFilter:istio-system:stats-filter-1.13.
      Removed EnvoyFilter:istio-system:stats-filter-1.14.
      Removed EnvoyFilter:istio-system:stats-filter-1.15.
      Removed EnvoyFilter:istio-system:stats-filter-1.16.
      Removed EnvoyFilter:istio-system:tcp-stats-filter-1.13.
      Removed EnvoyFilter:istio-system:tcp-stats-filter-1.14.
      Removed EnvoyFilter:istio-system:tcp-stats-filter-1.15.
      Removed EnvoyFilter:istio-system:tcp-stats-filter-1.16.
      Removed PodDisruptionBudget:istio-system:istio-egressgateway.
      Removed PodDisruptionBudget:istio-system:istio-ingressgateway.
      Removed PodDisruptionBudget:istio-system:istiod.
      Removed MutatingWebhookConfiguration::istio-revision-tag-default.
      Removed MutatingWebhookConfiguration::istio-sidecar-injector.
      Removed ValidatingWebhookConfiguration::istio-validator-istio-system.
      Removed ValidatingWebhookConfiguration::istiod-default-validator.
      Removed ClusterRole::istio-reader-clusterrole-istio-system.
      Removed ClusterRole::istio-reader-istio-system.
      Removed ClusterRole::istiod-clusterrole-istio-system.
      Removed ClusterRole::istiod-gateway-controller-istio-system.
      Removed ClusterRole::istiod-istio-system.
      Removed ClusterRoleBinding::istio-reader-clusterrole-istio-system.
      Removed ClusterRoleBinding::istio-reader-istio-system.
      Removed ClusterRoleBinding::istiod-clusterrole-istio-system.
      Removed ClusterRoleBinding::istiod-gateway-controller-istio-system.
      Removed ClusterRoleBinding::istiod-istio-system.
      Removed CustomResourceDefinition::authorizationpolicies.security.istio.io.
      Removed CustomResourceDefinition::destinationrules.networking.istio.io.
      Removed CustomResourceDefinition::envoyfilters.networking.istio.io.
      Removed CustomResourceDefinition::gateways.networking.istio.io.
      Removed CustomResourceDefinition::istiooperators.install.istio.io.
      Removed CustomResourceDefinition::peerauthentications.security.istio.io.
      Removed CustomResourceDefinition::proxyconfigs.networking.istio.io.
      Removed CustomResourceDefinition::requestauthentications.security.istio.io.
      Removed CustomResourceDefinition::serviceentries.networking.istio.io.
      Removed CustomResourceDefinition::sidecars.networking.istio.io.
      Removed CustomResourceDefinition::telemetries.telemetry.istio.io.

  5. Reset microk8s to restore MicroK8s to the default install:

    $ sudo /var/lib/snapd/snap/bin/microk8s reset

    • The output is similar to the following:

      Disabling all addons
      Disabling addon : core/cert-manager
      Disabling addon : core/dashboard
      Disabling addon : core/dns
      Disabling addon : core/gpu
      Disabling addon : core/helm
      Disabling addon : core/helm3
      Disabling addon : core/host-access
      Disabling addon : core/hostpath-storage
      Disabling addon : core/ingress
      Disabling addon : core/kube-ovn
      Disabling addon : core/mayastor
      Disabling addon : core/metallb
      Disabling addon : core/metrics-server
      Disabling addon : core/minio
      Disabling addon : core/observability
      Disabling addon : core/prometheus
      Disabling addon : core/rbac
      Disabling addon : core/registry
      Disabling addon : core/storage
      All addons are disabled.
      Deleting the CNI
      Cleaning resources in namespace bookinfo
      Cleaning resources in namespace cert-manager
      Cleaning resources in namespace default
      Cleaning resources in namespace ejbca-cert-manager
      Cleaning resources in namespace ejbca-csr-signer
      Cleaning resources in namespace ejbca-k8s
      Cleaning resources in namespace ingress
      Cleaning resources in namespace istio-system
      Cleaning resources in namespace kube-node-lease
      Cleaning resources in namespace kube-public
      Cleaning resources in namespace kube-system
      Cleaning resources in namespace metallb-system
      Cleaning resources in namespace pkirules
      Cleaning resources in namespace vault
      Removing CRDs
      Removing PriorityClasses
      Removing StorageClasses
      Removing namespace/bookinfo
      Removing namespace/cert-manager
      Removing namespace/ejbca-cert-manager
      Removing namespace/ejbca-csr-signer
      Removing namespace/ejbca-k8s
      Removing namespace/ingress
      Removing namespace/istio-system
      Removing namespace/metallb-system
      Removing namespace/pkirules
      Removing namespace/vault
      Restarting cluster
      Stopped.
      Setting up the CNI

You have now reset MicroK8s back to the default. Continue to the next step to enable the required services in MicroK8s for deploying MariaDB and EJBCA.

Step 4 - Enable required services on MicroK8s

Enable DNS, Helm, and hostpath storage on the MicroK8s cluster which are used to deploy EJBCA and MariaDB and provide persistent storage for MariaDB. Follow the steps below to prepare the cluster:

  1. Enable the DNS, Helm, and hostpath-storage addons on the cluster:

    $ microk8s enable dns hostpath-storage helm

    • The output is similar to the following:

      Infer repository core for addon dns
      Infer repository core for addon hostpath-storage
      Infer repository core for addon helm
      WARNING: Do not enable or disable multiple addons in one command.
      This form of chained operations on addons will be DEPRECATED in the future.
      Please, enable one addon at a time: 'microk8s enable <addon>'
      Addon core/dns is already enabled
      Enabling default storage class.
      WARNING: Hostpath storage is not suitable for production environments.
      A hostpath volume can grow beyond the size limit set in the volume claim manifest.
       
      deployment.apps/hostpath-provisioner created
      storageclass.storage.k8s.io/microk8s-hostpath created
      serviceaccount/microk8s-hostpath created
      clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
      clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
      Storage will be available soon.
      Addon core/helm is already enabled

  2. Enable the Metal LB addon on the cluster using the IP range of the VM network the VM is on:

    $ microk8s enable metallb:172.16.170.240-172.16.170.245

    • The output is similar to the following:

      Infer repository core for addon metallb
      Enabling MetalLB
      Applying Metallb manifest
      customresourcedefinition.apiextensions.k8s.io/addresspools.metallb.io created
      customresourcedefinition.apiextensions.k8s.io/bfdprofiles.metallb.io created
      customresourcedefinition.apiextensions.k8s.io/bgpadvertisements.metallb.io created
      customresourcedefinition.apiextensions.k8s.io/bgppeers.metallb.io created
      customresourcedefinition.apiextensions.k8s.io/communities.metallb.io created
      customresourcedefinition.apiextensions.k8s.io/ipaddresspools.metallb.io created
      customresourcedefinition.apiextensions.k8s.io/l2advertisements.metallb.io created
      namespace/metallb-system created
      serviceaccount/controller created
      serviceaccount/speaker created
      clusterrole.rbac.authorization.k8s.io/metallb-system:controller created
      clusterrole.rbac.authorization.k8s.io/metallb-system:speaker created
      role.rbac.authorization.k8s.io/controller created
      role.rbac.authorization.k8s.io/pod-lister created
      clusterrolebinding.rbac.authorization.k8s.io/metallb-system:controller created
      clusterrolebinding.rbac.authorization.k8s.io/metallb-system:speaker created
      rolebinding.rbac.authorization.k8s.io/controller created
      secret/webhook-server-cert created
      service/webhook-service created
      rolebinding.rbac.authorization.k8s.io/pod-lister created
      daemonset.apps/speaker created
      deployment.apps/controller created
      validatingwebhookconfiguration.admissionregistration.k8s.io/validating-webhook-configuration created
      Waiting for Metallb controller to be ready.
      deployment.apps/controller condition met
      ipaddresspool.metallb.io/default-addresspool created
      l2advertisement.metallb.io/default-advertise-all-pools created
      MetalLB is enabled

  3. Create an alias for helm command:

    $ sudo snap alias microk8s.helm helm

    • The output is similar to the following:

      #Added:
      # - microk8s.helm as helm

  4. Review what is deployed in the cluster after the microk8s reset:

    $ kubectl get all,secret,pv,pvc,svc,no --all-namespaces

    • The output is similar to the following:

      NAMESPACE NAME READY STATUS RESTARTS AGE
      kube-system pod/calico-kube-controllers-7cb4fd5784-xs4db 1/1 Running 0 3m39s
      kube-system pod/calico-node-s42fj 1/1 Running 0 3m39s
      kube-system pod/coredns-85db789d58-2pfxz 1/1 Running 0 2m3s
      kube-system pod/hostpath-provisioner-7df77bc496-xcdbm 1/1 Running 0 50s
      metallb-system pod/controller-5f7bb57799-wxcvx 1/1 Running 0 50s
      metallb-system pod/speaker-rdn59 1/1 Running 0 50s
       
      NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 6m31s
      kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP,9153/TCP 2m3s
      metallb-system service/webhook-service ClusterIP 10.152.183.225 <none> 443/TCP 102s
       
      NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
      kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 4m44s
      metallb-system daemonset.apps/speaker 1 1 1 1 1 kubernetes.io/os=linux 102s
       
      NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
      kube-system deployment.apps/calico-kube-controllers 1/1 1 1 4m44s
      kube-system deployment.apps/coredns 1/1 1 1 2m3s
      kube-system deployment.apps/hostpath-provisioner 1/1 1 1 114s
      metallb-system deployment.apps/controller 1/1 1 1 102s
       
      NAMESPACE NAME DESIRED CURRENT READY AGE
      kube-system replicaset.apps/calico-kube-controllers-7cb4fd5784 1 1 1 3m39s
      kube-system replicaset.apps/coredns-85db789d58 1 1 1 2m3s
      kube-system replicaset.apps/hostpath-provisioner-7df77bc496 1 1 1 50s
      metallb-system replicaset.apps/controller-5f7bb57799 1 1 1 50s
       
      NAMESPACE NAME TYPE DATA AGE
      metallb-system secret/memberlist Opaque 1 49s
      metallb-system secret/webhook-server-cert Opaque 4 102s
       
      NAMESPACE NAME STATUS ROLES AGE VERSION
      node/microk8-01 Ready <none> 449d v1.29.4

You have now enabled the required services to deploy EJBCA and MariaDB using Helm. Continue to the next step to deploy MariaDB using Helm.

Step 5 - Deploy MariaDB with Helm & restore the EJBCA SQL backup

Before EJBCA is deployed you need to have the database (MariaDB) deployed and the SQL backup restored. That way when EJBCA is deployed you begin right where you left off from the last backup taken. Follow these steps to deploy MariaDB with Helm and restore the EJBCA database backup:

  1. Install Helm repository for MariaDB:

    $ helm repo add groundhog2k https://groundhog2k.github.io/helm-charts

    • The output is similar to the following:

      "groundhog2k" has been added to your repositories

  2. Create namespace for the EJBCA MariaDB deployment:

    $ kubectl create namespace ejbca-k8s

    • The output is similar to the following:

      namespace/ejbca-k8s created

  3. Create the MariaDB root password secret:

    $ kubectl -n ejbca-k8s create secret generic mariadb-secret --from-literal=MARIADB_ROOT_PASSWORD=ejbca

    • The output is similar to the following:

      secret/mariadb-secret created

  4. Change to the k81 directory:

    $ cd k81

  5. Create the mariadb-override.yaml file with settings to deploy MariaDB:

    $ cat > mariadb-override.yaml << EOF
    ## Default values for MariaDB deployment
     
    ## MariaDB docker image
    image:
    registry: "docker.io"
    repository: "mariadb"
    pullPolicy: IfNotPresent
    # Overrides the image tag whose default is the chart appVersion.
    tag: "latest"
     
    ## Optional service account
    serviceAccount:
    # Specifies whether a service account should be created
    create: true
    # Annotations to add to the service account
    annotations: {}
    # The name of the service account to use.
    # If not set and create is true, a name is generated using the fullname template
    name: ""
     
    # Resource limits and requests
    resources:
    limits:
    cpu: 2
    memory: 2048Mi
    requests:
    cpu: 1000m
    memory: 1024Mi
     
    ## Use Kubernetes Deployment instead of StatefulSet
    useDeployment: false
     
    ## Database configuration
    settings:
    ## Arguments for the container entrypoint process
    arguments:
    - --character-set-server=utf8mb4
    - --collation-server=utf8mb4_unicode_ci
    ## Optional existing secret for the MariaDB root password
    existingSecret: mariadb-secret
     
    ## Set true to skip loading timezone data during init
    skipTZInfo: false
     
    ## Storage parameters
    storage:
    ## Alternative set requestedSize to define a size for a dynamically created PVC
    requestedSize: 1Gi
     
    EOF

  6. Deploy MariaDB using helm chart with override:

    $ helm install -n ejbca-k8s ejbca-mariadb groundhog2k/mariadb -f mariadb-override.yaml

    • The output is similar to the following:

      NAME: ejbca-mariadb
      LAST DEPLOYED: Sun Jun 9 13:31:31 2024
      NAMESPACE: ejbca-k8s
      STATUS: deployed
      REVISION: 1
      TEST SUITE: None

  7. Check on the deployment after waiting for 1 min:

    $ kubectl -n ejbca-k8s get all

    • The output is similar to the following:

      NAME READY STATUS RESTARTS AGE
      pod/ejbca-mariadb-0 1/1 Running 0 85s
       
      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      service/ejbca-mariadb ClusterIP 10.152.183.95 <none> 3306/TCP 85s
       
      NAME READY AGE
      statefulset.apps/ejbca-mariadb 1/1 85s

  8. Connect to the database using the service/ejbca-mariadb to make sure the DB is accessible, enter the password for the root account when prompted:

    $ mariadb -h 10.152.183.234 -u root -p

  9. Enter the password for the root account.

    • The output is similar to the following:

      Enter password:
      Welcome to the MariaDB monitor. Commands end with ; or \g.
      Your MariaDB connection id is 51
      Server version: 11.4.2-MariaDB-ubu2404 mariadb.org binary distribution
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [(none)]>

  10. Enter exit to exit the database shell:

    MariaDB [(none)]> exit

    • The output is similar to the following:

      MariaDB [(none)]> exit
      Bye

  11. Import the EJBCA SQL backup into MariaDB, enter the password for the root account when prompted:

    $ mariadb -h 10.152.183.234 -u root -p < ../sql/20240610_BackupToMigrate.sql

  12. Enter the password for the root account.

  13. Connect to the ejbca database as the ejbca user:

    $ mariadb -h 10.152.183.234 -u ejbca -p ejbca

  14. Enter the password for the ejbca account.

    • The output is similar to the following:

      Enter password:
      Reading table information for completion of table and column names
      You can turn off this feature to get a quicker startup with -A
       
      Welcome to the MariaDB monitor. Commands end with ; or \g.
      Your MariaDB connection id is 185
      Server version: 11.4.2-MariaDB-ubu2404 mariadb.org binary distribution
       
      Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
       
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
       
      MariaDB [ejbca]>

  15. Verify the tables are present:

    MariaDB [ejbca]> show tables;

    • The output is similar to the following:

      MariaDB [ejbca]> show tables;
       
      # Output similar to the following:
      +-------------------------------+
      | Tables_in_ejbca |
      +-------------------------------+
      | AccessRulesData |
      | AcmeAccountData |
      | AcmeAuthorizationData |
      | AcmeChallengeData |
      | AcmeNonceData |
      | AcmeOrderData |
      | AdminEntityData |
      | AdminGroupData |
      | AdminPreferencesData |
      | ApprovalData |
      | AuditRecordData |
      | AuthorizationTreeUpdateData |
      | Base64CertData |
      | BlacklistData |
      | CAData |
      | CRLData |
      | CertReqHistoryData |
      | CertificateData |
      | CertificateProfileData |
      | CryptoTokenData |
      | EndEntityProfileData |
      | GlobalConfigurationData |
      | IncompleteIssuanceJournalData |
      | InternalKeyBindingData |
      | KeyRecoveryData |
      | NoConflictCertificateData |
      | OcspResponseData |
      | PeerData |
      | ProfileData |
      | PublisherData |
      | PublisherQueueData |
      | RoleData |
      | RoleMemberData |
      | SctData |
      | ServiceData |
      | UserData |
      | UserDataSourceData |
      +-------------------------------+
      37 rows in set (0.001 sec)

  16. Enter exit to exit the database:

    MariaDB [ejbca]> exit

    • The output is similar to the following:

      MariaDB [ejbca]> exit
      Bye

You have now deployed MariaDB using Helm and restored the EJBCA database backup. You then verified the tables created from restoring the backup. Continue to the next step to deploy EJBCA using Helm.

Step 6 - Deploy EJBCA with Helm

EJBCA is deployed after MariaDB is deployed and the database backup is restored. Follow the steps below to deploy EJBCA:

  1. Install the Helm repository for EJBCA:

    $ helm repo add ejbca-ce https://keyfactor.github.io/ejbca-community-helm

    • The output is similar to the following:

      "ejbca-ce" has been added to your repositories

  2. Create the ejbca-override.yaml file:

    $ cat > ejbca-overrides.yaml << EOF
    ejbca:
    useEphemeralH2Database: false
    useH2Persistence: false
    importExternalCas: false
    importAppserverKeystore: false
    importAppserverTruststore: false
    importEjbcaConfFiles: false
    env:
    TLS_SETUP_ENABLED: "simple"
    DATABASE_JDBC_URL: "jdbc:mariadb://ejbca-mariadb.ejbca-k8s:3306/ejbca?characterEncoding=utf8"
    LOG_AUDIT_TO_DB: true
    LOG_LEVEL_APP: "INFO"
    LOG_LEVEL_SERVER: "INFO"
    OBSERVABLE_BIND: "127.0.0.1"
    METRICS_ENABLED: false
    OCSP_CHECK_SIGN_CERT_VALIDITY: false
    OCSP_NON_EXISTING_IS_GOOD: true
     
    envRaw:
    - name: DATABASE_USER
    valueFrom:
    secretKeyRef:
    name: ejbcadbcredentials
    key: database_user
    - name: DATABASE_PASSWORD
    valueFrom:
    secretKeyRef:
    name: ejbcadbcredentials
    key: database_password
    # - name: PASSWORD_ENCRYPTION_KEY
    # valueFrom:
    # secretKeyRef:
    # name: ejbca-confidentiality-secrets
    # key: password_encrpytion_key
    envFrom: []
    initContainers: []
    sidecarContainers: []
    volumes: []
    volumeMounts: []
    services:
    directHttp:
    enabled: false
    proxyAJP:
    enabled: false
    proxyHttp:
    enabled: true
    type: LoadBalancer
    bindIP: 0.0.0.0
    httpPort: 80
    httpsPort: 443
    sidecarPorts: []
    nginx:
    enabled: true
    host: "ejbca-node1.ejbca-k8s"
    proxy_url_host: localhost
    mountInternalNginxCert: true
    secretInternalNginxCert: "internal-nginx-credential-secret-ca"
    service:
    enabled: false
    type: NodePort
    httpPort: 30080
    httpsPort: 30443
    ingress:
    enabled: false
    replicaCount: 1
    updateStrategy: {}
    image:
    repository: keyfactor/ejbca-ce
    pullPolicy: IfNotPresent
    tag: latest
    imagePullSecrets: []
    nameOverride: "ejbca-ca"
    fullnameOverride: "ejbca-ca"
    serviceAccount:
    create: true
    annotations: {}
    name: ""
    podLabels: {}
    podAnnotations: {}
    podSecurityContext: {}
    securityContext: {}
    resources:
    limits:
    cpu: 4
    memory: 4096Mi
    requests:
    cpu: 1000m
    memory: 2048Mi
    autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 80
    behavior: {}
    podDisruptionBudget:
    create: false
    minAvailable: 1
    maxUnavailable: ""
    nodeSelector: {}
    affinity: {}
    tolerations: []
    topologySpreadConstraints: []
    priorityClassName: ""
    schedulerName: ""
    terminationGracePeriodSeconds: ""
     
    EOF

  3. Create the secret for the EJBCA Database username and password:

    $ kubectl -n ejbca-k8s create secret generic ejbcadbcredentials --from-literal=database_user=ejbca --from-literal=database_password=ejbca

    • The output is similar to the following:

      secret/ejbcadbcredentials created

  4. Create the TLS secret for the certificate, key, and CA cert used by NGINX that is deployed in the EJBCA pod:

    $ kubectl -n ejbca-k8s create secret generic internal-nginx-credential-secret-ca \
    --from-file=ejbca-node1.ejbca-k8s.pem=ejbca-node1.ejbca-k8s.crt \
    --from-file=ejbca-node1.ejbca-k8s-Key.pem=ejbca-node1.ejbca-k8s.key \
    --from-file=ejbca-node1.ejbca-k8s-CA.pem=../ca.crt

    • The output is similar to the following:

      secret/internal-nginx-credential-secret-ca created

  5. Deploy EJBCA using the Helm chart and the override values:

    $ helm install -n ejbca-k8s ejbca-ca ejbca-ce/ejbca-community-helm -f ejbca-overrides.yaml

    • The output is similar to the following:

      NAME: ejbca-ca
      LAST DEPLOYED: Sun Jun 9 15:29:16 2024
      NAMESPACE: ejbca-k8s
      STATUS: deployed
      REVISION: 1
      TEST SUITE: None
      NOTES:
      Thank you for installing EJBCA CE!
       
      Now that you have deployed EJBCA, you should look over the docs on using
      EJBCA available here:
       
      <https://doc.primekey.com/>
       
       
      Your release is named ejbca-ca. To learn more about the release, try:
       
      $ helm status ejbca-ca
      $ helm get manifest ejbca-ca

  6. Review what has been deployed:

    kubectl -n ejbca-k8s get all

    • The output is similar to the following:

      NAME READY STATUS RESTARTS AGE
      pod/ejbca-ca-84649f7f7b-ck5bn 1/2 Running 0 15s
      pod/ejbca-mariadb-0 1/1 Running 0 83m
       
      NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
      service/ejbca-ca LoadBalancer 10.152.183.78 172.16.170.240 80:31313/TCP,443:30725/TCP 15s
      service/ejbca-mariadb ClusterIP 10.152.183.95 <none> 3306/TCP 83m
       
      NAME READY UP-TO-DATE AVAILABLE AGE
      deployment.apps/ejbca-ca 0/1 1 0 15s
       
      NAME DESIRED CURRENT READY AGE
      replicaset.apps/ejbca-ca-84649f7f7b 1 1 0 15s
       
      NAME READY AGE
      statefulset.apps/ejbca-mariadb 1/1 83m

  7. Check the EJBCA container logs and verify the container has properly started:

    $ kubectl -n ejbca-k8s logs pod/ejbca-ca-84649f7f7b-spwh7 -c ejbca-ca

    • The output is similar to the following:

      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) #######################################################################################################
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # Whenever you are ready for production: #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # Try out the commercially supported EJBCA Enterprise Cloud on AWS or Azure, fully featured with: #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # - Additional enrollment APIs such as a complete REST API, EST, ACME and more. #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # - Support for external Registration Authority and OCSP responders #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # - Hardware Security Module support #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # - Application updates #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # - ...and more! #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # Feel free to contact us directly for a cloud, on-prem, SaaS, embedded or hybrid solution that fits #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # your specific needs. #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # <https://aws.amazon.com/marketplace/seller-profile?id=7edf9048-58e6-4086-9d98-b8e0c1d78fce> #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # <https://azuremarketplace.microsoft.com/en-us/marketplace/apps/primekey.ejbca_enterprise_cloud_2> #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # <https://www.keyfactor.com> #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # sales@keyfactor.com #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) # #
      2024-06-09 19:29:50,009+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) #######################################################################################################
      2024-06-09 19:29:50,015+0000 INFO [/opt/keyfactor/bin/start.sh] (process:1) Health check now reports application status at /ejbca/publicweb/healthcheck/ejbcahealth

  8. Open a web browser and access EJBCA Administration.

You have now deployed EJBCA using Helm and logged into the AdminWeb to verify that all your configuration is present from the database restore. At this point, you can continue using EJBCA and try out some of the the other EJBCA integrations.

Next steps

In this tutorial, you learned how to clean up/reset the MicroK8s cluster and deploy MariaDB along with EJBCA using Helm.

Here are some next steps we recommend: