For the latest version, please use Certificate Lifecycle Manager 5.0.2!

Docker Compose Installation

Prerequisites

  • Docker engine 26.0.0 or later.

  • Docker compose 2.26.0 or later.

  • Zip file containing the MTG KMS in docker compose form (accessible via MTG Download Center).

  • Configured domain name resolution for the external IP address to be used.

  • Network access to the MTG docker registry "repo.mtg.de" and other public docker image registries.

  • A high entropy source. For OSes with linux kernel > 5.4 or Windows systems, no additional actions are needed. For older OSes the haveged service can be installed.

  • The 'sse2' cpu flag must be available in virtualized environment. Otherwise, some components cannot work correctly. To check if the flag is present and available, you can use:

if [[ ! -z $( cat /proc/cpuinfo | grep -E 'sse2' ) ]] ; then echo sse2 flag is presented ; else
echo no sse2 flags presented ; fi

Hardware Requirements

Minimum Requirements

  • 8 GB RAM

  • 4 CPU cores

  • 50 GB hard disk

  • 32 GB RAM

  • 8 CPU cores

  • 200 GB hard disk

MTG Docker Registry Configuration

Login to MTG docker repository with the username and password you were provided via:

docker login -u <MTG_DOCKER_REPO_USER> repo.mtg.de

If you want to verify access for MTG CPKI, make sure you are able to download a container image:

docker pull repo.mtg.de/releases/ers/clm/mtg-clm-ui:4.1.0

or if you want to verify access for MTG KMS:

docker pull repo.mtg.de/releases/ers/kms/kms-ui:3.3.0

First Time Installation

Unzipping

Unzip the contents of the provided zip file at a location of your choice and navigate to the "mtg-ers" folder.

Environment Variables

Rename the default ".template.env" to ".env", in order for it to be available for execution by the installation scripts. This file is located in the same directory as the "docker-compose.yml". Edit the default ".env" file according to your needs. This file is the main configuration file of the docker compose installation. This file also includes a plethora of passwords and secrets, used for various purposes within the application. All passwords have been randomly generated by MTG. Before continuing, you have the option to change them to values of your choice. Try to maintain the same password characteristics to avoid unforeseen errors (e.g., through special secret characters).

Choosing a Database

The docker compose installation provides two options for the database. You can either use an included "embedded" MariaDB or Postgres database, or connect to an external MariaDB or Postgres database. For simple installations, the embedded database method is recommended.

Configuring the Embedded Database

In order to use the embedded database, the compose profile "embedded-db" must be included in the "COMPOSE_PROFILES" configuration option. All "DB_*" options must also be set for the embedded database.

Configuring the External Database

The external database must be either a MariaDB 10.6 or a Postgres 14 one. It must be configured via the "DB_*" configuration options. There are two options for the external database:

  • Externally maintained; the required empty databases and database users are prepared before the initial installation.

  • Maintained by the provided tooling (found under "mtg-ers/tools"); to use those tools, a form of privileged access (root) to the database is needed. The privileged database containers must be configured via "DB_ROOT_USER" and "DB_ROOT_PASSWORD".

External Database Initialization

This procedure requires privileged access.

Execute the following:

  • Linux:

cd tools/create-dbs && sh create-dbs.sh && cd ../../

  • Windows:

cd tools\create-dbs
.\create-dbs.ps1
cd ..\..\

Database Deletion

This procedure requires privileged access.

Stop all running applications and execute the following:

  • Linux:

sh stop.sh && cd tools/delete-all && sh delete-all.sh && cd ../../

  • Windows:

.\stop.ps1
cd tools\delete-all
.\delete-all.ps1
cd ..\..\

Bootstrapping

The bootstrapping scripts will load and start all container images needed. They will also remove any temporary containers that are used exclusively in the bootstrap process. Eventually, you will only have the production containers running in your docker infrastructure.

Execute the following and wait until its finished:

  • Linux:

sh bootstrap.sh

  • Windows:

.\bootstrap.ps1

TLS Configuration

During the bootstrapping phase, installation specific data are generated and placed in the "ssl" folder.

In the root of the ssl folder:

  • "truststore.p12" ⇒ Truststore file which includes the CARA Default Root CA or the MiniCA Root in PKCS12 format.

  • "apache-truststore.pem" ⇒ Truststore file which includes the CARA RA Root, CARA Default Sub CA and KMS MiniCA Sub CA in PEM format. Used by the apache reverse proxy, to authenticate TLS client connections.

In cara folder (available only if "cpki" profile is active):

  • Management Root CA ("rootRa.pem") ⇒ for user certificates used to administer the PKI platform.

  • Management Admin User Certificate ("ra.p12") ⇒ for accessing the CA Admin UI under <domain-name>/cara-admin/. It must be imported in the browser, to be used for TLS client authentication. The password for this file can be found in ".env", under the "DEFAULT_PKI_MANAGEMENT_ADMIN_USER_P12_PASS" variable. Additionally, the Management Root CA must be imported and trusted in the browser/OS.

  • Default Root CA ("rootCA.pem") and one default Sub CA based on the provided configurations that will be used as the default chain for all CPKI use cases. This Root CA should be distributed as the root of trust for issued certificates.

In acme folder (available only if "cpki" profile is active):

  • Server Certificate ("server.*.pem") ⇒ for platform reverse proxy to secure the external TLS connections for UIs and APIs.

In kms folder (available only if "kms" profile is active):

  • "minica_root.pem" ⇒ Default MiniCA Root CA.

  • "minica_sub.pem" ⇒ Default MiniCA Sub CA.

  • "minica_server_cert.pem" ⇒ Default server certificate issued via MiniCA Sub CA, only in cases when "cpki" profile is deactivated.

  • "minica_server_key.pem" ⇒ Default server certificate key issued via MiniCA Sub CA, only in cases when "cpki" profile is deactivated.

  • "kms-tenant-users.json" ⇒ Credentials for the tenant users created for the "Default" KMS tenant.

  • "kms-client.json" ⇒ KMIP & Basic Auth credentials for the kms client user created in the "Default" KMS tenant.

  • "kms-client-ui.json" ⇒ GUI Keycloak credentials for the kms client user created in the "Default" KMS tenant.

  • "kms-client-keystore.p12" ⇒ TLS client auth credentials for the kms client user created in the "Default" KMS tenant.

  • "kms-client-keystore-pass.txt" ⇒ TLS client auth credentials for the kms client user created in the "Default" KMS tenant.

  • "kms-tcp2http-keystore.p12" ⇒ Keystore file containing the minica_server_cert and key in p12 format, to be used as a server certificate by the KMS TCP2HTTP server if the "tcp2http" is enabled.

Start MTG Docker Infrastructure

Execute the following and wait until its finished:

  • Linux:

sh start.sh

  • Windows:

.\start.ps1

You may check your running containers with:

docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"

You’ll get an output that looks like this and depends on the enabled profiles:

   mtg-ers~$ docker ps --format "table {{.ID}}\t{{.Status}}\t{{.Names}}"
   CONTAINER ID   STATUS                    NAMES
   5b3b48b32189   Up 8 minutes (healthy)    mtg-acme-server
   0e13f3d9841c   Up 8 minutes (healthy)    mtg-scep-server
   1e3d933cb72b   Up 8 minutes (healthy)    mtg-clm-frontend
   479e9b86855c   Up 9 minutes (healthy)    mtg-clm-backend
   d8ddf63cbe1a   Up 9 minutes (healthy)    reverse-proxy
   d928f4b81f21   Up 9 minutes (healthy)    cara-ris
   56213b2cae7f   Up 10 minutes (healthy)   cara-admin
   5521bb264fdc   Up 11 minutes (healthy)   cara-server
   74593095b12a   Up 11 minutes (healthy)   ers-keycloak
   9e0e9c9e91a2   Up 11 minutes (healthy)   ers-db

Stop MTG Docker Infrastructure

Execute the following and wait until its finished:

  • Linux:

sh stop.sh

  • Windows:

.\stop.ps1

Product Documentation

Online documentation can be found at docs.mtg.de.

Available URL Paths

Corporate PKI:

  • CARA Admin UI: /cara-admin/

  • CARA REST API: /cara-ws/

  • CLM UI: /clm-ui/

  • Keycloak Admin UI: /auth/

  • CLM API: /clm-api/swagger-ui.html

  • ACME: /acme/v2/directory

  • SCEP: /scep

  • CMP: /cmp

  • EST: /.well-known/est

  • AEC: /aec

Key Management System:

  • KMS UI: /kms-ui/

  • MINI CA UI: /mini-ca/

  • KMIP HTTP API: /kms/

  • KMIP TCP API: <domain-name>:5696

  • KMS REST API: /kms-crypto-api/

Log Files

To view application logs go from root to logs directory:

cd logs

Backups

Backup-relevant objects include the following:

  1. "ssl" folder that includes all platform TLS data.

  2. ".env" config file that includes all the configuration options.

  3. Database backup. Depending on the mode used (embedded or external) and the database type (MariaDB or Postgres), different methods apply.

External Postgres / MariaDB Backups

Backup your databases according to your provider’s instructions. The following databases must be backed up.

Corporate PKI:

  • keycloakdb

  • clmdb

  • acmedb

  • caradb

KMS:

  • keycloakdb

  • minicadb

  • kmsdb

Embedded Postgres DB

Local backups can be created using the following commands and stored in the mtg-ers/backup directory.

Make sure to transfer the created files to a secure location.

Corporate PKI:

For caradb execute:

docker exec -it ers-db pg_dump -U postgres caradb > caradb-backup.sql | gzip > backup/caradb-backup.sql.gz && rm caradb-backup.sql

For clmdb execute:

docker exec -it ers-db pg_dump -U postgres clmdb > clmdb-backup.sql | gzip > backup/clmdb-backup.sql.gz && rm clmdb-backup.sql

For acmedb execute:

docker exec -it ers-db pg_dump -U postgres acmedb > acmedb-backup.sql | gzip > backup/acmedb-backup.sql.gz && rm acmedb-backup.sql

For keycloakdb execute:

docker exec -it ers-db pg_dump -U postgres keycloakdb > keycloakdb-backup.sql | gzip > backup/keycloakdb-backup.sql.gz && rm keycloakdb-backup.sql

KMS:

For minicadb execute the following:

docker exec -it ers-db pg_dump -U postgres minicadb > minica-backup.sql | gzip > backup/minica-backup.sql.gz && rm minica-backup.sql

For kmsdb execute the following:

docker exec -it ers-db pg_dump -U postgres kmsdb > kmsdb-backup.sql | gzip > backup/kmsdb-backup.sql.gz && rm kmsdb-backup.sql

For keycloakdb execute the following:

docker exec -it ers-db pg_dump -U postgres keycloakdb > keycloakdb-backup.sql | gzip > backup/keycloakdb-backup.sql.gz && rm keycloakdb-backup.sql

Embedded MariaDB

If you use the MariaDB embedded-db option, the docker volume "mtg-ers_ersdbdata" includes all database content for the MTG infrastructure. Docker volumes can be backed up in multiple ways, one of which is described below.

Assuming you’ve chosen the MariaDB embedded-db option in your MTG ERS Docker infrastructure, and you’re using a Linux host. To back up your data, use following commands in your crontab entries:

docker exec -it ers-db bash -c 'mariadb-backup --backup --user root --password "$MARIADB_ROOT_PASSWORD" --stream=mbstream | gzip > /backup/$(date +%Y_%m%d_%H%M)_backup.stream.gz'.

The above command will fetch a full backup of MariaDB and place it upon the mounted directory backup (which should be under the mtg-ers directory).

zip -r ./backup/$(date +%Y_%m%d_%H%M)_ssl.zip ./ssl.

The above command will zip everything under the ssl directory recursively and place it in ./backup. For log backups, just add ./logs after ./ssl.

Restore

Restoring existing backups includes the following:

Starting from a clean mtg-ers folder (having the same version as the backups):

  1. Place the contents of the "ssl" inside the empty ssl folder under mtg-ers.

  2. Place the .env config file inside the mtg-ers folder, overwriting the default.

  3. Restore database backups.

If you’re using the MariaDB embedded-db option, you can find the whole process documented here.

During a restore process, bootstrap.sh should not be used, as it would create new data and overwrite existing data.

Upgrade

New version notifications are sent via the MTG Download Center. For this product, each update will include a new zip file with up-to-date:

  • docker infrastructure components

  • documentation

  • release notes

  • migration information (if applicable)

In order to upgrade the container images you use in your infrastructure, all you have to do is to modify the following values in the ".env" file:

  • CARA_VERSION

  • CLM_VERSION

  • KEYCLOAK_VERSION

  • KMS_VERSION

  • MARIADB_VERSION or MARIADB_VERSION

First, stop the infrastructure (Stop MTG Docker Infrastructure) and download the latest release version zip file at a location on your machine (preferably your home folder).

Extract the zip file.

Copy the contents of the mtg-ers/ssl folder and the mtg-ers/.env file from your old docker components directory into the new one which you just downloaded.

Make the change in the ".env" file, stating the latest versions of CARA, CLM, KEYCLOAK, KMS, POSTGRES or MARIADB, always according to the latest ERS Release Notes (mtg-ers-release-notes-doc-<version>.zip), and start it up again (Start MTG Docker Infrastructure).

The above should be enough for most upgrades. You should get notified when there’s a new version for ERS components, regardless of any upgrades in the Docker Compose Infrastructure zip file.

In cases where changes are also required in the docker-compose files, a new version for the Docker Compose Infrastructure will be released and instructions will be explicitly included in the migration documentation.

Purge MTG Docker Infrastructure

Execute the following command to clean up and reset your infrastructure, deleting all data and starting fresh.

Cοmplete all steps in the following sections in following order:

The above will:

  • stop and delete all containers

  • delete the mtg-ers_ersdbdata docker volume or the databases in your external configured server

  • delete the ssl directory

Appendix

Configure Custom Keycloak Truststore (outdated)

  1. Run cd mtg-ers

  2. Run docker compose down

  3. Add or uncomment the following section at the end of the file: mtg-ers/common/keycloak.env:

    ====== ######## Truststore properties for outgoing connections (LDAPs, ADFS, etc)

= more information can be found here: https://www.keycloak.org/server/keycloak-truststore

KC_SPI_TRUSTSTORE_FILE_FILE=/var/mtg/conf/truststore.p12
KC_SPI_TRUSTSTORE_FILE_TYPE=pkcs12
KC_SPI_TRUSTSTORE_FILE_PASSWORD=$TRUSTSTORE_PASS

= available values are: ANY, WILDCARD, STRICT

KC_SPI_TRUSTSTORE_FILE_HOSTNAME_VERIFICATION_POLICY=WILDCARD

  4. Add the following line in the file, if it’s missing: mtg-ers/common/common-compose.yml after line 60 (- ./keycloak-setup/realm-export-mtg-ers.json:/opt/keycloak/data/import/import.json):

    - ../ssl/:/var/mtg/conf/:ro

    The result should look like this:

      ers-keycloak:
    image: quay.io/keycloak/keycloak:$KEYCLOAK_VERSION
    container_name: ers-keycloak
    # command: ["start", "--import-realm"]
    # Read only rootfs issues see https://github.com/keycloak/keycloak/issues/11286
    # in order to overcome those issues we modified the entrypoint and command used
    entrypoint: ""
    command: >
      sh -c "cp -R --preserve=all /opt/keycloak /var/tmp
      && /var/tmp/keycloak/bin/kc.sh start --import-realm"
    restart: unless-stopped
    read_only: true
    tmpfs:
      - /var/tmp:exec
      - /tmp
    env_file:
      - keycloak.env
    volumes:
      - ../logs/keycloak/:/tmp/keycloak/logs/
      - ./keycloak-setup/realm-export-mtg-ers.json:/opt/keycloak/data/import/import.json
      - ../ssl/:/var/mtg/conf/:ro
    healthcheck:
      test: cat /proc/net/tcp*|grep 2139
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 1m
    networks:
      - ersnet
    depends_on:
      ers-db:
        condition: service_healthy

  5. Add the certificate to the existing truststore. Supposed there is the file cert.crt already, place it inside the folder mtg-ers/ssl/.

  6. In the following statement, replace $TRUSTSTORE_PASS with the correct value from the .env file and run it from the mtg-ers folder.

    docker compose -f docker-compose-bootstrap.yml run --no-deps --rm keystore-converter bash -c "keytool -import -alias trusted-cert -noprompt -file /tmp/cert.crt -keystore /tmp/truststore.p12 -storetype PKCS12 -storepass $TRUSTSTORE_PASS"

  7. Run docker compose up -d

Useful Administration Tools

Passwords and secrets autocompletion script for the environment file (.env)

Avoid executing this script after bootstrap process, since it will overwrite all your docker infrastructure’s passwords in the .env file! A lock protection has been implemented in order to protect your .env file by an accidental execution of the autocomplete.sh script.

This is a script for auto-completing .env file with auto-generated passwords and secrets which are needed the ERS bootstrap process

Procedure:

  • .env file must be located to the /mtg-ers directory

  • While inside the /mtg-ers directory, execute below command before proceeding with the bootstrap process:

cd tools/env-autocompletion/ && PASS_LENGTH=<password_character_length> SPECIAL_LENGTH=<special_character_length> docker compose -f docker-compose-env.yml up && cd ../../

Below you may find a common execution example:

cd tools/env-autocompletion/ && PASS_LENGTH=20 SPECIAL_LENGTH=2 docker compose -f docker-compose-env.yml up && cd ../../
This script will generate passwords according to the length you give as a first argument, including an amount of special characters you give as a second argument.

CASES:

Case 1: All following passwords are generated following the principle of the inputs - arguments:

    SMTPSERVER_PASS
    DB_ROOT_PASSWORD
    OPENSEARCH_ADMIN_PASSWORD
    DB_KEYCLOAK_PASS
    KEYCLOAK_ADMIN_USER_PASS
    ERS_ADMIN_API_CLIENT_SECRET
    TRUSTSTORE_PASS
    DEFAULT_PKI_MANAGEMENT_ADMIN_USER_P12_PASS
    DEFAULT_PKI_MANAGEMENT_ADMIN_USER_PASSWORD
    DB_CLM_PASS
    DB_ACME_PASS
    KEYCLOAK_CLM_SERVER_CLIENT_SECRET
    CLM_ACME_API_CLIENT_SECRET
    CLM_SCEP_API_CLIENT_SECRET
    CLM_EST_API_CLIENT_SECRET
    CLM_CMP_API_CLIENT_SECRET
    CLM_AEC_API_CLIENT_SECRET
    DB_CARA_PASS
    CARA_ADMIN_APP_SECRET
    CARA_ADMIN_SECRET
    CARA_ADMIN_BASIC_PASS
    CARA_CLM_APP_SECRET
    CARA_RIS_APP_SECRET
    DB_MINICA_PASS
    KMS_MINICA_ADMIN_PASSWORD
    KMS_MINICA_CLIENT_PASSWORD
    KMS_MINICA_HSM_KEYSTORE_PASSWORD
    DB_KMS_PASS
    KEYCLOAK_KMS_SERVER_CLIENT_SECRET
    KMS_HSM_KEYSTORE_PASSWORD
    KMS_TCP2HTTP_KEYSTORE_PASS

Case 2: The following is generated by hashing the matching passwords using bcrypt:

KMS_MINICA_ADMIN_PASSWORD_BCRYPT
KMS_MINICA_CLIENT_PASSWORD_BCRYPT

  • Command that is being used inside the script to perform the hashing:

htpasswd -bnBC 10 "" "$password" | tr -d ':\n' | sed 's/\$2y/\$2a/'

Case 3: The following is generated by using this command: openssl rand -base64 32:

KMS_MINICA_HSM_ENCRYPTION_AES_KEY=
KMS_DB_PROTECTION_SECRET=

Case 4: The following is generated by using this command: openssl rand -base64 64

KMS_MINICA_JWT_SERCET=
After this execution you may check the .env file and proceed with the bootstrap process.

Database backup and restoration scripts for Embedded Postgres and MariaDB

Execute these scripts (backup.sh, restore.sh) while in mtg-ers root folder, in case you need to use it for your embedded databases backups.

backup.sh - Execution Summary: Creates the databases backups and stores them as gzip compressed files on /backup directory (ers-db container volume).

# At least 2 arguments are needed to run this script
# 1st argument is the database type -> postgres or mariadb (Mandatory)
# 2nd argument is the database name -> caradb or clmdb or acmedb or keycloackdb or kmsdb or minicadb (Mandatory)
# ... (Optional)
# 7th argument is the database name -> caradb or clmdb or acmedb or keycloackdb or kmsdb or minicadb (Optional)

# Execution Example: ./tools/embedded/embedded-backup/backup.sh mariadb caradb clmdb acmedb keycloakdb kmsdb minicadb
# Execution Example: ./tools/embedded/embedded-backup/backup.sh postgres caradb clmdb acmedb keycloakdb kmsdb minicadb
# Execution Example: ./tools/embedded/embedded-backup/backup.sh postgres caradb minicadb
# Execution Example: ./tools/embedded/embedded-backup/backup.sh postgres keycloackdb acmedb

restore.sh - Execution Summary: Restores the databases previous states using the backup files which are already stored on /backup directory (ers-db container volume)

# At least 2 arguments are needed to run this script
# 1st argument is the database type -> postgres or mariadb (Mandatory)
# 2nd argument is the database name -> caradb or clmdb or acmedb or keycloackdb or kmsdb or minicadb (Mandatory)
# ... (Optional)
# 7th argument is the database name -> caradb or clmdb or acmedb or keycloackdb or kmsdb or minicadb (Optional)

# Execution Example: ./tools/embedded/embedded-restore/restore.sh mariadb caradb clmdb acmedb keycloakdb kmsdb minicadb
# Execution Example: ./tools/embedded/embedded-restore/restore.sh postgres caradb clmdb acmedb keycloakdb kmsdb minicadb
# Execution Example: ./tools/embedded/embedded-restore/restore.sh postgres caradb minicadb
# Execution Example: ./tools/embedded/embedded-restore/restore.sh postgres keycloackdb acmedb

Database backup and restoration scripts for External Postgres and MariaDB

Execute these scripts (external-backup.sh, external-restore.sh) while in mtg-ers root folder, in case you need to use it for your external databases backups. To run these scripts, Docker Compose files are needed as well.

external-backup.sh - Execution Summary: Creates the databases backups and stores them as gzip compressed files on /backup directory.

If you are using external Postgres and need backups, run below command while being in mtg-ers root folder:

cd tools/externaldb-backup/ && docker compose -f docker-compose-postgres.yml up && cd ../../

  • After usage, remove the container running the below command:

cd tools/externaldb-backup/ && docker compose -f docker-compose-postgres.yml down && cd ../../

If you are using external Postgres and need backups, run below command in mtg-ers root folder:

cd tools/externaldb-backup/ && docker compose -f docker-compose-maria.yml up && cd ../../

After usage, remove the container running the below command:

cd tools/externaldb-backup/ && docker compose -f docker-compose-maria.yml down && cd ../../

external-restore.sh - Execution Summary: Restores the databases' previous states using the backup files which are already stored on /backup directory.

If you are using external MariaDB and need backups, run below command in mtg-ers root folder:

cd tools/externaldb-restore/ && docker compose -f docker-compose-postgres.yml up && cd ../../

After usage, remove the container running the below command:

cd tools/externaldb-restore/ && docker compose -f docker-compose-postgres.yml down && cd ../../

If you are using external MariaDB and need to restore backups, run below command in mtg-ers root folder:

cd tools/externaldb-restore/ && docker compose -f docker-compose-maria.yml up && cd ../../

After usage, remove the container running below command:

cd tools/externaldb-restore/ && docker compose -f docker-compose-maria.yml down && cd ../../