Using Percona Server for MySQL 8.0 and Percona XtraBackup 8.0 with HashiCorp Vault Enterprise KMIP Secrets Engine

Mysql

KMIP (Key Management Interoperability Protocol) is an open standard developed by OASIS (Organization for Advancement of Structured Information Standards) for the encryption of stored data and cryptographic key management.

Percona Server for MySQL 8.0.27 and Percona XtraBackup 8.0.27 now include a KMIP keyring plugin to enable the exchange of cryptographic keys between a key management server and the database for encryption purposes. The procedure to use them with HashiCorp Vault Enterprise is described below.

Install Hashicorp Vault Enterprise

We will first install Hashicorp Vault Enterprise on Ubuntu Linux “Bionic” and then enable the KMIP secrets engine. The KMIP secrets engine is only available with the Enterprise version of HashiCorp Vault, hence a valid license for it is required.

Add HashiCorp repository and install enterprise vault package:

curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add –
sudo apt-add-repository “deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main”
sudo apt-get update && sudo apt-get install vault-enterprise

Export the license as an environment variable:

export VAULT_LICENSE=XXXX

Create a configuration file to be used with the vault, vault_config.hcl:

disable_mlock = true
default_lease_ttl = “24h”
max_lease_ttl = “24h”
storage “file” {
  path    = “/home/manish.chawla/vault/data”
}

listener “tcp” {
  address     = “127.0.0.1:8200”
  tls_cert_file = “/home/manish.chawla/test_mode/certificates/vault.crt”
  tls_key_file = “/home/manish.chawla/test_mode/certificates/vault.key”
}

Note: Vault root certificates and key need to be created separately and are not covered here.

Start vault server with the configuration file:

vault server -config=$HOME/vault_config.hcl 2>&1 &

Note: To configure and start the vault using systemd, refer to the instructions here.

Initialize the vault:

vault operator init -address=https://127.0.0.1:8200

This will generate five unseal keys and the initial root token.

Unseal Key 1: rf4C6gY87tN0UkJbJY96Aq4+Zext1YqgwaDnm+0gBBAH
Unseal Key 2: f9KA7EdlF391cIUzDqaS1N21JjML/36sZFl1x/OfCPn4
Unseal Key 3: 8Nh3EFYF7S0S9qqzdUIilPPRZRmWaDG+3El4rr4FmZNX
Unseal Key 4: iJaCKBIzxulLvA/6vbF3fRK1RXVZ0zLEZoVdlv/s13Sc
Unseal Key 5: rA4pwT6EZLwmVXPQJfU9fjgeGwPaHl260qM9CVNiUw13

Initial Root Token: s.WXEZ26Yb3MtvzbvNMMIG8bve

Unseal the vault.

Use any three unseal keys to unseal the vault. Three keys are required to unseal the vault.

vault operator unseal -address=https://127.0.0.1:8200 rf4C6gY87tN0UkJbJY96Aq4+Zext1YqgwaDnm+0gBBAH
Key                Value
—                —–
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       aa1a25c8-aa90-49b3-b127-875524de38f8
Version            1.9.3+ent
Storage Type       file
HA Enabled         false

vault operator unseal -address=https://127.0.0.1:8200 f9KA7EdlF391cIUzDqaS1N21JjML/36sZFl1x/OfCPn4
Key                Value
—                —–
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       aa1a25c8-aa90-49b3-b127-875524de38f8
Version            1.9.3+ent
Storage Type       file
HA Enabled         false

vault operator unseal -address=https://127.0.0.1:8200 8Nh3EFYF7S0S9qqzdUIilPPRZRmWaDG+3El4rr4FmZNX
Key             Value
—             —–
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.9.3+ent
Storage Type    file
Cluster Name    vault-cluster-7d9b43de
Cluster ID      c047dcb5-5038-5a29-f4af-47c1ad560f9c
HA Enabled      false

The vault is unsealed.

To use the vault in any terminal, run:

export VAULT_ADDR=https://127.0.0.1:8200
export VAULT_TOKEN=s.WXEZ26Yb3MtvzbvNMMIG8bve
export VAULT_CACERT=/home/manish.chawla/test_mode/certificates/root.cer

Configure KMIP Secrets Engine in Vault

Enable KMIP secrets engine:

vault secrets enable kmip
Success! Enabled the kmip secrets engine at: kmip/

View the secrets list:

vault secrets list
Path          Type         Accessor              Description
—-          —-         ——–              ———–
cubbyhole/    cubbyhole    cubbyhole_ec12856f    per-token private secret storage
identity/     identity     identity_12d9670d     identity store
kmip/         kmip         kmip_5fb3d4c6         n/a
sys/          system       system_1733eece       system endpoints used for control, policy and debugging

Change the kmip server listening address and port:

vault write kmip/config listen_addrs=0.0.0.0:5696
Success! Data written to: kmip/config

Note: Here kmip is the default path of the secret engine and not the type of the engine.

By default, the kmip generates certificates in EC(Elliptic Curve). We need RSA for MySQL, so specify the certificate type (tls_ca_key_type) and bits (tls_ca_key_bits) to configure the kmip server.

vault write kmip/config tls_ca_key_type=”rsa” tls_ca_key_bits=2048
Success! Data written to: kmip/config

vault read kmip/config
Key                            Value
—                            —–
default_tls_client_key_bits    256
default_tls_client_key_type    ec
default_tls_client_ttl         336h
listen_addrs                   [0.0.0.0:5696]
server_hostnames               [localhost]
server_ips                     [127.0.0.1 ::1]
tls_ca_key_bits                2048
tls_ca_key_type                rsa
tls_min_version                tls12

The KMIP secrets engine uses scopes to partition object storage into multiple named buckets. Within a scope, roles can be created with a set of allowed operations that the particular role can perform.

Create a scope:

vault write -f kmip/scope/my-service
Success! Data written to: kmip/scope/my-service

Create a role within the scope, specifying the set of operations to allow or deny.

vault write kmip/scope/my-service/role/admin operation_all=true
Success! Data written to: kmip/scope/my-service/role/admin

Client Certificate Generation for the scope and role created above.

Retrieve the generated CA certificate:

vault read kmip/ca

Copy and save the CA certificate as ca.pem.

Generate a certificate in PEM format, and save it in a JSON file named credential.json.

vault write -format=json
    kmip/scope/my-service/role/admin/credential/generate
    format=pem > credential.json

Extract the certificate from the credential.json using jq tool and save it in a file named cert.pem.

jq -r .data.certificate < credential.json > cert.pem

Extract the private key from the credential.json using jq tool and save it in a file named key.pem.

jq -r .data.private_key < credential.json > key.pem

The KMIP configuration is now complete.

Percona Server for MySQL 8.0.27 Configuration for KMIP

This section describes the KMIP configuration in Percona Server for MySQL. KMIP is configured as a component in Percona Server for MySQL.

Create the global manifest file(mysqld.my) in the mysqld installation directory.

{
  “components”: “file://component_keyring_kmip”
}

Create the global configuration file, component_keyring_kmip.cnf in the directory, where the component_keyring_kmip library resides.

{ “path”: “/home/manish.chawla/keyring_kmip”, “server_addr”: “0.0.0.0”, “server_port”: “5696”, “client_ca”: “/home/manish.chawla/cert.pem”, “client_key”: “/home/manish.chawla/key.pem”, “server_ca”: “/home/manish.chawla/ca.pem” }

Note: SElinux/AppArmor rules may have to be adjusted, so that Percona Server for MySQL and Percona XtraBackup can access the certificates.

Initialize and start mysqld with encryption options(add in my.cnf): 

–innodb-undo-log-encrypt –innodb-redo-log-encrypt –binlog-encryption –default-table-encryption=ON –log-replica-updates –gtid-mode=ON –enforce-gtid-consistency –binlog-format=row –source-verify-checksum=ON –binlog-checksum=CRC32 –table-encryption-privilege-check=ON

Check the KMIP component status:

8.0.27>SELECT * FROM performance_schema.keyring_component_status;
+———————+——————————+
| STATUS_KEY | STATUS_VALUE |
+———————+——————————+
| Component_name | component_keyring_kmip |
| Author | Percona Corporation |
| License | GPL |
| Implementation_name | component_keyring_kmip |
| Version | 1.0 |
| Component_status | Active |
| Server_addr | 0.0.0.0 |
| Server_port | 5696 |
| Client_ca | /home/manish.chawla/cert.pem |
| Client_key | /home/manish.chawla/key.pem |
| Server_ca | /home/manish.chawla/ca.pem |
| Object_group | <NONE> |
+———————+——————————+

Create some encrypted tables and add data in the Percona Server for MySQL.

Backup and Restore of Percona Server for MySQL 8.0.27 Using Percona XtraBackup 8.0.27

This section describes the procedure for taking backup and restore of Percona Server for MySQL 8.0.27 when the KMIP component is enabled and the KMIP vault server is running. Percona XtraBackup reads the KMIP configuration in Percona Server for MySQL automatically, and it is not required to pass this information separately.

Take full backup:

xtrabackup –user=backup –password=* –backup –target-dir=backup_directory

Prepare full backup:

xtrabackup –prepare –target_dir=backup_directory

Stop Percona Server for MySQL and move the data directory to another location. Disable SElinux/AppArmor before restoring the backup.

Restore full backup:

xtrabackup –copy-back –target-dir=backup_directory

Change the ownership of the copied files in the Percona Server for MySQL data directory to the MySQL user.

Start Percona Server for MySQL and check the data. Enable SElinux/AppArmor, if disabled previously.