Vault

Thanks to Tomas Srnec https://github.com/Tomasfire for bash script create-snap.sh.

https://www.vaultproject.io

HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services that are gated by authentication and authorization methods. Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing.

Architecture

Vault is an intricate system with numerous distinct components. This page details the system architecture and hopes to assist Vault users and developers to build a mental model while understanding the theory of operation.

Hashicorp Vault Architecture

Shamir’s algorithm

Shamir’s algorithm is used to generate keys to usealing operations after initialization of Vault service.

Hashicorp Vault Shamir's algorithm

User perspactive and process

User interaction is pretty simple. You can use UI (not the best) or use CLI, which suite my needs better then UI.

Hashicorp VaultUser Interaction

Also this give you answer to question: “How the overall process could looklike?”

Hashicorp Vault Process

Roles

Role Description Abbreviation
Developer Developer, or DevOps engineer, or application architect, that requires access to a set of secrets. dev
Application An Application, that requires access to a set of secrets. app
Security Officer A Security officer, that reviews the policy and ultimately approves access. sec
Secrets Admin A Secrets administrator, or Operator that generally implements the policy. dcc
Auditor A Compliance officer, or auditor, that needs full traceability on the what, how and when policies were changed. audit

Secrets storage paths

For various requirements we will need different Secrets engines then path may vary. Most secrets engines can be enabled, disabled, tuned, and moved via the CLI or API. Previous versions of Vault called these “mounts”, but that term was overloaded.

  • Enable - This enables a secrets engine at a given path. With few exceptions, secrets engines can be enabled at multiple paths. Each secrets engine is isolated to its path. By default, they are enabled at their “type” (e.g. “aws” enables at “aws/”).
  • Disable - This disables an existing secrets engine. When a secrets engine is disabled, all of its secrets are revoked (if they support it), and all of the data stored for that engine in the physical storage layer is deleted.
  • Move - This moves the path for an existing secrets engine. This process revokes all secrets, since secret leases are tied to the path they were created at. The configuration data stored for the engine persists through the move.
  • Tune - This tunes global configuration for the secrets engine such as the TTLs.
Path Description
secrets/{region}/{project_id}/{env_type} Environment type and distinguish between DEV, STG, PRE-PROD and PROD for purpose of revocations or other reasons.
secret/{region}/{project_id}/{env_type} Project ID identifies main logical structure for purpose of revocations or other reasons. Project id can ‘lichnakcz’, ‘00000000-0000-0000-ffff-000000029082’, or other IDs supporting easy management of Vault.
secret/{region}/{project_id}/{env_type}/{secret_role} Secret role should be top path where you secrets, keys or certificates are stored

Authenticatio paths

Path Description
auth/{auth_method} Authentication methods supported by your Vault installation.
auth/{auth_method}/role/{access_type} Access type is semantically following roles required for configuration in the field mainly will reflect Roles abbreviations.

Path policies

Policies are written in HCL or JSON and describe which paths in Vault a user or machine is allowed to access:

  1. Write ACL policies in HCL format
  2. Create policies
  3. View existing policies
  4. Check capabilities of a token
path "<PATH>" {
  capabilities = [ "<LIST_OF_CAPABILITIES>" ]
}
Capability Associated HTTP verbs
create POST/PUT
read GET
update POST/PUT
list LIST

Installation

Vault is being deployed to virtual machine hosted at with operating system Rocky Linux 8.4.

Setup of Vault will be to perform autounseal with unsealing backend in AWS KMS. Resources needed for autounsealing will be deployed with terraform.

AWS

  • IAM Account
  • IAM Group
  • IAM Policy
  • IAM Access Key
  • KMS Key

Terraform install Ubuntu

https://www.terraform.io/intro

Terraform is an open-source, infrastructure as code, software tool created by HashiCorp. Users define and provide data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language, or optionally JSON.

sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
wget -O- https://apt.releases.hashicorp.com/gpg | \
    gpg --dearmor | \
    sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
gpg --no-default-keyring \
    --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
    --fingerprint
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
    https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
    sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt-get install terraform

Terraform usage

terraform init -backend-config=”backend-dev.config”
terraform plan -var-file="parentvarvalues.tfvars"
terraform apply -var-file="parentvarvalues-team1.tfvars" -auto-approve

Terraform Enviroment

terraform {
  backend "local" {
  }
}

module "vault_kms_autoseal" {
  source                     = "../../manifests/vault_kms_autounseal"
  aws_region                 = "eu-central-1"
  app_name                   = "siem.lichnak.cz"
  app_id                     = "17"
  team                       = "lichnaci"
  iam_path                   =  "/lichnak/"
 }

Terraform Module

variable "aws_region" {
  type = string
}

variable "app_name" {
  type = string
}

variable "app_id" {
  type = string
}

variable "team" {
  type = string
}

variable "iam_path" {
  type = string
}

provider "aws" {
  #  version = "~> 2"
  region = var.aws_region
}

resource "random_pet" "env" {
  length    = 2
  separator = "_"
}

resource "aws_iam_user" "vault_kms_access" {
  name = "${var.app_name}-${random_pet.env.id}"
  path = var.iam_path

  tags = {
    Name     = "${var.app_name}-${random_pet.env.id}"
    app_id   = var.app_id
    app_name = var.app_name
    team     = var.team
  }
}

resource "aws_iam_access_key" "vault_kms_access_key" {
  user = aws_iam_user.vault_kms_access.name
}

resource "aws_iam_group" "vault_kms_access_group" {
  name = "${var.app_name}-${random_pet.env.id}"
  path = var.iam_path
}

# Policy documents needs adjuments to fit
data "aws_iam_policy_document" "vault_kms_access_policy_document" {
  statement {
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:DescribeKey",
    ]

    resources = [
      "arn:aws:kms:::key/*",
    ]
  }
}

resource "aws_iam_group_policy" "vault_kms_access_policy" {
  name   = "${var.app_name}-${random_pet.env.id}"
  group  = aws_iam_group.vault_kms_access_group.name
  policy = data.aws_iam_policy_document.vault_kms_access_policy_document.json
}

resource "aws_iam_group_membership" "vault_kms_access_group_membership" {
  name  = "${var.app_name}-${random_pet.env.id}"
  users = [aws_iam_user.vault_kms_access.name]
  group = aws_iam_group.vault_kms_access_group.name
  depends_on = [aws_iam_user.vault_kms_access, aws_iam_group.vault_kms_access_group]
}

resource "aws_kms_key" "vault_kms_key" {
  description = "Vault KMS Keys for Auto-Unsealing"
}

resource "aws_kms_alias" "vault_kms_key_alias" {
  target_key_id = aws_kms_key.vault_kms_key.key_id
  name          = "alias/${var.app_name}-${random_pet.env.id}"
}

output "vault_kms_key_id" {
  value = aws_kms_key.vault_kms_key.key_id
}

output "aws_iam_access_key_secret_id" {
  value = aws_iam_access_key.vault_kms_access_key.id
}

output "aws_iam_access_key_secret" {
  value = aws_iam_access_key.vault_kms_access_key.secret
#  value = nonsensitive(aws_iam_access_key.vault_kms_access_key.secret)
#   sensitive = true
}

Terraform apply

terraform init && terraform validate
terraform fmt
terraform plan -out=planned.tfstate
terraform apply planned.tfstate
terraform console
> module.vault_kms_autoseal.vault_kms_key_id
KMSKID
> module.vault_kms_autoseal.aws_iam_access_key_secret_id
AKID
> module.vault_kms_autoseal.aws_iam_access_key_secret
(sensitive)
> exit

VPSFree

vpsFree.cz is an association that was founded in 2008 for the purpose of hosting virtual servers (VPS) for its members. A non-profit association expresses by its very nature that every member has the right to participate in the running of the association and to influence its operation. This is in contrast to classic hosting, where every customer must respect the rules set by them or leave to the competition.

  • OS Rocky Linux
  • Installed software: iptables, nginx, docker-ce, vault
# Installation
dnf install -y nginx iptables certbot python3-certbot-nginx yum-utils
systemctl enable --now nginx
certbot certonly --rsa-key-size 4096 --email kuk@kuk.com --nginx -d lichnak.cz -d www.lichnak.cz -d vault.lichnak.cz -d blog.lichnak.cz

echo "vault soft nofile 65536" >> /etc/security/limits.conf
echo "vault hard nofile 65536" >> /etc/security/limits.conf
echo "vault soft memlock infinity" >> /etc/security/limits.conf
echo "vault hard memlock infinity" >> /etc/security/limits.conf
sysctl -p
sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
ulimit -n 20000000
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384

yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
dnf -y install vault
cat >/usr/lib/systemd/system/vault.service <<"EOF"
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=15
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF
cat > /etc/vault.d/vault.env <<"EOF"
VAULT_SKIP_VERIFY=true
EOF
cat > /etc/vault.d/vault.hcl <<"EOF"
# Full configuration options can be found at https://www.vaultproject.io/docs/configuration

ui = true

# relates to https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/#proxy_protocol

#proxy_protocol_behavior = use_always

#mlock = true
disable_mlock = true

#storage "file" {
#  path = "/opt/vault/data"
#}

storage "raft" {
 path = "/opt/vault/data"
 node_id = "raft_01"
}

#storage "consul" {
#  address = "127.0.0.1:8500"
#  path    = "vault"
#}

# HTTP listener
#listener "tcp" {
#  address = "127.0.0.1:8200"
#  tls_disable = 1
#}

# HTTPS listener
#listener "tcp" {
#  address       = "0.0.0.0:8200"
#  tls_cert_file = "/opt/vault/tls/tls.crt"
#  tls_key_file  = "/opt/vault/tls/tls.key"
#}

listener "tcp" {
  address       = "127.0.0.1:8200"
  tls_cert_file = "/opt/vault/tls/fullchain.pem"
  tls_key_file  = "/opt/vault/tls/privkey.pem"
}

cluster_addr = "https://127.0.0.1:8201"
api_addr = "https://127.0.0.1:8200"
#api_addr = "https://vault.lichnak.cz"

#license_path = "/etc/vault.d/vault.hclic"

# Example AWS KMS auto unseal
seal "awskms" {
  region = "eu-central-1"
  kms_key_id = "kmskid"
}

# Example HSM auto unseal
#seal "pkcs11" {
#  lib            = "/usr/vault/lib/libCryptoki2_64.so"
#  slot           = "0"
#  pin            = "AAAA-BBBB-CCCC-DDDD"
#  key_label      = "vault-hsm-key"
#  hmac_key_label = "vault-hsm-hmac-key"
#}
EOF
cat /opt/vault/tls/tls.crt
cat <<"EOF" | sudo tee /etc/systemd/system/vault.service.d/override.conf > /dev/null
[Service]
Environment="AWS_ACCESS_KEY_ID=AKID"
Environment="AWS_SECRET_ACCESS_KEY=ASAK"
Environment="AWS_REGION=eu-central-1"
EOF
vault -version
vault -autocomplete-install
whereis vault
complete -C /usr/bin/vault vault

#with symbolic link you will need more adjuments
ln -s /etc/letsencrypt/live/lichnak.cz/fullchain.pem /opt/vault/tls/fullchain.pem
ln -s /etc/letsencrypt/live/lichnak.cz/privkey.pem /opt/vault/tls/privkey.pem

#or you can simply copy certs
cp /etc/letsencrypt/live/lichnak.cz/privkey.pem /opt/vault/tls/privkey.pem
cp /etc/letsencrypt/live/lichnak.cz/fullchain.pem /opt/vault/tls/fullchain.pem

chown -R vault:vault /opt/vault/tls/
systemctl enable --now vault
export VAULT_SKIP_VERIFY=true
export VAULT_ADDR="https://127.0.0.1:8200"
vault operator init -key-shares=1 -key-threshold=1

cat > /etc/nginx/sites-available/vault.lichnak.cz.conf <<"EOF"
upstream vault {
  server 127.0.0.1:8200 fail_timeout=0;
}

server {
  server_name vault.lichnak.cz;
  access_log /var/log/nginx/vhost/vault.lichnak.cz-access.log;
  error_log /var/log/nginx/vhost/vault.lichnak.cz-error.log;

  location / {
    proxy_read_timeout 300;
    proxy_connect_timeout 300;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header Host $host;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass https://vault;
    proxy_redirect off;
  }

  location /ui/ {
    proxy_read_timeout 300;
    proxy_connect_timeout 300;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header Host $host;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass https://vault;
    proxy_redirect off;
  }

  location /v1/ {
    proxy_read_timeout 300;
    proxy_connect_timeout 300;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header Host $host;
    proxy_set_header X-NginX-Proxy true;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass https://vault;
    proxy_redirect off;
  }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/lichnak.cz/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/lichnak.cz/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = vault.lichnak.cz) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  server_name vault.lichnak.cz;
    listen 80;
    return 404; # managed by Certbot


}
EOF
ln -s /etc/nginx/sites-available/vault.lichnak.cz.conf /etc/nginx/sites-enabled/vault.lichnak.cz.conf
nginx -t
systemctl restart nginx

Security Checklist

  • Authentication backends: FreeIPA or Azure AD will be used to authenticate users via ldap to provide administrative priviledges.
  • End-to-End TLS: Vault should always be used with TLS in production. If intermediate load balancers or reverse proxies are used to front Vault, they should not terminate TLS. This way traffic is always encrypted in transit to Vault and minimizes risks introduced by intermediate layers. Are certificates prepared? Yes.
  • Single Tenancy: Vault should be the only main process running on a machine. This reduces the risk that another process running on the same machine is compromised and can interact with Vault. Similarly, running on bare metal should be preferred to a VM, and a VM preferred to a container. This reduces the surface area introduced by additional layers of abstraction and other tenants of the hardware. Both VM and container based deployments work, but should be avoided when possible to minimize risk. Do we use single node deployment? Yes.
  • Firewall traffic: Vault listens on well known ports, use a local firewall to restrict all incoming and outgoing traffic to Vault and essential system services like NTP. This includes restricting incoming traffic to permitted subnets and outgoing traffic to services Vault needs to connect to, such as databases. Do we have firewall rules in place for server firewall instance and for internet facing firewall instance? Yes.
  • Disable SSH / Remote Desktop. When running a Vault as a single tenant application, users should never access the machine directly. Instead, they should access Vault through its API over the network. Use a centralized logging and telemetry solution for debugging. Be sure to restrict access to logs as need to know. Do we have ssh enabled and opened? Yes. – Residual risk is on acceptable level.
  • Disable Swap. Vault encrypts data in transit and at rest, however it must still have sensitive data in memory to function. Risk of exposure should be minimized by disabling swap to prevent the operating system from paging sensitive data to disk. Vault attempts to “memory lock” to physical memory automatically, but disabling swap adds another layer of defense. Do we have disabled swap? Yes.
  • Don’t Run as Root. Vault is designed to run as an unprivileged user, and there is no reason to run Vault with root or Administrator privileges, which can expose the Vault process memory and allow access to Vault encryption keys. Running Vault as a regular user reduces its privilege. Configuration files for Vault should have permissions set to restrict access to only the Vault user. Do we create vault user? Yes.
  • Turn Off Core Dumps. A user or administrator that can force a core dump and has access to the resulting file can potentially access Vault encryption keys. Preventing core dumps is a platform-specific process; on Linux setting the resource limit RLIMIT_CORE to 0 disables core dumps. This can be performed by process managers and is also exposed by various shells; in Bash ulimit -c 0 will accomplish this. Do we disabled core memory dump? Yes.
  • Immutable Upgrades. Vault relies on an external storage backend for persistence, and this decoupling allows the servers running Vault to be managed immutably. When upgrading to new versions, new servers with the upgraded version of Vault are brought online. They are attached to the same shared storage backend and unsealed. Then the old servers are destroyed. This reduces the need for remote access and upgrade orchestration which may introduce security gaps. Do we have persistent storage decoupled with Vault instance? Yes.
  • Avoid Root Tokens. Vault provides a root token when it is first initialized. This token should be used to setup the system initially, particularly setting up auth methods so that users may authenticate. We recommend treating Vault configuration as code, and using version control to manage policies. Once setup, the root token should be revoked to eliminate the risk of exposure. Root tokens can be generated when needed, and should be revoked as soon as possible. Do we disabled root token for login to vault? Yes.
  • Enable Auditing. Vault supports several audit devices. Enabling auditing provides a history of all operations performed by Vault and provides a forensics trail in the case of misuse or compromise. Audit logs securely hash any sensitive data, but access should still be restricted to prevent any unintended disclosures. Do we setup auditing? Yes. - Syslog-ng forward to ELK

Configuration

# Configuration
# Basic Admins role and userpass
cat > vaultadmin-policy.hcl <<"EOF"

# Manage auth methods broadly across Vault
path "auth/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Create, update, and delete auth methods
path "sys/auth/*"
{
  capabilities = ["create", "update", "delete", "sudo"]
}

# List auth methods
path "sys/auth"
{
  capabilities = ["read"]
}

# List existing policies via CLI
path "sys/policy"
{
  capabilities = ["read"]
}

# Create and manage ACL policies via CLI
path "sys/policy/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Create and manage ACL policies via API
path "sys/policies/acl/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List, create, update, and delete key/value secrets
path "secret/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage secret engines broadly across Vault
path "sys/mounts/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List existing secret engines
path "sys/mounts"
{
  capabilities = ["read"]
}

# Read health checks
path "sys/health"
{
  capabilities = ["read", "sudo"]
}

EOF

vault policy write vaultadmins vaultadmin-policy.hcl
vault auth enable userpass

vault write auth/userpass/users/lichnak \
    password=foo \
    policies=vaultadmins

# Gitlab CI/CD integration
vault auth enable jwt

vault policy write myproject-staging - <<EOF
# Policy name: myproject-staging
#
# Read-only permission on 'secret/data/myproject/staging/*' path
path "secret/data/myproject/staging/*" {
  capabilities = [ "read" ]
}
EOF

vault policy write myproject-production - <<EOF
# Policy name: myproject-production
#
# Read-only permission on 'secret/data/myproject/production/*' path
path "secret/data/myproject/production/*" {
  capabilities = [ "read" ]
}
EOF

vault write auth/jwt/role/myproject-staging - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-staging"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims": {
    "project_id": "22",
    "ref": "master",
    "ref_type": "branch"
  }
}
EOF

$ vault write auth/jwt/role/myproject-production - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-production"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims_type": "glob",
  "bound_claims": {
    "project_id": "22",
    "ref_protected": "true",
    "ref_type": "branch",
    "ref": "auto-deploy-*"
  }
}
EOF

vault write auth/jwt/config \
    jwks_url="https://gitlab.example.com/-/jwks" \
    bound_issuer="gitlab.example.com"

# Azure AD OAuthv2 authentication
#!/usr/bin/env bash

# flags
set -o errexit # make your script exit when a command fails
set -o pipefail # exit status of the last command that threw a non-zero exit code is returned
set -o nounset # exit when your script tries to use undeclared variables

export _CONST_SECRET_PATH="secrets"
export _CONST_AUTH_PATH="lichnakdevops"
export _CONST_GROUP_NAME="lichnakdevops"
export _CONST_POLICY_NAME="lichnakdevops"

export VAULT_ADDR="https://vault.lichak.cz"
#export VAULT_LOGIN_METHOD="ldap"
#export VAULT_LOGIN_USER="cz010377"
export VAULT_LOGIN_ROLE=${_CONST_GROUP_NAME}
export VAULT_TOKEN=<your_token>

# Vault login
#vault login -method="${VAULT_LOGIN_METHOD}" username="${VAULT_LOGIN_USER}"
vault login

# Azure cli login
az login --allow-no-subscriptions

export AD_AZURE_DISPLAY_NAME="${_CONST_GROUP_NAME}"

export AD_VAULT_APP_ID=$(az ad app create \
   --display-name ${AD_AZURE_DISPLAY_NAME} \
   --reply-urls "http://localhost:8250/oidc/callback" \
   "${VAULT_ADDR}/ui/vault/auth/${_CONST_AUTH_PATH}/oidc/callback" | \
   jq -r '.appId')

export AD_MICROSOFT_GRAPH_API_ID=$(az ad sp list \
   --filter "displayName eq 'Microsoft Graph'" \
   --query '[].appId' -o tsv)

export AD_PERMISSION_GROUP_MEMBER_READ_ALL_ID=$(az ad sp show \
   --id ${AD_MICROSOFT_GRAPH_API_ID} \
   --query "oauth2Permissions[?value=='GroupMember.Read.All'].id" -o tsv)

az ad app permission add \
   --id ${AD_VAULT_APP_ID} \
   --api ${AD_MICROSOFT_GRAPH_API_ID} \
   --api-permissions ${AD_PERMISSION_GROUP_MEMBER_READ_ALL_ID}=Scope

az ad sp create --id ${AD_VAULT_APP_ID}

az ad app permission grant --id ${AD_VAULT_APP_ID} \
    --api ${AD_MICROSOFT_GRAPH_API_ID}

export AD_TENANT_ID=$(az ad sp show --id ${AD_VAULT_APP_ID} \
   --query 'appOwnerTenantId' -o tsv)

export AD_CLIENT_SECRET=$(az ad app credential reset \
    --id ${AD_VAULT_APP_ID} | jq -r '.password')

cat > manifest.json << "EOF"
{
    "idToken": [
        {
            "name": "groups",
            "additionalProperties": []
        }
    ]
}
EOF

az ad app update --id ${AD_VAULT_APP_ID} \
    --set groupMembershipClaims=SecurityGroup \
    --optional-claims @manifest.json

cat > "${_CONST_POLICY_NAME}"-policy.hcl << "EOF"
path "${_CONST_SECRET_PATH}/*" {
    capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

vault policy write "${_CONST_POLICY_NAME}" "${_CONST_POLICY_NAME}"-policy.hcl

vault secrets enable -path="${_CONST_SECRET_PATH}" -version=2 kv

vault kv enable-versioning "${_CONST_SECRET_PATH}"/

vault auth enable -path=${_CONST_AUTH_PATH} oidc

vault write auth/${_CONST_AUTH_PATH}/config \
    oidc_client_id="${AD_VAULT_APP_ID}" \
    oidc_client_secret="${AD_CLIENT_SECRET}" \
    default_role="${VAULT_LOGIN_ROLE}" \
    oidc_discovery_url="https://login.microsoftonline.com/${AD_TENANT_ID}/v2.0"

vault write auth/f13devops/config \
    oidc_client_id="6f1f6a80-0695-4ba8-addd-651e846b4f50" \
    oidc_client_secret="${AD_CLIENT_SECRET}" \
    default_role="f13devops" \
    oidc_discovery_url="https://login.microsoftonline.com/5f58b220-c237-4744-b456-3bbab5fba4f8/v2.0"

vault write auth/${_CONST_AUTH_PATH}/role/${VAULT_LOGIN_ROLE} \
   allowed_redirect_uris="http://localhost:8250/oidc/callback" \
   allowed_redirect_uris="${VAULT_ADDR}/ui/vault/auth/${_CONST_AUTH_PATH}/oidc/callback" \
   user_claim="email" \
   groups_claim="groups" \
   policies="${_CONST_POLICY_NAME}" \
   oidc_scopes="https://graph.microsoft.com/.default"

az ad group create \
   --display-name ${AD_AZURE_DISPLAY_NAME} \
   --mail-nickname=${AD_AZURE_DISPLAY_NAME}

export AD_GROUP_ID=$(az ad group show \
   --group ${AD_AZURE_DISPLAY_NAME} \
   --query objectId -o tsv)

export VAULT_GROUP_ID=$(vault write \
   -field=id -format=table \
   identity/group \
   name="${VAULT_LOGIN_ROLE}" \
   type="external" \
   policies="${_CONST_POLICY_NAME}")

export VAULT_OIDC_ACCESSOR_ID=$(vault auth list -format=json | \
   jq -r '."${_CONST_AUTH_PATH}/".accessor')

export VAULT_GROUP_ALIAS_ID=$(vault write \
   -field=id -format=table \
   identity/group-alias \
   name="${AD_GROUP_ID}" \
   mount_accessor="${VAULT_OIDC_ACCESSOR_ID}" \
   canonical_id="${VAULT_GROUP_ID}")

vault login -method=oidc -path="${_CONST_AUTH_PATH}" role="${_CONST_GROUP_NAME}"

vault kv list "${_CONST_SECRET_PATH}"

export VAULT_ADDR="https://vault.lichnak.cz"                                                                                                            
vault login -method=oidc -path=your_path role=your_role
vault kv list secrets
vault kv get secrets/wwwmysitecz/demo
vault kv get secrets/riskmgmt/test/aad_client_credential
vault kv get secrets/riskmgmt/test/aad_client_id
vault kv get secrets/riskmgmt/test/aad_authority

Backup and restore

# Backups
## Vault Automated bakups
# Check for automated snapshots
vault list sys/storage/raft/snapshot-auto/config

# Local
vault list sys/storage/raft/snapshot-auto/config/hourly \
interval=1h
retain=336
storage_type=local
local_max_space=1073741824 #1GiB in bytes

# AWS S3 bucket
vault list sys/storage/raft/snapshot-auto/config/hourly \
interval=1h
retain=336
storage_type=aws-s3
aws_s3_bucket=lichnak-vault-snapshot
aws_s3_regiont=eu-central-1
aws_s3_enable_kms=true
aws_s3_kms_key=00000000-0000-0000-0000-000000000000

## Vault Manual snapshots
# Vault server preparation
sudo adduser vault-snapshot #password stored in KeePass
sudo su - vault-snapshot
echo "export VAULT_ADDR=https://vault.lichnak.cz" >> .bashrc
. .bashrc
mkdir vault-snap
cd vault-snap

# Creation of policy and generation of credentials
vault login

cat <<EOF > snapshot_policy.hcl
path "/sys/storage/raft/snapshot"
{
  capabilities = ["read"]
}
EOF
vault policy write snapshot_policy snapshot_policy.hcl
vault write auth/approle/role/snapshot-role token_policies="snapshot_policy"
vault read -format=json auth/approle/role/snapshot-role/role-id > ./.role-id
vault write -format=json -f auth/approle/role/snapshot-role/secret-id > ./.secret-id

# Create create-snap.sh script
cat > create-snap.sh <<"EOF"
#!/usr/bin/env bash

# flags
#set -o errexit # make your script exit when a command fails
#set -o pipefail # exit status of the last command that threw a non-zero exit code is returned
#set -o nounset # exit when your script tries to use undeclared variables

export VAULT_ADDR=https://vault.lichnak.cz
SNAP_PATH="/home/vault-snapshot/vault-snap/"
NOW=$(date +'%Y%m%d%H%M')
SECRET_PATH="${SNAP_PATH}.secret-id"
ROLE_PATH="${SNAP_PATH}.role-id"

SECRET_ID_TOKEN=$(jq -r '.data.secret_id' ${SECRET_PATH})
ROLE_ID_TOKEN=$(jq -r '.data.role_id' ${ROLE_PATH})

#echo ${SECRET_ID_TOKEN}
#echo ${ROLE_ID_TOKEN}

throw_error () {
if [ $1 -ne 0 ]; then
        echo "${NOW} - $2"
        exit 1
fi
}

LOGIN_TOKEN=$(vault write -field=token auth/approle/login role_id=${ROLE_ID_TOKEN} secret_id=${SECRET_ID_TOKEN})
login_ret_code=$?
throw_error ${login_ret_code} "Vault login error!"

vault login ${LOGIN_TOKEN} > /dev/null
login_ret_code=$?
throw_error ${login_ret_code} "Vault login error!"

vault operator raft snapshot save ${SNAP_PATH}/vlt${NOW}.snap
snapshot_ret_code=$?
throw_error ${snapshot_ret_code} "Vault snapshot creation error!"
EOF

# Test functionality
./create-snap.sh
ll
# Crontab setup
sudo vi /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed
#17 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
#25 6    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
#47 6    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
#52 6    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )

# Run snapshot script every hour
10 * * * * vault-snapshot /home/vault-snapshot/vault-snap/create-snap.sh &>> /home/vault-snapshot/vault-snap/crontab.log

# Logrotate setup
mkdir /home/vault-snapshot/vault-snap/archive/
cd /etc/logrotate.d/
sudo cat <<EOF > vault-snap
/home/vault-snapshot/vault-snap/*.snap {
        daily
        compress
        olddir /home/vault-snapshot/vault-snap/archive/
        nocreate
        maxage 60
}
EOF
# Restore

Examples

# Create secrets storage
# secret/{project_id}/{env_type}/{secret_role} → secrets/lichnakcz/dev/ssh
#Create secretes storage with KV-v2  
vault secrets enable -version=2 kv
vault kv enable-versioning secret/
vault kv put secrets/lichnakcz/dev/ssh \
  ssh-root-key=AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAArF6nttsDco8LolAOa4paErvRQp2vw4p5KzwaVkABPA=
vault kv get secrets/lichnakcz/dev/ssh
vault kv delete secrets/lichnakcz/dev/ssh
vault kv undelete secrets/lichnakcz/dev/ssh
vault kv destroy -versions=2 secrets/lichnakcz/dev/ssh
vault kv metadata get secrets/lichnakcz/dev/ssh

# Create policies
# vaultadmin policy will be applied to provide access for auth/oidc/role/dcc
# Admin Policy deployment  

cat > vaultadmin-policy.hcl <<"EOF"
# Manage auth methods broadly across Vault
path "auth/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Create, update, and delete auth methods
path "sys/auth/*"
{
  capabilities = ["create", "update", "delete", "sudo"]
}

# List auth methods
path "sys/auth"
{
  capabilities = ["read"]
}

# List existing policies via CLI
path "sys/policy"
{
  capabilities = ["read"]
}

# Create and manage ACL policies via CLI
path "sys/policy/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Create and manage ACL policies via API
path "sys/policies/acl/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List, create, update, and delete key/value secrets
path "secret/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage secret engines broadly across Vault
path "sys/mounts/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List existing secret engines
path "sys/mounts"
{
  capabilities = ["read"]
}

# Read health checks
path "sys/health"
{
  capabilities = ["read", "sudo"]
}
EOF

vault policy write vaultadmin vaultadmin-policy.hcl

vault auth enable oidc
vault login -method=oidc port=8400 role=dcc
#Complete the login via your OIDC provider. Launching browser to:
#
#    https://myco.auth0.com/authorize?redirect_uri=http%3A%2F%2Flocalhost%3A8400%2Foidc%2Fcallback&client_id=r3qXc2bix9eF...

vault write auth/oidc/config \
    oidc_discovery_url="https://myco.auth0.com/" \
    oidc_client_id="m5i8bj3iofytj" \
    oidc_client_secret="f4ubv72nfiu23hnsj" \
    default_role="dcc"

# Create a named role:
vault write auth/oidc/role/dcc \
    bound_subject="r3qX9DljwFIWhsiqwFiu38209F10atW6@clients" \
    bound_audiences="https://vault.plugin.auth.jwt.test" \
    user_claim="https://vault/user" \
    groups_claim="https://vault/groups" \
    policies=vaultadmin \
    ttl=1h
me

My name is Adam Lichonvsky and I'm proud father and researcher.