Vault
Thanks to Tomas Srnec https://github.com/Tomasfire for bash script create-snap.sh.
HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates. Vault provides encryption services that are gated by authentication and authorization methods. Vault secures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing.
Architecture
Vault is an intricate system with numerous distinct components. This page details the system architecture and hopes to assist Vault users and developers to build a mental model while understanding the theory of operation.
Shamir’s algorithm
Shamir’s algorithm is used to generate keys to usealing operations after initialization of Vault service.
User perspactive and process
User interaction is pretty simple. You can use UI (not the best) or use CLI, which suite my needs better then UI.
Also this give you answer to question: “How the overall process could looklike?”
Roles
Role | Description | Abbreviation |
---|---|---|
Developer | Developer, or DevOps engineer, or application architect, that requires access to a set of secrets. | dev |
Application | An Application, that requires access to a set of secrets. | app |
Security Officer | A Security officer, that reviews the policy and ultimately approves access. | sec |
Secrets Admin | A Secrets administrator, or Operator that generally implements the policy. | dcc |
Auditor | A Compliance officer, or auditor, that needs full traceability on the what, how and when policies were changed. | audit |
Secrets storage paths
For various requirements we will need different Secrets engines then path may vary. Most secrets engines can be enabled, disabled, tuned, and moved via the CLI or API. Previous versions of Vault called these “mounts”, but that term was overloaded.
- Enable - This enables a secrets engine at a given path. With few exceptions, secrets engines can be enabled at multiple paths. Each secrets engine is isolated to its path. By default, they are enabled at their “type” (e.g. “aws” enables at “aws/”).
- Disable - This disables an existing secrets engine. When a secrets engine is disabled, all of its secrets are revoked (if they support it), and all of the data stored for that engine in the physical storage layer is deleted.
- Move - This moves the path for an existing secrets engine. This process revokes all secrets, since secret leases are tied to the path they were created at. The configuration data stored for the engine persists through the move.
- Tune - This tunes global configuration for the secrets engine such as the TTLs.
Path | Description |
---|---|
secrets/{region}/{project_id}/{env_type} | Environment type and distinguish between DEV, STG, PRE-PROD and PROD for purpose of revocations or other reasons. |
secret/{region}/{project_id}/{env_type} | Project ID identifies main logical structure for purpose of revocations or other reasons. Project id can ‘lichnakcz’, ‘00000000-0000-0000-ffff-000000029082’, or other IDs supporting easy management of Vault. |
secret/{region}/{project_id}/{env_type}/{secret_role} | Secret role should be top path where you secrets, keys or certificates are stored |
Authenticatio paths
Path | Description |
---|---|
auth/{auth_method} | Authentication methods supported by your Vault installation. |
auth/{auth_method}/role/{access_type} | Access type is semantically following roles required for configuration in the field mainly will reflect Roles abbreviations. |
Path policies
Policies are written in HCL or JSON and describe which paths in Vault a user or machine is allowed to access:
- Write ACL policies in HCL format
- Create policies
- View existing policies
- Check capabilities of a token
path "<PATH>" {
capabilities = [ "<LIST_OF_CAPABILITIES>" ]
}
Capability | Associated HTTP verbs |
---|---|
create | POST/PUT |
read | GET |
update | POST/PUT |
list | LIST |
Installation
Vault is being deployed to virtual machine hosted at
Setup of Vault will be to perform autounseal with unsealing backend in AWS KMS. Resources needed for autounsealing will be deployed with terraform.
AWS
- IAM Account
- IAM Group
- IAM Policy
- IAM Access Key
- KMS Key
Terraform install Ubuntu
https://www.terraform.io/intro
Terraform is an open-source, infrastructure as code, software tool created by HashiCorp. Users define and provide data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language, or optionally JSON.
sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
wget -O- https://apt.releases.hashicorp.com/gpg | \
gpg --dearmor | \
sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
gpg --no-default-keyring \
--keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg \
--fingerprint
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt-get install terraform
Terraform usage
terraform init -backend-config=”backend-dev.config”
terraform plan -var-file="parentvarvalues.tfvars"
terraform apply -var-file="parentvarvalues-team1.tfvars" -auto-approve
Terraform Enviroment
terraform {
backend "local" {
}
}
module "vault_kms_autoseal" {
source = "../../manifests/vault_kms_autounseal"
aws_region = "eu-central-1"
app_name = "siem.lichnak.cz"
app_id = "17"
team = "lichnaci"
iam_path = "/lichnak/"
}
Terraform Module
variable "aws_region" {
type = string
}
variable "app_name" {
type = string
}
variable "app_id" {
type = string
}
variable "team" {
type = string
}
variable "iam_path" {
type = string
}
provider "aws" {
# version = "~> 2"
region = var.aws_region
}
resource "random_pet" "env" {
length = 2
separator = "_"
}
resource "aws_iam_user" "vault_kms_access" {
name = "${var.app_name}-${random_pet.env.id}"
path = var.iam_path
tags = {
Name = "${var.app_name}-${random_pet.env.id}"
app_id = var.app_id
app_name = var.app_name
team = var.team
}
}
resource "aws_iam_access_key" "vault_kms_access_key" {
user = aws_iam_user.vault_kms_access.name
}
resource "aws_iam_group" "vault_kms_access_group" {
name = "${var.app_name}-${random_pet.env.id}"
path = var.iam_path
}
# Policy documents needs adjuments to fit
data "aws_iam_policy_document" "vault_kms_access_policy_document" {
statement {
actions = [
"kms:Encrypt",
"kms:Decrypt",
"kms:DescribeKey",
]
resources = [
"arn:aws:kms:::key/*",
]
}
}
resource "aws_iam_group_policy" "vault_kms_access_policy" {
name = "${var.app_name}-${random_pet.env.id}"
group = aws_iam_group.vault_kms_access_group.name
policy = data.aws_iam_policy_document.vault_kms_access_policy_document.json
}
resource "aws_iam_group_membership" "vault_kms_access_group_membership" {
name = "${var.app_name}-${random_pet.env.id}"
users = [aws_iam_user.vault_kms_access.name]
group = aws_iam_group.vault_kms_access_group.name
depends_on = [aws_iam_user.vault_kms_access, aws_iam_group.vault_kms_access_group]
}
resource "aws_kms_key" "vault_kms_key" {
description = "Vault KMS Keys for Auto-Unsealing"
}
resource "aws_kms_alias" "vault_kms_key_alias" {
target_key_id = aws_kms_key.vault_kms_key.key_id
name = "alias/${var.app_name}-${random_pet.env.id}"
}
output "vault_kms_key_id" {
value = aws_kms_key.vault_kms_key.key_id
}
output "aws_iam_access_key_secret_id" {
value = aws_iam_access_key.vault_kms_access_key.id
}
output "aws_iam_access_key_secret" {
value = aws_iam_access_key.vault_kms_access_key.secret
# value = nonsensitive(aws_iam_access_key.vault_kms_access_key.secret)
# sensitive = true
}
Terraform apply
terraform init && terraform validate
terraform fmt
terraform plan -out=planned.tfstate
terraform apply planned.tfstate
terraform console
> module.vault_kms_autoseal.vault_kms_key_id
KMSKID
> module.vault_kms_autoseal.aws_iam_access_key_secret_id
AKID
> module.vault_kms_autoseal.aws_iam_access_key_secret
(sensitive)
> exit
VPSFree
vpsFree.cz is an association that was founded in 2008 for the purpose of hosting virtual servers (VPS) for its members. A non-profit association expresses by its very nature that every member has the right to participate in the running of the association and to influence its operation. This is in contrast to classic hosting, where every customer must respect the rules set by them or leave to the competition.
- OS Rocky Linux
- Installed software: iptables, nginx, docker-ce, vault
# Installation
dnf install -y nginx iptables certbot python3-certbot-nginx yum-utils
systemctl enable --now nginx
certbot certonly --rsa-key-size 4096 --email kuk@kuk.com --nginx -d lichnak.cz -d www.lichnak.cz -d vault.lichnak.cz -d blog.lichnak.cz
echo "vault soft nofile 65536" >> /etc/security/limits.conf
echo "vault hard nofile 65536" >> /etc/security/limits.conf
echo "vault soft memlock infinity" >> /etc/security/limits.conf
echo "vault hard memlock infinity" >> /etc/security/limits.conf
sysctl -p
sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
ulimit -n 20000000
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
dnf -y install vault
cat >/usr/lib/systemd/system/vault.service <<"EOF"
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=15
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
EOF
cat > /etc/vault.d/vault.env <<"EOF"
VAULT_SKIP_VERIFY=true
EOF
cat > /etc/vault.d/vault.hcl <<"EOF"
# Full configuration options can be found at https://www.vaultproject.io/docs/configuration
ui = true
# relates to https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/#proxy_protocol
#proxy_protocol_behavior = use_always
#mlock = true
disable_mlock = true
#storage "file" {
# path = "/opt/vault/data"
#}
storage "raft" {
path = "/opt/vault/data"
node_id = "raft_01"
}
#storage "consul" {
# address = "127.0.0.1:8500"
# path = "vault"
#}
# HTTP listener
#listener "tcp" {
# address = "127.0.0.1:8200"
# tls_disable = 1
#}
# HTTPS listener
#listener "tcp" {
# address = "0.0.0.0:8200"
# tls_cert_file = "/opt/vault/tls/tls.crt"
# tls_key_file = "/opt/vault/tls/tls.key"
#}
listener "tcp" {
address = "127.0.0.1:8200"
tls_cert_file = "/opt/vault/tls/fullchain.pem"
tls_key_file = "/opt/vault/tls/privkey.pem"
}
cluster_addr = "https://127.0.0.1:8201"
api_addr = "https://127.0.0.1:8200"
#api_addr = "https://vault.lichnak.cz"
#license_path = "/etc/vault.d/vault.hclic"
# Example AWS KMS auto unseal
seal "awskms" {
region = "eu-central-1"
kms_key_id = "kmskid"
}
# Example HSM auto unseal
#seal "pkcs11" {
# lib = "/usr/vault/lib/libCryptoki2_64.so"
# slot = "0"
# pin = "AAAA-BBBB-CCCC-DDDD"
# key_label = "vault-hsm-key"
# hmac_key_label = "vault-hsm-hmac-key"
#}
EOF
cat /opt/vault/tls/tls.crt
cat <<"EOF" | sudo tee /etc/systemd/system/vault.service.d/override.conf > /dev/null
[Service]
Environment="AWS_ACCESS_KEY_ID=AKID"
Environment="AWS_SECRET_ACCESS_KEY=ASAK"
Environment="AWS_REGION=eu-central-1"
EOF
vault -version
vault -autocomplete-install
whereis vault
complete -C /usr/bin/vault vault
#with symbolic link you will need more adjuments
ln -s /etc/letsencrypt/live/lichnak.cz/fullchain.pem /opt/vault/tls/fullchain.pem
ln -s /etc/letsencrypt/live/lichnak.cz/privkey.pem /opt/vault/tls/privkey.pem
#or you can simply copy certs
cp /etc/letsencrypt/live/lichnak.cz/privkey.pem /opt/vault/tls/privkey.pem
cp /etc/letsencrypt/live/lichnak.cz/fullchain.pem /opt/vault/tls/fullchain.pem
chown -R vault:vault /opt/vault/tls/
systemctl enable --now vault
export VAULT_SKIP_VERIFY=true
export VAULT_ADDR="https://127.0.0.1:8200"
vault operator init -key-shares=1 -key-threshold=1
cat > /etc/nginx/sites-available/vault.lichnak.cz.conf <<"EOF"
upstream vault {
server 127.0.0.1:8200 fail_timeout=0;
}
server {
server_name vault.lichnak.cz;
access_log /var/log/nginx/vhost/vault.lichnak.cz-access.log;
error_log /var/log/nginx/vhost/vault.lichnak.cz-error.log;
location / {
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://vault;
proxy_redirect off;
}
location /ui/ {
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://vault;
proxy_redirect off;
}
location /v1/ {
proxy_read_timeout 300;
proxy_connect_timeout 300;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header Host $host;
proxy_set_header X-NginX-Proxy true;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass https://vault;
proxy_redirect off;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/lichnak.cz/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/lichnak.cz/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = vault.lichnak.cz) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name vault.lichnak.cz;
listen 80;
return 404; # managed by Certbot
}
EOF
ln -s /etc/nginx/sites-available/vault.lichnak.cz.conf /etc/nginx/sites-enabled/vault.lichnak.cz.conf
nginx -t
systemctl restart nginx
Security Checklist
- Authentication backends: FreeIPA or Azure AD will be used to authenticate users via ldap to provide administrative priviledges.
- End-to-End TLS: Vault should always be used with TLS in production. If intermediate load balancers or reverse proxies are used to front Vault, they should not terminate TLS. This way traffic is always encrypted in transit to Vault and minimizes risks introduced by intermediate layers. Are certificates prepared? Yes.
- Single Tenancy: Vault should be the only main process running on a machine. This reduces the risk that another process running on the same machine is compromised and can interact with Vault. Similarly, running on bare metal should be preferred to a VM, and a VM preferred to a container. This reduces the surface area introduced by additional layers of abstraction and other tenants of the hardware. Both VM and container based deployments work, but should be avoided when possible to minimize risk. Do we use single node deployment? Yes.
- Firewall traffic: Vault listens on well known ports, use a local firewall to restrict all incoming and outgoing traffic to Vault and essential system services like NTP. This includes restricting incoming traffic to permitted subnets and outgoing traffic to services Vault needs to connect to, such as databases. Do we have firewall rules in place for server firewall instance and for internet facing firewall instance? Yes.
- Disable SSH / Remote Desktop. When running a Vault as a single tenant application, users should never access the machine directly. Instead, they should access Vault through its API over the network. Use a centralized logging and telemetry solution for debugging. Be sure to restrict access to logs as need to know. Do we have ssh enabled and opened? Yes. – Residual risk is on acceptable level.
- Disable Swap. Vault encrypts data in transit and at rest, however it must still have sensitive data in memory to function. Risk of exposure should be minimized by disabling swap to prevent the operating system from paging sensitive data to disk. Vault attempts to “memory lock” to physical memory automatically, but disabling swap adds another layer of defense. Do we have disabled swap? Yes.
- Don’t Run as Root. Vault is designed to run as an unprivileged user, and there is no reason to run Vault with root or Administrator privileges, which can expose the Vault process memory and allow access to Vault encryption keys. Running Vault as a regular user reduces its privilege. Configuration files for Vault should have permissions set to restrict access to only the Vault user. Do we create vault user? Yes.
- Turn Off Core Dumps. A user or administrator that can force a core dump and has access to the resulting file can potentially access Vault encryption keys. Preventing core dumps is a platform-specific process; on Linux setting the resource limit RLIMIT_CORE to 0 disables core dumps. This can be performed by process managers and is also exposed by various shells; in Bash ulimit -c 0 will accomplish this. Do we disabled core memory dump? Yes.
- Immutable Upgrades. Vault relies on an external storage backend for persistence, and this decoupling allows the servers running Vault to be managed immutably. When upgrading to new versions, new servers with the upgraded version of Vault are brought online. They are attached to the same shared storage backend and unsealed. Then the old servers are destroyed. This reduces the need for remote access and upgrade orchestration which may introduce security gaps. Do we have persistent storage decoupled with Vault instance? Yes.
- Avoid Root Tokens. Vault provides a root token when it is first initialized. This token should be used to setup the system initially, particularly setting up auth methods so that users may authenticate. We recommend treating Vault configuration as code, and using version control to manage policies. Once setup, the root token should be revoked to eliminate the risk of exposure. Root tokens can be generated when needed, and should be revoked as soon as possible. Do we disabled root token for login to vault? Yes.
- Enable Auditing. Vault supports several audit devices. Enabling auditing provides a history of all operations performed by Vault and provides a forensics trail in the case of misuse or compromise. Audit logs securely hash any sensitive data, but access should still be restricted to prevent any unintended disclosures. Do we setup auditing? Yes. - Syslog-ng forward to ELK
Configuration
# Configuration
# Basic Admins role and userpass
cat > vaultadmin-policy.hcl <<"EOF"
# Manage auth methods broadly across Vault
path "auth/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Create, update, and delete auth methods
path "sys/auth/*"
{
capabilities = ["create", "update", "delete", "sudo"]
}
# List auth methods
path "sys/auth"
{
capabilities = ["read"]
}
# List existing policies via CLI
path "sys/policy"
{
capabilities = ["read"]
}
# Create and manage ACL policies via CLI
path "sys/policy/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Create and manage ACL policies via API
path "sys/policies/acl/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# List, create, update, and delete key/value secrets
path "secret/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Manage secret engines broadly across Vault
path "sys/mounts/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# List existing secret engines
path "sys/mounts"
{
capabilities = ["read"]
}
# Read health checks
path "sys/health"
{
capabilities = ["read", "sudo"]
}
EOF
vault policy write vaultadmins vaultadmin-policy.hcl
vault auth enable userpass
vault write auth/userpass/users/lichnak \
password=foo \
policies=vaultadmins
# Gitlab CI/CD integration
vault auth enable jwt
vault policy write myproject-staging - <<EOF
# Policy name: myproject-staging
#
# Read-only permission on 'secret/data/myproject/staging/*' path
path "secret/data/myproject/staging/*" {
capabilities = [ "read" ]
}
EOF
vault policy write myproject-production - <<EOF
# Policy name: myproject-production
#
# Read-only permission on 'secret/data/myproject/production/*' path
path "secret/data/myproject/production/*" {
capabilities = [ "read" ]
}
EOF
vault write auth/jwt/role/myproject-staging - <<EOF
{
"role_type": "jwt",
"policies": ["myproject-staging"],
"token_explicit_max_ttl": 60,
"user_claim": "user_email",
"bound_claims": {
"project_id": "22",
"ref": "master",
"ref_type": "branch"
}
}
EOF
$ vault write auth/jwt/role/myproject-production - <<EOF
{
"role_type": "jwt",
"policies": ["myproject-production"],
"token_explicit_max_ttl": 60,
"user_claim": "user_email",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "22",
"ref_protected": "true",
"ref_type": "branch",
"ref": "auto-deploy-*"
}
}
EOF
vault write auth/jwt/config \
jwks_url="https://gitlab.example.com/-/jwks" \
bound_issuer="gitlab.example.com"
# Azure AD OAuthv2 authentication
#!/usr/bin/env bash
# flags
set -o errexit # make your script exit when a command fails
set -o pipefail # exit status of the last command that threw a non-zero exit code is returned
set -o nounset # exit when your script tries to use undeclared variables
export _CONST_SECRET_PATH="secrets"
export _CONST_AUTH_PATH="lichnakdevops"
export _CONST_GROUP_NAME="lichnakdevops"
export _CONST_POLICY_NAME="lichnakdevops"
export VAULT_ADDR="https://vault.lichak.cz"
#export VAULT_LOGIN_METHOD="ldap"
#export VAULT_LOGIN_USER="cz010377"
export VAULT_LOGIN_ROLE=${_CONST_GROUP_NAME}
export VAULT_TOKEN=<your_token>
# Vault login
#vault login -method="${VAULT_LOGIN_METHOD}" username="${VAULT_LOGIN_USER}"
vault login
# Azure cli login
az login --allow-no-subscriptions
export AD_AZURE_DISPLAY_NAME="${_CONST_GROUP_NAME}"
export AD_VAULT_APP_ID=$(az ad app create \
--display-name ${AD_AZURE_DISPLAY_NAME} \
--reply-urls "http://localhost:8250/oidc/callback" \
"${VAULT_ADDR}/ui/vault/auth/${_CONST_AUTH_PATH}/oidc/callback" | \
jq -r '.appId')
export AD_MICROSOFT_GRAPH_API_ID=$(az ad sp list \
--filter "displayName eq 'Microsoft Graph'" \
--query '[].appId' -o tsv)
export AD_PERMISSION_GROUP_MEMBER_READ_ALL_ID=$(az ad sp show \
--id ${AD_MICROSOFT_GRAPH_API_ID} \
--query "oauth2Permissions[?value=='GroupMember.Read.All'].id" -o tsv)
az ad app permission add \
--id ${AD_VAULT_APP_ID} \
--api ${AD_MICROSOFT_GRAPH_API_ID} \
--api-permissions ${AD_PERMISSION_GROUP_MEMBER_READ_ALL_ID}=Scope
az ad sp create --id ${AD_VAULT_APP_ID}
az ad app permission grant --id ${AD_VAULT_APP_ID} \
--api ${AD_MICROSOFT_GRAPH_API_ID}
export AD_TENANT_ID=$(az ad sp show --id ${AD_VAULT_APP_ID} \
--query 'appOwnerTenantId' -o tsv)
export AD_CLIENT_SECRET=$(az ad app credential reset \
--id ${AD_VAULT_APP_ID} | jq -r '.password')
cat > manifest.json << "EOF"
{
"idToken": [
{
"name": "groups",
"additionalProperties": []
}
]
}
EOF
az ad app update --id ${AD_VAULT_APP_ID} \
--set groupMembershipClaims=SecurityGroup \
--optional-claims @manifest.json
cat > "${_CONST_POLICY_NAME}"-policy.hcl << "EOF"
path "${_CONST_SECRET_PATH}/*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF
vault policy write "${_CONST_POLICY_NAME}" "${_CONST_POLICY_NAME}"-policy.hcl
vault secrets enable -path="${_CONST_SECRET_PATH}" -version=2 kv
vault kv enable-versioning "${_CONST_SECRET_PATH}"/
vault auth enable -path=${_CONST_AUTH_PATH} oidc
vault write auth/${_CONST_AUTH_PATH}/config \
oidc_client_id="${AD_VAULT_APP_ID}" \
oidc_client_secret="${AD_CLIENT_SECRET}" \
default_role="${VAULT_LOGIN_ROLE}" \
oidc_discovery_url="https://login.microsoftonline.com/${AD_TENANT_ID}/v2.0"
vault write auth/f13devops/config \
oidc_client_id="6f1f6a80-0695-4ba8-addd-651e846b4f50" \
oidc_client_secret="${AD_CLIENT_SECRET}" \
default_role="f13devops" \
oidc_discovery_url="https://login.microsoftonline.com/5f58b220-c237-4744-b456-3bbab5fba4f8/v2.0"
vault write auth/${_CONST_AUTH_PATH}/role/${VAULT_LOGIN_ROLE} \
allowed_redirect_uris="http://localhost:8250/oidc/callback" \
allowed_redirect_uris="${VAULT_ADDR}/ui/vault/auth/${_CONST_AUTH_PATH}/oidc/callback" \
user_claim="email" \
groups_claim="groups" \
policies="${_CONST_POLICY_NAME}" \
oidc_scopes="https://graph.microsoft.com/.default"
az ad group create \
--display-name ${AD_AZURE_DISPLAY_NAME} \
--mail-nickname=${AD_AZURE_DISPLAY_NAME}
export AD_GROUP_ID=$(az ad group show \
--group ${AD_AZURE_DISPLAY_NAME} \
--query objectId -o tsv)
export VAULT_GROUP_ID=$(vault write \
-field=id -format=table \
identity/group \
name="${VAULT_LOGIN_ROLE}" \
type="external" \
policies="${_CONST_POLICY_NAME}")
export VAULT_OIDC_ACCESSOR_ID=$(vault auth list -format=json | \
jq -r '."${_CONST_AUTH_PATH}/".accessor')
export VAULT_GROUP_ALIAS_ID=$(vault write \
-field=id -format=table \
identity/group-alias \
name="${AD_GROUP_ID}" \
mount_accessor="${VAULT_OIDC_ACCESSOR_ID}" \
canonical_id="${VAULT_GROUP_ID}")
vault login -method=oidc -path="${_CONST_AUTH_PATH}" role="${_CONST_GROUP_NAME}"
vault kv list "${_CONST_SECRET_PATH}"
export VAULT_ADDR="https://vault.lichnak.cz"
vault login -method=oidc -path=your_path role=your_role
vault kv list secrets
vault kv get secrets/wwwmysitecz/demo
vault kv get secrets/riskmgmt/test/aad_client_credential
vault kv get secrets/riskmgmt/test/aad_client_id
vault kv get secrets/riskmgmt/test/aad_authority
Backup and restore
# Backups
## Vault Automated bakups
# Check for automated snapshots
vault list sys/storage/raft/snapshot-auto/config
# Local
vault list sys/storage/raft/snapshot-auto/config/hourly \
interval=1h
retain=336
storage_type=local
local_max_space=1073741824 #1GiB in bytes
# AWS S3 bucket
vault list sys/storage/raft/snapshot-auto/config/hourly \
interval=1h
retain=336
storage_type=aws-s3
aws_s3_bucket=lichnak-vault-snapshot
aws_s3_regiont=eu-central-1
aws_s3_enable_kms=true
aws_s3_kms_key=00000000-0000-0000-0000-000000000000
## Vault Manual snapshots
# Vault server preparation
sudo adduser vault-snapshot #password stored in KeePass
sudo su - vault-snapshot
echo "export VAULT_ADDR=https://vault.lichnak.cz" >> .bashrc
. .bashrc
mkdir vault-snap
cd vault-snap
# Creation of policy and generation of credentials
vault login
cat <<EOF > snapshot_policy.hcl
path "/sys/storage/raft/snapshot"
{
capabilities = ["read"]
}
EOF
vault policy write snapshot_policy snapshot_policy.hcl
vault write auth/approle/role/snapshot-role token_policies="snapshot_policy"
vault read -format=json auth/approle/role/snapshot-role/role-id > ./.role-id
vault write -format=json -f auth/approle/role/snapshot-role/secret-id > ./.secret-id
# Create create-snap.sh script
cat > create-snap.sh <<"EOF"
#!/usr/bin/env bash
# flags
#set -o errexit # make your script exit when a command fails
#set -o pipefail # exit status of the last command that threw a non-zero exit code is returned
#set -o nounset # exit when your script tries to use undeclared variables
export VAULT_ADDR=https://vault.lichnak.cz
SNAP_PATH="/home/vault-snapshot/vault-snap/"
NOW=$(date +'%Y%m%d%H%M')
SECRET_PATH="${SNAP_PATH}.secret-id"
ROLE_PATH="${SNAP_PATH}.role-id"
SECRET_ID_TOKEN=$(jq -r '.data.secret_id' ${SECRET_PATH})
ROLE_ID_TOKEN=$(jq -r '.data.role_id' ${ROLE_PATH})
#echo ${SECRET_ID_TOKEN}
#echo ${ROLE_ID_TOKEN}
throw_error () {
if [ $1 -ne 0 ]; then
echo "${NOW} - $2"
exit 1
fi
}
LOGIN_TOKEN=$(vault write -field=token auth/approle/login role_id=${ROLE_ID_TOKEN} secret_id=${SECRET_ID_TOKEN})
login_ret_code=$?
throw_error ${login_ret_code} "Vault login error!"
vault login ${LOGIN_TOKEN} > /dev/null
login_ret_code=$?
throw_error ${login_ret_code} "Vault login error!"
vault operator raft snapshot save ${SNAP_PATH}/vlt${NOW}.snap
snapshot_ret_code=$?
throw_error ${snapshot_ret_code} "Vault snapshot creation error!"
EOF
# Test functionality
./create-snap.sh
ll
# Crontab setup
sudo vi /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
#17 * * * * root cd / && run-parts --report /etc/cron.hourly
#25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
#47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
#52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
# Run snapshot script every hour
10 * * * * vault-snapshot /home/vault-snapshot/vault-snap/create-snap.sh &>> /home/vault-snapshot/vault-snap/crontab.log
# Logrotate setup
mkdir /home/vault-snapshot/vault-snap/archive/
cd /etc/logrotate.d/
sudo cat <<EOF > vault-snap
/home/vault-snapshot/vault-snap/*.snap {
daily
compress
olddir /home/vault-snapshot/vault-snap/archive/
nocreate
maxage 60
}
EOF
# Restore
Examples
# Create secrets storage
# secret/{project_id}/{env_type}/{secret_role} → secrets/lichnakcz/dev/ssh
#Create secretes storage with KV-v2
vault secrets enable -version=2 kv
vault kv enable-versioning secret/
vault kv put secrets/lichnakcz/dev/ssh \
ssh-root-key=AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAArF6nttsDco8LolAOa4paErvRQp2vw4p5KzwaVkABPA=
vault kv get secrets/lichnakcz/dev/ssh
vault kv delete secrets/lichnakcz/dev/ssh
vault kv undelete secrets/lichnakcz/dev/ssh
vault kv destroy -versions=2 secrets/lichnakcz/dev/ssh
vault kv metadata get secrets/lichnakcz/dev/ssh
# Create policies
# vaultadmin policy will be applied to provide access for auth/oidc/role/dcc
# Admin Policy deployment
cat > vaultadmin-policy.hcl <<"EOF"
# Manage auth methods broadly across Vault
path "auth/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Create, update, and delete auth methods
path "sys/auth/*"
{
capabilities = ["create", "update", "delete", "sudo"]
}
# List auth methods
path "sys/auth"
{
capabilities = ["read"]
}
# List existing policies via CLI
path "sys/policy"
{
capabilities = ["read"]
}
# Create and manage ACL policies via CLI
path "sys/policy/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Create and manage ACL policies via API
path "sys/policies/acl/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# List, create, update, and delete key/value secrets
path "secret/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# Manage secret engines broadly across Vault
path "sys/mounts/*"
{
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
# List existing secret engines
path "sys/mounts"
{
capabilities = ["read"]
}
# Read health checks
path "sys/health"
{
capabilities = ["read", "sudo"]
}
EOF
vault policy write vaultadmin vaultadmin-policy.hcl
vault auth enable oidc
vault login -method=oidc port=8400 role=dcc
#Complete the login via your OIDC provider. Launching browser to:
#
# https://myco.auth0.com/authorize?redirect_uri=http%3A%2F%2Flocalhost%3A8400%2Foidc%2Fcallback&client_id=r3qXc2bix9eF...
vault write auth/oidc/config \
oidc_discovery_url="https://myco.auth0.com/" \
oidc_client_id="m5i8bj3iofytj" \
oidc_client_secret="f4ubv72nfiu23hnsj" \
default_role="dcc"
# Create a named role:
vault write auth/oidc/role/dcc \
bound_subject="r3qX9DljwFIWhsiqwFiu38209F10atW6@clients" \
bound_audiences="https://vault.plugin.auth.jwt.test" \
user_claim="https://vault/user" \
groups_claim="https://vault/groups" \
policies=vaultadmin \
ttl=1h