Watch ETCD change as Vault’s backend

Install Vault & Etcd locally

brew install etcd vault

➜  ~ etcd --version
etcd Version: 3.4.15
Git SHA: Not provided (use ./build instead of go build)
Go Version: go1.16
Go OS/Arch: darwin/amd64

Start Vault & Etcd

• Start ETCD

rm -fr rm default.etcd/

ETCDCTL_API=3 etcd Log:

lsof -i :2379
COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
etcd    29633 chenxi    9u  IPv4 0xf55da3372e132b95      0t0  TCP localhost:2379 (LISTEN)
etcd    29633 chenxi   16u  IPv4 0xf55da3372530d17d      0t0  TCP localhost:60987->localhost:2379 (ESTABLISHED)
etcd    29633 chenxi   17u  IPv4 0xf55da3372e25ab95      0t0  TCP localhost:2379->localhost:60987 (ESTABLISHED)

cat config.hcl

disable_mlock = true
ui            = true

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}


storage "etcd" {
  address  = "http://localhost:2379"
  etcd_api = "v3"
  path  = "vault/"
}
vault server -config vault-server.hcl

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_SKIP_VERIFY=true

vault status
vault operator init
vault operator unseal
vault secrets enable -version=2 kv

Create secret via Vault and Watch Etcd update

• Dump etcd as etcd_empty: $etcdctl get / --prefix --keys-only > etcd_empty

• Create:

$vault vault kv put kv/mytest1 vaule=t1
Key              Value
---              -----
created_time     2021-04-05T00:06:31.866362Z
deletion_time    n/a
destroyed        false
version          1

• Dump etcd as etcd_create:

  $etcdctl get / --prefix --keys-only > create_v1

• Compare:

$diff etcd_empty create_v1
30a31,32
> /vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/metadata/p0EkgERGrRtqJdqNyTQZ5lglckrte4t8ctdqvXIQb41YhDKIpaSb3JRbl6B6308MqcJ
>
36a39,40
> /vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/versions/dfd/ba0013b48ade4bd11c77467f07d79b6b9793b57f03c0b740db5418eaaa095
>

• Create v2:

$vault kv put kv/mytest1 vaule=t2
Key              Value
---              -----
created_time     2021-04-05T00:07:41.371068Z
deletion_time    n/a
destroyed        false
version          2

• Dump etcd data again:

  $etcdctl get / --prefix --keys-only > create_v2

• Compare:

$diff create_v1 create_v2
38a39,40
> /vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/versions/20b/00b788b6bcb45ade8f14e622af05f6b7b81d1da90f9893d5f6cbc3067ca23
>
>

• Watch the event when create:

$etcdctl watch --prefix /vault/logical -- printf "Path /vault/logical was changed.\n"

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/versions/752/e9b11a7dffc563cc18b8dc3a4a883cfc0d3ef3800407d0a70d7541c31bc69
���9;Gv�:�!ӟ0�3�l�0K�
                     �k"�0s���X����9�g0�`��P�]�pZ
Path /vault/logical was changed.
PUT
/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/metadata/p0EkgERGrRtqJdqNyTQZ5lglckrte4t8ctdqvXIQb41YhDKIpaSb3JRbl6B6308MqcJ
5 �@֍�0��~a�%9;v��쩑̔ٻ��s%ҡ{�kO#(S�$�'�#4�#�@������)�֨�����
۾s�QH͒W�                                                  ����g�^/���ն9m7T��Bw�T��0(V�.u�yF
Path /vault/logical was changed.

Log: Dumped ETCD data:

$etcdctl get / --prefix --keys-only > create_v2

/vault/core/audit

/vault/core/auth

/vault/core/cluster/feature-flags

/vault/core/cluster/local/info

/vault/core/hsm/barrier-unseal-keys

/vault/core/keyring

/vault/core/local-audit

/vault/core/local-auth

/vault/core/local-mounts

/vault/core/master

/vault/core/mounts

/vault/core/seal-config

/vault/core/shamir-kek

/vault/core/wrapping/jwtkey

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/archive/metadata

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/metadata/p0EkgERGrRtqJdqNyTQZ5lglckrte4t8ctdqvXIQb41YhDKIpaSb3JRbl6B6308MqcJ

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/policy/metadata

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/salt

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/upgrading

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/versions/20b/00b788b6bcb45ade8f14e622af05f6b7b81d1da90f9893d5f6cbc3067ca23

/vault/logical/095c5343-9469-a79d-c27d-6aa3328d65fa/9995d978-eee3-2a2b-6b5b-be038d1edc29/versions/dfd/ba0013b48ade4bd11c77467f07d79b6b9793b57f03c0b740db5418eaaa095

/vault/logical/cd7ae0db-b74a-8056-635c-4ec0e35cbb90/casesensitivity

/vault/sys/counters/requests/2021/04

/vault/sys/policy/control-group

/vault/sys/policy/default

/vault/sys/policy/response-wrapping

/vault/sys/token/accessor/3bd8eaedf31ae56c1f03acc2a29cc30e8215220d

/vault/sys/token/id/h107a9088af318ae8de021be76fb3f75f96c351ea28aaf0a36505748a1c0bdb2c

/vault/sys/token/salt