Kurrent Capacitor Shared Memory for Humans & Coding Agents Shared Memory for Humans & Coding Agents Get It Now >
Kurrent logo
Kurrent logo
Login Sign up
Back to all posts

DNS discovery and Certificate Management for EventStoreDB versions 21.10.5 and beyond

Yves Lorphelin avatar Yves Lorphelin
•
DNS discovery and Certificate Management for EventStoreDB versions 21.10.5 and beyond

We've been working to simplify wildcard certificate management, so here is some new guidance on certificate management from EventStoreDB version 21.10.5 and beyond.

These instructions explain how to configure EventStoreDB with a wildcard certificate (from a public or private Certificate Authority (CA)) and cluster DNS discovery, how to rotate the certificates, and how to replace a node with this setup.

Be aware that these instructions will only work as of version 21.10.5.

There are also two assumptions made about your setup:

  • Each node has a single IP address used for node-to-node and client-to-cluster communication
  • Clients have the root Certificate Authority certificate installed
ES Diagram

Note: see this post for more information .

Initial Setup

The initial setup of EventStoreDB will be a 3-node cluster. All nodes and the cluster DNS will use *.project.example.com as a wildcard certificate example.

Each node has 1 IP address used both for internal (node to node) communication and client communication.

DNS Records

The following table lists the necessary DNS records

**Record **

Purpose

node1.project.example.com. 300 IN A 127.0.20.1

Node 1

node2.project.example.com. 300 IN A 127.0.20.2

Node 2

node3.project.example.com. 300 IN A 127.0.20.3

Node 3

cluster.project.example.com. 300 IN A 127.0.20.1

cluster.project.example.com. 300 IN A 127.0.20.2

cluster.project.example.com. 300 IN A 127.0.20.3

Cluster DNS name

Note that the records TTL are low (minutes, not hours) in order to allow more rapid propagation in case of a node being replaced.

Certificate

Public CA:

The certificate private key must be in RSA format. The certificate needs to have the following attributes:

  • Subject Name - Common Name (CN): *.project.example.com
  • Subject Alternative Name (SAN): *.project.example.com

Assuming that for each node the public key is in a file called node.crt and the private key in a file called node.key, copy the certificates to their respective nodes.

The relevant configuration for the certificates for all nodes is:

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/ssl/certs
CertificateReservedNodeCommonName: "*.project.example.com"

Note: The TrustedRootCertificatesPathis the standard location of trusted root certificates in Linux-based systems and is used in this case since the root certificate is a public Certificate Authority (CA).

Private CA:

This assumes you are using The Event Store gencert-cli .

Generate the root CA

./es-gencert-cli create-ca

Generate each node certificates

./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n1
./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n2
./es-gencert-cli.exe create-node --dns-names *.project.example.com -out ./certs/n3

Copy each node.crt and node.key files to /etc/eventstore/certs on each respective node. Copy the public key of the root certificate authority, the ca.crt file, to /etc/eventstore/certs/ca on each node.

The relevant configuration for the certificates for each node is:

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

Note: The TrustedRootCertificatesPath in this case is not the standard location of trusted root certificates in Linux-based systems.

Node Configuration

Each node configuration is then, in the case of private CA: (for public CA issued certificates, replace the certificate configuration section)

Node1

---
# Paths
Db: / var/lib/eventstore
Index: / var/lib/eventstore/index
Log: / var/log/eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
IntIp: 127.0.20.1
ExtIp: 127.0.20.1
AdvertiseHostToClientAs: node1.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: true
ClusterDns: cluster.project.example.com

# Projections configuration
RunProjections: All

Node2

---
# Paths
Db: / var/lib/eventstore
Index: / var/lib/eventstore/index
Log: / var/log/eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
IntIp: 127.0.20.2
ExtIp: 127.0.20.2
AdvertiseHostToClientAs: node2.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: true
ClusterDns: cluster.project.example.com

# Projections configuration
RunProjections: All

Node3

---
# Paths
Db: / var/lib/eventstore
Index: / var/lib/eventstore/index
Log: / var/log/eventstore

# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.key
TrustedRootCertificatesPath: /etc/eventstore/certs/ca

# Network configuration
IntIp: 127.0.20.3
ExtIp: 127.0.20.3
AdvertiseHostToClientAs: node3.project.example.com
HttpPort: 2113
IntTcpPort: 1112
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster gossip
ClusterSize: 3
DiscoverViaDns: true
ClusterDns: cluster.project.example.com

# Projections configuration
RunProjections: All

Client connection string

Note that when using certificates signed by a private CA, the root CA public key must be trusted by the client. Typically, it is placed in the trusted store of the user under which the application is running.

gRPC

esdb+discover://cluster.project.example.com:2113

TCP

ConnectTo=discover://cluster.project.example.com:2113;UseSslConnection=true

Renewing a node certificate

In both cases of certificates issued by a public and private CA the procedure is the same.

  • Issue the new certificates
  • Copy the public key to the same locations as the CertificateFile configuration setting e.g.: /etc/eventstore/certs/node.new.crt
  • Copy the private key to the same location as the CertificatePrivateKeyFile configuration setting e.g.: /etc/eventstore/certs/node.new.key
  • Change the configuration file:
# Certificates configuration
CertificateFile: /etc/eventstore/certs/node.new.crt
CertificatePrivateKeyFile: /etc/eventstore/certs/node.new.key
  • Force a reload of the certificate configuration

Note: the -k forces curl to skip verification of the certificate revocation

curl -k -X POST -u [user]:[password] --basic https://node1.project.examples.com:2113/admin/reloadconfig -d '' -v

Once the configuration has been reloaded, the server will contain log entries like:

[14476,74,19:15:23.591,INF] Reloading the node's configuration since a request has been received on /admin/reloadconfig.
[14476,23,19:15:23.593,INF] Loading the node's certificate(s) from file: "/etc/eventstore/certs/node.new.crt"
[14476,23,19:15:23.628,INF] Loading the node's certificate. Subject: "CN=eventstoredb-node", Previous thumbprint: "DA090ED6844C15F11031A1CE8A3841ECAEA5AA6C", New thumbprint: "DA090ED6844C15F11031A1CE8A3841ECAEA5AA6C"
[14476,23,19:15:23.628,INF] Loading trusted root certificates.
[14476,23,19:15:23.631,INF] Loading trusted root certificate file: "/etc/eventstore/certs/ca/ca.crt"
[14476,23,19:15:23.631,INF] Loading trusted root certificate. Subject: "CN=EventStoreDB CA 8a9b87f0cedb5230569473bc8bcca45c, O=Event Store Ltd, C=UK", Thumbprint: "24BEB8402D30DD7B62FC3DAF4F877D8911FD8CED"
[14476,23,19:15:23.635,INF] Certificate chain verification successful.
[14476,23,19:15:23.635,INF] All certificates successfully loaded.
[14476,23,19:15:23.635,INF] The node's configuration was successfully reloaded

Replacing a node

In order to replace a node in this configuration, follow these steps:

cluster.project.example.com. 300 IN A 127.0.20.1
cluster.project.example.com. 300 IN A 127.0.20.2
cluster.project.example.com. 300 IN A 127.0.20.4

At this point, the old node will still show up as DEAD in the gossip endpoint and the UI for some time, depending on the DeadMemberRemovalPeriodSec configuration setting. The default setting is 30 minutes.

No changes are needed in the other nodes since they are using DiscoverViaDns and ClusterDns. The client applications do not need any changes as well if they use the cluster DNS discovery as described in the “client connection string” section.

Back to all posts