Networking

EventStoreDB provides two interfaces:

• HTTP(S) for gRPC communication and REST APIs
• TCP for cluster replication (internal)

Nodes in the cluster replicate with each other using the TCP protocol, but use gRPC for discovering other cluster nodes.

Server nodes separate internal and external TCP communication explicitly, but use a single HTTP binding. Replication between the cluster nodes is considered internal, and all the TCP clients that connect to the database are external. EventStoreDB allows you to separate network configurations for internal and external communication. For example, you can use different network interfaces on the node. All the internal communication can go over the isolated private network, but access for external clients can be configured on another interface, which is connected to a more open network. You can set restrictions on those external interfaces to make your deployment more secure.

For gRPC and HTTP, there's no internal vs external separation of traffic.

HTTP configuration

HTTP is the primary protocol for EventStoreDB. It is used in gRPC communication and HTTP APIs (management, gossip and diagnostics). The HTTP endpoint always binds to the IP address configured in the NodeIp setting (previously referred to as ExtIp).

Format	Syntax
Command line	--node-ip
YAML	NodeIp
Environment variable	EVENTSTORE_NODE_IP

When the NodeIp setting is not provided, EventStoreDB will use the first available non-loopback address. You can also bind HTTP to all available interfaces using 0.0.0.0 as the setting value. If you do that, you'd need to configure the NodeHostAdvertiseAs (previously ExtHostAdvertiseAs) setting (read more here), since 0.0.0.0 is not a valid IP address to connect from the outside world.

Warning

Please note that the ExtIp parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodeIp parameter instead.

The default HTTP port is 2113. Depending on the security settings of the node, it either responds over plain HTTP or via HTTPS. There is no HSTS redirect, so if you try reaching a secure node via HTTP, you can get an empty response.

You can change the HTTP port using the NodePort setting (previously HttpPort setting) :

Format	Syntax
Command line	--node-port
YAML	NodePort
Environment variable	EVENTSTORE_NODE_PORT

Default: 2113

Warning

Please note that the HttpPort parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodePort parameter instead.

If your network setup requires any kind of IP address, DNS name and port translation for internal or external communication, you can use available address translation settings.

Keep-alive pings

The reliability of the connection between the client application and database is crucial for the stability of the solution. If the network is not stable or has some periodic issues, the client may drop the connection. Stability is essential for the stream subscriptions where a client is listening to database notifications. Having an existing connection open when an app resumes activity allows for the initial gRPC calls to be made quickly, without any delay caused by the reestablished connection.

EventStoreDB supports the built-in gRPC mechanism for keeping the connection alive. If the other side does not acknowledge the ping within a certain period, the connection will be closed. Note that pings are only necessary when there's no activity on the connection.

Keepalive pings are enabled by default, with the default interval set to 10 seconds. The default value is based on the gRPC proposal that suggests 10 seconds as the minimum. It's a compromise value to ensure that the connection is open and not making too many redundant network calls.

You can customise the following Keepalive settings:

KeepAliveInterval

After a duration of keepAliveInterval (in milliseconds), if the server doesn't see any activity, it pings the client to see if the transport is still alive.

Format	Syntax
Command line	--keep-alive-interval
YAML	KeepAliveInterval
Environment variable	EVENTSTORE_KEEP_ALIVE_INTERVAL

Default: 10000 (ms, 10 sec)

KeepAliveTimeout

After having pinged for keepalive check, the server waits for a duration of keepAliveTimeout (in milliseconds). If the connection doesn't have any activity even after that, it gets closed.

Format	Syntax
Command line	--keep-alive-timeout
YAML	KeepAliveTimeout
Environment variable	EVENTSTORE_KEEP_ALIVE_TIMEOUT

Default: 10000 (ms, 10 sec)

As a general rule, we do not recommend putting EventStoreDB behind a load balancer. However, if you are using it and want to benefit from the Keepalive feature, then you should make sure if the compatible settings are properly set. Some load balancers may also override the Keepalive settings. Most of them require setting the idle timeout larger/longer than the keepAliveTimeout. We suggest checking the load balancer documentation before using Keepalive pings.

AtomPub

The AtomPub application protocol over HTTP is disabled by default since v20. We plan to deprecate the AtomPub support in the future versions, but we aim to provide a replacement before we finally deprecate AtomPub.

In Event Store Cloud, the AtomPub protocol is enabled. For self-hosted instances, use the configuration setting to enable AtomPub.

Format	Syntax
Command line	--enable-atom-pub-over-http
YAML	EnableAtomPubOverHttp
Environment variable	EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP

Default: false (AtomPub is disabled)

HTTP caching

Tips

This section is about caching static resources for HTTP API. It does not affect the server performance directly and cannot be used with gRPC clients.

Most of the URIs that EventStoreDB emits are immutable (including the UI and Atom Feeds).

An Atom feed has a URI that represents an event, e.g., /streams/foo/0 which represents 'event 0'. The data for event 0 never changes. If this stream is open to public reads, then the URI is set to be 'cachable' for long periods of time.

You can see a similar example in reading a feed. If a stream has 50 events in it, the feed page 20/forward/10 never changes, it will always be events 20-30. Internally, EventStoreDB controls serving the right URIs by using rel links with feeds (for example prev/next).

This caching behavior is great for performance in a production environment and we recommended you use it, but in a developer environment it can become confusing.

For example, what happens if you started a database, wrote /streams/foo/0 and performed a GET request? The GET request is cachable and now in your cache. Since this is a development environment, you shut down EventStoreDB and delete the database. You then restart EventStoreDB and append a different event to /streams/foo/0. You open your browser and inspect the /streams/foo/0 stream, and you see the event appended before you deleted the database.

To avoid this during development it's best to run EventStoreDB with the --disable-http-caching command line option. This disables all caching and solves the issue.

The option can be set as follows:

Format	Syntax
Command line	--disable-http-caching
YAML	DisableHttpCaching
Environment variable	EVENTSTORE_DISABLE_HTTP_CACHING

Default: false, so the HTTP caching is enabled by default.

Kestrel Settings

It's generally not expected that you'll need to update the Kestrel configuration that EventStoreDB has set by default, but it's good to know that you can update the following settings if needed.

Kestrel uses the kestrelsettings.json configuration file. This file should be located in the default configuration directory.

MaxConcurrentConnections

Sets the maximum number of open connections. See the docs here.

This is configured with Kestrel.Limits.MaxConcurrentConnections in the settings file.

MaxConcurrentUpgradedConnections

Sets the maximum number of open, upgraded connections. An upgraded connection is one that has been switched from HTTP to another protocol, such as WebSockets. See the docs here.

This is configured with Kestrel.Limits.MaxConcurrentUpgradedConnections in the settings file.

Http2 InitialConnectionWindowSize

Sets how much request body data the server is willing to receive and buffer at a time aggregated across all requests (streams) per connection. Note requests are also limited by KestrelInitialStreamWindowSize

The value must be greater than or equal to 65,535 and less than 2^31. See the docs here.

This is configured with Kestrel.Limits.Http2.InitialConnectionWindowSize in the settings file.

Http2 InitialStreamWindowSize

Sets how much request body data the server is willing to receive and buffer at a time per stream. Note connections are also limited by KestrelInitialConnectionWindowSize

Value must be greater than or equal to 65,535 and less than 2^31. See the docs here.

This is configured with Kestrel.Limits.Http2.InitialStreamWindowSize in the settings file.

Replication protocol

Replication between cluster nodes uses a proprietary TCP-based protocol. Options for configuring the internal replication protocol are described below.

Interface and port

Internal TCP binds to the IP address specified in the ReplicationIp setting (previously IntIp). It must be configured if you run a multi-node cluster.

By default, EventStoreDB binds its internal networking on the loopback interface only (127.0.0.1). You can change this behaviour and tell EventStoreDB to listen on a specific internal IP address. To do that set the ReplicationIp to 0.0.0.0 or the IP address of the network interface.

Format	Syntax
Command line	--replication-ip
YAML	ReplicationIp
Environment variable	EVENTSTORE_REPLICATION_IP

Default: 127.0.0.1 (loopback).

If you keep this setting to its default value, cluster nodes won't be able to talk to each other.

Warning

Please note that the IntIp parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationIp parameter instead.

By default, EventStoreDB uses port 1112 for internal TCP. You can change this by specifying the ReplicationPort setting (previously IntTcpPort setting).

Format	Syntax
Command line	--replication-port
YAML	ReplicationPort
Environment variable	EVENTSTORE_REPLICATION_PORT

Default: 1112

Warning

Please note that the IntTcpPort parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationPort parameter instead.

Security

When the node is secured (by default), all the TCP traffic will use TLS. You can disable TLS for TCP internally and externally using the settings described below.

Format	Syntax
Command line	--disable-internal-tcp-tls
YAML	DisableInternalTcpTls
Environment variable	EVENTSTORE_DISABLE_INTERNAL_TCP_TLS

Default: false

If your network setup requires any kind of IP address, DNS name and port translation for internal communication, you can use available address translation settings.

Network address translation

Due to NAT (network address translation), or other reasons a node may not be bound to the address it is reachable from other nodes. For example, the machine has an IP address of 192.168.1.13, but the node is visible to other nodes as 10.114.12.112.

Options described below allow you to tell the node that even though it is bound to a given address it should not gossip that address. When returning links over HTTP, EventStoreDB will also use the specified addresses instead of physical addresses, so the clients that use HTTP can follow those links.

Another case when you might want to specify the advertised address although there's no address translation involved. When you configure EventStoreDB to bind to 0.0.0.0, it will use the first non-loopback address for gossip. It might or might not be the address you want it to use. Whilst the best way to avoid such a situation is to configure the binding properly using the NodeIp and ReplicationIp settings, you can also use address translation setting with the correct IP address or DNS name.

Also, even if you specified the NodeIp and ReplicationIp settings in the configuration, you might still want to override the advertised address if you want to use hostnames and not IP addresses. That might be needed when running a secure cluster with certificates that only contain DNS names of the nodes.

The only place where these settings make any effect is the gossip endpoint response.

HTTP translations

By default, a cluster node will advertise itself using NodeIp and NodePort. You can override the advertised HTTP port using the NodePortAdvertiseAs setting (previously HttpPortAdvertiseAs setting).

Format	Syntax
Command line	--node-port-advertise-as
YAML	NodePortAdvertiseAs
Environment variable	EVENTSTORE_NODE_PORT_ADVERTISE_AS

Warning

Please note that the HttpPortAdvertiseAs parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodePortAdvertiseAs parameter instead.

If you want the node to advertise itself using the hostname rather than its IP address, use the NodeHostAdvertiseAs setting (previously ExtHostAdvertiseAs setting).

Format	Syntax
Command line	--node-host-advertise-as
YAML	NodeHostAdvertiseAs
Environment variable	EVENTSTORE_NODE_HOST_ADVERTISE_AS

Warning

Please note that the ExtHostAdvertiseAs parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodeHostAdvertiseAs parameter instead.

TCP translations

TCP ports used for replication can be advertised using custom values:

Format	Syntax
Command line	--replication-tcp-port-advertise-as
YAML	ReplicationTcpPortAdvertiseAs
Environment variable	EVENTSTORE_REPLICATION_TCP_PORT_ADVERTISE_AS

Warning

Please note that the IntTcpPortAdvertiseAs parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationTcpPortAdvertiseAs and NodeTcpPortAdvertiseAs parameters instead, respectively.

If you want to change how the node TCP address is advertised internally, use the ReplicationHostAdvertiseAs setting (previously IntHostAdvertiseAs setting). You can use an IP address or a hostname.

Format	Syntax
Command line	--replication-host-advertise-as
YAML	ReplicationHostAdvertiseAs
Environment variable	EVENTSTORE_REPLICATION_HOST_ADVERTISE_AS

Warning

Please note that the IntHostAdvertiseAs parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationHostAdvertiseAs parameter instead.

Advertise to clients

In some cases, the cluster needs to advertise itself to clients using a completely different set of addresses and ports. Usually, you need to do it because addresses and ports configured for the HTTP protocol are not available as-is to the outside world. One of the examples is running a cluster in Docker Compose. In such environment, HTTP uses internal hostnames in the Docker network, which isn't accessible on the host. So, in order to connect to the cluster from the host machine, you need to use localhost and translated HTTP ports to reach the cluster nodes.

To configure how the cluster nodes advertise to clients, use the Advertise<*>ToClient settings listed below.

Specify the advertised hostname or IP address:

Format	Syntax
Command line	--advertise-host-to-client-as
YAML	AdvertiseHostToClientAs
Environment variable	EVENTSTORE_ADVERTISE_HOST_TO_CLIENT_AS

Specify the advertised HTTP(S) port (previously AdvertiseHttpPortToClientAs setting):

Format	Syntax
Command line	--advertise-node-port-to-client-as
YAML	AdvertiseNodePortToClientAs
Environment variable	EVENTSTORE_ADVERTISE_NODE_PORT_TO_CLIENT_AS

Warning

Please note that the AdvertiseHttpPortToClientAs parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the AdvertiseNodePortToClientAs parameter instead.

Heartbeat timeouts

EventStoreDB uses heartbeats over all TCP connections to discover dead clients and nodes. Heartbeat timeouts should not be too short, as short timeouts will produce false positives. At the same time, setting too long timeouts will prevent discovering dead nodes and clients in time.

Each heartbeat has two points of configuration. The first is the interval; this represents how often the system should consider a heartbeat. EventStoreDB doesn't send a heartbeat for every interval, but only if it has not heard from a node within the configured interval. In a busy cluster, you may never see any heartbeats.

The second point of configuration is the timeout. This determines how long EventStoreDB server waits for a client or node to respond to a heartbeat request.

Different environments need different values for these settings. The defaults are likely fine on a LAN. If you experience frequent elections in your environment, you can try to increase both interval and timeout, for example:

• An interval of 5000ms.
• A timeout of 1000ms.

Tips

If in doubt, choose higher numbers. This adds a small period of time to discover a dead client or node and is better than the alternative, which is false positives.

Replication/Internal TCP heartbeat (between cluster nodes):

Format	Syntax
Command line	--replication-heartbeat-interval
YAML	ReplicationHeartbeatInterval
Environment variable	EVENTSTORE_REPLICATION_HEARTBEAT_INTERVAL

Default: 700 (ms)

Format	Syntax
Command line	--replication-heartbeat-timeout
YAML	ReplicationHeartbeatTimeout
Environment variable	EVENTSTORE_REPLICATION_HEARTBEAT_TIMEOUT

Default: 700 (ms)

Warning

Please note that the IntTcpHeartbeatInterval and IntTcpHeartbeatTimeout parameters have been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationHeartbeatInterval and ReplicationHeartbeatTimeout parameters instead, respectively.

gRPC heartbeats

For the gRPC heartbeats, EventStoreDB and its gRPC clients use the protocol feature called Keepalive ping. Read more about it on the HTTP configuration page.

Exposing endpoints

If you need to disable some HTTP endpoints on the external HTTP interface, you can change some settings below. It is possible to disable the Admin UI, stats and gossip port to be exposed externally.

You can disable the Admin UI on external HTTP by setting AdminOnExt setting to false.

Format	Syntax
Command line	--admin-on-ext
YAML	AdminOnExt
Environment variable	EVENTSTORE_ADMIN_ON_EXT

Default: true, Admin UI is enabled on the external HTTP.

Exposing the stats endpoint externally is required for the Admin UI and can also be useful if you collect stats for an external monitoring tool.

Format	Syntax
Command line	--stats-on-ext
YAML	StatsOnExt
Environment variable	EVENTSTORE_STATS_ON_EXT

Default: true, stats endpoint is enabled on the external HTTP.

You can also disable the gossip protocol in the external HTTP interface. If you do that, ensure that the internal interface is properly configured. Also, if you use gossip with DNS, ensure that the gossip port is set to the internal HTTP port.

Format	Syntax
Command line	--gossip-on-ext
YAML	GossipOnExt
Environment variable	EVENTSTORE_GOSSIP_ON_EXT

Default: true, gossip is enabled on the external HTTP.

External TCP

Commercial

The TCP client protocol plugin enables client applications based on the external TCP API to run without any changes. This provides developers a mechanism to migrate to gRPC clients. This bridge solution enables development teams to plan for the End Of Life (EOL) of the external TCP API. Past the EOL, the external TCP API will no longer be supported.

Enabling the plugin

Refer to the general plugins configuration guide to see how to configure plugins with JSON files and environment variables.

To enable the TCP API plugin, save the following configuration in a JSON file, for example <esdb-installation-directory>/config/tcp-plugin-config.json, in the EventStoreDB installation directory for a node:

{
  "EventStore": {
    "TcpPlugin": {
      "EnableExternalTcp": true
    }
  }
}

Once the plugin is enabled, the server will log a message similar to the one below:

[11212, 1,18:44:34.070,INF] "TcpApi" "24.6.0.0" plugin enabled.

Other parameters

The following options can be set in the json configuration:

1. NodeHeartbeatTimeout - defaults to: 1000
2. NodeHeartbeatInterval - defaults to: 2000
3. NodeTcpPort - defaults to: 1113
4. NodeTcpPortAdvertiseAs - defaults to: 1113

The above default values can be overridden, for example:

{
  "EventStore": {
    "TcpPlugin": {
      "EnableExternalTcp": true,
      "NodeTcpPort": 8113,
      "NodeTcpPortAdvertiseAs": 8113,
      "NodeHeartbeatInterval": 2000,
      "NodeHeartbeatTimeout": 1000
    }
  }
}

Troubleshooting

Plugin doesn't load

The plugin has to be located in a subdirectory of the server's plugins directory. To check this:

1. Go to the installation directory of a node, the directory containing the EventStoreDb executable.
2. In this directory, there should be a directory called plugins, create it if this is not the case.
3. The plugins directory should have a subdirectory for the TCP API plugin, for example EventStore.TcpPlugin. Create it if it doesn't exist.
4. The binaries of the plugin should be located in that same subdirectory.

You can verify which plugins have been found and loaded by searching for log entries similar to the following:

[11212, 1,18:44:30.420,INF] Plugins path: "C:\\EventStore\\plugins"
[11212, 1,18:44:30.420,INF] Adding: "C:\\EventStore\\plugins" to the plugin catalog.
[11212, 1,18:44:30.422,INF] Adding: "C:\\EventStore\\plugins\\EventStore.Licensing" to the plugin catalog.
[11212, 1,18:44:30.432,INF] Adding: "C:\\EventStore\\plugins\\EventStore.POC.ConnectedSubsystemsPlugin" to the plugin catalog.
[11212, 1,18:44:30.433,INF] Adding: "C:\\EventStore\\plugins\\EventStore.POC.ConnectorsPlugin" to the plugin catalog.
[11212, 1,18:44:30.436,INF] Adding: "C:\\EventStore\\plugins\\EventStore.TcpPlugin" to the plugin catalog.
[11212, 1,18:44:30.479,INF] Loaded SubsystemsPlugin plugin: "licensing" "24.10.0.0".
[11212, 1,18:44:30.483,INF] Loaded SubsystemsPlugin plugin: "connected" "0.0.5".
[11212, 1,18:44:30.496,INF] Loaded ConnectedSubsystemsPlugin plugin: "connectors" "0.0.5".
[11212, 1,18:44:30.504,INF] Loaded SubsystemsPlugin plugin: "tcp-api" "24.10.0.0".

Plugin doesn't start

The plugin has to be configured to be enabled. If you see the following log it means the plugin was found but not started:

[ 5104, 1,19:03:13.807,INF] "TcpApi" "24.6.0.0" plugin disabled. "Set 'EventStore:TcpPlugin:EnableExternalTcp' to 'true' to enable"

When the plugin starts, you should see a log similar to the following:

[11212, 1,18:44:34.070,INF] "TcpApi" "24.6.0.0" plugin enabled.