Networking
EventStoreDB provides two interfaces:
- HTTP(S) for gRPC communication and REST APIs
- TCP for cluster replication (internal)
Nodes in the cluster replicate with each other using the TCP protocol, but use gRPC for discovering other cluster nodes.
Server nodes separate internal and external TCP communication explicitly, but use a single HTTP binding. Replication between the cluster nodes is considered internal, and all the TCP clients that connect to the database are external. EventStoreDB allows you to separate network configurations for internal and external communication. For example, you can use different network interfaces on the node. All the internal communication can go over the isolated private network, but access for external clients can be configured on another interface, which is connected to a more open network. You can set restrictions on those external interfaces to make your deployment more secure.
For gRPC and HTTP, there's no internal vs external separation of traffic.
HTTP configuration
HTTP is the primary protocol for EventStoreDB. It is used in gRPC communication and HTTP APIs (management, gossip and diagnostics). The HTTP endpoint always binds to the IP address configured in the NodeIp
setting (previously referred to as ExtIp
).
Format | Syntax |
---|---|
Command line | --node-ip |
YAML | NodeIp |
Environment variable | EVENTSTORE_NODE_IP |
When the NodeIp
setting is not provided, EventStoreDB will use the first available non-loopback address. You can also bind HTTP to all available interfaces using 0.0.0.0
as the setting value. If you do that, you'd need to configure the NodeHostAdvertiseAs
(previously ExtHostAdvertiseAs
) setting (read more here), since 0.0.0.0
is not a valid IP address to connect from the outside world.
Warning
Please note that the ExtIp
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodeIp
parameter instead.
The default HTTP port is 2113
. Depending on the security settings of the node, it either responds over plain HTTP or via HTTPS. There is no HSTS redirect, so if you try reaching a secure node via HTTP, you can get an empty response.
You can change the HTTP port using the NodePort
setting (previously HttpPort
setting) :
Format | Syntax |
---|---|
Command line | --node-port |
YAML | NodePort |
Environment variable | EVENTSTORE_NODE_PORT |
Default: 2113
Warning
Please note that the HttpPort
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodePort
parameter instead.
If your network setup requires any kind of IP address, DNS name and port translation for internal or external communication, you can use available address translation settings.
Keep-alive pings
The reliability of the connection between the client application and database is crucial for the stability of the solution. If the network is not stable or has some periodic issues, the client may drop the connection. Stability is essential for the stream subscriptions where a client is listening to database notifications. Having an existing connection open when an app resumes activity allows for the initial gRPC calls to be made quickly, without any delay caused by the reestablished connection.
EventStoreDB supports the built-in gRPC mechanism for keeping the connection alive. If the other side does not acknowledge the ping within a certain period, the connection will be closed. Note that pings are only necessary when there's no activity on the connection.
Keepalive pings are enabled by default, with the default interval set to 10 seconds. The default value is based on the gRPC proposal that suggests 10 seconds as the minimum. It's a compromise value to ensure that the connection is open and not making too many redundant network calls.
You can customise the following Keepalive settings:
KeepAliveInterval
After a duration of keepAliveInterval
(in milliseconds), if the server doesn't see any activity, it pings the client to see if the transport is still alive.
Format | Syntax |
---|---|
Command line | --keep-alive-interval |
YAML | KeepAliveInterval |
Environment variable | EVENTSTORE_KEEP_ALIVE_INTERVAL |
Default: 10000
(ms, 10 sec)
KeepAliveTimeout
After having pinged for keepalive check, the server waits for a duration of keepAliveTimeout
(in milliseconds). If the connection doesn't have any activity even after that, it gets closed.
Format | Syntax |
---|---|
Command line | --keep-alive-timeout |
YAML | KeepAliveTimeout |
Environment variable | EVENTSTORE_KEEP_ALIVE_TIMEOUT |
Default: 10000
(ms, 10 sec)
As a general rule, we do not recommend putting EventStoreDB behind a load balancer. However, if you are using it and want to benefit from the Keepalive feature, then you should make sure if the compatible settings are properly set. Some load balancers may also override the Keepalive settings. Most of them require setting the idle timeout larger/longer than the keepAliveTimeout
. We suggest checking the load balancer documentation before using Keepalive pings.
AtomPub
The AtomPub application protocol over HTTP is disabled by default since v20. We plan to deprecate the AtomPub support in the future versions, but we aim to provide a replacement before we finally deprecate AtomPub.
In Event Store Cloud, the AtomPub protocol is enabled. For self-hosted instances, use the configuration setting to enable AtomPub.
Format | Syntax |
---|---|
Command line | --enable-atom-pub-over-http |
YAML | EnableAtomPubOverHttp |
Environment variable | EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP |
Default: false
(AtomPub is disabled)
HTTP caching
Tips
This section is about caching static resources for HTTP API. It does not affect the server performance directly and cannot be used with gRPC clients.
Most of the URIs that EventStoreDB emits are immutable (including the UI and Atom Feeds).
An Atom feed has a URI that represents an event, e.g., /streams/foo/0
which represents 'event 0'. The data for event 0 never changes. If this stream is open to public reads, then the URI is set to be 'cachable' for long periods of time.
You can see a similar example in reading a feed. If a stream has 50 events in it, the feed page 20/forward/10
never changes, it will always be events 20-30. Internally, EventStoreDB controls serving the right URIs by using rel
links with feeds (for example prev
/next
).
This caching behavior is great for performance in a production environment and we recommended you use it, but in a developer environment it can become confusing.
For example, what happens if you started a database, wrote /streams/foo/0
and performed a GET
request? The GET
request is cachable and now in your cache. Since this is a development environment, you shut down EventStoreDB and delete the database. You then restart EventStoreDB and append a different event to /streams/foo/0
. You open your browser and inspect the /streams/foo/0
stream, and you see the event appended before you deleted the database.
To avoid this during development it's best to run EventStoreDB with the --disable-http-caching
command line option. This disables all caching and solves the issue.
The option can be set as follows:
Format | Syntax |
---|---|
Command line | --disable-http-caching |
YAML | DisableHttpCaching |
Environment variable | EVENTSTORE_DISABLE_HTTP_CACHING |
Default: false
, so the HTTP caching is enabled by default.
Kestrel Settings
It's generally not expected that you'll need to update the Kestrel configuration that EventStoreDB has set by default, but it's good to know that you can update the following settings if needed.
Kestrel uses the kestrelsettings.json
configuration file. This file should be located in the default configuration directory.
MaxConcurrentConnections
Sets the maximum number of open connections. See the docs here.
This is configured with Kestrel.Limits.MaxConcurrentConnections
in the settings file.
MaxConcurrentUpgradedConnections
Sets the maximum number of open, upgraded connections. An upgraded connection is one that has been switched from HTTP to another protocol, such as WebSockets. See the docs here.
This is configured with Kestrel.Limits.MaxConcurrentUpgradedConnections
in the settings file.
Http2 InitialConnectionWindowSize
Sets how much request body data the server is willing to receive and buffer at a time aggregated across all requests (streams) per connection. Note requests are also limited by KestrelInitialStreamWindowSize
The value must be greater than or equal to 65,535 and less than 2^31. See the docs here.
This is configured with Kestrel.Limits.Http2.InitialConnectionWindowSize
in the settings file.
Http2 InitialStreamWindowSize
Sets how much request body data the server is willing to receive and buffer at a time per stream. Note connections are also limited by KestrelInitialConnectionWindowSize
Value must be greater than or equal to 65,535 and less than 2^31. See the docs here.
This is configured with Kestrel.Limits.Http2.InitialStreamWindowSize
in the settings file.
Replication protocol
Replication between cluster nodes uses a proprietary TCP-based protocol. Options for configuring the internal replication protocol are described below.
Interface and port
Internal TCP binds to the IP address specified in the ReplicationIp
setting (previously IntIp
). It must be configured if you run a multi-node cluster.
By default, EventStoreDB binds its internal networking on the loopback interface only (127.0.0.1
). You can change this behaviour and tell EventStoreDB to listen on a specific internal IP address. To do that set the ReplicationIp
to 0.0.0.0
or the IP address of the network interface.
Format | Syntax |
---|---|
Command line | --replication-ip |
YAML | ReplicationIp |
Environment variable | EVENTSTORE_REPLICATION_IP |
Default: 127.0.0.1
(loopback).
If you keep this setting to its default value, cluster nodes won't be able to talk to each other.
Warning
Please note that the IntIp
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationIp
parameter instead.
By default, EventStoreDB uses port 1112
for internal TCP. You can change this by specifying the ReplicationPort
setting (previously IntTcpPort
setting).
Format | Syntax |
---|---|
Command line | --replication-port |
YAML | ReplicationPort |
Environment variable | EVENTSTORE_REPLICATION_PORT |
Default: 1112
Warning
Please note that the IntTcpPort
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationPort
parameter instead.
Security
When the node is secured (by default), all the TCP traffic will use TLS. You can disable TLS for TCP internally and externally using the settings described below.
Format | Syntax |
---|---|
Command line | --disable-internal-tcp-tls |
YAML | DisableInternalTcpTls |
Environment variable | EVENTSTORE_DISABLE_INTERNAL_TCP_TLS |
Default: false
If your network setup requires any kind of IP address, DNS name and port translation for internal communication, you can use available address translation settings.
Network address translation
Due to NAT (network address translation), or other reasons a node may not be bound to the address it is reachable from other nodes. For example, the machine has an IP address of 192.168.1.13
, but the node is visible to other nodes as 10.114.12.112
.
Options described below allow you to tell the node that even though it is bound to a given address it should not gossip that address. When returning links over HTTP, EventStoreDB will also use the specified addresses instead of physical addresses, so the clients that use HTTP can follow those links.
Another case when you might want to specify the advertised address although there's no address translation involved. When you configure EventStoreDB to bind to 0.0.0.0
, it will use the first non-loopback address for gossip. It might or might not be the address you want it to use. Whilst the best way to avoid such a situation is to configure the binding properly using the NodeIp
and ReplicationIp
settings, you can also use address translation setting with the correct IP address or DNS name.
Also, even if you specified the NodeIp
and ReplicationIp
settings in the configuration, you might still want to override the advertised address if you want to use hostnames and not IP addresses. That might be needed when running a secure cluster with certificates that only contain DNS names of the nodes.
The only place where these settings make any effect is the gossip endpoint response.
HTTP translations
By default, a cluster node will advertise itself using NodeIp
and NodePort
. You can override the advertised HTTP port using the NodePortAdvertiseAs
setting (previously HttpPortAdvertiseAs
setting).
Format | Syntax |
---|---|
Command line | --node-port-advertise-as |
YAML | NodePortAdvertiseAs |
Environment variable | EVENTSTORE_NODE_PORT_ADVERTISE_AS |
Warning
Please note that the HttpPortAdvertiseAs
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodePortAdvertiseAs
parameter instead.
If you want the node to advertise itself using the hostname rather than its IP address, use the NodeHostAdvertiseAs
setting (previously ExtHostAdvertiseAs
setting).
Format | Syntax |
---|---|
Command line | --node-host-advertise-as |
YAML | NodeHostAdvertiseAs |
Environment variable | EVENTSTORE_NODE_HOST_ADVERTISE_AS |
Warning
Please note that the ExtHostAdvertiseAs
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the NodeHostAdvertiseAs
parameter instead.
TCP translations
TCP ports used for replication can be advertised using custom values:
Format | Syntax |
---|---|
Command line | --replication-tcp-port-advertise-as |
YAML | ReplicationTcpPortAdvertiseAs |
Environment variable | EVENTSTORE_REPLICATION_TCP_PORT_ADVERTISE_AS |
Warning
Please note that the IntTcpPortAdvertiseAs
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationTcpPortAdvertiseAs
and NodeTcpPortAdvertiseAs
parameters instead, respectively.
If you want to change how the node TCP address is advertised internally, use the ReplicationHostAdvertiseAs
setting (previously IntHostAdvertiseAs
setting). You can use an IP address or a hostname.
Format | Syntax |
---|---|
Command line | --replication-host-advertise-as |
YAML | ReplicationHostAdvertiseAs |
Environment variable | EVENTSTORE_REPLICATION_HOST_ADVERTISE_AS |
Warning
Please note that the IntHostAdvertiseAs
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationHostAdvertiseAs
parameter instead.
Advertise to clients
In some cases, the cluster needs to advertise itself to clients using a completely different set of addresses and ports. Usually, you need to do it because addresses and ports configured for the HTTP protocol are not available as-is to the outside world. One of the examples is running a cluster in Docker Compose. In such environment, HTTP uses internal hostnames in the Docker network, which isn't accessible on the host. So, in order to connect to the cluster from the host machine, you need to use localhost
and translated HTTP ports to reach the cluster nodes.
To configure how the cluster nodes advertise to clients, use the Advertise<*>ToClient
settings listed below.
Specify the advertised hostname or IP address:
Format | Syntax |
---|---|
Command line | --advertise-host-to-client-as |
YAML | AdvertiseHostToClientAs |
Environment variable | EVENTSTORE_ADVERTISE_HOST_TO_CLIENT_AS |
Specify the advertised HTTP(S) port (previously AdvertiseHttpPortToClientAs
setting):
Format | Syntax |
---|---|
Command line | --advertise-node-port-to-client-as |
YAML | AdvertiseNodePortToClientAs |
Environment variable | EVENTSTORE_ADVERTISE_NODE_PORT_TO_CLIENT_AS |
Warning
Please note that the AdvertiseHttpPortToClientAs
parameter has been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the AdvertiseNodePortToClientAs
parameter instead.
Heartbeat timeouts
EventStoreDB uses heartbeats over all TCP connections to discover dead clients and nodes. Heartbeat timeouts should not be too short, as short timeouts will produce false positives. At the same time, setting too long timeouts will prevent discovering dead nodes and clients in time.
Each heartbeat has two points of configuration. The first is the interval; this represents how often the system should consider a heartbeat. EventStoreDB doesn't send a heartbeat for every interval, but only if it has not heard from a node within the configured interval. In a busy cluster, you may never see any heartbeats.
The second point of configuration is the timeout. This determines how long EventStoreDB server waits for a client or node to respond to a heartbeat request.
Different environments need different values for these settings. The defaults are likely fine on a LAN. If you experience frequent elections in your environment, you can try to increase both interval and timeout, for example:
- An interval of 5000ms.
- A timeout of 1000ms.
Tips
If in doubt, choose higher numbers. This adds a small period of time to discover a dead client or node and is better than the alternative, which is false positives.
Replication/Internal TCP heartbeat (between cluster nodes):
Format | Syntax |
---|---|
Command line | --replication-heartbeat-interval |
YAML | ReplicationHeartbeatInterval |
Environment variable | EVENTSTORE_REPLICATION_HEARTBEAT_INTERVAL |
Default: 700
(ms)
Format | Syntax |
---|---|
Command line | --replication-heartbeat-timeout |
YAML | ReplicationHeartbeatTimeout |
Environment variable | EVENTSTORE_REPLICATION_HEARTBEAT_TIMEOUT |
Default: 700
(ms)
Warning
Please note that the IntTcpHeartbeatInterval
and IntTcpHeartbeatTimeout
parameters have been deprecated as of version 23.10.0 and will be removed in future versions. It is recommended to use the ReplicationHeartbeatInterval
and ReplicationHeartbeatTimeout
parameters instead, respectively.
gRPC heartbeats
For the gRPC heartbeats, EventStoreDB and its gRPC clients use the protocol feature called Keepalive ping. Read more about it on the HTTP configuration page.
Exposing endpoints
If you need to disable some HTTP endpoints on the external HTTP interface, you can change some settings below. It is possible to disable the Admin UI, stats and gossip port to be exposed externally.
You can disable the Admin UI on external HTTP by setting AdminOnExt
setting to false
.
Format | Syntax |
---|---|
Command line | --admin-on-ext |
YAML | AdminOnExt |
Environment variable | EVENTSTORE_ADMIN_ON_EXT |
Default: true
, Admin UI is enabled on the external HTTP.
Exposing the stats
endpoint externally is required for the Admin UI and can also be useful if you collect stats for an external monitoring tool.
Format | Syntax |
---|---|
Command line | --stats-on-ext |
YAML | StatsOnExt |
Environment variable | EVENTSTORE_STATS_ON_EXT |
Default: true
, stats endpoint is enabled on the external HTTP.
You can also disable the gossip protocol in the external HTTP interface. If you do that, ensure that the internal interface is properly configured. Also, if you use gossip with DNS, ensure that the gossip port is set to the internal HTTP port.
Format | Syntax |
---|---|
Command line | --gossip-on-ext |
YAML | GossipOnExt |
Environment variable | EVENTSTORE_GOSSIP_ON_EXT |
Default: true
, gossip is enabled on the external HTTP.
External TCP
The TCP client protocol plugin enables client applications based on the external TCP API to run without any changes. This provides developers a mechanism to migrate to gRPC clients. This bridge solution enables development teams to plan for the End Of Life (EOL) of the external TCP API. Past the EOL, the external TCP API will no longer be supported.
Enabling the plugin
Refer to the general plugins configuration guide to see how to configure plugins with JSON files and environment variables.
You require a license key with the tcp
entitlement to use this plugin.
To enable the TCP API plugin, save the following configuration in a JSON file, for example <esdb-installation-directory>/config/tcp-plugin-config.json
, in the EventStoreDB installation directory for a node:
{
"EventStore": {
"TcpPlugin": {
"EnableExternalTcp": true
}
}
}
Once the plugin is enabled, the server will log a message similar to the one below:
[11212, 1,18:44:34.070,INF] "TcpApi" "24.6.0.0" plugin enabled.
Other parameters
The following options can be set in the json configuration:
NodeHeartbeatTimeout
- defaults to: 1000NodeHeartbeatInterval
- defaults to: 2000NodeTcpPort
- defaults to: 1113NodeTcpPortAdvertiseAs
- defaults to: 1113
The above default values can be overridden, for example:
{
"EventStore": {
"TcpPlugin": {
"EnableExternalTcp": true,
"NodeTcpPort": 8113,
"NodeTcpPortAdvertiseAs": 8113,
"NodeHeartbeatInterval": 2000,
"NodeHeartbeatTimeout": 1000
}
}
}
Troubleshooting
Plugin doesn't load
The plugin has to be located in a subdirectory of the server's plugins directory. To check this:
- Go to the installation directory of a node, the directory containing the EventStoreDb executable.
- In this directory, there should be a directory called
plugins
, create it if this is not the case. - The
plugins
directory should have a subdirectory for the TCP API plugin, for exampleEventStore.TcpPlugin
. Create it if it doesn't exist. - The binaries of the plugin should be located in that same subdirectory.
You can verify which plugins have been found and loaded by searching for log entries similar to the following:
[11212, 1,18:44:30.420,INF] Plugins path: "C:\\EventStore\\plugins"
[11212, 1,18:44:30.420,INF] Adding: "C:\\EventStore\\plugins" to the plugin catalog.
[11212, 1,18:44:30.422,INF] Adding: "C:\\EventStore\\plugins\\EventStore.Licensing" to the plugin catalog.
[11212, 1,18:44:30.432,INF] Adding: "C:\\EventStore\\plugins\\EventStore.POC.ConnectedSubsystemsPlugin" to the plugin catalog.
[11212, 1,18:44:30.433,INF] Adding: "C:\\EventStore\\plugins\\EventStore.POC.ConnectorsPlugin" to the plugin catalog.
[11212, 1,18:44:30.436,INF] Adding: "C:\\EventStore\\plugins\\EventStore.TcpPlugin" to the plugin catalog.
[11212, 1,18:44:30.479,INF] Loaded SubsystemsPlugin plugin: "licensing" "24.10.0.0".
[11212, 1,18:44:30.483,INF] Loaded SubsystemsPlugin plugin: "connected" "0.0.5".
[11212, 1,18:44:30.496,INF] Loaded ConnectedSubsystemsPlugin plugin: "connectors" "0.0.5".
[11212, 1,18:44:30.504,INF] Loaded SubsystemsPlugin plugin: "tcp-api" "24.10.0.0".
Plugin doesn't start
The plugin has to be configured to be enabled. If you see the following log it means the plugin was found but not started:
[ 5104, 1,19:03:13.807,INF] "TcpApi" "24.6.0.0" plugin disabled. "Set 'EventStore:TcpPlugin:EnableExternalTcp' to 'true' to enable"
When the plugin starts, you should see a log similar to the following:
[11212, 1,18:44:34.070,INF] "TcpApi" "24.6.0.0" plugin enabled.