Event streams
Event streams
EventStoreDB is purpose-built for event storage. Unlike traditional state-based databases, which retain only the most recent entity state, EventStoreDB allows you to store each state alteration as an independent event.
These events are logically organized into streams, typically only one stream per entity.
Metadata and reserved names
Stream metadata
In EventStoreDB, every stream and event is accompanied by metadata, distinguished by the $$
prefix. For example, the stream metadata for a stream named foo
is $$foo
. You can modify specific metadata values and write your data to the metadata, which can be referenced in your code.
Reserved names
EventStoreDB uses a $
prefix for all internal data, such as $maxCount
in a stream's metadata. Do not use a $
prefix for event names, metadata keys, or stream names, except as detailed below.
The supported internal settings are:
Property name | Description |
---|---|
$maxAge | Sets a sliding window based on dates. When data reaches a certain age, it automatically disappears from the stream and becomes eligible for scavenging. This value is set as an integer representing the number of seconds (must be >= 1). |
$maxCount | Sets a sliding window based on the number of items in the stream. When data reaches a certain length, it automatically disappears from the stream and becomes eligible for scavenging. This value is set as an integer representing the count of items (must be >= 1). |
$cacheControl | Controls the cache of the head of a stream. While most URIs in a stream are infinitely cacheable, the head, by default, will not cache. In some situations, it may be preferable to set a small amount of caching on the head to allow intermediaries to handle polls (e.g., 10 seconds). The argument is an integer representing the seconds to cache (must be >= 1). |
Tips
If you configure both $maxAge
and $maxCount
, events then become eligible for scavenging when either condition is met. For example, if you set $maxAge
to 10 seconds and $maxCount
to 50,000, events qualify for scavenging after either 10 seconds or the accumulation of 50,000 events. Affected items are removed once the scavenge process runs.
Security access control lists (ACLs) are part of the $acl
section in the stream metadata.
Property name | Description |
---|---|
$r | Users with read permissions |
$w | Users with write permissions |
$d | Users with delete permissions |
$mw | Users with write permissions to stream metadata |
$mr | Users with read permissions to stream metadata |
You can find more information on ACLs in the access control lists documentation.
Event metadata
All names starting with $
are reserved for internal use. The currently supported reserved internal names include:
Property name | Description |
---|---|
$correlationId | The application-level correlation ID for the message. |
$causationId | The application-level causation ID for the message. |
Projections within EventStoreDB adhere to both the correlationId
and causationId
patterns for any events they internally produce (e.g., through functions like linkTo
and emit
).
Deleting streams and events
In EventStoreDB, metadata can determine whether an event is deleted or not. You can use stream metadata elements such as TruncateBefore
, MaxAge
, and MaxCount
to filter events marked as deleted. During a stream read, the index references the stream's metadata to determine if any events have been deleted.
Note
In EventStoreDB, you cannot selectively delete events from the middle of a stream. It only allows truncating the stream.
When you delete a stream, EventStoreDB offers two options: soft delete or hard delete.
Soft delete triggers scavenging, removing all events from the stream during the subsequent scavenging process. This allows for the reopening of the stream by appending new events.
It's worth noting that the $all
stream circumvents index checking. Deleted events within this stream remain readable until a scavenging process removes them. To understand the prerequisites for successful event removal through scavenging, refer to the scavenging guide.
Even if a stream has been deleted, EventStoreDB retains one event within the stream to indicate the stream's existence and provide information about the last event version. As a best practice, consider appending a specific event like StreamDeleted
and then setting the MaxCount
to 1 to delete the stream. Keep this in mind, especially when dealing with streams containing sensitive data that you want to erase thoroughly and without a trace.
Soft delete and TruncateBefore
In event sourcing, TruncateBefore
(or $tb
) designates any event with an event number lower than its value as deleted. For instance, consider a stream with the following events:
0@test-stream
1@test-stream
2@test-stream
3@test-stream
Setting the TruncateBefore
(or $tb
) value to 3, results in reading only the last event:
3@test-stream
A soft delete leverages TruncateBefore
(or $tb
). When you delete a stream, TruncateBefore
(or $tb
) is set to the max long/Int64 value: 9223372036854775807.
If you attempt to read this soft-deleted stream the read returns a StreamNotFound
or 404
result.
Appending to the stream later adds events starting from the last stream revision before deletion, incremented by one. Continuing with our previous example, a new event would be appended with event number 4, and only events from event number 4 onwards are visible when you read this stream.
Max count and Max age
Max count ($maxCount
and MaxCount
) restricts the number of events you can read from a stream. Reading a stream with a max count of 5 allows you to retrieve only the last 5 events, regardless of the total events in the stream.
Max age ($maxAge
and MaxAge
) specifies the time an event can live in seconds. The age is calculated at the time of the read. For example, if you read a stream with a MaxAge
of 3 minutes, an event that has existed for 4 minutes won't be returned.
Hard delete
A hard delete involves appending a tombstone
event to the stream and permanently deleting it. This action prevents stream recreation or further appends. Tombstone events are appended with the event type $streamDeleted
. Any attempt to append or read a hard-deleted stream, the response is a StreamDeleted
or 410
result.
While events in the deleted stream are subject to removal during a scavenge, the tombstone event remains.
Note
Hard deletion of a stream is permanent; you cannot append or recreate it. It is recommended to opt for soft deletion for streams unless you have a specific requirement for permanent removal.
Deleted events and projections
If you plan to use projections and delete streams, there are some considerations:
In the context of
$all
, projections usingfromAll
read any deleted events that have not been scavenged. They also receive tombstone events from any hard deletions.System projections like by category or by event type produce new link events stored alongside the original event in the database. When original events are deleted, link events remain in the projected streams, but their links remain unresolved (with undefined values). You can ignore those events in the code logic.
System events and streams
$persistentSubscriptionConfig
$persistentSubscriptionConfig
is a specialized paged stream that stores all configuration events for all persistent subscriptions. It uses the $PersistentConfig
system event type (previously PersistentConfig1
), capturing configuration changes. The event data includes:
Property name | Description |
---|---|
version | Event data version |
updated | Date of the update |
updatedBy | User responsible for the configuration update |
maxCount | The number of configuration events to retain |
entries | Configuration items set by the event |
$all
$all
is a dedicated paged stream containing all events. You can employ the same paged reading method described earlier to read all events for a node by pointing the stream at /streams/$all. Like any other stream, you can perform all operations, except posting to it.
$settings
The $settings
stream hosts a special ACL serving as the default ACL. It governs the default ACL for streams lacking ACLs and controls who can create streams within the system.
Learn more about the default ACL in the access control lists documentation.
$stats
EventStoreDB offers debugging and statistical information about a cluster within the $stats
stream. Find out more in the stats guide.
$scavenges
$scavenges
represents a specialized paged stream for all scavenge-related events. It uses the following system event types:
$scavengeIndexInitialized
: An event that records the initialization of the scavenge index.$scavengeStarted
: An event that records the beginning of a scavenge process. The event data contains:scavengeId
: Scavenge event IDnodeEndpoint
: Node address
$scavengeCompleted
: An event that records the completion of a scavenge process. The event data contains:scavengeId
: Scavenge event IDnodeEndpoint
: Node addressresult
:Success
,Failed
,Stopped
error
: Error detailstimeTaken
: Duration of the scavenging process in millisecondsspaceSaved
: Space saved by scavenge process in bytes.