Indexing

EventStoreDB stores indexes separately from the main data files, accessing records by stream name.

Overview

EventStoreDB creates index entries as it processes commit events. It holds these in memory (called memtables) until it reaches the MaxMemTableSize and then persisted on disk in the index folder along with .bloomfilter files and an index map file. The index files are uniquely named, and the index map file called indexmap. The index map describes the order and the level of the index file as well as containing the data checkpoint for the last written file, the version of the index map file and a checksum for the index map file. The logs refer to the index files as a PTable.

Indexes are sorted lists based on the hashes of stream names. To speed up seeking the correct location in the file of an entry for a stream, EventStoreDB keeps midpoints to relate the stream hash to the physical offset in the file.

As EventStoreDB saves more files, they are automatically merged together whenever there are more than 2 files at the same level into a single file at the next level. Each index entry is 24 bytes and the index file size is approximately 24Mb per 1M events. The Bloom filter files are approximately 1% of the size of the rest of the index.

Level 0 is the level of the memtable that is kept in memory. Generally there is only 1 level 0 table unless an ongoing merge operation produces multiple level 0 tables.

Assuming the default MaxMemTableSize of 1M, the index files by level are:

LevelNumber of entriesSize
11M24MB
22M48MB
34M96MB
48M192MB
516M384MB
632M768MB
764M1536MB
8128M3072MB
n2^(n-1) * 1M2^(n-1) * 24Mb

Each index entry is 24 bytes and the index file size is approximately 24Mb per M events.