Integrations
Event Store Cloud offers integrations between internal sources such as cluster health, issue detection, notifications events and sinks which include external services such as Slack and OpsGenie.
Integration sources
"Sources" are driven by events or other mechanism inside the Event Store Cloud.
Currently, supported sources include:
Issues - issues represent a potentially problematic condition detected inside a cluster or other Event Store Cloud resource. Issues consist of multiple "open" events and a single "closed" event and have different levels of severity.
Notifications - notifications are noteworthy events detected within Event Store Cloud resources or the backend. For example notifications may be emitted when a cluster fails to provision.
Issues
Issues represent possibly problematic states detected within the Event Store Cloud. Below, you can find several issue examples.
NOTE
These examples are a subset of issues created by the system. The exact details of why issues are created are subject to change, but the cause of the issue and steps to resolve it will be clear in the messages associated with the related events.
Core load count
For each node of a cluster, the core load average is measured and divided by the number of physical cores on the node. If the result exceeds 2.0 an issue is opened. This issue is closed when the result consistently dips under 2.0.
If this happens consider increasing the size of the instance type for the cluster.
Disk usage
For each node of a cluster, the disk usage is measured several times a minute. If it starts to consistently exceed 80% an issue is opened. The issue is closed when the usage drops below 80%.
If this happens consider either removing data, running scavenge or increasing the disk size for the cluster.
Memory usage
For each node of a cluster, the memory usage is measured several times a minute. If it exceeds 90% an issue is opened. The issue is closed when memory usage consistently drops below 90%.
If this happens consider increasing the size of the instance type for the cluster.
Cluster consensus
Every node on a cluster has it's gossip status queried twice each minute. An issue is opened if either the query fails or if the reported gossip state for each node is not identical on a multi-node cluster.
The issue closes when the gossip status again returns expected values.
Notifications
Notifications represent noteworthy events which occur within the Event Store Cloud. Below you can find notifications examples.
NOTE
The following represent a subset of events which can lead to notifications.
Cluster provisioning failure
If, for some reason, the instances backing a cluster fail to provision the resource is marked as defunct by the API and a notification is sent with the message Cluster instances failed to provision
.
Volume expansion failure
If the volume fails to expand while expanding an instances size a notification event is created with the message Cluster volumes failed to provision
.
Integration sinks
"Sinks" are services outside the Event Store Cloud which events from sources can be forwarded to.
OpsGenie - OpsGenie is an alerting and incidence response tool. It is possible to set up integrations to create OpsGenie alerts when cluster health issues are detected.
Slack - Slack is a communication platform. It is possible to set up integrations which send Slack messages when issues and notifications are created or updated.
Slack sink
Slack is a business communications platform.
You can be notified of new events and notifications in your Slack workspace by creating a Slack App and setting up an integration with as described below.
Setup Slack
Before adding a Slack integration, you need to create a Slack App and obtain a token through slack.com.
- Go to
https://api.slack.com/apps
and clickCreate New App
. Give it any name you wish and select the target workspace, then clickCreate App
. - On the next screen, click on
OAuth & Permissions
. UnderScopes
,Bot Token Scopes
and add a scope ofchat:write
. - At the top of the page, click
Install to Workspace
. - After installing the app, the
OAuth & Permissions
page will have aBot User OAuth Token
shown at the top. Copy it for the next step. - Finally, you'll need to invite the Slack app user to the channel you want it to chat with. In Slack, go to the appropriate channel and write the following:
/invite @<name of bot>
Add a new Slack integration
- In the Event Store Cloud console, select an organization and then a project.
- Once viewing a project, you should see
Integrations
under the headingProject
in the sidebar to the left. Click it. - Click
New Integration
. Enter a name that will make it easier to find later, and then select Issues or Notifications. Then select the SinkSlack
as in the screen below.
- Under
Configuration
, next toChannel ID
, enter the channel you want the slack bot to communicate with (this must be the same as the place you invited the bot earlier). Remember to start the channel ID with a hash sign if appropriate. - In the box next to
Token
enter the OAuth token you copied from Slack's website. - Finally, click "Create Integration."
- Now back at the "Integrations" page, click on the row with recently created integration. In the bottom pane you should see a button marked
Test integration
as in the screen below. Click it.
- If your Slack App was created correctly you should see a test message in the channel you selected.
- If you get an error message double check that you copied the OAuth token and Channel ID into the integration correctly. You may also wish to review the previous step to ensure you gave the app appropriate permissions. You can edit the integration by clicking on the pencil icon in its row.
- If you see a Slack message, you can rest you will receive events from the configured source when they occur.
Details
If the source is Notifications
, a single message will be sent to the configured slack bot for each notification.
If the source is Issues
, a single message will be sent for each open issue and given a red bar. Open issues receive new events continuously, however the original Slack message will only be updated to reflect the latest event every five minutes.
When the issue is closed, the original message will be changed to reflect this, and the bar will be changed to green.
In order to send messages to multiple channels, create more integrations in Event Store Cloud using the same credentials.
OpsGenie sink
EventStore Cloud platform is using OpsGenie for its alerting system. Our minimal configuration requires an API key. For simplicity’ sake, we recommend the API key to belong to a responder team.
Those instructions assume you are starting from scratch and don't have a team set up yet. We also assume that you are currently on the landing page after logging in OpsGenie.
Create a team in OpsGenie
Log in to your OpsGenie instance, then complete the following steps to set up a new team. Skip this step if you already have a team.
- Click on the
Teams
tab up top. - Click on the
Add team
top left. - A popup should show up. Enter your team info like its name and members. Keep in mind that the team will be considered as the responder team in EventStore Cloud.
- Once you confirm your new team creation, you should be redirected to your new team dashboard page.
Generate the team API key
Follow these steps to generate an API key for the team, which should be alerted when issues happen in Event Store Cloud.
- Click on the
Teams
tab up top. - Select your team in the team table.
- By selecting your team, you should be redirected to your team dashboard.
- Click on
Integrations
, located in the left sidebar. - Click on the
Add integration
button. - In the integration list, click on
API
then theAdd
button. - By default, the form should be already pre-filled. Make sure that
Read Access
,Create and Update Access
,Delete Access
andEnabled
are checked. - Click on
Save Integration
at the bottom. - You can get your API key that should be located just below the
Name
property.
Complete the integration
- In the Event Store Cloud console, select an organization and then a project.
- Once viewing a project, you should see
Integrations
under the headingProject
in the sidebar to the left. Click it. - Click
New Integration
. Enter a name that will make it easier to find later, and then select Issues or Notifications. - Select the OpsGenie sink and put the API Key as in the screen below:
and click on the Create integration
button.
If all the details were entered correctly, the new integration should be set up. See the example on the screen below.