About time series monitoring
Time series monitoring jobs in Charts send emails for any set threshold breach. If there are several breaches within the minimum duration period, you'll only receive one email for the period. We recommend that you set a minimum duration period of 5 minutes to limit the number of email alerts.
The minimum expected data point frequency is 1 minute. If the data points have a lower frequency, alerts may not trigger as expected. A buffer time of 5 minutes is subtracted from each monitoring job's start time to make sure it overlaps with the previous job.
Monitoring jobs use a merge interval time for combining the breach notifications into one email. This the minimum time difference between breaches amd equals the value you set for minimum duration.
This example illustrates how emails are triggered. Note that activation interval refers to minimum duration.
-
First scenario: The threshold is breached twice for a period longer than the minimum duration. In this monitoring job, the time interval between the two breaches is larger than the merge interval time. These breaches aren't merged, and users receive two emails.
-
Second scenario: The threshold is breached twice for a period longer than the minimum duration. The time interval between the last breach in the previous monitoring job and the first breach in the current monitoring job is higher than the merge interval time. These emails aren't merged, and users receive two emails.
-
Third scenario: The threshold is breached twice, but there's no time interval between the last breach in the previous monitoring job and the first breach in the current monitoring job. These emails are merged, and users receive one email.
-
Fourth scenario: The threshold is breached once and the time interval between the previous breach and the current breach is lower than the merge interval. These emails are merged and users don't receive any emails for this monitoring job.
Emails triggered from threshold values
Example 1
- Threshold value = 0
- Threshold type = above
- Minimal duration value = 5 min.
Explanation: The time series value continuously exceeds the threshold. Only one email is generated since there is one continuous breach.
Example 2
- Threshold value = 0.5
- Threshold type = above
- Minimal duration value = 3 min
Explanation: The time series value has breached the threshold twice. The merge interval is 3 minutes, while the time interval between breaches is 30 minutes. These emails aren't merged, and users receive two emails.
Example 3
- Threshold value = 0.5
- Threshold type = below
- Minimal duration value = 5 min
Explanation: Three time series values breach the threshold. However, due to the set values, these time series only breach the threshold for 1 minute. This is less than the minimum duration value. Since the time series has a frequency larger than the minimum duration and merge interval, emails aren't merged.
Data granularity for monitoring and alerting
The monitoring and alerting system operates on continuous data. The minimum expected granularity is 1 minute, so data points must be ingested at a frequency finer than or equal to the selected activation interval. The activation interval is by default 1 minute.
If the ingested data has a lower frequency, e.g., data points arriving less frequently than every minute, alerts may not trigger as expected. This is the intended behavior of the system, which relies on timely and granular data to compute metrics and detect anomalies.