Skip to content

Commit

Permalink
throttle backend metrics to once every 5 seconds (#841)
Browse files Browse the repository at this point in the history
Instead of passing through backend metrics every second - let's just
take one metrics event every 5 seconds.
<!-- ELLIPSIS_HIDDEN -->

----

> [!IMPORTANT]
> Throttle backend metrics processing in `metrics_loop` to once every 5
seconds using `METRICS_INTERVAL`.
> 
>   - **Behavior**:
> - Throttle backend metrics processing in `metrics_loop` in
`metrics.rs` to once every 5 seconds using `METRICS_INTERVAL`.
>     - Metrics are skipped if processed within the last 5 seconds.
>   - **Constants**:
> - Add `METRICS_INTERVAL` constant set to 5 seconds in `metrics.rs`.
> 
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=jamsocket%2Fplane&utm_source=github&utm_medium=referral)<sup>
for 3d25cc1. It will automatically
update as commits are pushed.</sup>

<!-- ELLIPSIS_HIDDEN -->
  • Loading branch information
rolyatmax authored Nov 14, 2024
1 parent fc6e458 commit e2acc9c
Showing 1 changed file with 11 additions and 1 deletion.
12 changes: 11 additions & 1 deletion plane/src/drone/runtime/docker/metrics.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ use crate::{database::backend::BackendMetricsMessage, names::BackendName};
use bollard::{container::StatsOptions, Docker};
use futures_util::Stream;
use std::sync::{Arc, Mutex};
use tokio::time::{Duration, Instant};
use tokio_stream::StreamExt;

const METRICS_INTERVAL: Duration = Duration::from_secs(5);

fn stream_metrics(
docker: &Docker,
container_id: &ContainerId,
Expand All @@ -23,10 +26,17 @@ pub async fn metrics_loop(
callback: Arc<Mutex<Option<MetricsCallback>>>,
) {
let container_id = ContainerId::from(&backend_id);

let mut stream = stream_metrics(&docker, &container_id);
let mut last_processed = Instant::now() - METRICS_INTERVAL;

while let Some(stats) = stream.next().await {
// These metrics come from Docker every second, so let's throttle them to be less frequent.
let now = Instant::now();
if now.duration_since(last_processed) < METRICS_INTERVAL {
continue;
}
last_processed = now;

let stats = match stats {
Err(err) => {
tracing::error!(?err, "Error getting metrics for {container_id}");
Expand Down

0 comments on commit e2acc9c

Please sign in to comment.