Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] Consolidate various throughput calculations to a utility function #89578

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions x-pack/plugins/apm/server/lib/helpers/calculate_throughput.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

import { SetupTimeRange } from './setup_request';

export function calculateThroughput({
start,
end,
value,
}: SetupTimeRange & { value: number }) {
const durationAsMinutes = (end - start) / 1000 / 60;
return value / durationAsMinutes;
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import {
getProcessorEventForAggregatedTransactions,
getTransactionDurationFieldForAggregatedTransactions,
} from '../helpers/aggregated_transactions';
import { calculateThroughput } from '../helpers/calculate_throughput';

export async function getTransactionCoordinates({
setup,
Expand Down Expand Up @@ -63,12 +64,10 @@ export async function getTransactionCoordinates({
},
});

const deltaAsMinutes = (end - start) / 1000 / 60;

return (
aggregations?.distribution.buckets.map((bucket) => ({
x: bucket.key,
y: bucket.count.value / deltaAsMinutes,
y: calculateThroughput({ start, end, value: bucket.count.value }),
})) || []
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import { joinByKey } from '../../../../common/utils/join_by_key';
import { Setup, SetupTimeRange } from '../../helpers/setup_request';
import { getMetrics } from './get_metrics';
import { getDestinationMap } from './get_destination_map';
import { calculateThroughput } from '../../helpers/calculate_throughput';

export type ServiceDependencyItem = {
name: string;
Expand Down Expand Up @@ -51,7 +52,6 @@ export async function getServiceDependencies({
numBuckets: number;
}): Promise<ServiceDependencyItem[]> {
const { start, end } = setup;

const [allMetrics, destinationMap] = await Promise.all([
getMetrics({
setup,
Expand Down Expand Up @@ -134,8 +134,6 @@ export async function getServiceDependencies({
}
);

const deltaAsMinutes = (end - start) / 60 / 1000;

const destMetrics = {
latency: {
value:
Expand All @@ -150,11 +148,18 @@ export async function getServiceDependencies({
throughput: {
value:
mergedMetrics.value.count > 0
? mergedMetrics.value.count / deltaAsMinutes
? calculateThroughput({
start,
end,
value: mergedMetrics.value.count,
})
: null,
timeseries: mergedMetrics.timeseries.map((point) => ({
x: point.x,
y: point.count > 0 ? point.count / deltaAsMinutes : null,
y:
point.count > 0
? calculateThroughput({ start, end, value: point.count })
: null,
})),
},
errorRate: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import {
getProcessorEventForAggregatedTransactions,
getTransactionDurationFieldForAggregatedTransactions,
} from '../../helpers/aggregated_transactions';
import { calculateThroughput } from '../../helpers/calculate_throughput';

export async function getServiceInstanceTransactionStats({
setup,
Expand Down Expand Up @@ -112,8 +113,6 @@ export async function getServiceInstanceTransactionStats({
},
});

const deltaAsMinutes = (end - start) / 60 / 1000;

return (
response.aggregations?.[SERVICE_NODE_NAME].buckets.map(
(serviceNodeBucket) => {
Expand All @@ -135,10 +134,14 @@ export async function getServiceInstanceTransactionStats({
})),
},
throughput: {
value: count.value / deltaAsMinutes,
value: calculateThroughput({ start, end, value: count.value }),
timeseries: timeseries.buckets.map((dateBucket) => ({
x: dateBucket.key,
y: dateBucket.count.value / deltaAsMinutes,
y: calculateThroughput({
start,
end,
value: dateBucket.count.value,
}),
})),
},
latency: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import {
getLatencyAggregation,
getLatencyValue,
} from '../../helpers/latency_aggregation_type';
import { calculateThroughput } from '../../helpers/calculate_throughput';

export type ServiceOverviewTransactionGroupSortField =
| 'name'
Expand Down Expand Up @@ -64,8 +65,6 @@ export async function getTransactionGroupsForPage({
transactionType: string;
latencyAggregationType: LatencyAggregationType;
}) {
const deltaAsMinutes = (end - start) / 1000 / 60;

const field = getTransactionDurationFieldForAggregatedTransactions(
searchAggregatedTransactions
);
Expand Down Expand Up @@ -124,7 +123,11 @@ export async function getTransactionGroupsForPage({
latencyAggregationType,
aggregation: bucket.latency,
}),
throughput: bucket.transaction_count.value / deltaAsMinutes,
throughput: calculateThroughput({
start,
end,
value: bucket.transaction_count.value,
}),
errorRate,
};
}) ?? [];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

import { EVENT_OUTCOME } from '../../../../common/elasticsearch_fieldnames';
import { LatencyAggregationType } from '../../../../common/latency_aggregation_types';
import { calculateThroughput } from '../../helpers/calculate_throughput';
import { getLatencyValue } from '../../helpers/latency_aggregation_type';
import { TransactionGroupTimeseriesData } from './get_timeseries_data_for_transaction_groups';
import { TransactionGroupWithoutTimeseriesData } from './get_transaction_groups_for_page';
Expand All @@ -25,8 +26,6 @@ export function mergeTransactionGroupData({
latencyAggregationType: LatencyAggregationType;
transactionType: string;
}) {
const deltaAsMinutes = (end - start) / 1000 / 60;

return transactionGroups.map((transactionGroup) => {
const groupBucket = timeseriesData.find(
({ key }) => key === transactionGroup.name
Expand All @@ -52,7 +51,11 @@ export function mergeTransactionGroupData({
...acc.throughput,
timeseries: acc.throughput.timeseries.concat({
x: point.key,
y: point.transaction_count.value / deltaAsMinutes,
y: calculateThroughput({
start,
end,
value: point.transaction_count.value,
}),
}),
},
errorRate: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import {
getTransactionDurationFieldForAggregatedTransactions,
} from '../../helpers/aggregated_transactions';
import { getBucketSize } from '../../helpers/get_bucket_size';
import { calculateThroughput } from '../../helpers/calculate_throughput';
import {
calculateTransactionErrorPercentage,
getOutcomeAggregation,
Expand All @@ -35,16 +36,6 @@ interface AggregationParams {

const MAX_NUMBER_OF_SERVICES = 500;

function calculateAvgDuration({
value,
deltaAsMinutes,
}: {
value: number;
deltaAsMinutes: number;
}) {
return value / deltaAsMinutes;
}

export async function getServiceTransactionStats({
setup,
searchAggregatedTransactions,
Expand Down Expand Up @@ -139,8 +130,6 @@ export async function getServiceTransactionStats({
},
});

const deltaAsMinutes = (setup.end - setup.start) / 1000 / 60;

return (
response.aggregations?.services.buckets.map((bucket) => {
const topTransactionTypeBucket =
Expand Down Expand Up @@ -179,16 +168,18 @@ export async function getServiceTransactionStats({
),
},
transactionsPerMinute: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While you are at it: can you rename this to "throughput"?

Copy link
Member

@sorenlouv sorenlouv Jan 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above it says "avgResponseTime". Perhaps you can rename that to "averageLatency" - or simply "latency"

value: calculateAvgDuration({
value: calculateThroughput({
start,
end,
value: topTransactionTypeBucket.real_document_count.value,
deltaAsMinutes,
}),
timeseries: topTransactionTypeBucket.timeseries.buckets.map(
(dateBucket) => ({
x: dateBucket.key,
y: calculateAvgDuration({
y: calculateThroughput({
start,
end,
value: dateBucket.real_document_count.value,
deltaAsMinutes,
}),
})
),
Expand Down
12 changes: 6 additions & 6 deletions x-pack/plugins/apm/server/lib/services/get_throughput.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import {
getProcessorEventForAggregatedTransactions,
} from '../helpers/aggregated_transactions';
import { getBucketSize } from '../helpers/get_bucket_size';
import { calculateThroughput } from '../helpers/calculate_throughput';
import { Setup, SetupTimeRange } from '../helpers/setup_request';

interface Options {
Expand All @@ -27,16 +28,15 @@ interface Options {

type ESResponse = PromiseReturnType<typeof fetcher>;

function transform(response: ESResponse, options: Options) {
const { end, start } = options.setup;
const deltaAsMinutes = (end - start) / 1000 / 60;
function transform(options: Options, response: ESResponse) {
if (response.hits.total.value === 0) {
return [];
}
const { start, end } = options.setup;
const buckets = response.aggregations?.throughput.buckets ?? [];
return buckets.map(({ key: x, doc_count: y }) => ({
return buckets.map(({ key: x, doc_count: value }) => ({
x,
y: y / deltaAsMinutes,
y: calculateThroughput({ start, end, value }),
}));
}

Expand Down Expand Up @@ -87,6 +87,6 @@ async function fetcher({

export async function getThroughput(options: Options) {
return {
throughput: transform(await fetcher(options), options),
throughput: transform(options, await fetcher(options)),
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -106,9 +106,7 @@ export async function getThroughputCharts({
setup: Setup & SetupTimeRange;
searchAggregatedTransactions: boolean;
}) {
const { start, end } = setup;
const { bucketSize, intervalString } = getBucketSize({ start, end });
const durationAsMinutes = (end - start) / 1000 / 60;
const { bucketSize, intervalString } = getBucketSize(setup);

const response = await searchThroughput({
serviceName,
Expand All @@ -123,7 +121,7 @@ export async function getThroughputCharts({
throughputTimeseries: getThroughputBuckets({
throughputResultBuckets: response.aggregations?.throughput.buckets,
bucketSize,
durationAsMinutes,
setupTimeRange: setup,
}),
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,21 @@
import { sortBy } from 'lodash';
import { NOT_AVAILABLE_LABEL } from '../../../../common/i18n';
import { ThroughputChartsResponse } from '.';
import { calculateThroughput } from '../../helpers/calculate_throughput';
import { SetupTimeRange } from '../../helpers/setup_request';

type ThroughputResultBuckets = Required<ThroughputChartsResponse>['aggregations']['throughput']['buckets'];

export function getThroughputBuckets({
throughputResultBuckets = [],
bucketSize,
durationAsMinutes,
setupTimeRange,
}: {
throughputResultBuckets?: ThroughputResultBuckets;
bucketSize: number;
durationAsMinutes: number;
setupTimeRange: SetupTimeRange;
}) {
const { start, end } = setupTimeRange;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we could also update the calculation below (on line 31):

y: bucket.count.value / (bucketSize / 60),

Copy link
Member

@sorenlouv sorenlouv Jan 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw. this is a good candidate for using the rate agg that we also have an issue for.

Perhaps doing that could be a follow up to this PR (or something you could do as part of this one?)

const buckets = throughputResultBuckets.map(
({ key: resultKey, timeseries }) => {
const dataPoints = timeseries.buckets.map((bucket) => {
Expand All @@ -38,7 +41,7 @@ export function getThroughputBuckets({
.reduce((a, b) => a + b, 0);

// calculate average throughput
const avg = docCountTotal / durationAsMinutes;
const avg = calculateThroughput({ start, end, value: docCountTotal });

return { key, dataPoints, avg };
}
Expand Down