Skip to content

Commit

Permalink
[Log Stream] Support date nanos (elastic#170308)
Browse files Browse the repository at this point in the history
closes elastic#88290 

## 📝  Summary

As described in elastic#88290 we need to add `date_nanos` support to the Stream
UI page. In this PR the necessary changes have been made all over the
Stream UI and the usages of it.

## ✅  Testing

⚠️ Testing the Stream UI with old timestamp indices is important to make
sure that the behavior is still as before and not affected at all. This
can be done by running local env from the PR and simulating all
interactions on edge-lit cluster for example, to make sure that the
behavior is not changed.

For testing the new changes with support of `date_nano`:

1. You can use [the steps
here](elastic#88290 (comment))
to create and ingest documents with nano precision.
2. Navigate to the stream UI and the documents should be displayed
properly.
3. Sync with the URL state should be changed from timestamp to ISO
string date.
4. Changing time ranges should behave as before, as well as Text
Highlights.
5. Open the logs flyout and you should see the timestamp in nano
seconds.
6. Play around with the minimap, it should behave exactly as before.

### Stream UI:

<img width="2556" alt="Screenshot 2023-11-02 at 14 15 49"
src="https://github.com/elastic/kibana/assets/11225826/596966cd-0ee0-44ee-ba15-f387f3725f66">

- The stream UI has been affected in many areas:
- The logPosition key in the URL should now be in ISO string, but still
backward compatible incase the user has bookmarks with timestamps.
- The minimap should still behave as before in terms of navigation
onClick and also highlighting displayed areas

### Stream UI Flyout:

<img width="2556" alt="Screenshot 2023-11-02 at 14 15 23"
src="https://github.com/elastic/kibana/assets/11225826/6081533c-3bed-43e1-872d-c83fe78ab436">

- The logs flyout should now display the date in nanos format if the
document is ingested using a nano format.

### Anomalies:

<img width="1717" alt="Screenshot 2023-11-01 at 10 37 22"
src="https://github.com/elastic/kibana/assets/11225826/b6170d76-40a4-44db-85de-d8ae852bc17e">

-Anomalies tab under logs as a navigation to stream UI which should
still behave as before passing the filtration and time.

### Categories:

<img width="1705" alt="Screenshot 2023-11-01 at 10 38 19"
src="https://github.com/elastic/kibana/assets/11225826/c4c19202-d27f-410f-b94d-80507542c775">

-Categories tab under logs as a navigation to stream UI which should
still behave as before passing the filtration and time.

### External Links To Stream:
- All links to the Stream UI should still work as before:
   - APM Links for traces, containers, hosts
   - Infra links in Inventory and Hosts views

## 🎥 Demo


https://github.com/elastic/kibana/assets/11225826/9a39bc5a-ba37-49e0-b7f2-e73260fb01f0
  • Loading branch information
mohamedhamed-ahmed authored Nov 13, 2023
1 parent 19e43c8 commit ddc07c5
Show file tree
Hide file tree
Showing 60 changed files with 850 additions and 133 deletions.
8 changes: 8 additions & 0 deletions packages/kbn-io-ts-utils/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,11 @@ export {
export { datemathStringRt } from './src/datemath_string_rt';

export { createPlainError, decodeOrThrow, formatErrors, throwErrors } from './src/decode_or_throw';

export {
DateFromStringOrNumber,
minimalTimeKeyRT,
type MinimalTimeKey,
type TimeKey,
type UniqueTimeKey,
} from './src/time_key_rt';
53 changes: 53 additions & 0 deletions packages/kbn-io-ts-utils/src/time_key_rt/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import * as rt from 'io-ts';
import moment from 'moment';
import { pipe } from 'fp-ts/lib/pipeable';
import { chain } from 'fp-ts/lib/Either';

const NANO_DATE_PATTERN = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3,9}Z$/;

export const DateFromStringOrNumber = new rt.Type<string, number | string>(
'DateFromStringOrNumber',
(input): input is string => typeof input === 'string',
(input, context) => {
if (typeof input === 'string') {
return NANO_DATE_PATTERN.test(input) ? rt.success(input) : rt.failure(input, context);
}
return pipe(
rt.number.validate(input, context),
chain((timestamp) => {
const momentValue = moment(timestamp);
return momentValue.isValid()
? rt.success(momentValue.toISOString())
: rt.failure(timestamp, context);
})
);
},
String
);

export const minimalTimeKeyRT = rt.type({
time: DateFromStringOrNumber,
tiebreaker: rt.number,
});
export type MinimalTimeKey = rt.TypeOf<typeof minimalTimeKeyRT>;

const timeKeyRT = rt.intersection([
minimalTimeKeyRT,
rt.partial({
gid: rt.string,
fromAutoReload: rt.boolean,
}),
]);
export type TimeKey = rt.TypeOf<typeof timeKeyRT>;

export interface UniqueTimeKey extends TimeKey {
gid: string;
}
2 changes: 1 addition & 1 deletion x-pack/plugins/infra/common/locators/locators.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ const constructLogView = (logView?: LogViewReference) => {
};

const constructLogPosition = (time: number = 1550671089404) => {
return `(position:(tiebreaker:0,time:${time}))`;
return `(position:(tiebreaker:0,time:'${moment(time).toISOString()}'))`;
};

const constructLogFilter = ({
Expand Down
3 changes: 2 additions & 1 deletion x-pack/plugins/infra/common/time/time_key.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@
* 2.0.
*/

import { DateFromStringOrNumber } from '@kbn/io-ts-utils';
import { ascending, bisector } from 'd3-array';
import * as rt from 'io-ts';
import { pick } from 'lodash';

export const minimalTimeKeyRT = rt.type({
time: rt.number,
time: DateFromStringOrNumber,
tiebreaker: rt.number,
});
export type MinimalTimeKey = rt.TypeOf<typeof minimalTimeKeyRT>;
Expand Down
6 changes: 3 additions & 3 deletions x-pack/plugins/infra/common/url_state_storage_service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,18 @@ export const defaultLogViewKey = 'logView';

const encodeRisonUrlState = (state: any) => encode(state);

// Used by linkTo components
// Used by Locator components
export const replaceLogPositionInQueryString = (time?: number) =>
Number.isNaN(time) || time == null
? (value: string) => value
: replaceStateKeyInQueryString<PositionStateInUrl>(defaultPositionStateKey, {
position: {
time,
time: moment(time).toISOString(),
tiebreaker: 0,
},
});

// NOTE: Used by link-to components
// NOTE: Used by Locator components
export const replaceLogViewInQueryString = (logViewReference: LogViewReference) =>
replaceStateKeyInQueryString(defaultLogViewKey, logViewReference);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import {
LogEntryTime,
} from '@kbn/logs-shared-plugin/common';
import { scaleLinear } from 'd3-scale';
import moment from 'moment';
import * as React from 'react';
import { DensityChart } from './density_chart';
import { HighlightedInterval } from './highlighted_interval';
Expand Down Expand Up @@ -67,7 +68,7 @@ export class LogMinimap extends React.Component<LogMinimapProps, LogMinimapState

this.props.jumpToTarget({
tiebreaker: 0,
time: clickedTime,
time: moment(clickedTime).toISOString(),
});
};

Expand Down Expand Up @@ -142,9 +143,9 @@ export class LogMinimap extends React.Component<LogMinimapProps, LogMinimapState

{highlightedInterval ? (
<HighlightedInterval
end={highlightedInterval.end}
end={moment(highlightedInterval.end).valueOf()}
getPositionOfTime={this.getPositionOfTime}
start={highlightedInterval.start}
start={moment(highlightedInterval.start).valueOf()}
targetWidth={TIMERULER_WIDTH}
width={width}
target={target}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ const validateSetupIndices = async (
fields: [
{
name: timestampField,
validTypes: ['date'],
validTypes: ['date', 'date_nanos'],
},
{
name: partitionField,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ const validateSetupIndices = async (
fields: [
{
name: timestampField,
validTypes: ['date'],
validTypes: ['date', 'date_nanos'],
},
{
name: partitionField,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ export type LogStreamPageContext = LogStreamPageTypestate['context'];
export interface LogStreamPageCallbacks {
updateTimeRange: (timeRange: Partial<TimeRange>) => void;
jumpToTargetPosition: (targetPosition: TimeKey | null) => void;
jumpToTargetPositionTime: (time: number) => void;
jumpToTargetPositionTime: (time: string) => void;
reportVisiblePositions: (visiblePositions: VisiblePositions) => void;
startLiveStreaming: () => void;
stopLiveStreaming: () => void;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

import { IToasts } from '@kbn/core-notifications-browser';
import { IKbnUrlStateStorage } from '@kbn/kibana-utils-plugin/public';
import { convertISODateToNanoPrecision } from '@kbn/logs-shared-plugin/common';
import moment from 'moment';
import { actions, ActorRefFrom, createMachine, EmittedFrom, SpecialTargets } from 'xstate';
import { isSameTimeKey } from '../../../../common/time';
import { OmitDeprecatedState, sendIfDefined } from '../../xstate_helpers';
Expand Down Expand Up @@ -159,11 +161,19 @@ export const createPureLogStreamPositionStateMachine = (initialContext: LogStrea
updatePositionsFromTimeChange: actions.assign((_context, event) => {
if (!('timeRange' in event)) return {};

const {
timestamps: { startTimestamp, endTimestamp },
} = event;

// Reset the target position if it doesn't fall within the new range.
const targetPositionNanoTime =
_context.targetPosition && convertISODateToNanoPrecision(_context.targetPosition.time);
const startNanoDate = convertISODateToNanoPrecision(moment(startTimestamp).toISOString());
const endNanoDate = convertISODateToNanoPrecision(moment(endTimestamp).toISOString());

const targetPositionShouldReset =
_context.targetPosition &&
(event.timestamps.startTimestamp > _context.targetPosition.time ||
event.timestamps.endTimestamp < _context.targetPosition.time);
targetPositionNanoTime &&
(startNanoDate > targetPositionNanoTime || endNanoDate < targetPositionNanoTime);

return {
targetPosition: targetPositionShouldReset ? null : _context.targetPosition,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ import { IKbnUrlStateStorage, withNotifyOnErrors } from '@kbn/kibana-utils-plugi
import * as Either from 'fp-ts/lib/Either';
import { pipe } from 'fp-ts/lib/function';
import { InvokeCreator } from 'xstate';
import { replaceStateKeyInQueryString } from '../../../../common/url_state_storage_service';
import { minimalTimeKeyRT, pickTimeKey } from '../../../../common/time';
import { createPlainError, formatErrors } from '../../../../common/runtime_types';
import type { LogStreamPositionContext, LogStreamPositionEvent } from './types';
Expand Down Expand Up @@ -98,13 +97,3 @@ export type PositionStateInUrl = rt.TypeOf<typeof positionStateInUrlRT>;
const decodePositionQueryValueFromUrl = (queryValueFromUrl: unknown) => {
return positionStateInUrlRT.decode(queryValueFromUrl);
};

export const replaceLogPositionInQueryString = (time?: number) =>
Number.isNaN(time) || time == null
? (value: string) => value
: replaceStateKeyInQueryString<PositionStateInUrl>(defaultPositionStateKey, {
position: {
time,
tiebreaker: 0,
},
});
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import {
defaultPositionStateKey,
DEFAULT_REFRESH_INTERVAL,
} from '@kbn/logs-shared-plugin/common';
import moment from 'moment';
import {
getTimeRangeEndFromTime,
getTimeRangeStartFromTime,
Expand Down Expand Up @@ -159,8 +160,8 @@ export const initializeFromUrl =
Either.chain(({ position }) =>
position && position.time
? Either.right({
from: getTimeRangeStartFromTime(position.time),
to: getTimeRangeEndFromTime(position.time),
from: getTimeRangeStartFromTime(moment(position.time).valueOf()),
to: getTimeRangeEndFromTime(moment(position.time).valueOf()),
})
: Either.left(null)
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ export const CategoryExampleMessage: React.FunctionComponent<{
search: {
logPosition: encode({
end: moment(timeRange.endTime).format('YYYY-MM-DDTHH:mm:ss.SSSZ'),
position: { tiebreaker, time: timestamp },
position: { tiebreaker, time: moment(timestamp).toISOString() },
start: moment(timeRange.startTime).format('YYYY-MM-DDTHH:mm:ss.SSSZ'),
streamLive: false,
}),
Expand Down Expand Up @@ -128,7 +128,10 @@ export const CategoryExampleMessage: React.FunctionComponent<{
id,
index: '', // TODO: use real index when loading via async search
context,
cursor: { time: timestamp, tiebreaker },
cursor: {
time: moment(timestamp).toISOString(),
tiebreaker,
},
columns: [],
};
trackMetric({ metric: 'view_in_context__categories' });
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ export const LogEntryExampleMessage: React.FunctionComponent<Props> = ({
search: {
logPosition: encode({
end: moment(timeRange.endTime).format('YYYY-MM-DDTHH:mm:ss.SSSZ'),
position: { tiebreaker, time: timestamp },
position: { tiebreaker, time: moment(timestamp).toISOString() },
start: moment(timeRange.startTime).format('YYYY-MM-DDTHH:mm:ss.SSSZ'),
streamLive: false,
}),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ export const ConnectedStreamPageContent: React.FC = () => {
jumpToTargetPosition: (targetPosition: TimeKey | null) => {
logStreamPageSend({ type: 'JUMP_TO_TARGET_POSITION', targetPosition });
},
jumpToTargetPositionTime: (time: number) => {
jumpToTargetPositionTime: (time: string) => {
logStreamPageSend({ type: 'JUMP_TO_TARGET_POSITION', targetPosition: { time } });
},
reportVisiblePositions: (visiblePositions: VisiblePositions) => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import { EuiSpacer } from '@elastic/eui';
import type { Query } from '@kbn/es-query';
import { euiStyled } from '@kbn/kibana-react-plugin/common';
import { LogEntry } from '@kbn/logs-shared-plugin/common';
import { LogEntry, convertISODateToNanoPrecision } from '@kbn/logs-shared-plugin/common';
import {
LogEntryFlyout,
LogEntryStreamItem,
Expand Down Expand Up @@ -117,8 +117,12 @@ export const StreamPageLogsContent = React.memo<{

const isCenterPointOutsideLoadedRange =
targetPosition != null &&
((topCursor != null && targetPosition.time < topCursor.time) ||
(bottomCursor != null && targetPosition.time > bottomCursor.time));
((topCursor != null &&
convertISODateToNanoPrecision(targetPosition.time) <
convertISODateToNanoPrecision(topCursor.time)) ||
(bottomCursor != null &&
convertISODateToNanoPrecision(targetPosition.time) >
convertISODateToNanoPrecision(bottomCursor.time)));

const hasQueryChanged = filterQuery !== prevFilterQuery;

Expand Down
6 changes: 4 additions & 2 deletions x-pack/plugins/infra/public/test_utils/entries.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,16 @@ export function generateFakeEntries(
const timestampStep = Math.floor((endTimestamp - startTimestamp) / count);
for (let i = 0; i < count; i++) {
const timestamp = i === count - 1 ? endTimestamp : startTimestamp + timestampStep * i;
const date = new Date(timestamp).toISOString();

entries.push({
id: `entry-${i}`,
index: 'logs-fake',
context: {},
cursor: { time: timestamp, tiebreaker: i },
cursor: { time: date, tiebreaker: i },
columns: columns.map((column) => {
if ('timestampColumn' in column) {
return { columnId: column.timestampColumn.id, timestamp };
return { columnId: column.timestampColumn.id, time: date };
} else if ('messageColumn' in column) {
return {
columnId: column.messageColumn.id,
Expand Down
2 changes: 2 additions & 0 deletions x-pack/plugins/logs_shared/common/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ export {
// eslint-disable-next-line @kbn/eslint/no_export_all
export * from './log_entry';

export { convertISODateToNanoPrecision } from './utils';

// Http types
export type { LogEntriesSummaryBucket, LogEntriesSummaryHighlightsBucket } from './http_api';

Expand Down
4 changes: 2 additions & 2 deletions x-pack/plugins/logs_shared/common/log_entry/log_entry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
* 2.0.
*/

import { TimeKey } from '@kbn/io-ts-utils';
import * as rt from 'io-ts';
import { TimeKey } from '../time';
import { jsonArrayRT } from '../typed_json';
import { logEntryCursorRT } from './log_entry_cursor';

Expand Down Expand Up @@ -34,7 +34,7 @@ export type LogMessagePart = rt.TypeOf<typeof logMessagePartRT>;
* columns
*/

export const logTimestampColumnRT = rt.type({ columnId: rt.string, timestamp: rt.number });
export const logTimestampColumnRT = rt.type({ columnId: rt.string, time: rt.string });
export type LogTimestampColumn = rt.TypeOf<typeof logTimestampColumnRT>;

export const logFieldColumnRT = rt.type({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import * as rt from 'io-ts';
import { decodeOrThrow } from '../runtime_types';

export const logEntryCursorRT = rt.type({
time: rt.number,
time: rt.string,
tiebreaker: rt.number,
});
export type LogEntryCursor = rt.TypeOf<typeof logEntryCursorRT>;
Expand All @@ -29,7 +29,7 @@ export const logEntryAroundCursorRT = rt.type({
});
export type LogEntryAroundCursor = rt.TypeOf<typeof logEntryAroundCursorRT>;

export const getLogEntryCursorFromHit = (hit: { sort: [number, number] }) =>
export const getLogEntryCursorFromHit = (hit: { sort: [string, number] }) =>
decodeOrThrow(logEntryCursorRT)({
time: hit.sort[0],
tiebreaker: hit.sort[1],
Expand Down
20 changes: 1 addition & 19 deletions x-pack/plugins/logs_shared/common/time/time_key.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,8 @@
* 2.0.
*/

import type { TimeKey } from '@kbn/io-ts-utils';
import { ascending, bisector } from 'd3-array';
import * as rt from 'io-ts';

export const minimalTimeKeyRT = rt.type({
time: rt.number,
tiebreaker: rt.number,
});

export const timeKeyRT = rt.intersection([
minimalTimeKeyRT,
rt.partial({
gid: rt.string,
fromAutoReload: rt.boolean,
}),
]);
export type TimeKey = rt.TypeOf<typeof timeKeyRT>;

export interface UniqueTimeKey extends TimeKey {
gid: string;
}

export type Comparator = (firstValue: any, secondValue: any) => number;

Expand Down
Loading

0 comments on commit ddc07c5

Please sign in to comment.