Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x] 🌊 Move streams to platform #212113

Open
wants to merge 2 commits into
base: 8.x
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -957,8 +957,9 @@ src/platform/packages/shared/kbn-std @elastic/kibana-core
packages/kbn-stdio-dev-helpers @elastic/kibana-operations
src/platform/packages/shared/kbn-storage-adapter @elastic/observability-ui
packages/kbn-storybook @elastic/kibana-operations
x-pack/solutions/observability/plugins/streams_app @simianhacker @flash1293 @dgieselaar
x-pack/solutions/observability/plugins/streams @simianhacker @flash1293 @dgieselaar
x-pack/platform/plugins/shared/streams_app @elastic/streams-program-team
x-pack/solutions/observability/plugins/observability_streams_wrapper @elastic/streams-program-team
x-pack/platform/plugins/shared/streams @simianhacker @flash1293 @dgieselaar
x-pack/platform/packages/shared/kbn-streams-schema @elastic/streams-program-team
packages/kbn-styled-components-mapping-cli @elastic/kibana-operations @elastic/eui-team
x-pack/solutions/observability/plugins/synthetics/e2e @elastic/obs-ux-management-team
Expand Down
8 changes: 6 additions & 2 deletions docs/developer/plugin-list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -922,14 +922,18 @@ routes, etc.
|The stack_connectors plugin provides connector types shipped with Kibana, built on top of the framework provided in the actions plugin.


|{kib-repo}blob/{branch}/x-pack/solutions/observability/plugins/streams/README.md[streams]
|{kib-repo}blob/{branch}/x-pack/platform/plugins/shared/streams/README.md[streams]
|This plugin provides an interface to manage streams


|{kib-repo}blob/{branch}/x-pack/solutions/observability/plugins/streams_app/README.md[streamsApp]
|{kib-repo}blob/{branch}/x-pack/platform/plugins/shared/streams_app/README.md[streamsApp]
|Home of the Streams app plugin, which allows users to manage Streams via the UI.


|{kib-repo}blob/{branch}/x-pack/solutions/observability/plugins/observability_streams_wrapper/README.md[streamsAppWrapper]
|Observability-specific wrapper for the streams app. This is responsible for actually registering the app and making it accessible in observability contexts


|{kib-repo}blob/{branch}/x-pack/solutions/observability/plugins/synthetics/README.md[synthetics]
|The purpose of this plugin is to provide users of Heartbeat more visibility of what's happening
in their infrastructure.
Expand Down
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -937,8 +937,9 @@
"@kbn/status-plugin-b-plugin": "link:test/server_integration/plugins/status_plugin_b",
"@kbn/std": "link:src/platform/packages/shared/kbn-std",
"@kbn/storage-adapter": "link:src/platform/packages/shared/kbn-storage-adapter",
"@kbn/streams-app-plugin": "link:x-pack/solutions/observability/plugins/streams_app",
"@kbn/streams-plugin": "link:x-pack/solutions/observability/plugins/streams",
"@kbn/streams-app-plugin": "link:x-pack/platform/plugins/shared/streams_app",
"@kbn/streams-app-wrapper-plugin": "link:x-pack/solutions/observability/plugins/observability_streams_wrapper",
"@kbn/streams-plugin": "link:x-pack/platform/plugins/shared/streams",
"@kbn/streams-schema": "link:x-pack/platform/packages/shared/kbn-streams-schema",
"@kbn/synthetics-plugin": "link:x-pack/solutions/observability/plugins/synthetics",
"@kbn/task-manager-fixture-plugin": "link:x-pack/test/alerting_api_integration/common/plugins/task_manager_fixture",
Expand Down
1 change: 1 addition & 0 deletions packages/kbn-optimizer/limits.yml
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ pageLoadAssetSize:
stackConnectors: 67227
streams: 16742
streamsApp: 20537
streamsAppWrapper: 20537
synthetics: 55971
telemetry: 51957
telemetryManagementSection: 38586
Expand Down
10 changes: 6 additions & 4 deletions tsconfig.base.json
Original file line number Diff line number Diff line change
Expand Up @@ -1908,10 +1908,12 @@
"@kbn/storage-adapter/*": ["src/platform/packages/shared/kbn-storage-adapter/*"],
"@kbn/storybook": ["packages/kbn-storybook"],
"@kbn/storybook/*": ["packages/kbn-storybook/*"],
"@kbn/streams-app-plugin": ["x-pack/solutions/observability/plugins/streams_app"],
"@kbn/streams-app-plugin/*": ["x-pack/solutions/observability/plugins/streams_app/*"],
"@kbn/streams-plugin": ["x-pack/solutions/observability/plugins/streams"],
"@kbn/streams-plugin/*": ["x-pack/solutions/observability/plugins/streams/*"],
"@kbn/streams-app-plugin": ["x-pack/platform/plugins/shared/streams_app"],
"@kbn/streams-app-plugin/*": ["x-pack/platform/plugins/shared/streams_app/*"],
"@kbn/streams-app-wrapper-plugin": ["x-pack/solutions/observability/plugins/observability_streams_wrapper"],
"@kbn/streams-app-wrapper-plugin/*": ["x-pack/solutions/observability/plugins/observability_streams_wrapper/*"],
"@kbn/streams-plugin": ["x-pack/platform/plugins/shared/streams"],
"@kbn/streams-plugin/*": ["x-pack/platform/plugins/shared/streams/*"],
"@kbn/streams-schema": ["x-pack/platform/packages/shared/kbn-streams-schema"],
"@kbn/streams-schema/*": ["x-pack/platform/packages/shared/kbn-streams-schema/*"],
"@kbn/styled-components-mapping-cli": ["packages/kbn-styled-components-mapping-cli"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,10 @@
module.exports = {
preset: '@kbn/test',
rootDir: '../../../../..',
roots: ['<rootDir>/x-pack/solutions/observability/plugins/streams'],
coverageDirectory:
'<rootDir>/target/kibana-coverage/jest/x-pack/solutions/observability/plugins/streams',
roots: ['<rootDir>/x-pack/platform/plugins/shared/streams'],
coverageDirectory: '<rootDir>/target/kibana-coverage/jest/x-pack/platform/plugins/shared/streams',
coverageReporters: ['text', 'html'],
collectCoverageFrom: [
'<rootDir>/x-pack/solutions/observability/plugins/streams/{common,public,server}/**/*.{js,ts,tsx}',
'<rootDir>/x-pack/platform/plugins/shared/streams/{common,public,server}/**/*.{js,ts,tsx}',
],
};
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
"id": "@kbn/streams-plugin",
"owner": "@simianhacker @flash1293 @dgieselaar",
"description": "A manager for Streams",
"group": "observability",
"visibility": "private",
"group": "platform",
"visibility": "shared",
"plugin": {
"id": "streams",
"configPath": ["xpack", "streams"],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
import { SanitizedRule } from '@kbn/alerting-plugin/common';
import { RulesClient } from '@kbn/alerting-plugin/server';
import { SavedObject, SavedObjectsClientContract } from '@kbn/core/server';
import { termQuery } from '@kbn/observability-utils-server/es/queries/term_query';
import { IStorageClient } from '@kbn/storage-adapter';
import { keyBy } from 'lodash';
import objectHash from 'object-hash';
import pLimit from 'p-limit';
import type { QueryDslQueryContainer } from '@elastic/elasticsearch/lib/api/types';
import {
ASSET_TYPES,
Asset,
Expand All @@ -24,6 +24,22 @@ import {
import { ASSET_ENTITY_ID, ASSET_ENTITY_TYPE, ASSET_TYPE } from './fields';
import { AssetStorageSettings } from './storage_settings';

interface TermQueryOpts {
queryEmptyString: boolean;
}

function termQuery<T extends string>(
field: T,
value: string | boolean | number | undefined | null,
opts: TermQueryOpts = { queryEmptyString: true }
): QueryDslQueryContainer[] {
if (value === null || value === undefined || (!opts.queryEmptyString && value === '')) {
return [];
}

return [{ term: { [field]: value } }];
}

function sloSavedObjectToAsset(
sloId: string,
savedObject: SavedObject<{ name: string; tags: string[] }>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import type { estypes } from '@elastic/elasticsearch';
import { fromKueryExpression, toElasticsearchQuery } from '@kbn/es-query';

function excludeTiersQuery(
excludedDataTiers: Array<'data_frozen' | 'data_cold' | 'data_warm' | 'data_hot'>
): estypes.QueryDslQueryContainer[] {
return [
{
bool: {
must_not: [
{
terms: {
_tier: excludedDataTiers,
},
},
],
},
},
];
}

export function excludeFrozenQuery(): estypes.QueryDslQueryContainer[] {
return excludeTiersQuery(['data_frozen']);
}

export function kqlQuery(kql?: string): estypes.QueryDslQueryContainer[] {
if (!kql) {
return [];
}

const ast = fromKueryExpression(kql);
return [toElasticsearchQuery(ast)];
}

export function rangeQuery(
start?: number,
end?: number,
field = '@timestamp'
): estypes.QueryDslQueryContainer[] {
return [
{
range: {
[field]: {
gte: start,
lte: end,
format: 'epoch_millis',
},
},
},
];
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,11 @@
* 2.0.
*/

import { excludeFrozenQuery } from '@kbn/observability-utils-common/es/queries/exclude_frozen_query';
import { kqlQuery } from '@kbn/observability-utils-common/es/queries/kql_query';
import { rangeQuery } from '@kbn/observability-utils-common/es/queries/range_query';
import { UnparsedEsqlResponse, createTracedEsClient } from '@kbn/traced-es-client';
import { z } from '@kbn/zod';
import { isNumber } from 'lodash';
import { createServerRoute } from '../create_server_route';
import { excludeFrozenQuery, kqlQuery, rangeQuery } from './query_helpers';

export const executeEsqlRoute = createServerRoute({
endpoint: 'POST /internal/streams/esql',
Expand All @@ -37,7 +35,7 @@ export const executeEsqlRoute = createServerRoute({
}),
handler: async ({ params, request, logger, getScopedClients }): Promise<UnparsedEsqlResponse> => {
const { scopedClusterClient } = await getScopedClients({ request });
const observabilityEsClient = createTracedEsClient({
const tracedEsClient = createTracedEsClient({
client: scopedClusterClient.asCurrentUser,
logger,
plugin: 'streams',
Expand All @@ -47,7 +45,7 @@ export const executeEsqlRoute = createServerRoute({
body: { operationName, query, filter, kuery, start, end },
} = params;

const response = await observabilityEsClient.esql(
const response = await tracedEsClient.esql(
operationName,
{
query,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,6 @@
"@kbn/licensing-plugin",
"@kbn/server-route-repository-client",
"@kbn/object-utils",
"@kbn/observability-utils-server",
"@kbn/observability-utils-common",
"@kbn/alerting-plugin",
"@kbn/std",
"@kbn/safer-lodash-set",
Expand All @@ -39,6 +37,7 @@
"@kbn/server-route-repository-utils",
"@kbn/inference-plugin",
"@kbn/storage-adapter",
"@kbn/traced-es-client"
"@kbn/traced-es-client",
"@kbn/es-query"
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ import { coreMock } from '@kbn/core/public/mocks';
import type { DataPublicPluginStart } from '@kbn/data-plugin/public';
import type { DataViewsPublicPluginStart } from '@kbn/data-views-plugin/public';
import type { StreamsPluginStart } from '@kbn/streams-plugin/public';
import type { ObservabilitySharedPluginStart } from '@kbn/observability-shared-plugin/public';
import type { UnifiedSearchPublicPluginStart } from '@kbn/unified-search-plugin/public';
import type { SharePublicStart } from '@kbn/share-plugin/public/plugin';
import { NavigationPublicStart } from '@kbn/navigation-plugin/public/types';
Expand All @@ -27,7 +26,6 @@ export function getMockStreamsAppContext(): StreamsAppKibanaContext {
core,
dependencies: {
start: {
observabilityShared: {} as unknown as ObservabilitySharedPluginStart,
dataViews: {} as unknown as DataViewsPublicPluginStart,
data: {} as unknown as DataPublicPluginStart,
unifiedSearch: {} as unknown as UnifiedSearchPublicPluginStart,
Expand All @@ -40,6 +38,7 @@ export function getMockStreamsAppContext(): StreamsAppKibanaContext {
},
services: {
dataStreamsClient: Promise.resolve({} as unknown as DataStreamsStatsClient),
PageTemplate: () => null,
},
isServerless: false,
};
Expand Down
23 changes: 23 additions & 0 deletions x-pack/platform/plugins/shared/streams_app/jest.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

module.exports = {
preset: '@kbn/test',
rootDir: '../../../../..',
roots: [
'<rootDir>/x-pack/platform/plugins/shared/streams_app/public',
'<rootDir>/x-pack/platform/plugins/shared/streams_app/common',
'<rootDir>/x-pack/platform/plugins/shared/streams_app/server',
],
setupFiles: ['<rootDir>/x-pack/platform/plugins/shared/streams_app/.storybook/jest_setup.js'],
collectCoverage: true,
collectCoverageFrom: [
'<rootDir>/x-pack/platform/plugins/shared/streams_app/{public,common,server}/**/*.{js,ts,tsx}',
],

coverageReporters: ['html'],
};
Original file line number Diff line number Diff line change
@@ -1,17 +1,16 @@
{
"type": "plugin",
"id": "@kbn/streams-app-plugin",
"owner": "@simianhacker @flash1293 @dgieselaar",
"group": "observability",
"visibility": "private",
"owner": "@elastic/streams-program-team",
"group": "platform",
"visibility": "shared",
"plugin": {
"id": "streamsApp",
"server": true,
"browser": true,
"configPath": ["xpack", "streamsApp"],
"requiredPlugins": [
"streams",
"observabilityShared",
"data",
"dataViews",
"unifiedSearch",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,13 @@
* 2.0.
*/
import React from 'react';
import ReactDOM from 'react-dom';
import { APP_WRAPPER_CLASS, type AppMountParameters, type CoreStart } from '@kbn/core/public';
import { type AppMountParameters, type CoreStart } from '@kbn/core/public';
import { KibanaRenderContextProvider } from '@kbn/react-kibana-context-render';
import { css } from '@emotion/css';
import type { StreamsAppStartDependencies } from './types';
import { StreamsAppServices } from './services/types';
import { AppRoot } from './components/app_root';

export const renderApp = ({
export const StreamsApplication = ({
coreStart,
pluginsStart,
services,
Expand All @@ -25,15 +23,7 @@ export const renderApp = ({
services: StreamsAppServices;
isServerless: boolean;
} & { appMountParameters: AppMountParameters }) => {
const { element } = appMountParameters;

const appWrapperClassName = css`
overflow: auto;
`;
const appWrapperElement = document.getElementsByClassName(APP_WRAPPER_CLASS)[1];
appWrapperElement.classList.add(appWrapperClassName);

ReactDOM.render(
return (
<KibanaRenderContextProvider {...coreStart}>
<AppRoot
appMountParameters={appMountParameters}
Expand All @@ -42,11 +32,6 @@ export const renderApp = ({
services={services}
isServerless={isServerless}
/>
</KibanaRenderContextProvider>,
element
</KibanaRenderContextProvider>
);
return () => {
ReactDOM.unmountComponentAtNode(element);
appWrapperElement.classList.remove(APP_WRAPPER_CLASS);
};
};
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ import {
RouteRenderer,
RouterProvider,
} from '@kbn/typed-react-router-config';
import { HeaderMenuPortal } from '@kbn/observability-shared-plugin/public';
import { EuiFlexGroup, EuiFlexItem } from '@elastic/eui';
import { StreamsAppContextProvider } from '../streams_app_context_provider';
import { streamsAppRouter } from '../../routes/config';
import { StreamsAppStartDependencies } from '../../types';
import { StreamsAppServices } from '../../services/types';
import { HeaderMenuPortal } from '../header_menu';

export function AppRoot({
coreStart,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,24 @@ import {
} from '@elastic/charts';
import { EuiFlexGroup, EuiFlexItem, EuiIcon, EuiSpacer } from '@elastic/eui';
import { i18n } from '@kbn/i18n';
import { getTimeZone } from '@kbn/observability-utils-browser/utils/ui_settings/get_timezone';
import { css } from '@emotion/css';
import { AbortableAsyncState } from '@kbn/react-hooks';
import type { UnparsedEsqlResponse } from '@kbn/traced-es-client';
import { IUiSettingsClient } from '@kbn/core/public';
import { UI_SETTINGS } from '@kbn/data-plugin/public';
import { esqlResultToTimeseries } from '../../util/esql_result_to_timeseries';
import { useKibana } from '../../hooks/use_kibana';
import { LoadingPanel } from '../loading_panel';

function getTimeZone(uiSettings?: IUiSettingsClient) {
const kibanaTimeZone = uiSettings?.get<'Browser' | string>(UI_SETTINGS.DATEFORMAT_TZ);
if (!kibanaTimeZone || kibanaTimeZone === 'Browser') {
return 'local';
}

return kibanaTimeZone;
}

const END_ZONE_LABEL = i18n.translate('xpack.streams.esqlChart.endzone', {
defaultMessage:
'The selected time range does not include this entire bucket. It might contain partial data.',
Expand Down
Loading