Snapshot Repositories UI (#34407)

* [SR] Snapshot and restore plugin boilerplate (#32276)

* Initial plugin set up

* Set up client shell

* Add initial repository list routes

* Fix merge issues and some typings

* Decouple server from plugin.ts files, tighten up typings

* Use exported constant for required license

* Translate plugin name, more typings

* Fix more types, move list components under /home

* Remove unused var

* Change scss prefix

* Uncouple unmount logic from routing shim, and some other PR feedback

* [SR] Repository list and details UI (#33367)

* Initial pass at repositories list UI

* Add detail panel for file system repositories, and a generic detail panel with json settings view

* Add detail components for other types

* Add detail panel footer, rename `useStateValue` to `useAppState`

* Fix detail panel footer

* Fix unused vars

* PR feedback

* PR feedback

* [SR] Refactor proposal (#33690)

* Move app dependencies to its own context provider

* Add index.ts barrel file for common types

* Move Enums to constants.ts file

* Refactor function component using `React.FunctionComponent<Props>`

* Refactor service folder structure

* Fix type import

* Move REPOSITORY_DOC_PATHS from common to public constants

* Move AppCore and AppPlugins interfaces back to shim and re-export them from app types

* [SR] Create and edit repositories UI (#34020)

* Add routing and placeholder form

* Fix typings

* Set up edit repository route, and basic form UI

* Add typings for wrapCustomError, and copy extractCausedByChain from CCR wrapEsError

* Throw errors that are already boomified

* Create and edit for basic repository types (fs, url, source)

* Add repository verification UI to table and details

* Create and edit for plugin repository types (hdfs, azure, s3, gcs)

* Fix linting

* Fix test

* Fix test

* Remove unused import

* Fix duplicate i18n key

* Fix details opening on cancel edit, remove unnecessary Fragments, definition file for some EUI components to x-pack, rename saveError

* Remove breaks

* Adjust add and edit repo routes so they don't conflict with list route

* Add repo plugin and types doc links to form

* Bootstrap documentation service

* Bootstrap text service and replace RepositoryTypeName component with it

* Bootstrap breadcrumb service and replace usages

* Bootstrap httpService, remove chrome and http from app dependencies(!)

* Add request creator and replace all instances of useRequest and sendRequest with it

* Fix typo

* Simplify update repository and update repository setting methods

* Adjust copy

* Lint

* Remove unused var

* Remove unused import

* [SR] Add API for retrieving snapshots. (#34598)

* [SR] Single and multiple repository delete (#34593)

* Add single/multi repository delete API and UI

* Address PR feedback

* [SR] Add SnapshotTable and SnapshotDetails. (#34837)

* Remove associations between multiple repositories with a single snapshot.
* Retrieve complete snapshot details in getAllHandler.
* Fix cleanup function bug in useRequest hook.
* Fix bug in useRequest which prevented old data from being cleared when subsequent requests returned errors.
* Add initialValue config option to useRequest.
* Add formatDate service to text module.

* [SR] Fix linting and add (de)serialization for repositories (#35031)

* Fix eslint issues and add (de)serialization for repositories

* Add comment about flattening settings

* [SR] Surface repository errors and index failures more prominently (#35042)

* Add links to repositories from Snapshot Table and Snapshot Details.
- Rename services/breadcrumbs to services/navigation and add linkToRepository function.
- Refactor home component to update active tab when URL was changed.
* Add warning callout to let user know when their repositories contain errors.
* Sort failures by shard and add test for snapshot serialization.
* Sort failures and indices.
* Add filter for filtering snapshots by their repository.
* Surface states with humanized text, icons, and tooltips where necessary.
* Fix pluralization of seconds.
* Surface failures tab even if there are none.
- Display a '-' for missing times and durations.
- Create DataPlaceholder component.

* [SR] Polish repositories UX (#35123)

* Refactor repository detail panel to load repository based directly on route param.
* Display repository detail panel while table is loading.
* Make 'Edit repository' table action a link instead of a button.
* Render disabled EuiSelect as a readonly EuiFieldText.
* Prepend HDFS URI with hdfs:// protocol.
* Present scheme options for Read-Only URL repository as a select.

* [SR] Add client-side validation to repository form and link to snapshots from details (#35238)

* Add client side repository form validation, extract `flatten` into common lib

* Add snapshot count to repository details and link to snapshot list

* Reset validation when changing repository type

* Fix snapshot list filter deep linking for repository names with slashes and spaces

* Fix imports

* PR feedback

* [SR] Design and copywriting fixes (#35591)

* Split repository form into two steps; move `clean_settings.ts` to server

* Default to snapshots tab, adjust snapshot empty prompt, add app description

* Add minimum timeout to list view requests to avoid flicker, use EuiEmptyPrompt for loading screen, add doc link to settings step

* Add information about snapshots to delete repository behavior, add doc link for source only toggle, add size notation help text

* Add main doc link

* Copywriting and i18n fixes, and add some common settings to third party repo types

* Add fields to third party repo detail panel

* More copywriting fixes

* Use spinner for duration and end time if snapshotting is still in progress

* Show all repository type options, mark missing plugins

* Revert "Show all repository type options, mark missing plugins"

This reverts commit e34ee47.

* Fix space

* [SR] Add permissions UI and Cloud-specific repository type UI branch (#35833)

* Add missing permissions UI and cloud-specific repository type UI branch

* Add ES UI as owners of /snapshot_restore directory

* Add no repository types callout for Cloud edge case

* Redirect invalid section param to repositories

* Add warning empty prompt if all repositories have errrors

* Replace repository cards with EuiCard

* Add snapshot doc link to repository error empty prompt

* Remove auto-verification from list and get routes, add separate verification route, add manual verification to repository detail panel

* Update copy and remove obsolete test

* Remove unused scss files

* Final changes to repository cards

.github/CODEOWNERS

@@ -46,6 +46,7 @@
 /x-pack/plugins/remote_clusters/  @elastic/es-ui
 /x-pack/plugins/rollup/  @elastic/es-ui
 /x-pack/plugins/searchprofiler/  @elastic/es-ui
+/x-pack/plugins/snapshot_restore/  @elastic/es-ui
 /x-pack/plugins/watcher/  @elastic/es-ui
 
 # Kibana TSVB external contractors

.i18nrc.json

@@ -45,6 +45,7 @@
     "xpack.siem": "x-pack/plugins/siem",
     "xpack.security": "x-pack/plugins/security",
     "xpack.server": "x-pack/server",
+    "xpack.snapshotRestore": "x-pack/plugins/snapshot_restore",
     "xpack.spaces": "x-pack/plugins/spaces",
     "xpack.upgradeAssistant": "x-pack/plugins/upgrade_assistant",
     "xpack.uptime": "x-pack/plugins/uptime",

src/legacy/ui/public/management/index.d.ts

@@ -28,4 +28,8 @@ declare module 'ui/management' {
     allowOverride: boolean
   ): void;
   export const management: any; // TODO - properly provide types
+  export const MANAGEMENT_BREADCRUMB: {
+    text: string;
+    href: string;
+  };
 }

x-pack/index.js

@@ -40,6 +40,7 @@ import { upgradeAssistant } from './plugins/upgrade_assistant';
 import { uptime } from './plugins/uptime';
 import { ossTelemetry } from './plugins/oss_telemetry';
 import { encryptedSavedObjects } from './plugins/encrypted_saved_objects';
+import { snapshotRestore } from './plugins/snapshot_restore';
 
 module.exports = function (kibana) {
   return [
@@ -79,5 +80,6 @@ module.exports = function (kibana) {
     uptime(kibana),
     ossTelemetry(kibana),
     encryptedSavedObjects(kibana),
+    snapshotRestore(kibana),
   ];
 };

x-pack/plugins/cross_cluster_replication/server/routes/api/ccr.js

@@ -64,8 +64,8 @@ export const registerCcrRoutes = (server) => {
       }
 
       const securityInfo = (xpackInfo && xpackInfo.isAvailable() && xpackInfo.feature('security'));
-      if (!securityInfo || !securityInfo.isEnabled()) {
-        // If security isn't enabled, let the user use CCR.
+      if (!securityInfo || !securityInfo.isAvailable() || !securityInfo.isEnabled()) {
+        // If security isn't enabled or available (in the case where security is enabled but license reverted to Basic) let the user use CCR.
         return {
           hasPermission: true,
           missingClusterPrivileges: [],

x-pack/plugins/infra/types/eui.d.ts

@@ -116,6 +116,8 @@ declare module '@elastic/eui' {
     loading?: any;
     hasActions?: any;
     message?: any;
+    rowProps?: any;
+    cellProps?: any;
   };
   export const EuiInMemoryTable: React.SFC<EuiInMemoryTableProps>;
 }

x-pack/plugins/snapshot_restore/common/constants.ts

@@ -0,0 +1,53 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+import { LICENSE_TYPE_BASIC, LicenseType } from '../../../common/constants';
+import { RepositoryType } from './types';
+
+const PLUGIN_NAME = 'Snapshot Repositories';
+
+export const PLUGIN = {
+  ID: 'snapshot_restore',
+  MINIMUM_LICENSE_REQUIRED: LICENSE_TYPE_BASIC as LicenseType,
+  getI18nName: (translate: (key: string, config: object) => string): string => {
+    return translate('xpack.snapshotRestore.appName', {
+      defaultMessage: PLUGIN_NAME,
+    });
+  },
+};
+
+export const API_BASE_PATH = '/api/snapshot_restore/';
+
+export enum REPOSITORY_TYPES {
+  fs = 'fs',
+  url = 'url',
+  source = 'source',
+  s3 = 's3',
+  hdfs = 'hdfs',
+  azure = 'azure',
+  gcs = 'gcs',
+}
+
+// Deliberately do not include `source` as a default repository since we treat it as a flag
+export const DEFAULT_REPOSITORY_TYPES: RepositoryType[] = [
+  REPOSITORY_TYPES.fs,
+  REPOSITORY_TYPES.url,
+];
+
+export const PLUGIN_REPOSITORY_TYPES: RepositoryType[] = [
+  REPOSITORY_TYPES.s3,
+  REPOSITORY_TYPES.hdfs,
+  REPOSITORY_TYPES.azure,
+  REPOSITORY_TYPES.gcs,
+];
+
+export const REPOSITORY_PLUGINS_MAP: { [key: string]: RepositoryType } = {
+  'repository-s3': REPOSITORY_TYPES.s3,
+  'repository-hdfs': REPOSITORY_TYPES.hdfs,
+  'repository-azure': REPOSITORY_TYPES.azure,
+  'repository-gcs': REPOSITORY_TYPES.gcs,
+};
+
+export const APP_PERMISSIONS = ['monitor', 'create_snapshot', 'cluster:admin/repository'];

x-pack/plugins/snapshot_restore/common/lib/flatten.test.ts

@@ -0,0 +1,31 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+import { flatten } from './flatten';
+
+describe('flatten()', () => {
+  it('should return an empty object', () => {
+    expect(flatten({})).toEqual({});
+  });
+
+  it('should flatten a nested object', () => {
+    expect(
+      flatten({
+        foo: 'bar',
+        nested: {
+          test: 'value',
+          list: [{ fruit: 'apple' }, { fruit: 'banana' }],
+        },
+        'with.dot': 123,
+      })
+    ).toEqual({
+      foo: 'bar',
+      'nested.test': 'value',
+      'nested.list.0.fruit': 'apple',
+      'nested.list.1.fruit': 'banana',
+      'with.dot': 123,
+    });
+  });
+});

x-pack/plugins/snapshot_restore/common/lib/flatten.ts

@@ -0,0 +1,21 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export const flatten = (source: any, path: any[] = []): { [key: string]: any } => {
+  if (!(source instanceof Object)) {
+    return {
+      [path.join('.')]: source,
+    };
+  }
+
+  return Object.keys(source).reduce((result, key) => {
+    const flattened: any = flatten(source[key], [...path, key]);
+    return {
+      ...result,
+      ...flattened,
+    };
+  }, {});
+};

x-pack/plugins/snapshot_restore/common/lib/index.ts

@@ -0,0 +1,6 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+export { flatten } from './flatten';

x-pack/plugins/snapshot_restore/common/types/index.ts

@@ -0,0 +1,8 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export * from './repository';
+export * from './snapshot';

x-pack/plugins/snapshot_restore/common/types/repository.ts

@@ -0,0 +1,159 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export type FSRepositoryType = 'fs';
+export type ReadonlyRepositoryType = 'url';
+export type SourceRepositoryType = 'source';
+export type S3RepositoryType = 's3';
+export type HDFSRepositoryType = 'hdfs';
+export type AzureRepositoryType = 'azure';
+export type GCSRepositoryType = 'gcs';
+
+export type RepositoryType =
+  | FSRepositoryType
+  | ReadonlyRepositoryType
+  | SourceRepositoryType
+  | S3RepositoryType
+  | HDFSRepositoryType
+  | AzureRepositoryType
+  | GCSRepositoryType;
+
+export interface FSRepository {
+  name: string;
+  type: FSRepositoryType;
+  settings: {
+    location: string;
+    compress?: boolean;
+    chunkSize?: string | null;
+    maxRestoreBytesPerSec?: string;
+    maxSnapshotBytesPerSec?: string;
+    readonly?: boolean;
+  };
+}
+
+export interface ReadonlyRepository {
+  name: string;
+  type: ReadonlyRepositoryType;
+  settings: {
+    url: string;
+  };
+}
+
+export interface S3Repository {
+  name: string;
+  type: S3RepositoryType;
+  settings: {
+    bucket: string;
+    client?: string;
+    basePath?: string;
+    compress?: boolean;
+    chunkSize?: string | null;
+    serverSideEncryption?: boolean;
+    bufferSize?: string;
+    cannedAcl?: string;
+    storageClass?: string;
+    maxRestoreBytesPerSec?: string;
+    maxSnapshotBytesPerSec?: string;
+    readonly?: boolean;
+  };
+}
+
+export interface HDFSRepository {
+  name: string;
+  type: HDFSRepositoryType;
+  settings: {
+    uri: string;
+    path: string;
+    loadDefaults?: boolean;
+    compress?: boolean;
+    chunkSize?: string | null;
+    maxRestoreBytesPerSec?: string;
+    maxSnapshotBytesPerSec?: string;
+    readonly?: boolean;
+    ['security.principal']?: string;
+    [key: string]: any; // For conf.* settings
+  };
+}
+
+export interface AzureRepository {
+  name: string;
+  type: AzureRepositoryType;
+  settings: {
+    client?: string;
+    container?: string;
+    basePath?: string;
+    locationMode?: string;
+    compress?: boolean;
+    chunkSize?: string | null;
+    maxRestoreBytesPerSec?: string;
+    maxSnapshotBytesPerSec?: string;
+    readonly?: boolean;
+  };
+}
+
+export interface GCSRepository {
+  name: string;
+  type: GCSRepositoryType;
+  settings: {
+    bucket: string;
+    client?: string;
+    basePath?: string;
+    compress?: boolean;
+    chunkSize?: string | null;
+    maxRestoreBytesPerSec?: string;
+    maxSnapshotBytesPerSec?: string;
+    readonly?: boolean;
+  };
+}
+
+export interface EmptyRepository {
+  name: string;
+  type: null;
+  settings: {
+    [key: string]: any;
+  };
+}
+
+export interface SourceRepository<T> {
+  name: string;
+  type: SourceRepositoryType;
+  settings: SourceRepositorySettings<T>;
+}
+
+export type SourceRepositorySettings<T> = T extends FSRepositoryType
+  ? FSRepository['settings']
+  : T extends S3RepositoryType
+  ? S3Repository['settings']
+  : T extends HDFSRepositoryType
+  ? HDFSRepository['settings']
+  : T extends AzureRepositoryType
+  ? AzureRepository['settings']
+  : T extends GCSRepositoryType
+  ? GCSRepository['settings']
+  : any & {
+      delegateType: T;
+    };
+
+export type Repository<T = null> =
+  | FSRepository
+  | ReadonlyRepository
+  | S3Repository
+  | HDFSRepository
+  | AzureRepository
+  | GCSRepository
+  | SourceRepository<T>;
+
+export interface ValidRepositoryVerification {
+  valid: true;
+  response: object;
+}
+
+export interface InvalidRepositoryVerification {
+  valid: false;
+  error: object;
+}
+
+export type RepositoryVerification = ValidRepositoryVerification | InvalidRepositoryVerification;

x-pack/plugins/snapshot_restore/common/types/snapshot.ts

@@ -0,0 +1,31 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License;
+ * you may not use this file except in compliance with the Elastic License.
+ */
+
+export interface SnapshotDetails {
+  repository: string;
+  snapshot: string;
+  uuid: string;
+  versionId: number;
+  version: string;
+  indices: string[];
+  includeGlobalState: number;
+  state: string;
+  /** e.g. '2019-04-05T21:56:40.438Z' */
+  startTime: string;
+  startTimeInMillis: number;
+  /** e.g. '2019-04-05T21:56:45.210Z' */
+  endTime: string;
+  endTimeInMillis: number;
+  durationInMillis: number;
+  indexFailures: any[];
+  shards: SnapshotDetailsShardsStatus;
+}
+
+interface SnapshotDetailsShardsStatus {
+  total: number;
+  failed: number;
+  successful: number;
+}