[SIEM][Detection Engine] Switches actions/alerting on by default with…

… SIEM (elastic#51985) (elastic#52039)

## Summary

* Removes environment variables to have alerting/actions be defaulted on within SIEM
* Updates documentation
* Removes unit tests for it

### Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~

~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~

~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~

### For maintainers

~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

- [x] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)

x-pack/legacy/plugins/siem/index.ts

@@ -28,36 +28,13 @@ import {
 } from './common/constants';
 import { defaultIndexPattern } from './default_index_pattern';
 
-// This is VERY TEMPORARY as we need a way to turn on alerting and actions
-// for the server without having to manually edit this file. Once alerting
-// and actions has their enabled true by default this can be removed.
-// 'alerting', 'actions' are hidden behind feature flags at the moment so if you turn
-// these on without the feature flags turned on then Kibana will crash since we are a legacy plugin
-// and legacy plugins cannot have optional requirements.
-// This returns ['alerting', 'actions', 'kibana', 'elasticsearch'] iff alertingFeatureEnabled is true
-// or if the developer signalsIndex is setup. Otherwise this returns ['kibana', 'elasticsearch']
-export const getRequiredPlugins = (
-  alertingFeatureEnabled: string | null | undefined,
-  signalsIndex: string | null | undefined
-) => {
-  const baseRequire = ['kibana', 'elasticsearch'];
-  if (
-    (signalsIndex != null && signalsIndex.trim() !== '') ||
-    (alertingFeatureEnabled && alertingFeatureEnabled.toLowerCase() === 'true')
-  ) {
-    return [...baseRequire, 'alerting', 'actions'];
-  } else {
-    return baseRequire;
-  }
-};
-
 // eslint-disable-next-line @typescript-eslint/no-explicit-any
 export const siem = (kibana: any) => {
   return new kibana.Plugin({
     id: APP_ID,
     configPrefix: 'xpack.siem',
     publicDir: resolve(__dirname, 'public'),
-    require: getRequiredPlugins(process.env.ALERTING_FEATURE_ENABLED, process.env.SIGNALS_INDEX),
+    require: ['kibana', 'elasticsearch', 'alerting', 'actions'],
     uiExports: {
       app: {
         description: i18n.translate('xpack.siem.securityDescription', {

x-pack/legacy/plugins/siem/server/kibana.index.ts

@@ -42,21 +42,15 @@ export const initServerWithKibana = (
 
   const libs = compose(kbnServer, mode);
   initServer(libs);
-  if (
-    kbnServer.config().has('xpack.actions.enabled') &&
-    kbnServer.config().get('xpack.actions.enabled') === true &&
-    kbnServer.config().has('xpack.alerting.enabled') &&
-    kbnServer.config().has('xpack.alerting.enabled') === true
-  ) {
-    logger.info(
-      'Detected feature flags for actions and alerting and enabling detection engine API endpoints'
-    );
-    createRulesRoute(kbnServer);
-    readRulesRoute(kbnServer);
-    updateRulesRoute(kbnServer);
-    deleteRulesRoute(kbnServer);
-    findRulesRoute(kbnServer);
-  }
+
+  // Signals/Alerting Rules routes for
+  // routes such as ${DETECTION_ENGINE_RULES_URL}
+  // that have the REST endpoints of /api/detection_engine/rules
+  createRulesRoute(kbnServer);
+  readRulesRoute(kbnServer);
+  updateRulesRoute(kbnServer);
+  deleteRulesRoute(kbnServer);
+  findRulesRoute(kbnServer);
 
   const xpackMainPlugin = kbnServer.plugins.xpack_main;
   xpackMainPlugin.registerFeature({

x-pack/legacy/plugins/siem/server/lib/detection_engine/README.md

@@ -1,62 +1,20 @@
-Temporary README.md for users and developers working on the backend detection engine
-for how to get started.
+README.md for developers working on the backend detection engine on how to get started
+using the CURL scripts in the scripts folder.
 
-# Setup for Users
+The scripts rely on CURL and jq:
+* [CURL](https://curl.haxx.se)
+* [jq](https://stedolan.github.io/jq/)
 
-If you're just a user and want to enable the REST interfaces and UI screens do the following.
-NOTE: this is very temporary and once alerting and actions is enabled by default you will no
-longer have to do these steps
-
-Set the environment variable ALERTING_FEATURE_ENABLED to be true in your .profile or your windows
-global environment variable.
-
-```sh
-export ALERTING_FEATURE_ENABLED=true
-```
-
-In your `kibana.yml` file enable alerting and actions like so:
-
-```sh
-# Feature flag to turn on alerting
-xpack.alerting.enabled: true
-
-# Feature flag to turn on actions which goes with alerting
-xpack.actions.enabled: true
-```
-
-Start Kibana and you will see these messages indicating detection engine is activated like so:
-
-```sh
-server    log   [11:39:05.561] [info][siem] Detected feature flags for actions and alerting and enabling detection engine API endpoints
-```
-
-If you see crashes like this:
-
-```ts
- FATAL  Error: Unmet requirement "alerting" for plugin "siem"
-```
-
-It is because Kibana is not picking up your changes from `kibana.yml` and not seeing that alerting and actions is enabled.
-
-# For Developers
-
-See these two other pages for references:
-https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/alerting/README.md
-https://github.com/elastic/kibana/tree/master/x-pack/legacy/plugins/actions
-
-Since there is no UI yet and a lot of backend areas that are not created, you
-should install the kbn-action and kbn-alert project from here:
-https://github.com/pmuellr/kbn-action
-
-The scripts rely on CURL and jq, ensure both of these are installed:
 
+Install curl and jq
 ```sh
 brew update
 brew install curl
 brew install jq
 ```
 
-Open up your .zshrc/.bashrc and add these lines with the variables filled in:
+Open `$HOME/.zshrc` or `${HOME}.bashrc` depending on your SHELL output from `echo $SHELL`
+and add these environment variables:
 
 ```sh
 export ELASTICSEARCH_USERNAME=${user}
@@ -66,52 +24,30 @@ export KIBANA_URL=http://localhost:5601
 export SIGNALS_INDEX=.siem-signals-${your user id}
 export TASK_MANAGER_INDEX=.kibana-task-manager-${your user id}
 export KIBANA_INDEX=.kibana-${your user id}
-
-# This is for the kbn-action and kbn-alert tool
-export KBN_URLBASE=http://${user}:${password}@localhost:5601
 ```
 
-source your .zhsrc/.bashrc or open a new terminal to ensure you get the new values set.
-
-Optional env var when set to true will utilize `reindex` api for reindexing
-instead of the scroll and bulk index combination.
+source `$HOME/.zshrc` or `${HOME}.bashrc` to ensure variables are set:
 
 ```sh
-export USE_REINDEX_API=true
-```
-
-Add these lines to your `kibana.dev.yml` to turn on the feature toggles of alerting and actions:
-
-```sh
-# Feature flag to turn on alerting
-xpack.alerting.enabled: true
-
-# Feature flag to turn on actions which goes with alerting
-xpack.actions.enabled: true
+source ~/.zshrc
 ```
 
 Restart Kibana and ensure that you are using `--no-base-path` as changing the base path is a feature but will
 get in the way of the CURL scripts written as is. You should see alerting and actions starting up like so afterwards
 
 ```sh
-server    log   [22:05:22.277] [info][status][plugin:alerting@8.0.0] Status changed from uninitialized to green - Ready
-server    log   [22:05:22.270] [info][status][plugin:actions@8.0.0] Status changed from uninitialized to green - Ready
-```
-
-You should also see the SIEM detect the feature flags and start the API endpoints for detection engine
-
-```sh
-server    log   [11:39:05.561] [info][siem] Detected feature flags for actions and alerting and enabling detection engine API endpoints
+server log [22:05:22.277] [info][status][plugin:alerting@8.0.0] Status changed from uninitialized to green - Ready
+server log [22:05:22.270] [info][status][plugin:actions@8.0.0] Status changed from uninitialized to green - Ready
 ```
 
 Go into your SIEM Advanced settings and underneath the setting of `siem:defaultSignalsIndex`, set that to the same
-value as you did with the environment variable of SIGNALS_INDEX, which should be `.siem-signals-${your user id}`
+value as you did with the environment variable of `${SIGNALS_INDEX}`, which should be `.siem-signals-${your user id}`
 
 ```
 .siem-signals-${your user id}
 ```
 
-Open a terminal and go into the scripts folder `cd kibana/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts` and run:
+Go to the scripts folder `cd kibana/x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts` and run:
 
 ```sh
 ./hard_reset.sh
@@ -124,7 +60,7 @@ which will:
 - Delete any existing alerts you have
 - Delete any existing alert tasks you have
 - Delete any existing signal mapping you might have had.
-- Add the latest signal index and its mappings using your settings from `SIGNALS_INDEX` environment variable.
+- Add the latest signal index and its mappings using your settings from `${SIGNALS_INDEX}` environment variable.
 - Posts the sample rule from `rules/root_or_admin_1.json` by replacing its `output_index` with your `SIGNALS_INDEX` environment variable
 - The sample rule checks for root or admin every 5 minutes and reports that as a signal if it is a positive hit
 
@@ -181,21 +117,20 @@ You should see the new rules created like so:
 Every 5 minutes if you get positive hits you will see messages on info like so:
 
 ```sh
-server    log   [09:54:59.013] [info][plugins][siem] Total signals found from signal rule "id: a556065c-0656-4ba1-ad64-a77ca9d2013b", "ruleId: rule-1": 10000
+server log [09:54:59.013] [info][plugins][siem] Total signals found from signal rule "id: a556065c-0656-4ba1-ad64-a77ca9d2013b", "ruleId: rule-1": 10000
 ```
 
-Rules are space aware and default to the "default" space for these scripts if you do not export
-the variable of SPACE_URL. For example, if you want to post rules to the space `test-space` you would
-set your SPACE_URL to be:
+Rules are [space aware](https://www.elastic.co/guide/en/kibana/master/xpack-spaces.html) and default
+to the "default" (empty) URL space if you do not export the variable of `SPACE_URL`. Example, if you want to
+post rules to `test-space` you set `SPACE_URL` to be:
 
 ```sh
 export SPACE_URL=/s/test-space
 ```
 
-So that the scripts prepend a `/s/test-space` in front of all the APIs to correctly create, modify, delete, and update
-them from within that space.
-
-See the scripts folder and the tools for more command line fun.
+The  `${SPACE_URL}` is in front of all the APIs to correctly create, modify, delete, and update
+them from within the defined space. If this variable is not defined the default which is the url of an
+empty string will be used. 
 
 Add the `.siem-signals-${your user id}` to your advanced SIEM settings to see any signals
 created which should update once every 5 minutes at this point.
@@ -216,3 +151,7 @@ logging.events:
     ops: __no-ops__,
   }
 ```
+
+See these two README.md's pages for more references on the alerting and actions API:
+https://github.com/elastic/kibana/blob/master/x-pack/legacy/plugins/alerting/README.md
+https://github.com/elastic/kibana/tree/master/x-pack/legacy/plugins/actions