Merge branch 'master' into decrypted_check

README.md

@@ -3,7 +3,7 @@
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) 
 [![GitHub release](https://img.shields.io/github/v/tag/splunk/ansible-role-for-splunk?sort=semver&label=Version)](https://github.com/splunk/ansible-role-for-splunk/releases)
 
-This repository contains Splunk's official Ansible role for performing Splunk administration of remote hosts over SSH. This role can manage Splunk Enterprise and Universal Forwarders that are on Linux-based platforms (CentOS/Redhat/Ubuntu), as well as deploy configurations from Git repositories. Example playbooks and inventory files are also provided to help new Ansible users make the most out of this project.
+This repository contains Splunk's official Ansible role for performing Splunk administration of remote hosts over SSH. This role can manage Splunk Enterprise and Universal Forwarders that are on Linux-based platforms (CentOS/Redhat/Ubuntu/Amazon Linux/OpenSUSE), as well as deploy configurations from Git repositories. Example playbooks and inventory files are also provided to help new Ansible users make the most out of this project.
 
 ansible-role-for-splunk is used by the Splunk@Splunk team to manage Splunk's corporate deployment of Splunk.
 
@@ -125,7 +125,7 @@ This section contains additional reference documentation.
 
 Note: Any task with an **adhoc** prefix means that it can be used independently as a `deployment_task` in a playbook. You can use the tasks to resolve various Splunk problems or perform one-time activities, such as decommissioning an indexer from an indexer cluster.
 
-- **adhoc_clean_dispatch.yml** - This task is intended to be used for restoring service to search heads should the dispatch directory become full. You should need to use this task in a healthy environment, but it is at your disposal should the need arise. The task will stop splunk, remove all files in the dispatch directory, and then start splunk.
+- **adhoc_clean_dispatch.yml** - This task is intended to be used for restoring service to search heads should the dispatch directory become full. You should not need to use this task in a healthy environment, but it is at your disposal should the need arise. The task will stop splunk, remove all files in the dispatch directory, and then start splunk.
 - **adhoc_configure_hostname** - Configure a Splunk server's hostname using the value from inventory_hostname. It configures the system hostname, serverName in server.conf and host in inputs.conf. All Splunk configuration changes are made using the ini_file module, which will preserve any other existing configurations that may exist in server.conf and/or inputs.conf.
 - **adhoc_decom_indexer.yml** - Executes a splunk offline --enforce-counts command. This is useful when decommissioning one or more indexers from an indexer cluster.
 - **adhoc_fix_mongo.yml** - Use when Splunk is in a stopped state to fix mongodb/kvstore issues. This task ensures that permissions are set correctly on mongo's splunk.key file and deletes mongod.lock if it exists.
@@ -152,7 +152,10 @@ Note: Any task with an **adhoc** prefix means that it can be used independently
 - **configure_splunk_secret.yml** - Configures a common splunk.secret file from the files/authentication/splunk.secret so that pre-hashed passwords can be securely deployed. Note that changing splunk.secret will require re-encryption of any passwords that were encrypted using the previous splunk.secret since Splunk will no longer be able to decrypt them successfully.
 - **configure_systemd.yml** - Updates Splunk's systemd file using best practices and tips from the community. Also allows Splunk to start successfully using systemd after an upgrade without the need to run `splunk ftr --accept-license`.
 - **configure_thp.yml** - Installs a new systemd service (disable-thp) that disables THP for RedHat|CentOS systems 6.0+. This task is automatically called by the configure_os.yml task. Optionally, you can set `use_tuned_thp` to configure THP via `tuned` instead of a service. Default is `false`. Mote: Make sure your host does not require a specific `tuned` profile before applying this one.
-- **download_and_unarchive.yml** - Downloads the appropriate Splunk package to the Ansible host using `splunk_package_url` (derived automatically from the values of `splunk_package_url_full` or `splunk_package_url_uf` variables). The package is then installed to `splunk_install_path` (derived automatically in main.yml using the `splunk_install_path` and the host's membership of either a `uf` or `full` group in the inventory).
+- **download_and_unarchive.yml** - Downloads the appropriate Splunk package using `splunk_package_url` (derived automatically from the values of `splunk_package_url_full` or `splunk_package_url_uf` variables). The package is then installed to `splunk_install_path` (derived automatically in main.yml using the `splunk_install_path` and the host's membership of either a `uf` or `full` group in the inventory).  
+  You can set if the download/unarchive process uses the Ansible host or if each host downloads and unarchives the package individually by setting `splunk_download_local`.  
+  Default is `true` which will download the package to the Ansible host once and unarchive to each host from there.  
+  If set to `false` the package will be downloaded and unarchived to each host individually. Immediately after unarchive the package will be removed from the host.
 - **install_apps.yml** -  *Do not call install_apps.yml directly! Use configure_apps.yml* - Called by configure_apps.yml to perform app installation on the Splunk host.
 - **install_splunk.yml** - *Do not call install_splunk.yml directly! Use check_splunk.yml* - Called by check_splunk.yml to install/upgrade Splunk and Splunk Universal Forwarders, as well as perform any initial configurations. This task is called by check_splunk.yml when the check determines that Splunk is not currently installed. This task will create the splunk user and splunk group, configure the bash profile for the splunk user (by calling configure_bash.yml), configure THP and ulimits (by calling configure_os.ym), download and install the appropriate Splunk package (by calling download_and_unarchive.yml), configure a common splunk.secret (by calling configure_splunk_secret.yml, if configure_secret is defined), create a deploymentclient.conf file with the splunk_ds_uri and clientName (by calling configure_deploymentclient.yml, if clientName is defined), install a user-seed.conf with a prehashed admin password (if used_seed is defined), and will then call the post_install.yml task. See post_install.yml entry for details on post-installation tasks.
 - **install_utilities.yml** - Installs Linux packages that are useful for troubleshooting Splunk-related issues when `install_utilities: true` and `linux_packages` is defined with a list of packages to install.
@@ -165,7 +168,7 @@ Note: Any task with an **adhoc** prefix means that it can be used independently
 - **splunk_restart.yml**  - Restarts splunk via the service module. Used when waiting for a handler to run at the end of the play would be inappropriate.
 - **splunk_start.yml** - Starts splunk via the service module. Used when waiting for a handler to run at the end of the play would be inappropriate.
 - **splunk_stop.yml** - Stops splunk via the service module.  Used when waiting for a handler to run at the end of the play would be inappropriate.
-- **upgrade_splunk.yml** -  *Do not call upgrade_splunk.yml directly! Use check_splunk.yml* - Called by check_splunk.yml. Performs an upgrade of an existing splunk installation. Configures .bash_profile and .bashrc for splunk user (by calling configure_bash.yml), disables THP and increases ulimits (by calling configure_os.yml), kills any stale splunkd processes present (by calling adhoc_kill_splunkd.yml). Note: You should NOT run the upgrade_splunk.yml task directly from a playbook. check_splunk.yml will call upgrade_splunk.yml if it determines that an upgrade is needed; It will then download and unarchive the new version of Splunk (by calling download_and_unarchive.yml), ensure that mongod is in a good stopped state (by calling adhoc_fix_mongo.yml), and will then perform post-installation tasks using the post_install.yml task.
+- **upgrade_splunk.yml** -  *Do not call upgrade_splunk.yml directly! Use check_splunk.yml* - Called by check_splunk.yml. Performs an upgrade of an existing splunk installation. Configures .bash_profile and .bashrc for splunk user (by calling configure_bash.yml), disables THP and increases ulimits (by calling configure_os.yml), kills any stale splunkd processes present when `splunk_force_kill` is set to `True` (by calling adhoc_kill_splunkd.yml). Note: You should NOT run the upgrade_splunk.yml task directly from a playbook. check_splunk.yml will call upgrade_splunk.yml if it determines that an upgrade is needed; It will then download and unarchive the new version of Splunk (by calling download_and_unarchive.yml), ensure that mongod is in a good stopped state (by calling adhoc_fix_mongo.yml), and will then perform post-installation tasks using the post_install.yml task.
 
 ## Frequently Asked Questions
 **Q:** What is the difference between this and splunk-ansible?

environments/production/group_vars/heavyforwarder.yml

@@ -0,0 +1,6 @@
+---
+splunk_firewall_ports:
+  - "{{ splunkweb_port }}"
+  - "{{ splunkapi_port }}"
+  - "{{ splunktcpin_port }}"
+  - "{{ splunkhec_port }}"

environments/production/group_vars/indexer.yml

@@ -0,0 +1,7 @@
+---
+splunk_firewall_ports:
+  - "{{ splunkweb_port }}"
+  - "{{ splunkapi_port }}"
+  - "{{ splunktcpin_port }}"
+  - "{{ splunkhec_port }}"
+  - "{{ splunkidxcrep_port }}"

environments/production/group_vars/search.yml

@@ -0,0 +1,5 @@
+---
+splunk_firewall_ports:
+  - "{{ splunkweb_port }}"
+  - "{{ splunkapi_port }}"
+  - "{{ splunkshcrep_port }}"

roles/splunk/defaults/main.yml

@@ -6,15 +6,18 @@ slack_token: undefined
 splunk_home: auto_determined # This gets set by main.yml but we have to define it here or Ansible will complain that it is undefined
 splunk_package_url: auto_determined # This gets set by main.yml but we have to define it here or Ansible will complain that it is undefined
 splunk_package_path: ~/
-splunk_package_version: 9.0.0.1 # The default version to install or update to. This can be set in `group_vars` or `host_vars`.
-build_id: 9e907cedecb1 # The default build to install or update to. This can be set in `group_vars` or `host_vars`.
+splunk_package_version: 9.0.4 # The default version to install or update to. This can be set in `group_vars` or `host_vars`.
+build_id: de405f4a7979 # The default build to install or update to. This can be set in `group_vars` or `host_vars`.
 splunk_package_url_full: "https://download.splunk.com/products/splunk/releases/{{ splunk_package_version }}/linux/splunk-{{ splunk_package_version }}-{{ build_id }}-Linux-x86_64.tgz"
 splunk_package_url_uf: "https://download.splunk.com/products/universalforwarder/releases/{{ splunk_package_version }}/linux/splunkforwarder-{{ splunk_package_version }}-{{ build_id }}-Linux-x86_64.tgz"
+splunk_download_local: true  # This defines how the download process works. If `true` it will download to localhost and copy around to hosts from there. If `false` each host will download the package individually.
 splunk_install_type: undefined # There are two ways to configure this. The easiest way is to nest hosts under either a "full" group or a "uf" group in your inventory and main.yml will handle it for you. Or, you can also set the value via a group_vars or host_vars file.
 splunk_install_path: /opt # Base directory on the operating system to which splunk should be installed
 least_privileged: false # Do not change. This get automatically set in `tasks/main.yml` based on the version and install type.
 splunk_nix_user: splunk
 splunk_nix_group: splunk
+local_os_user: false # Whenther or not to force creation of a user using the `luseradd` or not.
+local_os_group: false # Whether or not to force creation of a group using the `lgroupadd` or not.
 splunk_uri_lm: undefined
 splunk_license_file: [] # This can be a list of license files to copy to the host.
 splunk_license_group: Trial # The default matches with the group splunk ships with. You can also set the value via a group_vars or host_vars file.
@@ -35,6 +38,7 @@ splunk_authenticationconf: authentication.conf.j2
 splunk_create_polkit: 0  # If set to 1 `enable boot-start` will create a polkit rules file allowing the 'splunk_nix_user' to restart the splunk service without authentication.
 splunk_use_initd: false  # If set to true, the system will use init.d. Default false
 splunk_use_systemd: true # DO NOT EDIT. To use init.d, set `splunk_use_initd` to true.
+splunk_force_kill: False
 systemd_unit_full: Splunkd # You can change this in `host_vars` or `group_vars` to customize the service name.
 systemd_unit_uf: SplunkForwarder # You can change this in `host_vars` or `group_vars` to customize the service name.
 splunk_disable_mgmt_port: false # If set to true, will disable splunkd management port during installation
@@ -72,19 +76,16 @@ add_pstack_script: false # Set to true to install a pstack generation script for
 configure_dmesg: false
 install_utilities: false # Set to true to install the list of packages defined in the linux_packages var after installing splunk
 use_tuned_thp: false
-linux_packages:
-  - nload
-  - iotop
-  - iftop
-  - sysstat
-  - telnet
-  - tcpdump
-  - htop
-  - atop
-  - lsof
-  - policycoreutils-python
-  - policycoreutils
-  - setroubleshoot
-  - nethogs
-  - gdb
-  - bind-utils
+# Firewall configs
+configure_firewall: false # Whether or not to configure the firewall service on your machine, if set to true, opens firewall ports using UFW (default) or Firewalld depending on OS
+splunk_firewall_service: splunk # The name of the Splunk firewall service to install for firewalld
+# Firewall port presets - reference these in group_vars to assign them to splunk
+splunkweb_port: {desc: "Splunk Web", protocol: "tcp", number: 8000}
+splunkhec_port: {desc: "Splunk HEC", protocol: "tcp", number: 8088}
+splunktcpin_port: {desc: "Splunk TCPIN", protocol: "tcp", number: 9997}
+splunkapi_port: {desc: "Splunk API", protocol: "tcp", number: "{{ splunkd_port }}"}
+splunkidxcrep_port: {desc: "Splunk Indexer Clustering Replication", protocol: "tcp", number: "{{ splunk_idxc_rep_port }}"}
+splunkshcrep_port: {desc: "Splunk Search Head Clustering Replication", protocol: "tcp", number: "{{ splunk_shc_rep_port }}"}
+splunk_firewall_ports: # List of ports to allow through local firewall in dict form
+  - "{{ splunkweb_port }}"
+  - "{{ splunkapi_port }}"

roles/splunk/handlers/main.yml

@@ -16,6 +16,7 @@
   when: splunk_use_initd and ansible_os_family == "RedHat" and ansible_distribution_major_version|int > 6
 
 - name: reload systemctl daemon
+  when: ansible_service_mgr == "systemd"
   systemd:
     daemon_reload: true
   become: true
@@ -72,7 +73,9 @@
   when: not start_splunk_handler_fired
 
 - name: restart redhat auditd service
-  command: service auditd condrestart
+  shell: |
+    service auditd stop
+    service auditd start
   become: true
   when: ansible_os_family == 'RedHat'
 
@@ -88,3 +91,7 @@
     port: "{{ splunkd_port }}"
     state: started
     delay: 5
+
+- name: reload firewalld
+  command: firewall-cmd --reload
+  become: true

roles/splunk/tasks/configure_authentication.yml

@@ -10,3 +10,4 @@
     - splunk_authenticationconf is defined
     - ad_bind_password != 'undefined'
   notify: restart splunk
+  no_log: true

roles/splunk/tasks/configure_deploymentclient.yml

@@ -59,3 +59,4 @@
   when: 
     - splunk_ds_key != 'undefined'
     - encrypted_secret_value.stdout == "" or (splunk_ds_key != decrypted_secret_value.stdout | default(''))
+  no_log: true

roles/splunk/tasks/configure_dmc.yml

@@ -12,6 +12,7 @@
   loop: "{{ query('inventory_hostnames', 'all:!indexer') }}"
   become: true
   become_user: "{{ splunk_nix_user }}"
+  no_log: true
 
 - name: Configure monitoring console in auto mode
   community.general.ini_file:

roles/splunk/tasks/configure_facl.yml

@@ -1,6 +1,11 @@
 ---
 - name: Configure file access control list (facl) settings for splunk user
   block:
+    - name: Install acl package
+      package:
+        name: "acl"
+        state: present
+      become: True
     - name: Set default facl to allow splunk user to read /var/log
       acl:
         path: /var/log
@@ -15,13 +20,12 @@
         - true
         - false
 
-    - name: Add logrotate script to enforce splunk user facls
-      template:
-        src: splunk_facl.j2
-        dest: /etc/logrotate.d/splunk_facl
-        owner: root
-        group: root
-      become: true
+    - name: Add setfacl to logrotate script
+      lineinfile:
+        path: "{{ logrotate_file }}"
+        insertbefore: 'endscript'
+        line: '        /usr/bin/setfacl -Rm u:{{ splunk_nix_user }}:rx /var/log'
+      become: True
 
     - name: Check if auditd.conf is present
       stat:
@@ -41,6 +45,4 @@
         - restart non-redhat auditd service
       when: result_auditd_conf.stat.exists
 
-  when:
-    - least_privileged == false or "'full' in group_names"
-    - splunk_nix_user != 'root'
+  when: not least_privileged

roles/splunk/tasks/configure_firewall.yml

@@ -0,0 +1,62 @@
+---
+- name: Ensure {{ firewall_service }} package is installed
+  ansible.builtin.package:
+    name: "{{ firewall_service }}"
+    state: present
+  become: true
+
+- name: Configure firewalld for Splunk
+  block:
+    - name: Ensure firewalld is Started and Enabled
+      ansible.builtin.systemd:
+        name: "{{ firewall_service }}"
+        state: started
+        enabled: true
+      become: true
+
+    - name: Add splunk firewalld service
+      ansible.builtin.template:
+        src: firewalld_service.xml.j2
+        dest: /etc/firewalld/services/{{ splunk_firewall_service }}.xml
+        backup: true
+        mode: 0644
+        owner: root
+        group: root
+      become: true
+      register: firewalld
+
+    - name: reload firewalld
+      command: firewall-cmd --reload
+      become: true
+      when: firewalld.changed
+
+    - name: Activate splunk firewalld service
+      ansible.posix.firewalld:
+        service: "{{ splunk_firewall_service }}"
+        permanent: true
+        state: enabled
+        immediate: true
+      notify: reload firewalld
+      become: true
+  when: firewall_service == "firewalld"
+
+- name: Configure UFW for Splunk
+  block:
+    - name: Ensure SSH is enabled
+      community.general.ufw:
+        port: 22
+        proto: tcp
+        rule: allow
+        state: enabled
+      become: true
+
+    - name: Add splunk port to UFW
+      community.general.ufw:
+        port: "{{ item.number }}"
+        proto: "{{ item.protocol }}"
+        rule: allow
+        state: reloaded
+        comment: "{{ item.desc | default('') }}"
+      become: true
+      loop: "{{ splunk_firewall_ports }}"
+  when: firewall_service == "ufw"

roles/splunk/tasks/configure_idxc_manager.yml

@@ -23,6 +23,7 @@
   notify: 
     - restart splunk
     - wait for splunkd
+  no_log: true
   loop:
     - { option: "mode", value: "{{ mode_value}}" }
     - { option: "replication_factor", value: "{{ splunk_idxc_rf }}" }

roles/splunk/tasks/configure_license.yml

@@ -5,7 +5,7 @@
 
 - name: Setting licensing mode based on Splunk version number
   set_fact:
-    mode_option: "{% if splunk_version_release | float < 9.0 %}master{% else %}manager{% endif %}_uri"
+    mode_option: "{{ splunk_version_release is version('9.0', '<') | ternary('master_uri', 'manager_uri') }}"
 
 - name: Configure Local License
   block:
@@ -30,7 +30,7 @@
     become: yes
     when:
       - splunk_license_group=="Enterprise"
-  - name: Remove master_uri when using local license
+  - name: "Remove {{ mode_option }} when using local license"
     ini_file:
       path: "{{ splunk_home }}/etc/system/local/server.conf"
       section: license
@@ -56,7 +56,7 @@
 
 - name: Configure License Peer
   block:
-  - name: Set license master_uri
+  - name: "Set license {{ mode_option }}"
     ini_file:
       path: "{{ splunk_home }}/etc/system/local/server.conf"
       section: license
@@ -71,11 +71,12 @@
       path: "{{ splunk_home }}/etc/system/local/server.conf"
       section: general
       option: pass4SymmKey
-      value: "{{ pass4SymmKey }}"
+      value: "{{ splunk_general_key }}"
       owner: "{{ splunk_nix_user }}"
       group: "{{ splunk_nix_group }}"
     become: yes
     notify: restart splunk
+    no_log: true
   when:
     - splunk_license_group=="Peer"
     - splunk_install_type=="full"