[READY] - openwrt: disable logd and enable prometheus #643

sarcasticadmin · 2023-11-28T06:06:42Z

Description of PR

Requires: #642

Relates to: #567 #33

Closes: #641 #613

Previous Behavior

openwrt: zabbix agent config was present
openwrt: logd was enabled
openwrt: lldp listening only on eth1

New Behavior

openwrt: prometheus is enabled
openwrt: logd is disabled
openwrt: full logger instead of busybox logger
openwrt: zabbix agent config removed
openwrt: lldp listening on all link interfaces
openwrt: ensure uhttpd default web server is disabled
serverspec: appropriate tests added for each new behavior

Tests

prometheus exporter working http://192.168.254.119:9100/metrics:

# TYPE node_scrape_collector_duration_seconds gauge
# TYPE node_scrape_collector_success gauge
# TYPE node_nf_conntrack_entries gauge
node_nf_conntrack_entries 0
# TYPE node_nf_conntrack_entries_limit gauge
node_nf_conntrack_entries_limit 15360
node_scrape_collector_duration_seconds{collector="conntrack"} 0.0032827854156494
node_scrape_collector_success{collector="conntrack"} 1
# TYPE node_boot_time_seconds gauge
node_boot_time_seconds 1700373908
# TYPE node_context_switches_total counter
node_context_switches_total 5273842
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="cpu0",mode="user"} 147.97
node_cpu_seconds_total{cpu="cpu0",mode="nice"} 0
node_cpu_seconds_total{cpu="cpu0",mode="system"} 137.69
node_cpu_seconds_total{cpu="cpu0",mode="idle"} 2167.55
node_cpu_seconds_total{cpu="cpu0",mode="iowait"} 0
node_cpu_seconds_total{cpu="cpu0",mode="irq"} 0
node_cpu_seconds_total{cpu="cpu0",mode="softirq"} 41.08
node_cpu_seconds_total{cpu="cpu0",mode="steal"} 0
node_cpu_seconds_total{cpu="cpu0",mode="guest"} 0
node_cpu_seconds_total{cpu="cpu0",mode="guest_nice"} 0
...

output is truncated

lldpcli show neighbors details

Interface:    enp7s0f4u2, via: LLDP, RID: 7, Time: 0 day, 00:36:49
  Chassis:
    ChassisID:    mac c4:04:15:a1:19:81
    SysName:      OpenWrt
    SysDescr:     OpenWrt System
    MgmtIP:       192.168.255.76
    MgmtIface:    24
    MgmtIP:       fe80::c604:15ff:fea1:1981
    MgmtIface:    2
    Capability:   Bridge, on
    Capability:   Router, on
    Capability:   Wlan, off
    Capability:   Station, off
  Port:
    PortID:       mac c4:04:15:a1:19:81
    PortDescr:    eth0
    TTL:          120
    PMD autoneg:  supported: yes, enabled: yes
      Adv:          1000Base-T, HD: no, FD: yes
      MAU oper type: 1000BaseTFD - Four-pair Category 5 UTP, full duplex mode
  VLAN:         103, pvid: no vlan103
  VLAN:         503, pvid: no vlan503
  VLAN:         101, pvid: no vlan101
  VLAN:         501, pvid: no vlan501
  VLAN:         100, pvid: no vlan100
  VLAN:         500, pvid: no vlan500
  VLAN:         108, pvid: no vlan108
  VLAN:         508, pvid: no vlan508
  LLDP-MED:
    Device Type:  Network Connectivity Device
    Capability:   Capabilities, yes
    Capability:   Policy, yes
    Capability:   Location, yes
    Capability:   MDI/PSE, yes
    Capability:   MDI/PD, yes
    Capability:   Inventory, yes
    Inventory:
      Software Revision: 5.15.138

serverspec mt7622 (belkin)

$ rake spec TEST_TYPE=openwrt TARGET_HOST=192.168.254.107
/nix/store/5xbwwbyjmc1xvjzhghk6r89rn4ylidv8-ruby-3.1.4/bin/ruby -I/nix/store/ca0nxdmn1qmaz7bravi0wdhyzgs9hh3a-serverspec/lib/ruby/gems/3.1.0/gems/rspec-core-3.12.2/lib:/nix/store/ca0nxdmn1qmaz7bravi0wdhyzgs9hh3a-serverspec/lib/ruby/gems/3.1.0/gems/rspec-support-3.12.1/lib /nix/store/ipdqksw2g3hphfazb35ffnrdnlgy86cv-ruby3.1.4-rspec-core-3.12.2/lib/ruby/gems/3.1.0/gems/rspec-core-3.12.2/exe/rspec --pattern spec/openwrt/\*_spec.rb

Image info:
Linux OpenWrt 5.15.138 #0 SMP Thu Jan 11 08:14:23 2024 aarch64 GNU/Linux
SCALE_VER=c2213841c9ada51535faf261d09111ca02abf60e
OPENWRT_VER=9062e5faaedc03823ee419fe34de1de73f48babc
BUILD_ID="r0-2da2393"
OPENWRT_BOARD="mediatek/mt7622"
OPENWRT_ARCH="aarch64_cortex-a53"

shared
  Command "which apinger 2> /dev/null"
    exit_status
      is expected to eq 0
  Command "which awk 2> /dev/null"
    exit_status
      is expected to eq 0
  Command "which bash 2> /dev/null"
    exit_status
      is expected to eq 0
  Command "which logrotate 2> /dev/null"
    exit_status
      is expected to eq 0
  Command "which rsyslogd 2> /dev/null"
    exit_status
      is expected to eq 0
  Command "which tcpdump 2> /dev/null"
    exit_status
      is expected to eq 0
  Command "which snmpd 2> /dev/null"
    exit_status
      is expected to eq 1
  Command "which dropbear 2> /dev/null"
    exit_status
      is expected to eq 1
  Command "which logd 2> /dev/null"
    exit_status
      is expected to eq 1
  Command "pgrep apinger"
    exit_status
      is expected to eq 0
  Command "pgrep crond"
    exit_status
      is expected to eq 0
  Command "pgrep rsyslogd"
    exit_status
      is expected to eq 0
  Command "pgrep lldpd"
    exit_status
      is expected to eq 0
  Command "pgrep ntpd"
    exit_status
      is expected to eq 0
  Port "80"
    is expected not to be listening
  Port "9100"
    is expected to be listening
  Command "rsyslogd -N1"
    exit_status
      is expected to eq 0
  Command "logger "serverspec test msg""
    exit_status
      is expected to eq 0
  File "/root/bin/wifi-details.sh"
    is expected to exist
    is expected to be mode 750
    is expected to be owned by "root"
    is expected to be grouped into "root"
  File "/root/bin/config-version.sh"
    is expected to exist
    is expected to be mode 750
    is expected to be owned by "root"
    is expected to be grouped into "root"
  Command "/root/bin/config-version.sh"
    exit_status
      is expected to eq 0
  Command "/root/bin/config-version.sh -c 9999"
    exit_status
      is expected to eq 1
  File "/etc/scale-release"
    is expected to exist
    is expected to be mode 644
    is expected to be owned by "root"
    is expected to be grouped into "root"
  Command "source /etc/scale-release && test -z $SCALE_VER"
    exit_status
      is expected to eq 1
  Command "source /etc/scale-release && test -z $OPENWRT_VER"
    exit_status
      is expected to eq 1
  File "/tmp/resolv.conf.d/resolv.conf.auto"
    is expected to exist
    is expected to be mode 644
    is expected to be owned by "root"
    is expected to be grouped into "root"
  File "/etc/resolv.conf"
    is expected to exist
    is expected to be symlink
    is expected to be owned by "root"
    is expected to be grouped into "root"
  File "/etc/config/network"
    is expected to exist
    is expected to be symlink
    is expected to be owned by "root"
    is expected to be grouped into "root"
  File "/etc/config/wireless"
    is expected to exist
    is expected to be symlink
    is expected to be owned by "root"
    is expected to be grouped into "root"
  correct_num_configs
    should always be equal
  ensure_dhcp_client_options
    should contain the following options
  Command "cat /etc/apinger.conf | grep "^target \"$(ip route | grep default | cut -d ' ' -f 3)\"""
    exit_status
      is expected to eq 0
  Command "wifi status | jq '.[] | select(.up == false )' | wc -l"
    stdout
      is expected to eq "0\n"
  Command "awk -F: -v user='root' '$1 == user {print $NF}' /etc/passwd"
    stdout
      is expected to match /\/bin\/bash/
  ensure_admin_ssh_key_present
    should match the following key fingerprint

Finished in 9.34 seconds (files took 1.1 seconds to load)
56 examples, 0 failures

serverspec ar71xx (netgear)

rake spec TEST_TYPE=openwrt TARGET_HOST=192.168.254.119
/nix/store/5xbwwbyjmc1xvjzhghk6r89rn4ylidv8-ruby-3.1.4/bin/ruby -I/nix/store/ca0nxdmn1qmaz7bravi0wdhyzgs9hh3a-serverspec/lib/ruby/gems/3.1.0/gems/rspec-core-3.12.2/lib:/nix/store/ca0nxdmn1qmaz7bravi0wdhyzgs9hh3a-serverspec/lib/ruby/gems/3.1.0/gems/rspec-support-3.12.1/lib /nix/store/ipdqksw2g3hphfazb35ffnrdnlgy86cv-ruby3.1.4-rspec-core-3.12.2/lib/ruby/gems/3.1.0/gems/rspec-core-3.12.2/exe/rspec --pattern spec/openwrt/\*_spec.rb

Image info:
Linux OpenWrt 5.15.138 #0 Sun Nov 19 06:05:22 2023 mips GNU/Linux
SCALE_VER=c2213841c9ada51535faf261d09111ca02abf60e
OPENWRT_VER=9062e5faaedc03823ee419fe34de1de73f48babc
BUILD_ID="r0-2da2393"
OPENWRT_BOARD="ath79/generic"
OPENWRT_ARCH="mips_24kc"

shared
 Command "which apinger 2> /dev/null"
   exit_status
     is expected to eq 0
 Command "which awk 2> /dev/null"
   exit_status
     is expected to eq 0
 Command "which bash 2> /dev/null"
   exit_status
     is expected to eq 0
 Command "which logrotate 2> /dev/null"
   exit_status
     is expected to eq 0
 Command "which rsyslogd 2> /dev/null"
   exit_status
     is expected to eq 0
 Command "which tcpdump 2> /dev/null"
   exit_status
     is expected to eq 0
 Command "which snmpd 2> /dev/null"
   exit_status
     is expected to eq 1
 Command "which dropbear 2> /dev/null"
   exit_status
     is expected to eq 1
 Command "which logd 2> /dev/null"
   exit_status
     is expected to eq 1
 Command "pgrep apinger"
   exit_status
     is expected to eq 0
 Command "pgrep crond"
   exit_status
     is expected to eq 0
 Command "pgrep rsyslogd"
   exit_status
     is expected to eq 0
 Command "pgrep lldpd"
   exit_status
     is expected to eq 0
 Command "pgrep ntpd"
   exit_status
     is expected to eq 0
 Port "80"
   is expected not to be listening
 Port "9100"
   is expected to be listening
 Command "rsyslogd -N1"
   exit_status
     is expected to eq 0
 Command "logger "serverspec test msg""
   exit_status
     is expected to eq 0
 File "/root/bin/wifi-details.sh"
   is expected to exist
   is expected to be mode 750
   is expected to be owned by "root"
   is expected to be grouped into "root"
 File "/root/bin/config-version.sh"
   is expected to exist
   is expected to be mode 750
   is expected to be owned by "root"
   is expected to be grouped into "root"
 Command "/root/bin/config-version.sh"
   exit_status
     is expected to eq 0
 Command "/root/bin/config-version.sh -c 9999"
   exit_status
     is expected to eq 1
 File "/etc/scale-release"
   is expected to exist
   is expected to be mode 644
   is expected to be owned by "root"
   is expected to be grouped into "root"
 Command "source /etc/scale-release && test -z $SCALE_VER"
   exit_status
     is expected to eq 1
 Command "source /etc/scale-release && test -z $OPENWRT_VER"
   exit_status
     is expected to eq 1
 File "/tmp/resolv.conf.d/resolv.conf.auto"
   is expected to exist
   is expected to be mode 644
   is expected to be owned by "root"
   is expected to be grouped into "root"
 File "/etc/resolv.conf"
   is expected to exist
   is expected to be symlink
   is expected to be owned by "root"
   is expected to be grouped into "root"
 File "/etc/config/network"
   is expected to exist
   is expected to be symlink
   is expected to be owned by "root"
   is expected to be grouped into "root"
 File "/etc/config/wireless"
   is expected to exist
   is expected to be symlink
   is expected to be owned by "root"
   is expected to be grouped into "root"
 correct_num_configs
   should always be equal
 ensure_dhcp_client_options
   should contain the following options
 Command "cat /etc/apinger.conf | grep "^target \"$(ip route | grep default | cut -d ' ' -f 3)\"""
   exit_status
     is expected to eq 0
 Command "wifi status | jq '.[] | select(.up == false )' | wc -l"
   stdout
     is expected to eq "0\n"
 Command "awk -F: -v user='root' '$1 == user {print $NF}' /etc/passwd"
   stdout
     is expected to match /\/bin\/bash/
 ensure_admin_ssh_key_present
   should match the following key fingerprint

Finished in 11.15 seconds (files took 2.22 seconds to load)
56 examples, 0 failures

rsyslog is already being used, this was incorrectly re-enabled after the shuffling of our configs to support the mt7622.

This way we dont have another regression and accidentally find logd installed.

Some pkgs from opkg we require but dont need their default service to start. There isnt a clear way to do this at build time so setting it via rc.local seems to be the most straight forward approach uhttpd is an example of such a service where its need for prometheus but not by itself.

the default busybox logger was erroring: logger: applet not found This could have been due to disabled logd but regardless going to opt to have a full logger via the busybox variant.

Removed various other openwrt default config options since many of the lldp-like modes do not apply to our network. location option also didnt seem to show up in my "lldpcli neighbors show detail" output so removing for now.

sarcasticadmin · 2024-01-11T22:45:15Z

This is ready to go.

Ill probably have more follow up regarding the wifi configs for the mt7622 but Id like to get these in first since its getting pretty large.

openwrt/files/etc/config/lldpd

owendelong

Excellent work, Rob.

sarcasticadmin added 3 commits November 29, 2023 04:29

openwrt: enable prometheus exporter

b9fafdb

openwrt: disable busybox logd

722c5f9

rsyslog is already being used, this was incorrectly re-enabled after the shuffling of our configs to support the mt7622.

openwrt: serverspec test to ensure logd is removed

a262c1f

This way we dont have another regression and accidentally find logd installed.

sarcasticadmin force-pushed the rh/1701148847prom branch from a24d3a4 to a262c1f Compare November 29, 2023 04:29

sarcasticadmin added 6 commits January 10, 2024 08:58

openwrt: prometheus listen on mgmt interface

deff7d5

openwrt: init logger over busybox

1b89985

the default busybox logger was erroring: logger: applet not found This could have been due to disabled logd but regardless going to opt to have a full logger via the busybox variant.

openwrt: remove zabbix agent config

7ef6107

openwrt: lldpd listen on all interfaces

0819c3f

Removed various other openwrt default config options since many of the lldp-like modes do not apply to our network. location option also didnt seem to show up in my "lldpcli neighbors show detail" output so removing for now.

openwrt: small note about mt7622 upgrade behavior

30607f2

sarcasticadmin force-pushed the rh/1701148847prom branch from 67ff0f8 to 30607f2 Compare January 11, 2024 03:22

sarcasticadmin added 4 commits January 11, 2024 08:08

serverspec: prometheus exporter and logger test

d2c9cf8

openwrt: logd service disabled

4096a11

openwrt-golden: make test and update behaviors exclusive

945488e

openwrt-golden: updates to ar71xx output

0f1646d

sarcasticadmin force-pushed the rh/1701148847prom branch from 8f22a2a to 0f1646d Compare January 11, 2024 22:42

sarcasticadmin changed the title ~~[WAIT] - openwrt: disable logd and enable prometheus~~ [ READY] - openwrt: disable logd and enable prometheus Jan 11, 2024

sarcasticadmin requested review from owendelong, kylerisse and davidelang January 11, 2024 22:43

sarcasticadmin changed the title ~~[ READY] - openwrt: disable logd and enable prometheus~~ [READY] - openwrt: disable logd and enable prometheus Jan 11, 2024

owendelong reviewed Jan 11, 2024

View reviewed changes

openwrt/files/etc/config/lldpd Show resolved Hide resolved

owendelong approved these changes Jan 11, 2024

View reviewed changes

owendelong merged commit c675dc2 into master Jan 11, 2024
3 checks passed

owendelong deleted the rh/1701148847prom branch January 11, 2024 23:50

This was referenced Jan 12, 2024

[READY] - nix pkg for serverspec #646

Merged

disable logd in openwrt builds #613

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[READY] - openwrt: disable logd and enable prometheus #643

[READY] - openwrt: disable logd and enable prometheus #643

sarcasticadmin commented Nov 28, 2023 •

edited

Loading

sarcasticadmin commented Jan 11, 2024

owendelong left a comment

[READY] - openwrt: disable logd and enable prometheus #643

[READY] - openwrt: disable logd and enable prometheus #643

Conversation

sarcasticadmin commented Nov 28, 2023 • edited Loading

Description of PR

Previous Behavior

New Behavior

Tests

sarcasticadmin commented Jan 11, 2024

owendelong left a comment

Choose a reason for hiding this comment

sarcasticadmin commented Nov 28, 2023 •

edited

Loading