for idaholab#415, beginning work on using DNS for populating hostname…

…s in netbox (work in progress, probably broken)

docs/asset-interaction-analysis.md

@@ -98,7 +98,7 @@ If the `NETBOX_AUTO_POPULATE` [environment variable in `./config/netbox-common.e
 
 However, careful consideration should be made before enabling this feature: the purpose of an asset management system is to document the intended state of a network: with Malcolm configured to populate NetBox with the live network state, a network misconfiguration fault could result in an **incorrect documented configuration**.
 
-Devices created using this autopopulate method will have their `status` field set to `staged`. It is recommended that users periodically review automatically-created devices for correctness and to fill in known details that couldn't be determined from network traffic. For example, the `manufacturer` field for automatically-created devices will be set based on the organizational unique identifier (OUI) determined from the first three bytes of the observed MAC address, which may not be accurate if the device's traffic was observed across a router. If possible, observed hostnames will be used in the naming of the automatically-created devices, falling back to the device manufacturer otherwise (e.g., `MYHOSTNAME @ 10.10.0.123` vs. `Schweitzer Engineering @ 10.10.0.123`).
+Devices created using this autopopulate method will include a `tags` value of `Autopopulated`. It is recommended that users periodically review automatically-created devices for correctness and to fill in known details that couldn't be determined from network traffic. For example, the `manufacturer` field for automatically-created devices will be set based on the organizational unique identifier (OUI) determined from the first three bytes of the observed MAC address, which may not be accurate if the device's traffic was observed across a router. If possible, observed hostnames will be used in the naming of the automatically-created devices, falling back to the device manufacturer otherwise (e.g., `MYHOSTNAME @ 10.10.0.123` vs. `Schweitzer Engineering @ 10.10.0.123`).
 
 Since device autocreation is based on IP address, information about network segments (IP [prefixes](https://docs.netbox.dev/en/stable/models/ipam/prefix/)) must be first [manually specified](#NetBoxPopManual) in NetBox in order for devices to be automatically populated. Users should populate the `description` field in the NetBox IPAM Prefixes data model to specify a name to be used for NetBox network segment autopopulation and enrichment, otherwise the IP prefix itself will be used.
 

logstash/pipelines/enrichment/21_netbox.conf

@@ -33,20 +33,19 @@ filter {
     # before we do the actual enrichments, we've got a few log types we can use to map IP addresses
     # to hostnames in autopopulation
 
-    # the "planned" status indicates that while we'll create the device
-    #   entry with an IP address and hostname, additional details (such
-    #   as the manufacturer based on MAC address) will need to be updated
-    #   later (also note the blank "target" which means this record will be
-    #   used to populate the netbox database, but nothing will actually
-    #   be stored in a field of the record itself as a result of this filter)
+    # for these ones while we'll create the device entry with an IP address
+    #   and hostname, additional details (such as the manufacturer based
+    #   on MAC address) will need to be updated later (also note the blank
+    #   "target" which means this record will be used to populate the netbox
+    #   database, but nothing will actually be stored in a field of the
+    #   record itself as a result of this filter)
 
     if ([dns][question][name]) and ([dns][resolved_ip]) {
       ruby {
         id => "ruby_netbox_enrich_dns_ip_to_host"
         path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
         script_params => {
           "lookup_type" => "ip_device"
-          "default_status" => "planned"
           "source" => "[dns][resolved_ip]"
           "source_hostname" => "[dns][question][name]"
           "enabled_env" => "NETBOX_ENRICHMENT"
@@ -69,7 +68,6 @@ filter {
         path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
         script_params => {
           "lookup_type" => "ip_device"
-          "default_status" => "planned"
           "source" => "[source][ip]"
           "source_hostname" => "[zeek][ntlm][hostname]"
           "enabled_env" => "NETBOX_ENRICHMENT"
@@ -93,7 +91,6 @@ filter {
           path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
           script_params => {
             "lookup_type" => "ip_device"
-            "default_status" => "planned"
             "source" => "[destination][ip]"
             "source_hostname" => "[zeek][ntlm][server_nb_computer_name]"
             "enabled_env" => "NETBOX_ENRICHMENT"
@@ -114,7 +111,6 @@ filter {
           path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
           script_params => {
             "lookup_type" => "ip_device"
-            "default_status" => "planned"
             "source" => "[destination][ip]"
             "source_hostname" => "[zeek][ntlm][server_dns_computer_name]"
             "enabled_env" => "NETBOX_ENRICHMENT"
@@ -139,7 +135,6 @@ filter {
           path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
           script_params => {
             "lookup_type" => "ip_device"
-            "default_status" => "planned"
             "source" => "[zeek][dhcp][assigned_addr]"
             "source_hostname" => "[zeek][dhcp][client_fqdn]"
             "enabled_env" => "NETBOX_ENRICHMENT"
@@ -160,7 +155,6 @@ filter {
           path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
           script_params => {
             "lookup_type" => "ip_device"
-            "default_status" => "planned"
             "source" => "[zeek][dhcp][assigned_addr]"
             "source_hostname" => "[zeek][dhcp][host_name]"
             "enabled_env" => "NETBOX_ENRICHMENT"
@@ -204,7 +198,6 @@ filter {
         path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
         script_params => {
           "lookup_type" => "ip_device"
-          "default_status" => "staged"
           "source" => "[source][ip]"
           "target" => "[source][device]"
           "source_oui" => "[source][oui]"
@@ -247,7 +240,6 @@ filter {
         path => "/usr/share/logstash/malcolm-ruby/netbox_enrich.rb"
         script_params => {
           "lookup_type" => "ip_device"
-          "default_status" => "staged"
           "source" => "[destination][ip]"
           "target" => "[destination][device]"
           "source_oui" => "[destination][oui]"

logstash/ruby/netbox_enrich.rb

@@ -253,12 +253,14 @@ def register(
 
   @nb_headers = { 'Content-Type': 'application/json' }.freeze
 
-  # for ip_device hash lookups, if a device is pulled out that has this status then
+  @device_tag_autopopulated = { 'slug': 'malcolm-autopopulated' }.freeze
+  # for ip_device hash lookups, if a device is pulled out that has one of these tags
   #   it should be *updated* instead of just created. this allows us to create even less-fleshed
   #   out device entries from things like DNS entries but then give more information (like
-  #   manufacturer) later on when actual traffic is observed
-  # TODO: this part is not done yet
-  @hostname_only_device_status = 'planned'.freeze
+  #   manufacturer) later on when actual traffic is observed. these values should match
+  #   what's in netbox/preload/tags.yml
+  @device_tag_manufacturer_unknown = { 'slug': 'manufacturer-unknown' }.freeze
+  @device_tag_hostname_unknown = { 'slug': 'hostname-unknown' }.freeze
 
 end
 
@@ -639,6 +641,7 @@ def autopopulate_devices(
   _autopopulate_oui = autopopulate_oui
   _autopopulate_manuf = nil
   _autopopulate_site = nil
+  _autopopulate_tags = [ @device_tag_autopopulated ]
 
   # if MAC is set but OUI is not, do a quick lookup
   if (!autopopulate_mac.nil? && !autopopulate_mac.empty?) &&
@@ -691,9 +694,15 @@ def autopopulate_devices(
                             :match => 0.0,
                             :vm => false,
                             :id => nil}
+    _autopopulate_tags << @device_tag_manufacturer_unknown
   end
   # puts('2. %{key}: %{found}' % { key: _autopopulate_oui, found: JSON.generate(_autopopulate_manuf) })
 
+
+  if autopopulate_hostname.to_s.empty?
+    _autopopulate_tags << @device_tag_hostname_unknown
+  end
+
   # make sure the site and role exists
   _autopopulate_site = lookup_or_create_site(autopopulate_default_site_name, nb)
   _autopopulate_role = lookup_or_create_role(autopopulate_default_role_name, nb)
@@ -709,6 +718,7 @@ def autopopulate_devices(
         _device_name = autopopulate_hostname.to_s.empty? ? "#{_autopopulate_manuf[:name]} @ #{ip_str}" : "#{autopopulate_hostname} @ #{ip_str}"
         _device_data = { :name => _device_name,
                          :site => _autopopulate_site[:id],
+                         :tags => _autopopulate_tags,
                          :status => autopopulate_default_status }
         if (_device_create_response = nb.post('virtualization/virtual-machines/', _device_data.to_json, @nb_headers).body) &&
            _device_create_response.is_a?(Hash) &&
@@ -739,6 +749,7 @@ def autopopulate_devices(
         if !_autopopulate_manuf.fetch(:id, nil)&.nonzero?
           # the manufacturer is still not found, create it
           _manuf_data = { :name => _autopopulate_manuf[:name],
+                          :tags => _autopopulate_tags,
                           :slug => _autopopulate_manuf[:name].to_url }
           if (_manuf_create_response = nb.post('dcim/manufacturers/', _manuf_data.to_json, @nb_headers).body) &&
              _manuf_create_response.is_a?(Hash)
@@ -769,6 +780,7 @@ def autopopulate_devices(
             # the device type is not found, create it
             _dtype_data = { :manufacturer => _autopopulate_manuf[:id],
                             :model => autopopulate_default_dtype,
+                            :tags => _autopopulate_tags,
                             :slug => autopopulate_default_dtype.to_url }
             if (_dtype_create_response = nb.post('dcim/device-types/', _dtype_data.to_json, @nb_headers).body) &&
                _dtype_create_response.is_a?(Hash) &&
@@ -787,6 +799,7 @@ def autopopulate_devices(
                              :device_type => _autopopulate_dtype[:id],
                              :role => _autopopulate_role[:id],
                              :site => _autopopulate_site[:id],
+                             :tags => _autopopulate_tags,
                              :status => autopopulate_default_status }
             if (_device_create_response = nb.post('dcim/devices/', _device_data.to_json, @nb_headers).body) &&
                _device_create_response.is_a?(Hash) &&
@@ -831,6 +844,8 @@ def autopopulate_prefixes(
   autopopulate_default_status,
   nb
 )
+  _autopopulate_tags = [ @device_tag_autopopulated ]
+
   _prefix_data = nil
   # TODO: IPv6?
   _private_ip_subnet = @private_ip_subnets.find { |subnet| subnet.include?(ip_obj) }
@@ -843,6 +858,7 @@ def autopopulate_prefixes(
     _autopopulate_site = lookup_or_create_site(autopopulate_default_site, nb)
     _prefix_post = { :prefix => _new_prefix_name,
                      :description => _new_prefix_name,
+                     :tags => _autopopulate_tags,
                      :site => _autopopulate_site&.fetch(:id, nil),
                      :status => autopopulate_default_status }
     begin

netbox/preload/tags.yml

@@ -0,0 +1,9 @@
+- name: Autopopulated
+  slug: malcolm-autopopulated
+  color: Light Blue
+- name: Manufacturer Unknown
+  slug: manufacturer-unknown
+  color: Light Grey
+- name: Hostname Unknown
+  slug: hostname-unknown
+  color: Light Grey