Sanitycheck: Support filtering based on DTS configuration data

[nashif]

Cleanup some hardcoded samples and depend on DTS data instead making the sample
more generic.

[zephyrbot]

All checks are passing now.

Review history of this comment for details about previous failed status.
Note that some checks might have not completed yet.

[codecov-io]

Codecov Report

Merging #11800 into master will increase coverage by 0.24%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #11800      +/-   ##
==========================================
+ Coverage   48.05%    48.3%   +0.24%     
==========================================
  Files         281      280       -1     
  Lines       43413    43326      -87     
  Branches    10404    10373      -31     
==========================================
+ Hits        20864    20929      +65     
+ Misses      18400    18245     -155     
- Partials     4149     4152       +3

Impacted Files	Coverage Δ
subsys/shell/shell_utils.h	50% <0%> (-30%)	⬇️
drivers/clock_control/nrf5_power_clock.c	40.17% <0%> (-11.93%)	⬇️
lib/cmsis_rtos_v1/cmsis_semaphore.c	60% <0%> (-10%)	⬇️
subsys/shell/shell_ops.c	19.11% <0%> (-3.96%)	⬇️
subsys/shell/shell_utils.c	40.41% <0%> (-3.59%)	⬇️
subsys/logging/log_core.c	71.24% <0%> (-3.43%)	⬇️
drivers/ethernet/eth_native_posix.c	23.62% <0%> (-3.17%)	⬇️
subsys/net/ip/ipv4.c	64.1% <0%> (-1.33%)	⬇️
subsys/net/ip/ipv6.c	54.23% <0%> (-0.69%)	⬇️
include/logging/log_backend.h	75% <0%> (-0.68%)	⬇️
... and 46 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fbcf03...f3f741b. Read the comment docs.

[andrewboie]

LGTM, would like @galak blessing

[galak]

I want to think about this a little before approving/merging, not sure what the implications are we remove dt fixup, move to codegen, etc.

[nashif]

I want to think about this a little before approving/merging, not sure what the implications are we remove dt fixup, move to codegen, etc.

we can always change to whatever method later and when such changes come, right now we are just checking for variables that are being used in samples anyways and those would need to be changed in any case depending on how we move forward.
Right now the hardcoding in samples and coverage of tests is very limited because we have no other way of filtering.

[dbkinder]

This is contradictory, so let's simplify and say:

This sample may need some code changes to run on specific
boards by setting the PORT, LED0, and LED1 according to
the board's GPIO configuration.  As examples, see the instructions
below for some of our supported boards.

addressed

[galak]

Is there some way we can convey these filters are coming from DT data? Would be helpful when we replace this with eDTS info that we know that. Not sure if adding a 'dt_filter' in addition to 'filter'.

[nashif]

Is there some way we can convey these filters are coming from DT data? Would be helpful when we replace this with eDTS info that we know that. Not sure if adding a 'dt_filter' in addition to 'filter'.

no, there is no way I can think of, the filter also supports environment variables, some variables from cmake, Kconfig etc. So DT is not special here.

We should not block this because of EDTS, unless we have a clear idea and a plan about when EDTS is landing.

[nashif]

Is there some way we can convey these filters are coming from DT data? Would be helpful when we replace this with eDTS info that we know that. Not sure if adding a 'dt_filter' in addition to 'filter'.

also, Kconfig is always prefix with CONFIG_, DT stuff will hopefully all move to DT_, there will be some aliases that will have neither, those will be assumed to be either from DTS or env. The only problematic ones will be CONFIG_ variables that come from DT, but those are going away slowly.

addressed