testament spec nimout too lax

[timotheecour]

nimout spec seems to just check that each line in nimout is a contiguous substring of a line in compilation output. This is way too lax for being usable, as shown here:

Example 1

discard """
  nimout: '''
foobar2
foobar
foobar
foobar2
oobar
ba
'''
"""

static: echo "foobar1"
static: echo "foobar2"

XDG_CONFIG_HOME= nim r testament/testament.nim r tests/misc/t11682b.nim

Current Output

PASS: tests/misc/t11682b.nim c

Expected Output

fails

Example 2

this can go horribly wrong, see #16698 (comment)

Possible Solution

• each line in nimout should be a complete line in compilation output (no substring match)
• it's ok to have unmatched lines in output (useful for new warnings/hints etc)
• order cannot change
• each matched line should only be used once

EDIT: this is maybe the thing to fix (in testament.nim):

proc greedyOrderedSubsetLines*(lhs, rhs: string): bool =
  ## returns true if each stripped line in `lhs` appears in rhs, using a greedy matching.
  let rhs = rhs.strip
  var currentPos = 0
  for line in lhs.strip.splitLines:
    currentPos = rhs.find(line.strip, currentPos)
    if currentPos < 0:
      return false
  return true

Additional Information

• 1.5.1 fd5c8ef
• marking high priority because test suite should have sane behavior
• not caused by testament: nimout now consistently uses nimoutCheck #16189 since I verified this was the case right before that PR too
• odd behavior noticed by @xflywind in some other PR

[ringabout]

Related

proc greedyOrderedSubsetLines*(lhs, rhs: string): bool =
  ## returns true if each stripped line in `lhs` appears in rhs, using a greedy matching.
  let rhs = rhs.strip
  var currentPos = 0
  for line in lhs.strip.splitLines:
    currentPos = rhs.find(line.strip, currentPos)
    if currentPos < 0:
      return false
  return true

[timotheecour]

indeed, was using an outdated proc before greedyOrderedSubsetLines was introduced; I fixed issue message.