Associate Log File With Pester Output

[thealanagrace]

Summary of the feature request

I have a script that generates a log file. I want to associaite that log file with a specific test case.

My pester tests call my script and I pass in different inputs. The script returns a data object that I assert on. It also returns information about the log file. I would like to be able to include the log file as a property for the test-case in my output. I'm currently using NUnitXml but am open to switching if something like this exists somewhere else.

I'm running my pester tests as part of an Azure DevOps pipeline. After I run the Publish Test Results task, I would like to run another script that adds the log files as an attachment to each test. This way, when the a test fails I can easily view the logs for the script from the ADO test tab. Right now, it's not possible to know which log is associated with which test.

[fflaten]

Support for NUnit3 XML-report is being worked on. That schema includes an output-element for standard output from tests.

Would that work for you? You'd simply output it like:

It 'test1' {
    # do stuff
    
    # output path/filename for report
    "See Mylog123.txt for details"
    
    1 | Should -Be 1
}

You can already see this in StandardOutput-property per test in the pester result object.

[thealanagrace]

So, "See Mylog123.txt for details" would be included in the NUnit xml output? That would be great! Is there a time frame for when we can expect that support?

[nohwnd]

I think Pester 5.4 is where we could achieve NUnit 3 output, so probably with stable release in 3 months, but we can release a preview as soon as the code is in.

[thealanagrace]

Great! Please tag this feature when this is available to try!

[bormm]

So, "See Mylog123.txt for details" would be included in the NUnit xml output? That would be great! Is there a time frame for when we can expect that support?

How this would help you @alanafrankly with the Azure DevOps integration? The file is still not part of the test result.

If I see correctly, Pester 5.4 was until now not released so file attachment and also output this message is not possible still. I am right? @nohwnd @fflaten ?

[fflaten]

If I see correctly, Pester 5.4 was until now not released so file attachment and also output this message is not possible still. I am right? @nohwnd @fflaten ?

That's unfortunately correct. We've been occupied with work, so mostly answered issues the past year. Will hopefully get more time to work on Pester in the coming months.

[bormm]

I really appreciate your hard work and can totally understand that if no company is willing to pay you for Pester, its hard to work on it. I could add some support for stuff of course, like this Add-ItFileAttachment based on the code of Set-ItResult if I understand correctly, but if the current used xml format disallows it for other reasons, it would be hard for me to get that deep into Pester.

[bormm]

I checked the source of Set-ItResult and it seems not a good starting point. As its uses throw and not some kind of "current test context", that's completely different to what Add-ItFileAttachment need to do. Add-ItFileAttachment should not terminate the current test in any kind.

[nohwnd]

@bormm The initial idea was imho to add the name of the log file into NUnitXML as standard output so the file that is later uploaded can be correlated with a test result in the NUnitXML file. You could also output the whole log into the file.

[nohwnd]

I don't think there is a good starting point for uploading files into azdo that would exist in Pester. Azdo has multiple ways to upload files. You are most likely looking for this one, that works just by outputting a command to the host: https://docs.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash#addattachment-attach-a-file-to-the-build

[fflaten]

While looking into NUnit3 I made a quick PoC with NUnit3-attachment nodes:

<test-case name="foo-bar.my 2.two" fullname="foo-bar.my 2.two" methodname="foo-bar.my 2.two" classname="foo-bar.my 2.two" start-time="2022-07-07T10:20:02.2068015Z" end-time="2022-07-07T10:20:02.2188375Z" duration="0.012" result="Failed">
    <failure>
        <message>
            <![CDATA[ Expected 1, but got 2. ]]>
        </message>
        <stack-trace>
            <![CDATA[ at It 'two' -Tag 'special' -TestCases @( @{Test = 1; Expected = 1 },@{Test = 2; Expected = 1 }) { $Test | Should -Be $Expected }, /workspaces/Pester/Samples/demoTagFilter.tests.ps1:4 at <ScriptBlock>, /workspaces/Pester/Samples/demoTagFilter.tests.ps1:4 ]]>
        </stack-trace>
    </failure>
    <attachments>
        <attachment>
            <filePath>D:\a\1\s\demo.tests.ps1</filePath>
            <description>My Abosolute demo</description>
        </attachment>
        <attachment>
            <filePath>README.md</filePath>
            <description>My relative demo. Shouldn't work since it's not rooted, but it does..</description>
        </attachment>
    </attachments>
</test-case>

Shows up and download works in AzDO

So technically we could support it, but it would require some extension of the result-objects + new function or parameters to add them during the run (with absolute paths). So it will probably not be part of initial NUnit3-support.