Set-output Truncates Multiline Strings

[jacderida]

Hi,

I am using this code to create a release for one of our repositories:

- shell: bash
        id: release_description
        run: |
          description=$(./resources/get_release_description.sh ${{ steps.versioning.outputs.cli_version }})
          echo $description
          echo "::set-output name=description::$description"
        if: startsWith(steps.commit_message.outputs.commit_message, 'Version change')
      - shell: bash
        run: |
          echo ${{ steps.release_description.outputs.description }}
        if: startsWith(steps.commit_message.outputs.commit_message, 'Version change')
      - uses: csexton/create-release@add-body
        id: create_release
        with:
          tag_name: ${{ steps.versioning.outputs.cli_version }}
          release_name: safe-cli
          draft: false
          prerelease: false
          body: ${{ steps.release_description.outputs.description }}

This works without any errors. However, the problem is, the description (the body parameter of the create_release action) is basically a small markdown document, so it’s a multiline string. The problem is that ::set-output is truncating everything except the first line. I can confirm that by echoing the description in the same action where it’s assigned and then referencing the description output variable in the next action.

Could someone please advise how I can get this to work with a multiline string?

Thanks. 

[Yanjingzhu]

I have reported your question to the appropriate engineering team for further evaluation. The team will review the feedback and notify me about the next steps. I will update here in time. Thank you for your understanding.

[Yanjingzhu]

I got response from the team. 

% and \n and \r can be escaped like below, the runner will unescape in reverse.

content="${content//'%'/'%25'}"
content="${content//$'\n'/'%0A'}"
content="${content//$'\r'/'%0D'}"

Please try to add next three red lines to your yml, kindly let me know whether this could help.

- shell: bash
        id: release_description
        run: |
          description=$(./resources/get_release_description.sh ${{ steps.versioning.outputs.cli_version }})
          echo $description
<font color="#FF0000"> description="${description//'%'/'%25'}"
          description="${description//$'\n'/'%0A'}"
          description="${description//$'\r'/'%0D'}"</font>
echo "::set-output name=description::$description"
if: startsWith(steps.commit_message.outputs.commit_message, 'Version change')

 I tested in my side, after adding these lines, I can use echo " ${{ steps.release_description.outputs.description1 }} " to output multiple line value. Please pay attention to “”. 

[jacderida]

Sorry, I just noticed your reply!

Thanks very much for this, it resolves the issue.

[andreasplesch]

It took me an hour to realize that multi-line strings were the issue for me, and then a few more hours to find this solution.

It would be great if multi-line strings as values could be better supported, and documented.

Here are a few observation. First, it is important to suppress word-splitting upon expansion in bash. This is easiest done by enclosing with double quotes:

REPORT="$(cat logfile)"

Similarly, word splitting also has to be suppressed when reading back:

echo "$REPORT"

The double quotes are necessary.

However, Workflows gets confused/truncates when dealing with an expanded multi-line value:

echo "::set-env name=REPORT::$REPORT"

This does not work for multi-line variables.

The solution above uses a standard bash feature to expand and substitute characters:

REPORT="${REPORT//'%'/'%25'}"
REPORT="${REPORT//$'\n'/'%0A'}"
REPORT="${REPORT//$'\r'/'%0D'}"

This essentially makes the value into a single line string.

The github engineers then also knew that when reading back the variable in the workflow these escape values get substituted back to the actual characters:

${{ env.REPORT }}

This results actually in a multi-line string. Unfortunately, this is not documented anywhere and comes across a bit as magic. If there are other substitutions it would be good to also document those.

[andreasplesch]

Thanks so much posting this. I came across the same issue and found this rather obscure solution, after a lot of trial and error.

A couple observation dealing with multiline values, mostly as a reminder on how this works.

If using bash, it is important to suppress word splitting upon parameter expansion, to keep the line breaks. It is easiest with double quotes:

REPORT="$(cat log.out)"

Accessing the variable also invokes word splitting, which needs to be suppressed:

echo "$REPORT"

However, set-output or set-env does not work with expanded multiline values:

echo "::set-env name=REPORT::$REPORT"

This does does not work, as Workflows somehow truncates or ignores the value.

The solution above is to escape the newlines and other apparently unhandled characters using bash expansion and substitution. This makes the value effectively single line.

The thing to notice is that Workflows substitutes the escaped characters back when the parameters is used in ${{ }}:

body: |
  eslint and diff report:
  ```
  ${{ env.REPORT }}
  ```

Here the final value includes actual newlines.

It would be great if this behaviour cold be documented somewhere. For example, are there other substitutions happening ?

[Yanjingzhu]

@andreasplesch  Thank you for your further investigation. I would recommand you to create an issue in the action toolkit repo . You could ask for adding an example for set-env and set-output with multiline values in this document. 

[andreasplesch]

Thanks. Good to know that there is more in depth documentation available in the toolkit repo. I think a link from the main help page at https://help.github.com/en/actions to this documentation could be helpful.

Hm, it looks like newlines should already be escaped automatically:

https://github.com/actions/toolkit/blob/master/packages/core/src/command.ts#L76

Anyways, here is the new issue in the repo:

actions/toolkit#403

and a related issue:

actions/toolkit#193

[andreasplesch]

Escaping of special characters already happened for javascript actions but does not for bash scripts. There is a plan to update documentation.

[bewuethr]

I have bumped into this exact issue, and found another solution: we can use jq to do the conversion into a valid JSON string for us, and then use fromJSON to unpack it again.

For example: the variable $mystr holds the (multiline) contents of file myfile; we convert to a one-line JSON string with jq --raw-input --slurp '.', and then unpack in a later step.

- id: mystep
  run: |
    mystr=$(< myfile)
    msg=$(printf '%s' "$mystr" | jq --raw-input --slurp '.')
    echo "::set-output name=msg::$msg"

- run: |
    echo "${{ fromJSON(steps.mystep.outputs.msg) }}"

Notice that this isn’t fool-proof, either: if the expanded string contains double quotes, the echo in the second step might get confused. Set the value as an environment variable to avoid.

[AraHaan]

I also have an issue where I would like to output a list of paths and other data only if the action processes multiple files (to push them to nuget.org or any nuget feeds).

[pestophagous]

Here is yet another potential work-around.

This is working in a CI Action job of mine, which takes the content of a text file that contains test results and posts the whole content (which is multiline content) back to the relevant PR as a comment. The resulting comment accurately reflects the original newlines of the test result output.

name: build_for_PR
on:

pull_request:

branches:

- main
jobs:

linux_publish_test_output:
runs-on: ubuntu-18.04

steps:
- uses: actions/checkout@v1

- name: run_all_linux_ci
  run: ./linux_build_all_and_test.sh

- name: get_legacy_gui_test_output
  id: legacy_out_01
  run: echo "::set-output name=testconfout::$(cat path_from_my_git_root/to/build/artifacts/db_mgr_test_output.txt | python3 -c 'import sys; print(sys.stdin.read().replace("\\","\\\\").replace("\"","\\\"").replace("\n","\\n"))' )\n"

- uses: actions/github-script@v3
  with:
    script: |
      var theContent = "${{join(steps.legacy_out_01.outputs.*, '\n')}}";
      github.issues.createComment({
      issue_number: context.issue.number,
      owner: context.repo.owner,
      repo: context.repo.repo,
      body: 'LEGACY GUI TEST OUTPUT (from most recent CI job on this PR):\n\n' + theContent
      })

This was inspired by the approach posted by @ bewuethr. As far as I can tell in a couple days of usage, the python-based solution I am sharing does manage to handle double-quotes in the text (in addition to newlines).

[PedroMartinSteenstrup]

Thanks a lot for sharing, this unblocked me :slight_smile:

[Simran-B]

For completeness, I want to point out that you can also use an environment variable via environment files for multiline text: Workflow commands for GitHub Actions - GitHub Docs

steps:
  - name: Set the value
    shell: bash
    run: |
      echo 'JSON_RESPONSE<<EOF' >> $GITHUB_ENV
      curl https://httpbin.org/json >> $GITHUB_ENV
      echo 'EOF' >> $GITHUB_ENV
  - name: Print multiline response
    shell: bash
    run: |
      echo "$JSON_RESPONSE"

[John4650-hub]

Thanks alot, this worked for me.

[Starlight220]

Has this been fixed? I built a function to work around this, and now it displays the escape sequences.

[MolotovCherry]

Here’s my multi-line output solution which requires no changing of the data, however you do have to write it to a text file first.

In this example I’m using it for stderr, but you can adapt it to any kind of multi-line output too

# on windows, powershell fails on stderr output and improperly
# handles putting it in the file. It's required to run as cmd /c
# if not using windows, then change this command appropriately
- name: build
  id: build
  continue-on-error: true
  run: cmd /c "build 2> errors.txt"

name: Set error-log to var

uses: actions/github-script@v4

id: error-log

if: steps.build.outcome != 'success'

with:

script: |

const fs = require('fs');

return fs.readFileSync('errors.txt','utf8').toString();

result-encoding: string

you can access the output variable in like so

name: Some action

uses: whateveryouwant/whatever@4

if: steps.build.outcome != 'success'

with:

something: ${{ steps.error-log.outputs.result }}

print errors.txt to stdout + fail workflow since build failed
use cat if not using windows

name: Show build errors

if: steps.build.outcome != 'success'

run: |

type errors.txt

exit 1

[MolotovCherry]

Note: You can change the output variable name by using core.setOutput as well.

[mirrorcult]

For any lost souls in the future: If you’re setting output in a Docker container (like if your action is a .NET binary), not in the action YML, by just printing ::set-output name=whatever::whatever, all you need to do to fix this is replace any \n you have with %0A

[elb-slack]

Also for anyone in the future with this issue, I JSON-encoded the content I needed to put into GITHUB_ENV like so:

cat_results=$(cat results/results_blob.txt | python3 -c 'import json; import sys; print(json.dumps(sys.stdin.read()))')
echo "RESULTS=${cat_results}" >> $GITHUB_ENV

I kept getting errors with the multiline $GITHUB_ENV. This worked perfectly since I wanted the content JSON-encoded anyway. It was sent over the Github API to create a comment on a PR. You can decode the JSON in another step rather than hoping you caught all url-encoded characters with replace().

[bdovh]

Wow, I had this problem for a year and I couldn't solve it. I believed this one the problem with slack action that wasn't supporting multiline comments. Now I have finally found this thread and found a solution for a problem in my script that existed more than a year

[ffried]

For future reference:
With GitHub's new GITHUB_OUTPUT file, multiline outputs don't need to be escaped anymore. They simply need to use a delimiter:

delimiter="$(openssl rand -hex 8)"
echo "output-name<<${delimiter}" >> "${GITHUB_OUTPUT}"
echo "Some\nMultiline\nOutput" >> "${GITHUB_OUTPUT}"
echo "${delimiter}" >> "${GITHUB_OUTPUT}"

Note I'm using openssl rand -hex 8 to get a unique delimiter as suggested in the docs.

[karfau]

It would be really nice to add those instractions to https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs

[bewuethr]

It's documented in workflow commands.

[plocket]

When I used this in a composite action and created an output with it, my mutliline string lost all of its new lines. When I tried to echo it later, it was all on one line. How do I preserve the new lines? Or did I do something wrong?

In the composite action it echos as a multiline string:

outputs:
  TEXT:
    description: 'Text from action'
    value: ${{ steps.text_output.outputs.TEXT }}
# Lower down in a step in a job
id: text_output
run: |
  echo "$TEXT"
  echo 'TEXT<<EOF' >> $GITHUB_OUTPUT
  echo $TEXT >> $GITHUB_OUTPUT
  echo 'EOF' >> $GITHUB_OUTPUT

In the workflow that uses the composite action, it echos the right text, but all on a single line:

run: |
  echo "TEXT=${{ steps.the_text.outputs.TEXT }}" >> $GITHUB_ENV

When I put it in the body of an issue using the GitHub cli, it also appears as just one line of text.

[ffried]

This is probably because you don't quote the $TEXT variable when passing it to echo the second time. This will lead to multiple arguments passed to echo which in turn looses the newlines.

Try using echo "$TEXT" >> "$GITHUB_OUTPUT" instead.
(btw. it's generally good practice to "quote everything" in shell scripts - with a few specific exceptions of course).

[bewuethr]

See this Stack Overflow Q&A: I just assigned a variable, but echo $variable shows something else

Else, any time you have multi-line values you want to set in the environment or as an output, you have to use the <<EOF style syntax in Actions.