Zsh/Fish completion failing

[amenzhinsky]

Hi there,

zsh completion is not working, I'm getting _arguments:comparguments:325: invalid option definition: --log-format[The log format [text,color,json].]:

Generated completion looks like:

function _buf {
  local -a commands

  _arguments -C \
    '--log-format[The log format [text,color,json].]:' \
    '--log-level[The log level [debug,info,warn,error].]:' \
    '--timeout[The duration until timing out.]:' \
    "1: :->cmnds" \
    "*::arg:->args"

It seems cobra doesn't escape flag description properly: https://github.com/spf13/cobra/blob/v1.0.0/zsh_completions.go#L334 and braces from flag descriptions with permitted values break it.

I'm not sure whether it's done on purpose for some flexibility or not.

% zsh --version
zsh 5.8 (x86_64-pc-linux-gnu)

[bufdev]

See spf13/cobra#1129

[bufdev]

See spf13/cobra#1129 (comment) - we're going to update to cobra master for now.

[amenzhinsky]

@bufdev the error is fixed, but completion is gone now btw :)
Didn't have much time to figure out why, but generated code does nothing now, it doesn't contain nether command names nor flags.

#compdef _buf buf

# zsh completion for buf                                  -*- shell-script -*-

__buf_debug()
{
    local file="$BASH_COMP_DEBUG_FILE"
    if [[ -n ${file} ]]; then
        echo "$*" >> "${file}"
    fi
}

_buf()
{
    local shellCompDirectiveError=1
    local shellCompDirectiveNoSpace=2
    local shellCompDirectiveNoFileComp=4
    local shellCompDirectiveFilterFileExt=8
    local shellCompDirectiveFilterDirs=16

    local lastParam lastChar flagPrefix requestComp out directive compCount comp lastComp
    local -a completions

    __buf_debug "\n========= starting completion logic =========="
    __buf_debug "CURRENT: ${CURRENT}, words[*]: ${words[*]}"

    # The user could have moved the cursor backwards on the command-line.
    # We need to trigger completion from the $CURRENT location, so we need
    # to truncate the command-line ($words) up to the $CURRENT location.
    # (We cannot use $CURSOR as its value does not work when a command is an alias.)
    words=("${=words[1,CURRENT]}")
    __buf_debug "Truncated words[*]: ${words[*]},"

    lastParam=${words[-1]}
    lastChar=${lastParam[-1]}
    __buf_debug "lastParam: ${lastParam}, lastChar: ${lastChar}"

    # For zsh, when completing a flag with an = (e.g., buf -n=<TAB>)
    # completions must be prefixed with the flag
    setopt local_options BASH_REMATCH
    if [[ "${lastParam}" =~ '-.*=' ]]; then
        # We are dealing with a flag with an =
        flagPrefix="-P ${BASH_REMATCH}"
    fi

    # Prepare the command to obtain completions
    requestComp="${words[1]} __complete ${words[2,-1]}"
    if [ "${lastChar}" = "" ]; then
        # If the last parameter is complete (there is a space following it)
        # We add an extra empty parameter so we can indicate this to the go completion code.
        __buf_debug "Adding extra empty parameter"
        requestComp="${requestComp} \"\""
    fi

    __buf_debug "About to call: eval ${requestComp}"

    # Use eval to handle any environment variables and such
    out=$(eval ${requestComp} 2>/dev/null)
    __buf_debug "completion output: ${out}"

    # Extract the directive integer following a : from the last line
    local lastLine
    while IFS='\n' read -r line; do
        lastLine=${line}
    done < <(printf "%s\n" "${out[@]}")
    __buf_debug "last line: ${lastLine}"

    if [ "${lastLine[1]}" = : ]; then
        directive=${lastLine[2,-1]}
        # Remove the directive including the : and the newline
        local suffix
        (( suffix=${#lastLine}+2))
        out=${out[1,-$suffix]}
    else
        # There is no directive specified.  Leave $out as is.
        __buf_debug "No directive found.  Setting do default"
        directive=0
    fi

    __buf_debug "directive: ${directive}"
    __buf_debug "completions: ${out}"
    __buf_debug "flagPrefix: ${flagPrefix}"

    if [ $((directive & shellCompDirectiveError)) -ne 0 ]; then
        __buf_debug "Completion received error. Ignoring completions."
        return
    fi

    compCount=0
    while IFS='\n' read -r comp; do
        if [ -n "$comp" ]; then
            # If requested, completions are returned with a description.
            # The description is preceded by a TAB character.
            # For zsh's _describe, we need to use a : instead of a TAB.
            # We first need to escape any : as part of the completion itself.
            comp=${comp//:/\\:}

            local tab=$(printf '\t')
            comp=${comp//$tab/:}

            ((compCount++))
            __buf_debug "Adding completion: ${comp}"
            completions+=${comp}
            lastComp=$comp
        fi
    done < <(printf "%s\n" "${out[@]}")

    if [ $((directive & shellCompDirectiveFilterFileExt)) -ne 0 ]; then
        # File extension filtering
        local filteringCmd
        filteringCmd='_files'
        for filter in ${completions[@]}; do
            if [ ${filter[1]} != '*' ]; then
                # zsh requires a glob pattern to do file filtering
                filter="\*.$filter"
            fi
            filteringCmd+=" -g $filter"
        done
        filteringCmd+=" ${flagPrefix}"

        __buf_debug "File filtering command: $filteringCmd"
        _arguments '*:filename:'"$filteringCmd"
    elif [ $((directive & shellCompDirectiveFilterDirs)) -ne 0 ]; then
        # File completion for directories only
        local subDir
        subdir="${completions[1]}"
        if [ -n "$subdir" ]; then
            __buf_debug "Listing directories in $subdir"
            pushd "${subdir}" >/dev/null 2>&1
        else
            __buf_debug "Listing directories in ."
        fi

        _arguments '*:dirname:_files -/'" ${flagPrefix}"
        if [ -n "$subdir" ]; then
            popd >/dev/null 2>&1
        fi
    elif [ $((directive & shellCompDirectiveNoSpace)) -ne 0 ] && [ ${compCount} -eq 1 ]; then
        __buf_debug "Activating nospace."
        # We can use compadd here as there is no description when
        # there is only one completion.
        compadd -S '' "${lastComp}"
    elif [ ${compCount} -eq 0 ]; then
        if [ $((directive & shellCompDirectiveNoFileComp)) -ne 0 ]; then
            __buf_debug "deactivating file completion"
        else
            # Perform file completion
            __buf_debug "activating file completion"
            _arguments '*:filename:_files'" ${flagPrefix}"
        fi
    else
        _describe "completions" completions $(echo $flagPrefix)
    fi
}

[gbrlsnchs]

Neither ZSH nor Fish completions are working for me. No error is output, they simply don't work.

[bufdev]

Can you provide any more details than that, for example how you've installed it?

[gbrlsnchs]

I built the program normally with Go compiler, then used the binary with its zsh-completion subcommand to generate completion (I dropped it at /usr/share/zsh/site-functions as _buf, like I would with any other completion). I did the same adapted for fish, no success too. I didn't test with bash.

[bufdev]

@gbrlsnchs can you try installing from master and see if this is fixed? go get github.com/bufbuild/buf/cmd/buf@master will do it.

[bufdev]

Also @amenzhinsky

[bufdev]

I believe this is fixed in v0.26.0, let us know if there are any other issues, sorry for the trouble.

[gbrlsnchs]

Hi, @bufdev, thanks for your effort. However, no luck, completion is still not working in v0.26.0, neither for ZSH nor Fish.

[bufdev]

Well, shoot.

[bufdev]

The issue with both zsh and fish completion here is going to be within the github.com/spf13/cobra library - they've been doing a lot of refactors to shell completion recently https://github.com/spf13/cobra/commits/master and there's clearly some bugs. Unfortunately, we don't have the time to dive into these right now, but this is where the bug would be. Will keep this open, and will re-test as improvements in spf13/cobra are available.

[bufdev]

@amenzhinsky do you think you could file an issue on github.com/spf13/cobra with what you see? We're trying to debug but we're not zsh completion experts heh

[amenzhinsky]

I'll try to find some time this weekend to figure out what's the problem.

[bufdev]

Hey, any chance you could check in on this again? There's been a lot of random work in spf13/cobra on the issue, wondering if it got magically fixed!

[marckhouzam]

I've taken the liberty to investigate this. First, it is important to note that the new completion system provided by Cobra no longer hard-codes the completions within the completion shell script (except for bash). Instead, it delegates the completion logic to the Cobra program itself. For example you can request completions yourself like this:

# $HOME/.cache/buf/Darwin/x86_64/gobin/buf __complete ""
beta	Beta commands. Unstable and will likely change.
breaking	Check that the input location has no breaking changes compared to the against location.
build	Build all files from the input location and output an image.
config	Interact with the configuration of Buf.
generate	Generate stubs for protoc plugins using a template.
help	Help about any command
lint	Check that the input location passes lint checks.
ls-files	List all Protobuf files for the input.
protoc	High-performance protoc replacement.
:4
Completion ended with directive: ShellCompDirectiveNoFileComp

# $HOME/.cache/buf/Darwin/x86_64/gobin/buf __complete beta r
registry	Interact with the Buf Schema Registry.
:4
Completion ended with directive: ShellCompDirectiveNoFileComp

The problem now is that the completion scripts read this output from stdout and purposefully ignore stderr. However, buf sends all its output to stderr as I noticed here:

buf/internal/pkg/app/appcmd/appcmd.go

Line 149 in 7f21551

cobraCommand.SetOut(container.Stderr())

Since Cobra's completion scripts for zsh and fish (and powershell) ignore stderr, no completions are shown.

Is there a reason buf outputs everything to stderr?

If it is necessary, a way to work around this would be to output to stdout when the command is Cobra's __complete, which can be referred to using cobra.ShellCompRequestCmd
(https://github.com/spf13/cobra/blob/9a432671fd847f0faa5a5e4d9f9350ae289db2ac/completions.go#L14)

[ZymoticB]

Hey @marckhouzam, thanks for investigating this and all of the information! We've actually already patched this internally and it will go out with our next release We should have updated this issue when we landed the change internally, sorry about that.

[bufdev]

This should be fixed - please let us know if this comes up again! We won't be surprised :-)