fix(agent): normalize indentation in code completion postprocess

[Sma1lboy]

#3263

Fixing wrong indent in inline completion predict solution

Demo

Screen.Recording.2024-11-15.at.09.24.21.mov

Before

Only first line has extra spaces:

exactly one extra space in the line indent

After

exactly one extra space in the line indent

Implementation

Using whether the current indent is even as the judgment basis. If it's even, the prediction doesn't need any extra spaces and the prediction context will be processed accordingly.

Potential risks:

Some users might use odd indent size

Why

This approach is simple and doesn't require adding additional LSP or obtaining the best indent through client operations.

Issue didn't fix

Due to some concern this post-processing not going to fix this situation

All lines in prediction have extra spaces:

[Sma1lboy]

Follow up: This is kind of a post-processing filter to prevent inconsistent extra spaces due to different model capabilities. The above bad example happens when using the demo website model, but sometimes works fine with StarCoder-1B without this modification.

[icycodes]

As all bad cases in #3263 have exactly one extra space in the line indent, I suggest we should only process the case of exactly one extra space in the line indent, not extra indent.
It should be allowed to have extra indents in the completion text. Consider the case of

function foo() {
║
}

and the completion is

function foo() {
├    bar();┤
}

[Sma1lboy]

As all bad cases in #3263 have exactly one extra space in the line indent, I suggest we should only process the case of exactly one extra space in the line indent, not extra indent. It should be allowed to have extra indents in the completion text. Consider the case of

function foo() {
║
}

and the completion is

function foo() {
├    bar();┤
}

The current version should work fine with this.

[sboys3]

This merged pull request mangles auto-completions when tabs are used instead of spaces.

It replaces parts of correctly tabbed responses with incorrect amounts of spaces. This has been driving me insane for a while, and today after reviewing completion logs I realized it was entirely client side. I had thought it was due to a mix of spaced type definitions and tabbed code when using javascript, but when I also saw it in a small c++ project, I looked into it deeper.

As a quick test I replaced return item.withText(normalizedLines.join("")); with return item; in the agent's bundle in my installed extension, and the problem completely went away with this code bypassed.
To do that in the current production bundle located at %userprofile%.vscode\extensions\tabbyml.vscode-tabby-1.18.0\dist\tabby-agent\node\index.js replace e.withText(n.join("")) with e/*indent fix*/

Here is an example of what has been happening. It has been extremely frustrating having to constantly fix indention.

And here is the result with this pull request bypassed

[Sma1lboy]

@sboys3 I am very sorry for causing this problem, I will fix the bug caused by this PR as quickly as possible