Autocomplete in IEx broken (on Windows)

[Hades32]

The documentation says

   ## Autocomplete   
   To discover all available functions for a module, type the module name 
   followed by a dot, then press tab to trigger autocomplete. For example: 
        Enum. 

but this does not work on Windows 10, neither in cmd.exe nor in ConEmu and the MSys Bash (from Git).

[josevalim]

I think it only works if you start it with iex --werl. I will update the docs.

[Hades32]

You're right, it works in the special 'werl' console...

But I can't imagine anybody wanting to use 'werl'... It's ugly as hell, doesn't integrate well with other consoles, doesn't support high-DPI and so on...

Is there no way to support this without 'werl'?

[josevalim]

@Hades32 afaik, no. :( But I am not a windows specialist :)

[Hades32]

I still don't know HOW it works, but I now know that it is possible.

The Haskel interactive compiler "ghci.exe" as TAB-completion on Windows. (When running under cmd.exe and also in Msys Bash)

So, maybe it would be worth to have a look there...

PS: My personal guess is that they don't use the systems "read line" function but a "read char" function and emulate line reading. That way they can capture the tab char...

[CharlesOkwuagwu]

@Hades32 any luck on your research into this issue?

Where can we start looking for a solution?

How does the autocomplete feature currently work in Linux, what's missing from Windows?

[josevalim]

@CharlesOkwuagwu you need the terminal to emit the tab escape code whenever you press tab instead of inserting a literal tab. I am not sure what is required to make such happen on Windows. I believe they may have introduced that on Windows 10 TH2 (since they also introduced ansi escapes).

[amkoba]

If anyone is looking for Tab selection with mix

https://elixirforum.com/t/enabling-iex-tab-completion-in-windows/3032/2?u=amkoba

[thiagomajesk]

I've tested iex with the following terminals without success for auto-completion:

Native:

• Powershell (with and without PSReadLine)
• Cmd (with and without clink)
• The new Windows Terminal (That supposedly supports ANSI escapes)

Emulators:

• Cmder/ ConEmu (with and without this configurations)
• Alacritty
• Terminus (which supports clink by default)
• Fluent Terminal

Funny thing is that IRB and Pry both work out of the box on Windows without much trouble - maybe there's something that could be done on the iex side?

[Nekkowe]

For the reasons that Hades32 mentioned, I'm surprised this issue remains closed.
It's an ongoing problem, and iex is (as far as I'm aware) alone with this issue.

[JakaBac]

iex is not alone, erl is doing the same thing (under the hood it actually IS the same thing)

This behavior comes from Erlang's implementation of its ttysl driver. On unix-like systems it uses tcsetattr to configure the console and then it does its own low level i/o and line editing.

On Windows the ttysl driver talks to the Erlang Windows console. That was implemented by the Erlang/OTP team.
If you start "console" shell on windows it starts as if there is no console. It just reads/writes to stdin/stdout.

To fix this a new ttysl driver would need to be written for Windows, which uses Windows Console API under the hood.

[vladrmnv]

@JakaBac thanks for the info, any pointers to the driver spec? This needs to be implemented

[JakaBac]

I am not aware of any public written specification. The posix version of the driver is here:
https://github.com/erlang/otp/blob/master/erts/emulator/drivers/unix/ttsl_drv.c

And the Windows GUI console driver is here:
https://github.com/erlang/otp/blob/master/erts/emulator/drivers/win32/ttsl_drv.c
That one is calling into the console gui implementation:
https://github.com/erlang/otp/blob/master/erts/emulator/drivers/win32/win_con.c

I think that the Windows ttsl_drv.c could be changed to use the Windows Console API instead of the win_con.c GUI.

If the ttsl_drv is not started I think that everything falls back to simply reading a line from stdin. So in Windows you get the standard legacy Windows console implementation with its own line editing

[cognociente]

The IEX autocomplete functionality still doesn't seem to work in Windows.

[josevalim]

As mentioned above, it works under iex --werl, which is a console specific under Windows. You can also set USE_WERL=1 in your environment variables to it always uses werl.

[Fl4m3Ph03n1x]

While I appreciate the hard work @josevalim and his folks have been doing, I will still voice my support for Windows Users and I will further state that using werl is not a proper solution.

While most people probably develop on Linux, not everyone is so fortunate. Furthermore, as Elixir grows, it is likely to attract more Windows users, so having proper terminal integration would benefit everyone.

To support my claim, please notice the following image:

Here a few things to notice:

• The iex bash is colored and I can see the output from expressions. Even if it was not colored, the mere fact I can read the output from my expressions would already be a massive step up from the werl alternative.
• The werl terminal is unreadable. Finding the result of my 1*1 expression is practically impossible

Furthermore, werl is not compatible with IEx.configure. The Elixir team has invested a lot of effort into making the iex bash configurable, it seems like a waste that only a portion of the user base can actually benefit from it.

As my last argument, I also want to point out the effort the community and the team have given to Windows support, installation and setup, it again looks like counter-intuitive that something as simple as autocomplete cannot be available for Windows users when it is considered a basic functionality in most programming languages.

I wish to clarify that I am not attacking anyone here nor am I disgruntled. I very much support Elixir and admire the work you folks have been doing. I do however want to make a statement, which is "I do not agree that werl is a viable solution, not in the long term, nor in the short one."

[josevalim]

@Fl4m3Ph03n1x as mentioned above, we are relying on the foundation provided by Erlang so this would need to be addressed upstream. I can think of two options here:

1. Implement a tty for Windows using the Windows Console API mentioned by @JakaBac

2. Find a way to enable ANSI coloring on werl

It seems you prefer option 1 but we need to find someone interested in doing the work. The foundation may also be glad to sponsor this work if someone is interested (and I can help someone go through the process of getting funding, just reach out to me).

Edit: Oh, just now I noticed this issue is quite old, so I will go ahead and mention the ANSI escape support is itself relatively new (~1 year old) because that was also only made enabled by default on Windows Terminal (which is ~2 years old).

[simonmcconnell]

On a side note, @Fl4m3Ph03n1x there's definitely something weird going on with your werl. You should have legible output there.

[JakaBac]

@josevalim since Microsoft is improving their console subsystem in Windows, I believe that that the first option is the way to go.
Adding ANSI coloring to werl would also not be a small task.

I am interested in working on option 1 and I already have a hacked together implementation that works. But it needs performance improvements (printing lots of text is noticably slower than in werl).

But even with my hacked together implementation I managed to hit some bugs in windows terminal implementation (microsoft/terminal#6546)

However I would have to work on this in my spare time, so the progress may be slow.
I also need to get an opinion from the Erlang/OTP team how they would like this to fit in (probably depending on how ancient Windows versions they need to support)

[josevalim]

Hi @JakaBac, that's great to hear!

I have been reading about Windows Console vs Virtual Terminals, and it seems Microsoft is now recommending virtual terminals instead of Windows Console. Could it be that adapting the existing Unix code is the best way forward?

[JakaBac]

@josevalim Yes, that is Microsoft's recommendation. But that only appeared in certain Windows 10 update. So using that would mean no support for older Windows versions (fine by me if Erlang/OTP team and everyone else is happy).
Older versions could fallback to no ttysl port driver which is basically what you get on windows now if you run erl.exe.

I quickly glanced over Unix ttysl implementation and it is relying on termcap and is using signals (at least SIGWINCH for notification when the terminal window is resized). I don't think that a termcap implementation which knows about the new windows console exists yet and there are no signals in Windows (of course there are other means of getting notified when the console window is resized).

So for for implementing this on Windows there are basically 2 options:

1. Use Windows specific console functions.
   In this case older Windows versions are supported, but the implementation is still Windows specific (like werl). This is my current hack

2. Use the Unix/Posix code
   In this case the Unix/Posix code would get ifdefs for alternatives to signals and probably some "lightweight windows console specific termcap" that would be just enough to make the Erlang ttysl driver happy.

I'll ask about what the Erlang community would prefer on the Erlang mailing list tomorrow. But my guess is that Erlang/OTP team will prefer an option with least impact on the existing implementation(s)

If anyone has any comments/suggestions or opinions, I would be happy to hear them

[josevalim]

I am thinking 2 may be the best option because that's what we would want long term? In general, the current tty implementation:

1. Uses port drivers, which are generally being phased out in favor of NIFs

2. Uses an approach that is marked as outdated by Windows

3. Overrelies on C/NIF code. If we could move a reasonable amount of the logic to Erlang by relying on ANSI that would be a net positive too

The tty stuff is only handled by the user_drv module and that can likely be encapsulated so I am thinking there could be a new NIF-based implementation and keep the existing one around for werl and as a fallback. When the OTP team is ready, they would get rid of the port driver and werl.

At the same time, a working solution is better than an ideal solution, so if Windows Console is the fastest way to achieve it, then why not? But this is just my opinion. :)

I'll ask about what the Erlang community would prefer on the Erlang mailing list tomorrow. But my guess is that Erlang/OTP team will prefer an option with least impact on the existing implementation(s)

You can also consider using the ErlangForums. And as mentioned above, if you think funding can be helpful, let me know!

[JakaBac]

I completely agree.
I was too focused on the existing code structure and completely missed the option of replacing the port driver with a NIF.

Let me experiment a bit more

[Fl4m3Ph03n1x]

@simonmcconnell I can share my setup with you. I have a .iex.exs with the following content:

IEx.configure(
  colors: [
    enabled: true,
    eval_result: [ :cyan, :bright ],
    eval_error:  [ :light_magenta ],
  ],
  default_prompt: [
    "\r\e[38;5;220m",         # a pale gold
    "%prefix",                # IEx context
    "\e[38;5;112m(%counter)", # forest green expression count
    "\e[38;5;220m>",          # gold ">"
    "\e[0m",                  # and reset to default color
  ]
  |> IO.chardata_to_string    # (1)
)

And then I enabled ANSI Colors by making this change in the Windows registry:
https://superuser.com/a/1300251/222770

Both command prompt and powershell work almost fine (there are still some characters that dont show up properly, like *) but nothing like with werl.

@JakaBac I am super happy you decided to pick up the mantle here. I was going to post a request in the forum looking for people, but you beat me to it!

I wish you the best of luck !

[makarichevss]

I confirm - with the latest Windows latest update 22H2 IEx still not supports autocomplete.
Maybe it's now time to find Windows specialist and finally add this option?
(all consoles (Rust, Closure and many more languages) I've seen so far have ANSI support - and somehow IEx still does not...)
Using --werl with its horrible escape symbols is not the solution for me.
And I'm 99% sure Erlang guys definitely wouldn't make anything with ANSI in the next years, so we have to find that guy. I'm even ready to pay this guy... hm, 50 USD for this fix)

I would like to make another issue, and lets leave it open until this option is finally added, is that OK?

[simonmcconnell]

@makarichevss I'm pretty sure this is solved in the next Erlang release, which I think is due around March 2023.

See this topic for reference https://erlangforums.com/t/the-erlang-shell/1307.

[josevalim]

Correct, this will be addressed in the next Erlang/OTP release (Erlang/OTP 26). I have even already tested the main version to confirm it works as expected (in case you want to give it a try too). Locking this for now as we just need to wait a bit more.