Cannot support proxy?

[liupgd]

I tried with examples. If no proxy used, it succeeded. But if I use proxy, 'Bad request' returned. My test code is:

using WebSockets
WebSockets.open("wss://echo.websocket.org", proxy="http://127.0.0.1:10809") do ws
    println("ok")
end

Any help?

[hustf]

Hi,
I'm afraid you'll have to dive a bit deeper in, here. Stack traces are boring, but often the key. You can confirm if this is what you get, too:

ERROR: WebSocketClosedError(" while open ws|client: connect: connection refused (ECONNREFUSED)")
Stacktrace:
 [1] open(f::Function, url::String; verbose::Bool, subprotocol::String, kw::Base.Iterators.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:proxy,), Tuple{String}}})
   @ WebSockets ~\.julia\packages\WebSockets\QcswW\src\HTTP.jl:99
 [2] top-level scope
   @ REPL[5]:1

caused by: IOError(Base.IOError("connect: connection refused (ECONNREFUSED)", -4078) during request(http://127.0.0.1:10809))

Stacktrace:
  [1] getconnection(::Type{Sockets.TCPSocket}, host::SubString{String}, port::SubString{String}; keepalive::Bool, connect_timeout::Int64, readtimeout::Int64, kw::Base.Iterators.Pairs{Symbol, Any, NTuple{4, Symbol}, NamedTuple{(:require_ssl_verification, :iofunction, :reached_redirect_limit, :verbose), Tuple{Bool, WebSockets.var"#openstream#4"{var"#3#4", String}, Bool, Int64}}})
    @ HTTP.ConnectionPool ~\.julia\packages\HTTP\dCYNB\src\ConnectionPool.jl:702
  [2] newconnection(pod::HTTP.ConnectionPool.Pod, T::Type, host::SubString{String}, port::SubString{String}, pipeline_limit::Int64, require_ssl_verification::Bool, idle_timeout::Int64; kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :verbose), Tuple{WebSockets.var"#openstream#4"{var"#3#4", String}, Bool, Int64}}})
    @ HTTP.ConnectionPool ~\.julia\packages\HTTP\dCYNB\src\ConnectionPool.jl:624
  [3] getconnection(::Type{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}, host::SubString{String}, port::SubString{String}; connection_limit::Int64, pipeline_limit::Int64, idle_timeout::Int64, reuse_limit::Int64, require_ssl_verification::Bool, kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :verbose), Tuple{WebSockets.var"#openstream#4"{var"#3#4", String}, Bool, Int64}}})
    @ HTTP.ConnectionPool ~\.julia\packages\HTTP\dCYNB\src\ConnectionPool.jl:568
  [4] request(::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer{Union{}}}}, url::URIs.URI, req::HTTP.Messages.Request, body::Nothing; proxy::String, socket_type::Type, reuse_limit::Int64, kw::Base.Iterators.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:iofunction, :reached_redirect_limit, :verbose), Tuple{WebSockets.var"#openstream#4"{var"#3#4", String}, Bool, Int64}}})

What you are doing here is calling to a server you don't control. I suppose that server replies to an ordinary http connection with a response that includes a header, something like '407 Proxy Authentication Required'. Setting up a http connection is just the first step in establishing a websocket, and there's not much hope to do this if that server don't like proxies.

The headers will not be in scope in your function ('println'), but have a look at the 'guarded' examples. They are intended primarily for when you take the role as server and are reluctant to accept potentially malicious connections from outside. I hope this was helpful?

[liupgd]

@hustf Thanks for your reply.
Please, forgive me, I'm a novice in Julia. I've tried my best to dive into the request codes but no helpful clues found.
In fact, the ws link "wss://echo.websocket.org" is from repo's example codes. I've tried the example codes. If there is no proxy, they're working fine. However, my situation is, I need the proxy.
I see your ERROR info, it's probably because you do not have a vpn? I've installed a VPN, the 'http://127.0.0....' means my local vpn address. It cannot be accessed untill you have also setup a VPN in your own computer.

Some supplements:
Without proxy, the example code prints some info:

HTTP.Messages.Response:
"""
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Date: Wed, 28 Apr 2021 14:27:46 GMT
Sec-WebSocket-Accept: hg0skb/YlvPJTKJTn8KFcBBBDUo=
Server: Kaazing Gateway
Upgrade: websocket
"""

With proxy, the example code prints some info:

HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
Content-Type: text/plain
Connection: close
"""

I also debug the code with my very limited julia ablity. I debug the code in vscode. When the debugger step to line 108 in ConnectionRequest.jl of HTTP package, julia crashed. The crash happened so fast, I even can't get the error prints. The line crashed during debug is as bellow:

I don't know how to step deeper. Can you get some useful information from above? Thanks.

[hustf]

Perhaps this is an issue which is best sidestepped? First, a small, confusing fact (sorry!). There is a file in this package that is called HTTP.jl for historic reasons. The code in this file calls other functions, which are in the [package which is also called HTTP.jl]( https://github.com/JuliaWeb/HTTP.jl). HTTP.jl is a quite low-level 'framework', it assumes you know a good deal about very low-level stuff. And emphasis is placed on efficiency. There are some clever tricks with the compiler which makes it unsuitable for debugging by stepping through code. But it's good to know that when we do need to tweak those intricate details, there are no secrets, no black box, just a huge rabbit hole of knowledge. This really is complicated, so it's usually better to go through higher level packages. WebSockets.jl is one of the rather low level packages, too, but we have tried to make things a little bit easier and accessible here, by hiding the very long stacktraces you would get from HTTP.jl. Sometimes, errors have simple causes, so we don't show every tiny detail when that is the case. Your use case will hardly be to connect to a websocket placed on echo.websocket.org. That is a site which is only intended to check if your code is working. But that server doesn't work easily with proxies. That's fine - we know that the websockets package follows the protocols already, anyway. It would be better to spend your efforts getting things to work with another server. So what's your use case? If it is learning, I believe you can get quite far by setting up your own servers locally, like the examples do. The best way to use the examples is to run a server in one REPL, and a client in another REPL. You could also step through the files in the 'test' folder. The tests basically tries out all kinds of crazy ways to use websockets. As long as you don't get these errors, stepping through the test files (with F10, not F11 if you see what I mean) can be a good way to learn. Good luck, and sorry that I can't assist with solving the specific 'issue' with connecting to "wss://echo.websocket.org" through a proxy. If you describe your use case, I may perhaps (perhaps!) know resources to point to.

…

On Wed, Apr 28, 2021 at 5:34 PM liupgd ***@***.***> wrote: @hustf <https://github.com/hustf> Thanks for your reply. Please, forgive me, I'm a novice in Julia. I've tried my best to dive into the request codes but no helpful clues found. In fact, the ws link "wss://echo.websocket.org" is from repo's example codes. I've tried the example codes. If there is no proxy, they're working fine. However, my situation is, I need the proxy. I see your ERROR info, it's probably because you do not have a vpn? I've installed a VPN, the 'http://127.0.0....' means my local vpn address. It cannot be accessed untill you have also setup a VPN in your own computer. Some supplements: *Without* proxy, the example code prints some info: HTTP.Messages.Response: """ HTTP/1.1 101 Switching Protocols Connection: Upgrade Date: Wed, 28 Apr 2021 14:27:46 GMT Sec-WebSocket-Accept: hg0skb/YlvPJTKJTn8KFcBBBDUo= Server: Kaazing Gateway Upgrade: websocket """ *With* proxy, the example code prints some info: HTTP.Messages.Response: """ HTTP/1.1 400 Bad Request Content-Type: text/plain Connection: close """ I also debug the code with my very limited julia ablity. I debug the code in vscode. When the debugger step to line 108 in ConnectionRequest.jl of *HTTP* package, julia crashed. The crash happened so fast, I even can't get the error prints. The line crashed during debug is as bellow: [image: image] <https://user-images.githubusercontent.com/3350539/116430407-6392b400-a879-11eb-84c1-e10ea02a9074.png> I don't know how to step deeper. Can you get some useful information from above? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#172 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADKNVNGT7SJZUFELC4C3KHLTLATIRANCNFSM43W6BHMQ> .

[liupgd]

Thanks. My use case is I need to subscribe data stream from a platform. But that platform doesn't offer websocket service in my country. So a VPN is needed.
I tried to setup a websocket server for proxy test, it turns out to fail again. The server side never received requests after the proxy is used in the client side.
Here is my experiment code:

• server

using WebSockets
using Sockets
const LOCALIP = ip"127.0.0.1"
const PORT = 60100

handle(req) = "Hello..." |> WebSockets.Response

function gatekeeper(req, ws)
    orig = WebSockets.origin(req)
    @info "\nOrign: $orig Target: $(req.target) subprotocol: $(subprotocol(req))"
end

const server = WebSockets.ServerWS(handle, gatekeeper)

@async WebSockets.with_logger(WebSocketLogger()) do
    WebSockets.serve(server, LOCALIP, PORT)
end

• client

WebSockets.open("ws://127.0.0.1:60100", proxy="http://127.0.0.1:10809") do ws
       println("ok")
end

• failure info

HTTP.Messages.Response:
"""
HTTP/1.1 400 Bad Request
Content-Type: text/plain
Connection: close
"""

If there is no proxy, everything is fine.

I also tried python websocket, that works fine. Maybe I have to use python as data receiver.

[liupgd]

Hi @hustf Today, I try to modified HTTP.jl package(not the HTTP.jl of this repo).
HTTP.jl--> ConnectionRequest.jl line 96:

• Ther Original code:

try
    if proxy !== nothing && target_url.scheme == "https"
        # tunnel request
        target_url = merge(target_url, port=443)
        r = connect_tunnel(io, target_url, req)
        if r.status != 200
            close(io)
            return r
        end
        io = ConnectionPool.sslupgrade(io, target_url.host; kw...)
        req.headers = filter(x->x.first != "Proxy-Authorization", req.headers)
    end

    r =  request(Next, io, req, body; kw...)
......

• Now I changed to this:

try
    if proxy !== nothing# && target_url.scheme == "https"
        # tunnel request
        if target_url.scheme == "https" || target_url.scheme == "wss"
            target_url = merge(target_url, port=443)
        end
        r = connect_tunnel(io, target_url, req)
        if r.status != 200
            close(io)
            return r
        end
        if target_url.scheme == "https" || target_url.scheme == "wss"
            io = ConnectionPool.sslupgrade(io, target_url.host; kw...)
        end
        req.headers = filter(x->x.first != "Proxy-Authorization", req.headers)
    end

    r =  request(Next, io, req, body; kw...)
......

Now, it works for proxy now. I'm not sure whether my modification would affect other functions of HTTP. At least it works for me now.

[hustf]

Hi, and sorry for the late reply. Yes, the fix seems quite obvious, and could save quite some time for other people meeting this. I don't believe the use of websockets outside of local connections and with Julia have been very widely used. This is one example of the maturing which happens when we do that.

I suppose you use 'develop HTTP', and no, I don't see downsides expect that you'll miss updates. Please open an issue on HTTP.jl and send a pull request referring it!