Docker behind HAProxy, port 4222 issue #754
-
|
I've managed to get everything else working seemingly without issue, and was even able to cut out some of the somewhat redundant HAProxy (on PFSense) ACLs. However, I cannot get the agent to work properly because port 4222 refuses the connection. I've tried port forwarding with opening the port on the server firewall, and it still fails. The only thing I have t tried is setting up an HAProxy listener on 4222 to direct the traffic, only because I have no idea what URL it uses. Any thoughts? |
Beta Was this translation helpful? Give feedback.
Replies: 24 comments 20 replies
-
|
HAproxy shouldn't be needed for 4222 that should just be port forwarded directly. Are you using a custom certificate or the letsencrypt one? |
Beta Was this translation helpful? Give feedback.
-
|
I'm using a LetsEncrypt cert managed by the PFSense. Exported the cert and key, input them into the env file per instructions. Everything else seems fine. I can log in, go through set up, access the site, download the agent, etc, all using that imported cert info. So dumb question: when port forwarding, it should be forwarded to the host IP address, as docker listens for exposed ports on all interfaces, prior to any other rules, correct? |
Beta Was this translation helpful? Give feedback.
-
|
Yes I think it should be port forwarding to the docker host (I don't use docker). SSL setup sounds fine, was just checking. Feel free to give us the HAProxy setup steps you used if you don't mind to help others. |
Beta Was this translation helpful? Give feedback.
-
|
This is an unsupported configuration. You can use this for reference: Discussing in the Discord #unsupported channel is another option if you're still having problems |
Beta Was this translation helpful? Give feedback.
-
|
@silversword411 I'm aware it's an unsupported config. Unfortunately that example has nothing aside from making sure the port is forwarded included for 4222. It would be remarkably simple to forward it with HAProxy if I could simply find out what URL it's supposed to be directed to. |
Beta Was this translation helpful? Give feedback.
-
|
I thought that example had everything necessary for haproxy to work. There's only 3 URL's. rmm, api, and mesh. 4222 is api.yourdomain.com This may also be of help: https://github.com/wh1te909/tacticalrmm/blob/c4ffffeec8e32d1317272a1470c70899cb5eb3ed/docs/docs/unsupported_scripts.md#general-notes-on-proxies-and-tactical-rmm |
Beta Was this translation helpful? Give feedback.
-
|
It should be pointed at the host as it's a port forward URLs/haproxy etc are irrelevant |
Beta Was this translation helpful? Give feedback.
-
|
But yes dinger is right....a port forward is IP:port only, no URL's necessary :) |
Beta Was this translation helpful? Give feedback.
-
|
That's what I thought, but it always refused the connection during the agent install on a device. |
Beta Was this translation helpful? Give feedback.
-
|
And thank you for the clarification of which URL it's for. |
Beta Was this translation helpful? Give feedback.
-
|
Gotta be something else wrong. Do you have ufw enabled on the host? Maybe need to open ports on there? It's TCP port forward you need. To be honest the URL is irrelevant as it's all hosted on the same IP and just a TCP forward. Can you share your port forward tile? Blank out anything private |
Beta Was this translation helpful? Give feedback.
-
|
I use iptables with no management interface. I'll work on getting things ready to post images of current configs. |
Beta Was this translation helpful? Give feedback.
-
|
On your host? Can you check 4222 is open? Presumably you used pfsense for the port forward? |
Beta Was this translation helpful? Give feedback.
-
|
Problem solved by adding HAProxy listener on 4222 and forwarding it properly. Appear to have a fully working setup with ALL services proxied via HAProxy. It absolutely refused to forward via port forward. Working on editing HAProxy config and screen caps to post. |
Beta Was this translation helpful? Give feedback.
-
|
Not sure if proxying the nats traffic is going to work. It is a TLS connection, so I am guessing that the TLS handshake is failing. |
Beta Was this translation helpful? Give feedback.
-
|
It's fully functional using a let'sencrypt cert, exporting it, and importing it into the env file. I edited the docker-compose file to use different exposed ports (rather than 80 and 443) since I have multiple services running on my server. HAProxy config on my PFSense device, edited to remove actual URLs and other services: Automaticaly generated, dont edit manually.Generated on: 2021-10-09 14:34global listen HAProxyLocalStats frontend http_shared frontend https_shared-merged frontend api.example.com backend rmm.example.com_ipvANY backend mesh.example.com-websocket_ipvANY backend mesh.example.com_ipvANY backend api.example.com_ipvANY Photos of PFSense gui and functional rmm and mesh connection coming after editing. |
Beta Was this translation helpful? Give feedback.
-
|
Prereqs: PFSense, ACME cert management configured with valid cert, HAProxy, exported cert/key imported into docker env file per existing docker instructions. In HAProxy on PFSense, setup the standard http to https redirect frontend on port 80 (make sure to add all used URLs into the ACLs and Actions in config). Should be able to run this along side as many other sites and services as you want to.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks |
Beta Was this translation helpful? Give feedback.
-
|
Damn. After a while it looks like the connection drops. The agent originally pulled all the info (hardware, software, etc), but won't refresh after it sat idle for some time, and won't reconnect. I'll try playing with the timeouts and keep-alive settings. |
Beta Was this translation helpful? Give feedback.
-
|
Maybe check for av removing the agent |
Beta Was this translation helpful? Give feedback.
-
|
Looks like the installer added the exceptions correctly. I disabled all A/V except Windows Defender for the purpose of testing. The Tactical RPC service keeps pausing and restarting for some reason. To clarify, after initial install of the agent on a PC, it pulls all typical info, and remote access via mesh continues to work. The Tactical agent itself seems to lose connection after a while, and won't check-in. Not familiar enough with this to know if it's the app/services or the proxy, though if it's pulling the info in the beginning, I don't know why it'd stop checking in. |
Beta Was this translation helpful? Give feedback.
-
|
I'll try some further troubleshooting and dumpster dive into logs later this evening. Maybe I'll find something that'll give me a clue. There's got to be a way to get this working properly, and I'll find it if it kills me! Muahahahaha! |
Beta Was this translation helpful? Give feedback.
-
|
Was able to figure out a few issues by looking through logs. Port forwarding for 4222 wasn't working because I had to set up both lan and wan rules, as I'm testing on my lan (derp). Also determined that PFSense exports the cert only, not the full chain cert, which needs to be manually copied and pasted into a file to be imported. This was causing nats to fail. To get the fullchain cert from PFSense when using acme and Let'sEncrypt, ssh into PFSense, enter the shell (option 8), install nano with 'pkg install nano', then 'nano /conf/acme/fqdn.fullchain' and copy and paste the cert text into the CRT file you're going to import. Haven't worked out the port 4222 api proxy yet, going over previously posted backend configs. |
Beta Was this translation helpful? Give feedback.
-
|
4222 doesn't need proxied. It just needs a TCP port forward I have said this at least twice before to you. If mats was failing but working now I would go back and check the port forwards. First of all try telnet from inside your network and then try it from outside (with ports opened) |
Beta Was this translation helpful? Give feedback.
-
|
I'm aware it doesn't need to be proxied, I've got it all working now. Nats was failing due to the cert PFSense exports by default via the GUI not containing the full chain info, per the logs. Using the process I mentioned above solves the issue. |
Beta Was this translation helpful? Give feedback.
-
|
You mentioned "Haven't worked out the port 4222 api proxy yet, going over previously posted backend configs." In the previous post hence my confirming it doesn't need proxied. Glad it's working now. |
Beta Was this translation helpful? Give feedback.
-
|
Looks like I did figure out how to proxy (sort of) 4222 using HAProxy, no port forwarding needed. The reason I've been so on that is because I don't care for leaving exposed open ports as a rule without something filtering what gets through. I've tested everything I can think of, including installing notepad++ on the remote PC via the RMM web interface. |
Beta Was this translation helpful? Give feedback.
-
|
No errors showing in the logs either, so it appears to work. I'll post screenshots in a while. Also, while I'm aware that HAProxy is an unsupported config, the listed config causes issues. The one I've posted, with the upcoming changes, should work. If anyone else can run some tests on it to verify it's fully functional, I'd appreciate it. |
Beta Was this translation helpful? Give feedback.
-
|
Here is the current config I'm working with, fully proxied. Only posting screen captures of settings that have been changed: frontend https_shared-merged frontend api.example.com backend rmm.example.com_ipvANY backend mesh.example.com-websocket_ipvANY backend mesh.example.com_ipvANY backend api.example.com_ipvANY
|
Beta Was this translation helpful? Give feedback.
-
|
@silversword411 Ignore the 'frontend API' and 'backend Tactical-API_ipvANY' sections for port 4222. The rest in my last update is good. The 4222 TCP proxy works in and of itself, but the agent stops checking in after a random period of time (between 15min and 2hrs of install). Even though the network tests in the troubleshooting docs all pass, the RPC service stops communicating for some reason. If nothing else, I managed to build out a solid HAProxy config for the rest. |
Beta Was this translation helpful? Give feedback.
-
|
@silversword411 Just finished installing HAProxy on our hosted webserver. Process is a little different from going through PFSense. I'll write up some walkthroughs (doing it anyway for myself for reference) with copy/paste/edit configs. There's some special steps necessary if you've got a drop all by default iptables firewall like I do, so I'll include those as well as an addendum. Oh, and the docker install instructions themselves are currently incorrect. The command to grab the meshcentral exe should be 'docker', not 'docker-compose', and the container name needs to be changed to 'trmm-backend'. Ran into those during my installs. |
Beta Was this translation helpful? Give feedback.
-
|
@silversword411 Here's a quick write up for HAProxy on baremetal. Hopefully it doesn't get garbled. I don't know jack about markdown. HAProxy Docker config example HAProxy installed on host server This assumes you’ve done basic setup of your host server (I'm using Ubuntu 20.04 in this example), have at least a baseline install of Docker, HAProxy, Tactical-RMM, and have gotten your wildcard Let’sEncrypt cert via the host server with auto renewal, with no services listening on port 80 or 443. If you have services using those ports, change their listening ports to other available ports.
On Debian/Ubuntu it’s at /etc/haproxy/haproxy.cfg Use whatever editor you feel comfortable with. Add “tune.ssl.default-dh-param 2048” minus the quotes under the global heading. Copy and paste the following into haproxy.cfg after the defaults heading: Replace all entries for rmm.example.com, api.example.com, and mesh.example.com with the rmm, api, and mesh URLs you specified in your docker-compose file. Also replace "example" in under the frontend https_shared-merged heading with the base domain for your new subdomains. For example, if your base domain is marty.mcfly.com, it should read Edit the backend server ip:port entries to match your config. Save and close the file.
HAProxy wants your certs combined into a single file, but it will search for the private key if it's not in your certificate file by adding ".key" to the end of the cert you specified in haproxy config. To simplify setting up any other services you may set up directly on your server, This will keep you from having to copy over the certs for HAProxy and other services each time they renew. If you would rather not do that, then do the above commands, but instead of linking privkey.pem to fullchain.pem.key, do this: Then edit haproxy.cfg again, under the frontend https_shared-merged heading, changing to
It should return "Configuration file is valid". If it doesn't, figure out where the errors are and correct them before proceeding.
Make sure the HAProxy service is enabled to start at boot, as it sometimes isn't during installation. Verify that ports 80, 443, and 4222 are open on your firewall for new incoming traffic. If you haven't already, add the rmm, api, and mesh URLs to your hosts file. Entries should look like this: Start the HAProxy service, then test access from your browser. |
Beta Was this translation helpful? Give feedback.
-
|
Aaaaaaand it got garbled. Edited it until it looked fairly decent. |
Beta Was this translation helpful? Give feedback.
-
|
I'll work on some addendums for special situations like drop all by default IPTables firewall configs (which break localhost access to containers without rules to compensate in place), PFSense HAProxy config, and some tweaks to the docker-compose example file that should prevent a few quirks I ran into, as well as a way to automatically copy the certs after renewal. |
Beta Was this translation helpful? Give feedback.
-
|
@silversword411 I ran into random issues regarding the networking using the docker-compose file as it is. Changing the network section to this solved the issue: Haven't had a single issue since making that change. Obviously the subnets/gateways except for the proxy network are arbitrary, so long as they're defined. |
Beta Was this translation helpful? Give feedback.
-
|
@silversword411 An additional edge case config that could be useful to document is a drop all by default firewall, in my case, IPTables. The issue is that when INPUT and OUTPUT are set to DROP, it breaks accessing Docker containers by localhost:port. So, if you bind the exposed ports for containers to localhost (127.0.1.1 in my case) to expose as few ports to the outside world as possible, it won't work. This is what I do when using a VM hosted by Google, Amazon, or any other hosting service. To get around that, you have to add rules to allow traffic to the internal ports of the Docker container to the Docker network the container is on. That allows you to use a proxy on the same host as the docker containers and still use localhost:exposed port to forward the traffic to the containers. I've omitted most rules that weren't relevant, leaving a few for context: The applicable rules in the above list are basically cut and paste, just change the br-xxxxxxxxxxxx entries to match the correct network from the output of After saving the new rules, restart the IPTables service, then the docker service. |
Beta Was this translation helpful? Give feedback.
-
|
In most cases the above config for a drop all by default IPTables firewall will be fine, however, if you start having issues simply add rules to explicitly allow existing and related traffic to/from the Docker network. The vast majority of the time it isn't necessary, but I recently ran into an issue with Vaultwarden where email sent would fail at the end when the . to indicate the email was complete was sent. Connected and sent the content fine, but dropped at the end and would time out. |
Beta Was this translation helpful? Give feedback.
-
|
One other change I would recommend to the compose file default config: Add driver_opts: com.docker.network.bridge.name: trmmproxy to the proxy network config. That allows firewall rules to be assigned by that interface, which will remain the same, vs the default br-randomjunk that changes under certain circumstances. |
Beta Was this translation helpful? Give feedback.
-
|
It also makes troubleshooting easier, since readouts of the firewall rules would have interface names that indicated what they were for, rather than br-randomcrap. |
Beta Was this translation helpful? Give feedback.
Here is the current config I'm working with, fully proxied. Only posting screen captures of settings that have been changed:
frontend https_shared-merged
bind 0.0.0.0:443 name 0.0.0.0:443 ssl crt-list /var/etc/haproxy/https_shared.crt_list
mode http
log global
option socket-stats
option dontlognull
option http-server-close
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl aclcrt_https_shared var(txn.txnhost) -m reg -i ^([^\.]*).example.com(:([0-9]){1,5})?$
acl rmm var(txn.txnhost) -m str -i rmm.example.com
acl api var(txn.txnhost) -m str -i api.example.com
acl api422…