Unable to update firmware using port 443

[cdonovan]

Describe the Bug

I'm not totally sure if this is a bug with the container or controller software itself. I've been running the docker container and accessing it via Traefik using the default management port (8043) for a while. Using some redirect/header modification rules in Traefik, I don't have any issues updating device firmware. However, I recently switched to using port 443 as the management port to simplify my Traefik config. After the switch, device firmware updates stopped working. The file starts uploading but never seems to complete and eventually times out after 10 minutes.

I wound up reverting my Traefik/Omada updates, so it was using port 8043 again, and once I did that, the firmware update completed successfully. At this point, I feel like the issue is likely related to the Omada software, but when I brought it up on their forum, I was advised to come here instead.

Expected Behavior

Device firmware updates should complete successfully.

Steps to Reproduce

1. Run the docker container/omada software on port 443
2. Access via a reverse proxy such as Traefik
3. Start a device firmware update

How You're Launching the Container

omada:
  container_name: omada
  image: mbentley/omada-controller:5.6
  restart: unless-stopped
  ports:
    - 8088:8088
    - 8843:8843
    - 29810:29810/udp
    - 29811:29811
    - 29812:29812
    - 29813:29813
    - 29814:29814
  environment:
    - PUID=1000
    - PGID=1000
    - MANAGE_HTTPS_PORT=443
    - TZ=America/Los_Angeles
  volumes:
    - ./omada/data:/opt/tplink/EAPController/data
    - ./omada/work:/opt/tplink/EAPController/work
    - ./omada/logs:/opt/tplink/EAPController/logs
  sysctls:
    - net.ipv4.ip_unprivileged_port_start=0
  labels:
    traefik.http.routers.omada.rule: Host(`omada.home.lan`)
    traefik.http.routers.omada.entrypoints: web-secure
    traefik.http.services.omada.loadbalancer.server.scheme: https
    traefik.http.services.omada.loadbalancer.server.port: 443
    traefik.http.services.omada.loadbalancer.serversTransport: ignoreSSL@file

Container Logs

10-28-2022 10:56:50.622 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.DeviceUpgradeStaticTask(): upgrade task upgradeId [x] of OmadacId(x) is running...
10-28-2022 10:56:50.666 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.g(): wait for all switch check file complete, upgradeId[x] deviceCount[1]
10-28-2022 10:56:55.622 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.DeviceUpgradeStaticTask(): upgrade task upgradeId [x] of OmadacId(x) is running...
10-28-2022 10:56:55.666 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.g(): wait for all switch check file complete, upgradeId[x] deviceCount[1]
10-28-2022 10:57:00.622 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.DeviceUpgradeStaticTask(): upgrade task upgradeId [x] of OmadacId(x) is running...
10-28-2022 10:57:00.623 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.DeviceUpgradeStaticTask(): v2 device macx of omadacIdx upgradeIdx finished upgrade process in FIRMWARE_DOWNLOAD_FAILED
10-28-2022 10:57:00.631 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.c.c(): OmadacId(x)  upgradeId[x] DeviceMac(x) upgrade status change from[false] to [true]
10-28-2022 10:57:00.635 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.DeviceUpgradeStaticTask(): OmadacId(x) upgrade[id=x] finish.
10-28-2022 10:57:00.638 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.DeviceUpgradeStaticTask(): upgrade task upgradeId [x] of OmadacId(x) is finished
10-28-2022 10:57:00.641 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.c.c(): OmadacId(x) upgradeId[x] releaseUpgradeCatelog finished
10-28-2022 10:57:00.666 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.g(): wait for all switch check file complete, upgradeId[x] deviceCount[1]
10-28-2022 10:57:00.669 INFO [comm-pool-14] [] c.t.s.o.m.d.d.m.u.t.g(): all switch check file complete, upgradeId[x] check ok device count[0], start reboot

Additional Context

I'd like to try running the controller software natively, but on port 443 to try and rule out Docker or Traefik as the culprit, but am not sure how easy it would be to convert the docker install to bare-metal. Would it be just installing the native client (probably on a spare machine) and copying the three bind mount folders to the same locations after the install is complete?

[mbentley]

Well, when looking at this FAQ document, I am not sure what it means when it comes to port 8043 as it mentions the use for the port is:

1. When you visit the management page of Omada Software Controller via an HTTPS connection.
2. When upgrading the firmware for the Omada devices with Omada Software Controller.

The usage for number two there is what I am not exactly sure of. When I uploaded a firmware to perform a downgrade on one of my WAPs, it didn't start listening on any other ports so I don't think it needs to do that specifically during an upgrade or anything.

I use port 443 using macvlan on my docker container but do so directly without Traefik in front. What part of the upgrade does it fail on? I was able to upload a firmware fine and it applied successfully.

When you say that you're having problems uploading the firmware, do you get to the point where you can upload the firmware, it shows a uploaded and then you get a button to click to perform the upgrade or does it fail just uploading the file? If the problem is when uploading the file, could you open your browser's developer tools to see if there are any errors reported on the network tab while uploading?

You could try to run the controller natively - you would either need to try to do a backup/restore via the software or you could just try to copy over the persistent data directories to where the controller expects them and that ought to work, as long as you have matching OS versions of packages, especially MongoDB; not sure about how picky it might be with Java.

[mbentley]

Any update on where exactly it was failing?

[SamMousa]

I have the same issue. It likely has to do with the way the device tries to connect to the controller.

I'm using traefik reverse proxy with a certificate from LE. The management hostname is set to a public dns record that resolves to the IP of traefik.

[cdonovan]

I haven't had a chance to play with the network really the past few weeks. Glad (is that the right word???) that someone else has experienced it as well though. I might be able to find some time between work & family being home this week to bring the wifi down to test this.

[mbentley]

Also, do you have a link to the forum thread where you asked about this? I am curious if they can have someone take a look at the behavior of the update code to see the controller is advertising a non-routable address or a port that isn't exposed through the reverse proxy. The port documentation makes it sound like it should be using the https port for upgrades but it wouldn't surprise me if there is something untested with a reverse proxy in place.

[cdonovan]

Here's a link to the thread - https://community.tp-link.com/en/business/forum/topic/587358. Just to clarify though, I'm not too sure that the reverse proxy itself is causing the issue, since it works fine behind traefik using the "standard" management port (8043).

The proxy config for using port 8043 is a bit convoluted (has to adjust some headers & redirects for the auth flow), but in the end, firmware updates still work. It was when I changed (i.e. simplified) the config to remove the extra header/redirect stuff for the login and used port 443 directly by changing the management port in the Omada UI. When I did that, I removed the original port 8043 from being exposed to the host network. I wonder, even though the port was changed in the Omada software (and was indeed accessed via 443), if some things related to the firmware update flow were still looking to communicate on port 8043 (which now had access cut off). One of my planned experiments was to keep the new config (using port 443) but keeping 8043 still exposed. As soon as I get a change, I'll let you know how it goes.

[cdonovan]

Over the holiday break I finally had some time to think about this issue further and realize the likely problem. Traefik is currently exposed on port 443 and I'm mostly using http host rules to route to the correct container.

For the Omada controller, that's something like omada.internal, which (although used by me to manage the network via the ui), is likely not what is used by the actual devices 🤦. I can't expose port 443 for Omada specifically, because it's already being used by traefik, so I think the only way around this would be to set up additional routing rules. I'm not sure if the firmware update communication uses http or just tcp and what specific criteria could be used to correctly route to the omada container. Perhaps it's actually simpler to keep the login redirect hack (utilizing the unoccupied default port) instead of adding additional routing logic that could be just as fragile as the login redirect itself.

I think this can likely be closed, although if you do have any information on the actual firmware update logic (protocol, common headers, etc.) it could prove useful in trying to set up the proxy routing.

[mbentley]

This is really confusing as to where any why it would be failing. If I am understanding correctly, you're only exposing Traefik on 443 in both scenarios, right? I wasn't sure if your mention of it working when the controller is using 8043 was there you had 8043 also exposed on Traefik as well or if it was still only using 443 but with more complex rewrite rules.

I wish they would provide some more detailed information about how the upgrade process worked as it is hard to understand why it doesn't work.

[cdonovan]

I think the problem is primarily that the device itself needs to communicate back to the controller. When the controller is running on 8043 it can do so, since nothing else is using that port. However, in the scenario where the controller is running on port 443, traefik is already using that port. I imagine the packets go into traefik and die because traefik doesn't know where to send them; the only omada rules I have in place are based on the http request's hostname (e.g. send packets on 443 for host omada.internal to the omada container).

If the device uses http to communicate its update status, perhaps additional rules could be created to also route those particular packets to the omada container. The available rules for http are listed here https://doc.traefik.io/traefik/routing/routers/#rule (can use host, headers, path, etc.)

If the communication is straight TCP (or UDP) the matching is more limited and might not even be possible.

[mbentley]

So if the configuration is Traefik is listening on 443 and 8043 with only the Omada Controller on 8043, that would mean that the Omada controller should be the default server that traffic is sent to if no host match is found. Makes me curious if the device are just not passing a hostname or what exactly the devices are sending for a host header or SNI in the case of https. I suppose the quick test would be to set the Omada Controller as the default server backend to use in case of no matches and see if it succeeds then.

That's the only scenario where I can think of where it would succeed with Traefik as the reverse proxy to 8043 but it would fail using 443, common with other services.

[legonigel]

I also have hit this in my k3s cluster using Traefik ingress. I definitely remember doing upgrades to my devices sometime last year without issue, but it doesn't work any longer. I know that 5.7 added the "Device Management Hostname/IP" option, so I wonder if something changed with 5.7.

[dsander]

I might have/had a very similar problem, firmware updates used to work but then didn't. I am also using traefik to handle the SSL termination. To fix it on version 5.8 I have set Device Management Hostname/IP to the same value as Controller Hostname/IP and firmware updates started working again. I have tried to roll back the change to reproduce the problem but then it still worked, not sure if the change somehow persisted on the clients or what is up with that.

Nvm what I have written before. No idea if it worked somehow or I was just thinking it worked, I am now back to where everyone else is. tp-link must have done something really strange with one of their updates. One of my next steps will probably be to move the container to a separate host to test if it works without traefik.

[mbentley]

So while I know this isn't going to solve the issue for everyone, there is a help article related to failures of firmware upgrades that may explain some of the behaviors that people are seeing: https://community.tp-link.com/en/business/forum/topic/559150

[IndiGP]

Its very interesting, that only switches and routers are not capable of updating firmware. Accses Points are still getting updates from my controller behind nginx revers proxy (manager).. Very strange behaviour.
Tested with AP653 from version 1.0.0 --> 1.0.4

[mbentley]

I would assume that would be because certain (newer) firmwares will use the controller's HTTPS port for updates, other older firmwares will use a specific update port as documented in the linked forum post above.

[IndiGP]

I would assume that would be because certain (newer) firmwares will use the controller's HTTPS port for updates, other older firmwares will use a specific update port as documented in the linked forum post above.

Yes, but every port is open.. I set mgm port in controller to 443 and on reverse proxy on 443 and I am forwarding every port from 29810-14 to container instance itself. So very annoying thats its not working anymore. Hope there will be an update in future for that.

[dougmeredith]

Another data point on this problem:

I have the Omada container behind Traefik and encountered the same issue with not being able to update the firmware on a switch. The following confniguration allowed me to update the firmware:

• Omada conainer uses the defaults for all ports.
• Traefik listens on ports 80 and 443.
• Traefik is configured to use an upstream of https://omada:8043 (omada being the container name)
• I mapped port 8043 on the Omada container to port 8043 on the host.

That was it. I didn't even configure Omada with a valid certificate. I simply connected to the managment interface through Traefik and the firmware upgrade now worked.

[troyfontaine]

Just to add to this, if your Omada controller is sitting on two VLANs, i.e. one default and one for managing switches (and where they have their own addresses assigned from), then you need to expose 8043 on both VLANs.

[kaysond]

I just ran into this, but didn't want to needlessly expose port 8043 on my host. I did a little bit of digging to understand what a device does when you try to do a firmware upgrade, and it's really quite simple:

• sends an https request to inform -ip address- on the port specified in the HTTPS Port for Controller Management setting
• request is a GET with a path like /upgrade/files/62250191c7234d4ba757f75e0e818880/01fd6e30fbab569ae4247d971613a4b9
• the Host header is set to <inform ip>:<port>

This doesn't work for a number of reasons, depending on your exact setup:

• reverse proxy is probably listening on port 443, not 8043 which is the default
• you're not mapping port 8043 from the container to the host
• reverse proxy is not expecting the Host header that is sent, but probably a custom domain name

There are a number of ways to address this, including the above which bypasses the reverse proxy by allowing the device to communicate with the container directly on port 8043.

As mentioned, I didn't want to do that since it's unnecessary, so I did the following instead:

1. add a DNAT redirect from port 8043 to 443 on my router
2. configure my reverse proxy to forward requests to Host: ip:port to the controller container

This works just fine. It's annoying that it's a problem at all when Unifi deals with this seamlessly, so I submitted this feature request to allow the controller to handle reverse proxies: https://community.tp-link.com/en/business/forum/topic/706282?page=1

[morkyy]

Had this issue as well, in my opinion it's a major flaw to handle updates like this.

There's absolute no reason for the updates to go through the controller management port, and it gets very messy if you are working with gateways/switches in remote locations.

Simplest solution as pointed above would be to bypass the reverse proxy but this is very much against best practices and very insecure for dealing with remote device updating as it means we need to expose the controller management port (18043) to the internet.