Extremely high data upload!

[VIMVa]

Help!
Since installing the add-on yesterday evening after a system restore, my data upload between HA and OneDrive is at 143GB!!! That is reported by my router.
I see a lot of entries in the log file:
[24] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar
It is true that I had 2 backups with that same name, just because I had to restore the first one and re-upgrade to HA 2023.9.2.
I can but suspect that the add-on keeps uploading that 547MB file continuously in such case.
I deleted one of the backups within HA and on my OneDrive. Traffic is back to normal.

[lavinir]

Hi,
Not sure I completely follow that.
the 143GB was uploaded in a day ? You can see anything uploaded in the log. Looping on the same backup would be a bug if that's what happened.
Is this still happening ? Do you have logs you can share ?

[VIMVa]

In 1 day, yes. It was stopped by deleting one of the files with the same name.
Looks indeed like looping.

Most recent log:
09/14/2023 19:28:38 [36] Info: Found 1 backups to upload. 09/14/2023 19:28:38 [36] Info: Uploading core_2023.9.0 (09/11/2023 10:07:31) 09/14/2023 19:28:38 [36] Info: Fetching Local Backup (Slug:bcc0c0bb) 09/14/2023 19:28:43 [36] Info: Backup (bcc0c0bb) fetched successfully 09/14/2023 19:28:44 [36] Info: Starting file upload. (Size:573880320 bytes. Attempt: 1/3) 09/14/2023 19:31:23 [52] Info: Upload completed successfully 09/14/2023 19:31:23 [52] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:31:23 [52] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:33:39 [52] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:33:39 [52] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:33:39 [52] Info: Found 1 backups to upload. 09/14/2023 19:33:39 [52] Info: Uploading core_2023.9.0 (09/13/2023 15:54:18) 09/14/2023 19:33:39 [52] Info: Fetching Local Backup (Slug:7757f87b) 09/14/2023 19:33:45 [55] Info: Backup (7757f87b) fetched successfully 09/14/2023 19:33:45 [55] Info: Starting file upload. (Size:585523200 bytes. Attempt: 1/3) 09/14/2023 19:37:04 [31] Info: Upload completed successfully 09/14/2023 19:37:04 [31] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:37:05 [31] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:38:40 [14] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:38:40 [72] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:38:40 [72] Info: Found 1 backups to upload. 09/14/2023 19:38:40 [72] Info: Uploading core_2023.9.0 (09/11/2023 10:07:31) 09/14/2023 19:38:40 [72] Info: Fetching Local Backup (Slug:bcc0c0bb) 09/14/2023 19:38:46 [14] Info: Backup (bcc0c0bb) fetched successfully 09/14/2023 19:38:47 [14] Info: Starting file upload. (Size:573880320 bytes. Attempt: 1/3) 09/14/2023 19:41:25 [61] Info: Upload completed successfully 09/14/2023 19:41:25 [72] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:41:25 [26] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:43:41 [13] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:43:41 [13] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:43:41 [13] Info: Found 1 backups to upload. 09/14/2023 19:43:41 [24] Info: Uploading core_2023.9.0 (09/13/2023 15:54:18) 09/14/2023 19:43:41 [24] Info: Fetching Local Backup (Slug:7757f87b) 09/14/2023 19:43:47 [24] Info: Backup (7757f87b) fetched successfully 09/14/2023 19:43:48 [24] Info: Starting file upload. (Size:585523200 bytes. Attempt: 1/3) 09/14/2023 19:46:29 [26] Info: Upload completed successfully 09/14/2023 19:46:30 [13] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:46:30 [63] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:48:42 [55] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:48:42 [61] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:48:42 [61] Info: Found 1 backups to upload. 09/14/2023 19:48:42 [13] Info: Uploading core_2023.9.0 (09/11/2023 10:07:31) 09/14/2023 19:48:42 [13] Info: Fetching Local Backup (Slug:bcc0c0bb) 09/14/2023 19:48:48 [57] Info: Backup (bcc0c0bb) fetched successfully 09/14/2023 19:48:48 [57] Info: Starting file upload. (Size:573880320 bytes. Attempt: 1/3) 09/14/2023 19:51:28 [36] Info: Upload completed successfully 09/14/2023 19:51:29 [69] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:51:29 [36] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:53:43 [63] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:53:43 [20] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:53:43 [20] Info: Found 1 backups to upload. 09/14/2023 19:53:43 [20] Info: Uploading core_2023.9.0 (09/13/2023 15:54:18) 09/14/2023 19:53:43 [20] Info: Fetching Local Backup (Slug:7757f87b) 09/14/2023 19:53:55 [36] Info: Backup (7757f87b) fetched successfully 09/14/2023 19:53:55 [36] Info: Starting file upload. (Size:585523200 bytes. Attempt: 1/3) 09/14/2023 19:56:40 [56] Info: Upload completed successfully 09/14/2023 19:56:41 [24] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:56:41 [13] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:58:44 [14] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:58:44 [14] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 19:58:44 [14] Info: Found 1 backups to upload. 09/14/2023 19:58:44 [14] Info: Uploading core_2023.9.0 (09/11/2023 10:07:31) 09/14/2023 19:58:44 [14] Info: Fetching Local Backup (Slug:bcc0c0bb) 09/14/2023 19:58:57 [24] Info: Backup (bcc0c0bb) fetched successfully 09/14/2023 19:58:58 [24] Info: Starting file upload. (Size:573880320 bytes. Attempt: 1/3) 09/14/2023 20:01:40 [47] Info: Upload completed successfully 09/14/2023 20:01:41 [14] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:01:41 [13] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:03:45 [56] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:03:45 [56] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:03:45 [56] Info: Found 1 backups to upload. 09/14/2023 20:03:45 [69] Info: Uploading core_2023.9.0 (09/13/2023 15:54:18) 09/14/2023 20:03:45 [69] Info: Fetching Local Backup (Slug:7757f87b) 09/14/2023 20:03:56 [56] Info: Backup (7757f87b) fetched successfully 09/14/2023 20:03:57 [56] Info: Starting file upload. (Size:585523200 bytes. Attempt: 1/3) 09/14/2023 20:06:43 [56] Info: Upload completed successfully 09/14/2023 20:06:44 [56] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:06:44 [56] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:08:46 [61] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:08:46 [61] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:08:46 [61] Info: Found 1 backups to upload. 09/14/2023 20:08:46 [55] Info: Uploading core_2023.9.0 (09/11/2023 10:07:31) 09/14/2023 20:08:46 [55] Info: Fetching Local Backup (Slug:bcc0c0bb) 09/14/2023 20:08:53 [52] Info: Backup (bcc0c0bb) fetched successfully 09/14/2023 20:08:53 [52] Info: Starting file upload. (Size:573880320 bytes. Attempt: 1/3) 09/14/2023 20:11:33 [55] Info: Upload completed successfully 09/14/2023 20:11:33 [44] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:11:34 [55] Warning: Unrecognized file found in backup folder : core_2023_9_0.tar 09/14/2023 20:13:29 [14] Info: Downloading backup core_2023.9.0.tar 09/14/2023 20:19:15 [72] Info: core_2023.9.0.tar downloaded successfully 09/14/2023 20:19:15 [72] Info: Loading backup core_2023.9.0.tar to Home Assisant 09/14/2023 21:03:58 [65] Info: Creating new backup 09/14/2023 21:03:58 [65] Info: Starting full local backup 09/14/2023 21:23:54 [65] Info: Backup complete 09/14/2023 21:24:02 [40] Info: Found 1 backups to upload. 09/14/2023 21:24:02 [60] Info: Uploading HA OneDrive_2023-09-14-21-03 (09/14/2023 21:03:58) 09/14/2023 21:24:02 [60] Info: Fetching Local Backup (Slug:8392c377) 09/14/2023 21:24:08 [63] Info: Backup (8392c377) fetched successfully 09/14/2023 21:24:09 [63] Info: Starting file upload. (Size:596643840 bytes. Attempt: 1/3) 09/14/2023 21:26:56 [40] Info: Upload completed successfully

[lavinir]

Thank you. I see what's causing this. Is this still happening now though ?
Could you please go to that backup in OneDrive, look at the file details and send what you see under the File Info ? There should be metadata relevant to the backup :

[VIMVa]

I can give that info for one of the files only as I deleted the one in HA that was causing the continuous upload in order to stop that:
{"Slug":"bcc0c0bb","BackupDate":"2023-09-11T10:07:31.568053+02:00","InstanceName":null,"IsProtected":false,"Size":547.29,"BackupType":"partial","Addons":[],"Folders":[]}

[flyguy62n]

Not quite as stark as the 143GB report above, but I've noticed a big difference in data usage when this Add-on is running. I have two HA instances -- one at home where the data plan is unlimited, but the other is at a remote location with an LTE modem and a 15GB/mo data plan. At this second location this week, I noticed 8GB in data in a 12 hour period. I disabled the add-on and the data usage leveled off. When I turned it back on again, instant spike in data.

This has probably been the case for at least a couple of months as in October I noticed that my data plan was completely consumed in less than 2 days. But being super busy then, I couldn't really troubleshoot it other than turn everything off. It was only this week that I could get back to it and trace the consumption to the OneDrive Backup add-on.

I'm open to any ideas on how to limit data consumption. I've already tried cutting back the backups to weekly as well as limiting the number of copies to keep. But as soon as I start the Add-on again, I can watch the data usage spike. Thoughts? Or maybe there's a "Low Bandwidth" mode that could be added?

[lavinir]

I want to separate the discussion between something like the original issue above, which was a bug in the addon causing the same upgrade backup to be uploaded over and over again, and 'normal' usage which may be consuming a lot of bandwidth due to a combination of settings and backup size.

If you share your logs during the time you start the addon and see increased usage it would help get insights as to what the addon is doing. You can also take a look at the actual backup files in OneDrive, if you see their size is much less than the uploaded size in the timespan the addon was working this could indicate a problem / bug.

Also, if you believe there is an actual issue please open one see it's tracked as such.

[flyguy62n]

Thanks for the quick response! I had to wait for a couple of days before I tried anything as tomorrow is the last day before the LTE data usage resets, so if it sucked up the rest of the data, I would only be without for one day! Anyway, I just started it up and watched the logs. I also took a look at the files in the Apps folder on OneDrive. It uploaded a few things and then settled down right away. I'm guessing what it was doing was that it had a bunch of files to upload. I'm going to monitor it for a while on my new settings (only weekly backups -- not much changes at the hangar) and see if the data usage spikes again. Hmmm... gives me an idea for a Data Usage Rate sensor to implement and generate alerts when it starts climbing. Anyway, you can close the issue for now if you'd like. I'll reach out if there anything else that comes up while watching it!

…

-- Randy Bartels 602.738.3588

________________________________ From: lavinir ***@***.***> Sent: Saturday, December 9, 2023 1:36 PM To: lavinir/hassio-onedrive-backup ***@***.***> Cc: flyguy62n ***@***.***>; Comment ***@***.***> Subject: Re: [lavinir/hassio-onedrive-backup] Extremely high data upload! (Discussion #109) I want to separate the discussion between something like the original issue above, which was a bug in the addon causing the same upgrade backup to be uploaded over and over again, and 'normal' usage which may be consuming a lot of bandwidth due to a combination of settings and backup size. If you share your logs during the time you start the addon and see increased usage it would help get insights as to what the addon is doing. You can also take a look at the actual backup files in OneDrive, if you see their size is much less than the uploaded size in the timespan the addon was working this could indicate a problem / bug. — Reply to this email directly, view it on GitHub<#109 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO7HPUG3YALZOCTC3XCBO6LYISVTHAVCNFSM6AAAAAA4YSQXK6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMBYGM2DK>. You are receiving this because you commented.Message ID: ***@***.***>