Tesla Cloud Option #59
Conversation
|
Hi Jason, I have been testing this and checking that the cloud mode API responses attempt to replicate real TEG responses as closely as possible. Can you check what does your TEG return for the following API requests? For {
"real_mode": "self_consumption",
"backup_reserve_percent": 33.5
}My TEG doesn't include the items I have commented out below, but cloud mode is returning these - does your TEG include them? data = {
"real_mode": default_real_mode,
"backup_reserve_percent": backup_reserve_percent
# "freq_shift_load_shed_soe": 0,
# "freq_shift_load_shed_delta_f": 0
}Also for {
"site_name": "My powerwall",
"timezone": "Australia/Sydney"
}But the cloud mode response for this API is returning a lot more data, should it be? data = {
# "max_system_energy_kWh": nameplate_energy,
# "max_system_power_kW": nameplate_power,
"site_name": sitename,
"timezone": tz
# "max_site_meter_power_kW": max_site_meter_power_ac,
# "min_site_meter_power_kW": min_site_meter_power_ac,
# "nominal_system_energy_kWh": nameplate_energy,
# "nominal_system_power_kW": nameplate_power,
# "panel_max_current": None,
# "grid_code": {
# "grid_code": None,
# "grid_voltage_setting": None,
# "grid_freq_setting": None,
# "grid_phase_setting": None,
# "country": None,
# "state": None,
# "utility": utility
# }
} |
|
Ha! We are looking at the same thing. I spotted the mismatch on the APIs for /site_info - new commit And yes, my /api/operations is more verbose - which is odd. I'm basically doing side-by-side with one python session using the local connection, the other the cloud, and running the same |
Interesting - all good! Note I committed a fix too, for the backup reserve percent which needs scaling applied. |
|
I saw the SOE fix, awesome!! I was testing that about the same time your commit came through. One interesting thing I have noticed is that the frequency of SOE update on the cloud is a lot less than local (looks like it is close to 1 minute updates). But it seems like the power data ( |
|
Actually I haven't checked the soe ("percentage_charged" response value) yet, only the backup reserve percent... my Powerwall is at 100% and I need the sun to go down to test it!! 😄 I have noticed the update time differences as well. "SITE_DATA" (live_status) updates quite frequently, but the others like "SITE_CONFIG" could be delayed, even up to 30mins. If the value exists in the live_status response obviously it is best to use that. |
|
Published beta container image for anyone wanting to test: jasonacox/pypowerwall:0.7.0t33beta |
|
With the changes made regarding TTL I don't think this has fixed the root cause of the issue. I actually noticed this issue a few days ago but hadn't had time to work on a fix. I have worked out a good solution this afternoon however, so once I finish testing will commit the changes I recommend. This probably needs an explanation though.
One of the reasons this is occurring is because telegraf requests all of the input URLs simultaneously (every 5 seconds) and pypowerwall is running as a multi-threaded HTTP server. So, to service the telegraf requests, then we have multiple pypowerwall threads each sending a cloud API request to the Tesla servers - and this could be the same API request - and since they are all sent at the same time there has been no time for a response to be cached. I have a change almost ready that essentially adds a mutex lock around the cloud API requests. This means only one thread would send the API request, while the other threads will wait, and when the lock is released those waiting threads will return cached data. It's much friendlier on the Tesla servers, and would probably resolve the 429 error and may also mean the TTL changes are not required. I have tested with the TTL changes and confirmed that this issue is still present. Below is some log output where you can see multiple of the same cloud API request is sent in the same second (e.g. look for "Fetching new data for SITE_CONFIG" etc. which is being sent simultaneously in different threads). Here is the difference with the changes I'm working on... only 1 thread sends the SITE_CONFIG request (for example), the other threads wait and then return cached data once the 1st threads gets the response. It happens within a fraction of the second. Also I was testing what happens when the Internet drops, and am tweaking retries/timeouts as it was not great. |
This is great! ❤️ I do think the App is sending multiple concurrent requests as well, just nothing like the telegraf blast. The TTL hack did enough to eliminate the throttling from Tesla in my testing, but that doesn't mean it wasn't still extreme and an unkind level of requests (compared to the App). Getting this mutex solution in place should mitigate that and is a much cleaner solution than the TTL logic. Thanks! I didn't have a lot of time to spend on it today. I did push a few small changes to our stats API and logging to help indicate when the proxy is in cloud mode and to provide additional information related to that for troubleshooting help. Also, testing using this new "cloud mode" code is working for multiple Dashboard instances I'm testing on different OSs. Submit your mutex changes and I'll roll it out to my test platforms too. Also, I started updating Powerwall-Dashboard (mainly powerwall.yml to map the TeslaPy auth and site files) as needed for my testing. Added some bits to verify.sh as well. New Branch is v4.0.0: jasonacox/Powerwall-Dashboard@a0ed67c |
|
Hi Jason, Some notes on my changes as they might need further explanation. Hopefully I have not made any breaking changes. My testing shows all working well, but please test and let me know if you find any issues!
|
|
This is brilliant, @mcbirse !! I made a comment in your commit about abstracting the cache and mutex code in each of the get_*() functions into a central api call management function, but that is minor and could be handled later if it makes sense. Give it a try: jasonacox/pypowerwall:0.7.1t34 Testing now... ✅ MacOS - local mode, cloud mode |
|
I agree, putting that replicated code into a central api call management function definitely makes sense. I was considering that but left the changes in each function for easier comparison for now. |
|
I like keeping the separate get_*() functions named after their logical payload (the TeslaPy mapping isn't very intuitive IMHO). It also abstracts our API from TeslaPy changes. I created a central function def _site_api(self, name, ttl, **args):
"""
Get site data from Tesla Cloud
name - API name
ttl - cache time to live in seconds
args - Additional API arguments
Returns (response, cached)
"""
if self.tesla is None:
return (None, False)
# Check for lock and wait if api request already sent
if name in self.apilock:
locktime = time.perf_counter()
while self.apilock[name]:
time.sleep(0.2)
if time.perf_counter() >= locktime + self.timeout:
return (None, False)
# Check to see if we have cached data
if name in self.pwcache:
if self.pwcachetime[name] > time.perf_counter() - ttl:
return (self.pwcache[name], True)
try:
# Set lock
self.apilock[name] = True
response = self.site.api(name,args)
except Exception as err:
log.error(f"ERROR: Failed to retrieve {name} - {repr(err)}")
response = None
else:
self.pwcache[name] = response
self.pwcachetime[name] = time.perf_counter()
finally:
# Release lock
self.apilock[name] = False
return (response, False)
def get_battery(self):
"""
Get site battery data from Tesla Cloud
...
"""
# GET api/1/energy_sites/{site_id}/site_status
(response, cached) = self._site_api("SITE_SUMMARY",
self.pwcacheexpire, language="en")
return response
def get_site_power(self):
"""
Get site power data from Tesla Cloud
...
"""
# GET api/1/energy_sites/{site_id}/live_status?counter={counter}&language=en
(response, cached) = self._site_api("SITE_DATA",
self.pwcacheexpire, counter=self.counter, language="en")
if not cached:
self.counter = (self.counter + 1) % COUNTER_MAX
return response
def get_site_config(self):
"""
Get site configuration data from Tesla Cloud
...
"""
# GET api/1/energy_sites/{site_id}/site_info
(response, cached) = self._site_api("SITE_CONFIG",
SITE_CONFIG_TTL, language="en")
return response
def get_time_remaining(self):
"""
Get backup time remaining from Tesla Cloud
{'response': {'time_remaining_hours': 7.909122698326978}}
"""
# GET api/1/energy_sites/{site_id}/backup_time_remaining
(response, cached) = self._site_api("ENERGY_SITE_BACKUP_TIME_REMAINING",
self.pwcacheexpire, language="en")
return response
|
|
At first glance looks good to me! I like the idea that it returns whether the response was a cached response or not. Also I think it might be helpful to add debug log output showing when we actually return the response data, including whether that was a cached response or not - what do you think? ERROR log will currently output already on timeouts. i.e. DEBUG logs shows when pypowerwall receives a request, the required cloud request, but not when we get the response (or return cached response). I had added additionally logging for this when testing which was invaluable. |
|
I pushed the |
| if sites is None or len(sites) == 0: | ||
| print("\nERROR: No sites found for %s" % self.email) | ||
| return False | ||
|
|
I should remove the "ERROR" prefix in the log.error() calls. 😀 But otherwise, logging is working well. |
|
This is looking good! Successful tests using jasonacox/pypowerwall:0.7.1t35beta pre-release. ✅ RPi - cloud mode and local mode As a note, helpful page for confirmation of running proxy: http://localhost:8675/help I'm going to squash, merge to main and push pypowerwall v0.7.1 to PyPI for next round of non-beta testing. @mcbirse great job! Feel free to continue to branch and submit updates if we need to v0.7.2. |
|
v0.7.1 Released: https://pypi.org/project/pypowerwall/0.7.1/ |
This updates provides two updates: