An important change regarding Enphase Auto Uploader systems

[grannos]

Hi @tycho0. My daughter’s P.V. system has an Enphase Envoy which will be affected by Enphase’s decision to ‘retire’ their current API.

I have written and am currently testing a short script to poll the Envoy directly and then to push that data directly to PVO. A side benefit will be that the data will be updated at 5 minute intervals in the future rather than the current 15 minute interval.

She does NOT have a consumption meter so I will only be able to extract and push v1 and v2.

The critical API call, for her Envoy at least, is “http://”.$enphaseIP."/api/v1/production".

[EDIT]
I found this URL to be a source of useful information.

Enphase Envoy-S “Data Scraping”.

[/EDIT]

Grannos

[tycho0]

Hi @grannos,

That’s great news! I’m excited to hear it.

I do have consumption CTs installed on my system. Do you think you’d be able to add consumption data to your script? Do you need access to a system with consumption data to test against? I wonder if that’s something I could help out with?

It looks like http://envoy.local/production.json contains both production and consumption data? There’s also http://envoy.local/api/v1/consumption

And it looks like you’ll have to start generating a 6-month token when the IQ Gateway firmware is updated to v7.
https://enphase.com/download/iq-gateway-access-using-token-tech-brief

Thanks!

[bankstownbloke]

UPDATE: There has been an initial discussion with Enphase APAC to explore the possibility of making an exemption for PVOutput members with the new Enphase API v4 terms.

We should have a decision by July 2022.

[chilldev]

I just installed an IoTaWatt energy monitoring device on my main panel to monitor specific circuits/devices and to also add consumption monitoring as my Envoy is at the array 200 feet away and I never installed the consumption CT’s at install. I compared the data from both yesterday and they are spot on so I am just stopping the Enphase data pull. Instead I will push the data from IotaWatt instead.

Here is my post about my experience setting that up.

Enphase Solar Tracking with PVOutput

[rjodwyer]

Seems they are using the Microsoft playbook of get big from your partners then screw them all over.

I can’t find the link now but there was mention with the updated firmware that the refresh time would be going up even with the local API.

So even if we get to keep local API access the refresh time may make it less useful, assuming in future firmware the local API is still there(they seem to flip/flop regularly)

I’m a sucker and pay for Enlighten Manager so I could get per panel stats, by the sound of the above it seems even I would lose API access.

[mspohr]

I am also upset by this change. Enphase has always been difficult to work with if you are a residential customer. Years ago they made a change which resulted in a sawtooth graph and just ignored requests to fix it. That’s one of the reasons I’ve stopped installing them and don’t recommend them.
(Don’t think this is an American company thing, just universal corporate arrogance.)

[chilldev]

Weird, I have never seen this issue. Is this on the old pre IQ micros as you say years ago?

[chilldev]

I have had access to Enlighten Manager for free since I installed my system DIY in 2018. I always read that many people did not have access as their installer would not give them access. It will be interesting to see what happens in August but IMO the MyEnlighten shows enough data for 99% of people anyway.

[mspohr]

Top is from MyEnlighten, bottom from PVOutput.
Enphase switched from 5 minute intervals to 15 minute intervals in MyEnlighten years ago (I guess their servers couldn’t handle the data.). PVOutput used to get nice curves. Now I get a 15 minute sawtooth as Enphase does something weird with the 5 minute intervals that it reports to PVOutput.
These are M250 inverters.

[mspohr]

Here’s the original discussion of the change in reporting interval.

Enphase change reporting interval?

Enphase does still send “report” power (kW) data in 5min intervals, but mine is not correct, probably computed from stored 15min energy (kWh) data. Here’s my enlighten graph and a sample of the report data from yesterday. Note the strange repeating sequence… 2 equal values then a lower one resulting in a sawtooth graph. Graphs and data before 6/1/17 are smooth 5 min data (except for clouds!) and are all sawtooth after that. But “todays” graph (shown last) is ALWAYS stair-step, and that’s the d…

[chilldev]

Yeah I guess I sorta noticed the sawtooth although mine seems to be a bit less. I got the IoTaWatt a few days ago and pretty much immediately decided to get rid of the enphase data pull from PVoutput.

My arrary is 200 ft from the panel so it is prob more accurate as well and smooth like butter.

Check out that voltage spike around 4 that caused the micros to cut off. Weirdly I do not see a high voltage alert from Enphase for this though.

Sorry for hijacking this thread!

[vk2him]

@grannos - Note that the Envoys are currently running two main versions of Software/Firmware 5.x.x and 7.x.x

5.x.x allows access to the local API using the Installer user and the password generated by that Data Scraping" method you posted.

7.x.x changes this, access to the local API is via a cloud token that is valid for up to 6 months. It is however currently “broken” as most API url’s fail with 504 errors. The broken urls include the critical one you mention - “http://”.$enphaseIP."/api/v1/production".

I’m in Australia and I had 5.x.x and I wrote a number of scripts to scrape data from other api urls which turns the data into an MQTT stream https://github.com/vk2him/Enphase-Envoy-mqtt-json
Earlier this year Enphase remotely upgraded my Envoy to 7.x.x and my scripts stopped. I haven’t been able to get them working again as the 7.x.x api is broken.

There is a huge argument underway on the Enphase Community forum discussing the poor way that Enphase is rolling out 7.x.x as it breaks existing scripts and seems to be a forerunner to them charging for access to our own data. The announcement that PVOutput will no longer be able to use the API is conclusive proof about our fears of 7.x.x being all about charging users to access their own data.

Here’s a link to the (very) long Enphase Community thread.

https://support.enphase.com/s/question/0D53m00006ySLuRCAW/unimpressed-with-loss-of-local-api-connectivity-to-envoys?s1oid=00DA0000000aQ9D&s1nid=0DB2G000000Kz2X&emkind=chatterCommentNotification&s1uid=0053m00000CMsZ3&emtm=1648502326593&fromEmail=1&s1ext=0

[grannos]

Hi @vk2him

My daughter’s Envoy is an white coloured / oval shaped beast. It’s firmware appears to be R3.17.3 (0e8c7a) as viewed via its web GUI. It appears to be an ‘Envoy-R’. Oddly the Envoy’s API itself only returns updated data every 15 minutes. I wonder if the Envoy only polls the micro inverters every 15 minutes!

Thanks for the link to the Enphase Community Forum.

The [ PHP ] script that I’ve read can extract and upload the data via “http://”.$enphaseIP."/api/v1/production" without any authentication.

Thanks to @tycho0 I can now access http://".$enphaseIP."/api/v1/production/inverters using the installer account to extract the individual POWER output of the six panels that she has. I’ve just got that part of the script working as the suns sinks here ( Perth ). The data are uploading as v7 … v12.

Grannos

[madbrain]

I also have an Envoy-R, installed in 2010, which shows this currently. Serial and MAC address have been cut off.

I hit F5 in the browser on the page to refresh it over the course of several minutes. The kW data did not change. However, the “Last connection to website” time increased, and then reset back to 0, in much less than 15 minutes. So, it seems the firmware may be polling the micro-inverter data and average it over 15 minute periods. But it’s definitely trying to upload data more frequently than every 15 minutes.
The data being uploaded to Enlighten is more detailed.

This could be because two of the 40 micro-inverters have powerline communication issues. Even after hardware replacement with the newest hardware revision last December, they continue to do so. They eventually report the total kWh either late in the day after dark, or the next day. But power curves throughout the day are all wrong because of these 2.

I have my Envoy integrated in HomeAssistant, which just polls data from that page .

The data in HomeAssistant indeed looks like sawtooth graph, with data changing almost exactly at 15 minute intervals. It’s wrong, because it’s based only on the homepage real-time kW, which are wrong due to the PLC problems with 2 micro-inverters.

Funnily, Enlighten reports this - no communication with any of the micro-inverters (0 communicating), which is wrong :

If I check the “power today” page in Enlighten, I see that it hasn’t been received any real-time data update for the last couple hours, since about 1pm.

Going back to 12:50pm, I see this :

Note the 12.2 kW “system power” at the bottom left. Very interesting, given that my system is only 9.4 kW DC. This is not just for that particular time, too. Looks like it started going over 10 kW around 9:35am. In the very same screenshot, on the top right, under “Today”, peak is shown to be 6.49 kW at 12:45pm, which sounds correct. I looked at data for yesterday, and I see the same bug. I had never seen tat before.

Data in pvoutput shows like this. Production dta is coming from Enlighten API . Consumption is coming from Wattvision cloud, which has my Rainforest Eagle data, and updates much more frequently than every 15 minutes - 15 seconds from the Zigbee smartmeter, not sure what interval in Wattvision.
As a result, the “Power used” is always off. I know from the Eagle/Wattvision my house consumes at least 1.2 kW at night even when we sleep with everything off, and zero solar production. Yes, I’m working on reducing that …

At this point, all the production data from all sources appears to be wrong, both in Enlighten, probably because of display bugs, and locally on the Envoy device itself, because of two micro-inverters that fail to communicate.

The data in HomeAssistant appears to be more correct than in the Enlighten web site.

[grannos]

Hi @madbrain thanks for your post above. Out of curiosity I did a packet capture of traffic to-from the Envoy-R for a fifteen minute period. Within that period there was only one TLS session with a host 3.211.77.50 and a substantial transfer of data.

There were also a couple of brief http exchanges with very little payload.

0000 7c ff 4d eb e2 e6 00 1d c0 62 3a 6a 08 00 45 00 |.M…b:j…E.
0010 00 85 a8 89 40 00 40 06 0b 69 c0 a8 07 66 34 d9 …@.@…i…f4.
0020 89 99 dc dc 00 50 0f c4 61 79 0b c5 26 1b 50 18 …P…ay…&.P.
0030 01 6d 54 b8 00 00 48 45 41 44 20 2f 31 32 31 34 .mT…HEAD /1214
0040 31 35 30 32 37 36 37 34 20 48 54 54 50 2f 31 2e 15027674 HTTP/1.
0050 31 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 1…Accept: /…
0060 48 6f 73 74 3a 20 70 69 6e 67 2e 65 6e 70 68 61 Host: ping.enpha
0070 73 65 65 6e 65 72 67 79 2e 63 6f 6d 0d 0a 43 6f seenergy.com…Co
0080 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d nnection: close.
0090 0a 0d 0a …

@grannos

[madbrain]

Seems like that HTTP plaintext payload is just checking for web access to the Enphase website. The Envoy-R LCD displays shows “+Web” or “-Web”, likely based on this. Perhaps the frequent connections to the web site are solely to update the front display frequently enough, and not to transfer any micro-inverter data. Still weird they are using plaintext for this, though. Good that they are using TLS for the rest, though not for us trying to get access to the data.

I wish the “Lifetime generation” field on the Envoy-R home page showed kWh and not MWh. Would help with HomeAssistant integration a lot. Talk about a sawtooth graph when the data for the “Lifetime energy production” sensor in HA only increases about once a month.

[bankstownbloke]

@bankstownbloke

There has been an initial discussion with Enphase APAC to explore the possibility of making an exemption for PVOutput members with the new Enphase API v4 terms.

We now have a signed agreement. See original post update for details.

• An important change regarding Enphase Auto Uploader systems

[tycho0]

WOO!! That’s wonderful news!

I can’t express how happy I am that everyone was able to work out an amicable solution. Good things still can happen in this world.

[bankstownbloke]

Link to testing thread