Tesla Powerwall2 data to PVoutput?

bankstownbloke · July 17, 2018, 9:08am

It should not affect it, check pvoutput.log for requests to https://192.168.0.63/api/system_status/soe

The soc-url setting needs to be updated in powerwall.ini

soc-url=https://192.168.0.63/api/system_status/soe

willisave · July 17, 2018, 1:16pm

The soc-url setting was already working fine, and when I noticed the PW2 version had been updated I jumped in and added the ‘s’ into the http:// fields of the two URLs in powerwall2.ini and restarted. The main extended attributes came back again, the SOE did not.

See https://pvoutput.org/intraday.jsp?id=64310&sid=57175&dt=20180717, extended parameters - you’ll see the SOE and extended parameters from the PW2 are fine up to 8:30am (when Tesla updated the firmware), and then the extended parameters start again at 9:35am after I changed http to https in two places in powerwall.ini and restarted the PVO service. Adding those two ‘s’ characters was the only change.

The SOE is assigned to v11.

In the sample lines from pvoutput.log before the change, v11 is assigned a correct value 69.878 (as %):

2018-07-17 06:18:14,181 INFO [Thread-2] (WebClient.java:162) - >>> http://192.168.0.63:80/api/meters/aggregates
2018-07-17 06:18:14,213 INFO [Thread-2] (PowerwallLogReader.java:361) - >>> http://192.168.0.63/api/system_status/soe
2018-07-17 06:18:57,900 INFO [Thread-1] (Controller.java:1869) - >>> 20180717,06:20,-1,-1,-1.0,580,-1000.0,721.7,550,580.231,1949457.484,9.29,69.878,-6.37
2018-07-17 06:18:57,900 INFO [Thread-1] (WebClient.java:131) - >>> http://pvoutput.org:80/service/r2/addbatchstatus.jsp?data=20180717,06:20,-1,-1,-1.0,580,-1000.0,721.7,550,580.231,1949457.484,9.29,69.878,-6.37
2018-07-17 06:18:58,306 INFO [Thread-1] (Controller.java:1896) - <<< 20180717,06:20,1

However, after the version change, that v11 field is now empty on the upload:

2018-07-17 16:33:50,193 INFO [Thread-2] (WebClient.java:162) - >>> https://192.168.0.63:443/api/meters/aggregates
2018-07-17 16:33:50,303 INFO [Thread-2] (PowerwallLogReader.java:361) - >>> https://192.168.0.63/api/system_status/soe
2018-07-17 16:33:54,240 INFO [Thread-1] (Controller.java:1869) - >>> 20180717,16:35,-1,-1,-1.0,2334,-1000.0,721.4,2190,2334.626,1967550.373,23.53,154.7
2018-07-17 16:33:54,240 INFO [Thread-1] (WebClient.java:131) - >>> http://pvoutput.org:80/service/r2/addbatchstatus.jsp?data=20180717,16:35,-1,-1,-1.0,2334,-1000.0,721.4,2190,2334.626,1967550.373,23.53,,154.7
2018-07-17 16:33:54,646 INFO [Thread-1] (Controller.java:1896) - <<< 20180717,16:35,1

willisave · July 17, 2018, 1:21pm

OK - found the errors in service.log - turns out Java is unable to connect due to issues with the Tesla security certificate.
Every minute I’m now getting Java exceptions for SSL handling:

INFO | jvm 1 | 2018/07/17 18:37:16 | javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.0.63 found
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.Alerts.getSSLException(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.Handshaker.fatalSE(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.Handshaker.processLoop(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.Handshaker.process_record(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at java.net.HttpURLConnection.getResponseCode(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at org.pvoutput.integration.util.WebClient.get(WebClient.java:222)
INFO | jvm 1 | 2018/07/17 18:37:16 | at org.pvoutput.integration.reader.PowerwallLogReader$PowerwallFileWriter.addStateOfCharge(PowerwallLogReader.java:363)
INFO | jvm 1 | 2018/07/17 18:37:16 | at org.pvoutput.integration.reader.PowerwallLogReader$PowerwallFileWriter.parse(PowerwallLogReader.java:338)
INFO | jvm 1 | 2018/07/17 18:37:16 | at org.pvoutput.integration.reader.AHttpLogReader$ASimpleLogFileWriter.get(AHttpLogReader.java:479)
INFO | jvm 1 | 2018/07/17 18:37:16 | at org.pvoutput.integration.reader.AHttpLogReader$ASimpleLogFileWriter.run(AHttpLogReader.java:354)
INFO | jvm 1 | 2018/07/17 18:37:16 | Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 192.168.0.63 found
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.util.HostnameChecker.matchIP(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.util.HostnameChecker.match(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.X509TrustManagerImpl.checkIdentity(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.X509TrustManagerImpl.checkIdentity(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
INFO | jvm 1 | 2018/07/17 18:37:16 | … 19 more

Wonder why it can connect to the aggregates URL but not the SOE URL!

spludge · July 17, 2018, 1:33pm

Same results for me after updating this morning.

agiangonew · July 17, 2018, 9:21pm

Mine can’t connect to either. It seems to me that both require https now and I did change the .ini file accordingly but nothing gets written in the data stream file.
I guess I should turn on logging to see the same Java handshake error.

billz · July 17, 2018, 9:30pm

Check out this post Tesla Motors Club. Looks like there’s an issue that been discovered where you have to use a host name of Powerwall versus an IP to match the cert. I can’t test it as Tesla shutoff my PW2 web server. Might not work, but may be worth a try.
Here’s the URL:

willisave · July 18, 2018, 12:01am

That was the next thing to try - but it didn’t fix the issue.

Added an entry to the Windows host file (c:\windows\system32\drivers\etc\hosts):

192.168.0.63 powerwall

and now web browsing to https://powerwall/api/system_status/soe (after accepting all the warnings about insecure sites and self-signed certificates) gives the desired information:

{“percentage”:30.066038435853677}

However the Java VM can’t get through the invalid certificate authority checks, and still won’t access the information:
From pvoutput.log:

2018-07-18 09:39:17,280 INFO [Thread-2] (WebClient.java:162) - >>> https://powerwall:443/api/meters/aggregates
2018-07-18 09:39:17,561 INFO [Thread-2] (PowerwallLogReader.java:361) - >>> https://powerwall:443/api/system_status/soe
2018-07-18 09:39:18,967 INFO [Thread-1] (Controller.java:1869) - >>> 20180718,09:40,-1,-1,-1.0,3602,-1000.0,727.3,300,3602.145,1991118.164,1321.96,1973.01
2018-07-18 09:39:18,983 INFO [Thread-1] (WebClient.java:131) - >>> http://pvoutput.org:80/service/r2/addbatchstatus.jsp?data=20180718,09:40,-1,-1,-1.0,3602,-1000.0,727.3,300,3602.145,1991118.164,1321.96,,1973.01
2018-07-18 09:39:19,389 INFO [Thread-1] (Controller.java:1896) - <<< 20180718,09:40,1

(notice the second-last field, which is v11, is still empty).

Still wierd that the call for the ‘aggregates’ URL is getting results, but the call to the SOE URL is not.
Damn you Tesla, damn you all to heck!

willisave · July 18, 2018, 12:48am

BB, one thing I noticed:
With the ‘aggregates’ URL, changing ‘http’ to ‘https’ in pvoutput.ini caused the PVO service to automagically add the port ‘:443’ into the URL that was sent to the Internet.
However, there as no ‘:443’ added to the SOE URL.
powerwall.ini:

url=https://powerwall/api/meters/aggregates
soc-url=https://powerwall/api/system_status/soe
soc-parameter=v11

Produces in pvoutput.log:

2018-07-18 10:35:35,842 INFO [Thread-2] (WebClient.java:162) - >>> https://powerwall:443/api/meters/aggregates
2018-07-18 10:35:35,952 INFO [Thread-2] (PowerwallLogReader.java:361) - >>> https://powerwall/api/system_status/soe
2018-07-18 10:36:35,996 INFO [Thread-2] (WebClient.java:162) - >>> https://powerwall:443/api/meters/aggregates
2018-07-18 10:36:36,121 INFO [Thread-2] (PowerwallLogReader.java:361) - >>> https://powerwall/api/system_status/soe

Notice in the logs the ‘:443’ port override has been added to the aggregates URL, but not to the SOE URL.
Now the ‘:443’ doesn’t fix the issue - I added :443 into the SOE URL manually in the ini file and it still wouldn’t retrieve the data.
However, delving through the PVO source, this suggests to me that perhaps the SOE retrieval logic isn’t going through ‘getWebClient()’ in WebClient.java, which seems to have the logic for checking certificates and adding the ‘:443’ into the URL. Perhaps thats why the SOE retrieval is failing, but the other https retrieval is succeeding. But thats about the end of my Java-foo. Hope it helps.

bankstownbloke · July 18, 2018, 9:41am

Thanks for the logs.

Please download the following patch below -

https://bitbucket.org/pvoutput/pvoutput-integration-service/downloads/org.pvoutput.integration_1.5.3.1.jar

Rename to org.pvoutput.integration.jar
Stop the service
Copy this to the <pvoutput_install>/lib/ folder and replace the existing copy
Start the service

If this is working then it will be included in the next release.

spludge · July 18, 2018, 11:27am

Yep, SOE is back with that.

AnthonyB · July 19, 2018, 1:23am

Yep, same here.

Thanks bankstownbloke!

AnthonyB · July 19, 2018, 1:30am

Given Tesla’s direction since 1.20.0 making it increasingly difficult, and for some now actually impossible, to read the data directly from the Powerwall, is it possible to consider building Powerwall integration into the PVOutput on-line Auto Uploader (https://pvoutput.org/help.html#autoupload) functionality? I.e. Pull the data directly from Tesla’s web?

There are numerous community built applications already doing this (e.g. this Windows 10 app - https://www.microsoft.com/store/productId/9N3M45C4ZCJ4) and while it leads to slightly increased data latency, it would significantly improve reliability.

nswfugitive · July 19, 2018, 2:06am

Thanks BB - great work as always. Also, willisave for providing the logs to BB.

pv135 · July 19, 2018, 5:43am

I’ve done some work on this with that intention, as I agree it seems to be the way Tesla are heading. So far I can retrieve my SoC from the Tesla servers, a few issues to work out with API tokens as well as the remaining commands for the other relevant data. Plus side to this method is no need for a local PC, downside is their server seems pretty unreliable based on the number of times the app doesn’t work, however I would expect retrieving history should be possible in these instances like with an efergy hub.

When I get some more time to work on it I’ll update here.

bankstownbloke · July 19, 2018, 8:51am

This can be a problem resulting in gaps in the data for current/spot data.

We won’t be implementing this unless there is a history service API for the day’s 5-minute data.

pv135 · July 19, 2018, 10:06am

Completely understandable, I read somewhere there is a 48 hour history, unfortunately the api is not documented but will see what I can work out

Edit: thinking about it there must be a 48 hr history of 5min data the app uses it

willisave · July 20, 2018, 3:15am

I also confirm the patch works and restores the SOE collection for me.
Thanks BB for the rapid fix.

agiangonew · July 21, 2018, 2:29am

Thank you all. Two days out but now it is all good!

nswfugitive · July 25, 2018, 10:49pm

Well as of 5:15pm last night I have no access to the PW2, web page or access to the data. Nothing is working except for the stupid Tesla app. I can ping the PW2 - but no data.

dlj57 · August 14, 2018, 5:19am

Six days later it started working again - it took a phone call to Tesla support. Now with the latest version of the firmware (1.21.0) they state that customers will have access to wifi. Which is good news for us pvoutput org users.