Help Required Uploading Growatt Inverter To PVOutput

Inverters Growatt
#30

I had thought that there might be problems with Internet outages. I have a robust and fast Internet connection but there is always the chance of an outage or maybe PVOutput is taken down for maintenance.

I also have a SMA inverter and use SBFspot (https://github.com/SBFspot/SBFspot/wiki) to upload the data to PVOutput. SBFSpot writes the inverter data to a SQLite database and then a second program uploads the data from the database to PVOutput. If the upload isn’t successful then it keeps retrying until it succeeds and this works very well. I had thought that I might try something similar with your code although I’d have to learn quite a lot of new stuff so it would take some time.

To test this, do you know if I could have two programs querying the inverter at the same time without messing things up? If this is possible I’d leave your original script running and develop the part that writes the data to the database first. I could then test that was running ok before developing the database to PVOutput upload part.

Any thoughts?

Bob.

#31

The SQL Lite idea is not bad and not too difficult to implement; I thought of it a bit different by buffering requests on memory like a queue (a FIFO). But SQL Lite could be a good alternative that brings backup as bonus.

To test this, do you know if I could have two programs querying the inverter at the same time without messing things up?

No and yes!
No, serial ports are not meant to have concurrent access.

Yes, it will work but sometimes it may conflict and fail readings on both sides. I do this to test sometimes but you can try reading at same time the other program is doing its stuff. In this case both programs will result in invalid readings. For sporadic test it “ok” but for long running it is not good idea.

When you have one datasource like the inverter and want many consumers of that data, the best approach is to use a proxy agent in the middle. Most IoT home projects today uses MQTT protocol to buffer between datasources and data consumers. It would be simple to break the code into “inverter reader” that reads inverter and post data to somewhere (MQTT or SQLite or Redis) and another program that subscribe to that data, process and send it to another place. In fact, this architecture (using MQTT or Redis) would allow for fun things like real time dashboard, alarms, etc…

#32

In fact it was not your internet but the DNS server used by your network: “Temporary failure in name resolution”

#33

@bobboulby, I just pushed correction code, could you please give it a try? I could not push it to production yet, will do later today. I also use your suggestion of log only errors and caught most commons exceptions printing informative errors instead of crash :smiley:

Check tag v0.6.8

#34

Running it now. All ok so far but will monitor over the next few days.

Bob.

#35

Hi again @jrbenito

Just to report that the latest version of your code has been working fine for the last 12 days. I’ve had a couple of DNS errors reported in my log file and these have been trapped correctly by your code.

One thing I have noticed however is that each day I loose one 5 minute value (for an example see the attached image for 2:25PM today).

Annotation
Annotation897×149 17.3 KB

There is nothing in the log file to indicate any kind of error when this happens. It happens pretty much every day, usually in the afternoon.

Any ideas what might be happening?

Bob.

#36

Hi @bobboulby,

Does gaps match time and date of DNS errors? Notice that the way code is right now it does not retry send data to pvoutput; it just don´t crash in case of an error. Hence, if an error occurred while sending 2:25PM status, it will sleep 300 seconds and send new data. I do have a new version with a for loop doing 3 times retry before giving up. I will upload it as soon as possible.

I too have one gap, usually arround 13 localtime but sometimes around 17, almost every day. Actually 7 days with one gap in the last 12 days (08/24 to 09/05). I will look into logs later as arrive home.

Another issue:

I have some ideas to buffer data not sent (due any problem with internet) and sending than after. I am working on another project and as soon I have sometime I will try to implement it.

Benito.

#37

Please check if tag v0.6.9 helps

#38

No there is nothing in the log file at the same time as the gaps.

Thanks, will try this, probably at the weekend.

Bob.

#39

Hi @bobboulby,

Yesterday I went to check logs and see if my gaps leaved any trace behind but I noticed I did not update code to the latest version on my mom´s inverter. Guess what, I did not. Anyway, I put version v0.6.9 to run yesterday and tomorrow will be holiday here in Brazil so I will be at home while inverter is working. Good opportunity to do some debug.

BRs.

1 Like
#40

Back again :slight_smile:

I’ve just implemented v0.6.9 so will monitor over the next few days. Yesterday (while still running 0.6.8) I had two missed 5 minute values one at 12:40PM

1
1925×127 29.3 KB

and one at 3:35PM

2
2901×150 10.5 KB

My log file for yesterday was:

2018-09-06 20:00 - Next shift starts in 660 minutes
Error: %s
Error: %s
Error: %s
(‘2018-09-07 12:42’, ‘Error Connecting:’, ConnectionError(MaxRetryError(“HTTPConnectionPool(host=‘pvoutput.org’, port=80): Max retries exceeded with url: /service/r2/addstatus.jsp (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x75ea21b0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’,))”,),))
Error: %s
2018-09-07 20:00 - Next shift starts in 660 minutes

so there was an entry in the log for 12:40PM but not for 3:35PM. Not sure if this helps with debugging?

By the way I’ve not mentioned the “Error: %s” in the log file before - I get a few of these each day. Looking at the code it seems to be an error while getting the temperature from OWM so I’ve not really worried about it.

Thanks for your ongoing support by the way.

Bob

#41

I’ve rolled back to 0.6.8 this morning. Yesterday running 0.6.9 I had five missing 5 minute values plus the else condition in your new code is being activated every 5 minutes. This is a sample of my log file:

(‘2018-09-08 19:25’, ‘Could not send data after some attempts’)
(‘2018-09-08 19:30’, ‘Could not send data after some attempts’)
(‘2018-09-08 19:36’, ‘Could not send data after some attempts’)
(‘2018-09-08 19:41’, ‘Could not send data after some attempts’)

There were no DNS errors yesterday.

Bob.

#42

Hi Bob,

Sorry for late reply and thanks for detailed information.

First of all, Error: %s is a issue with srting formatting, please switch print "Error: %s" with print 'Error: {}' and let me know if it works. By the way, what python version are you using? I first used this with 2.7 but will migrate all code to 3.5+ soon due compatibility.

The missing 12:40 PM was clear an DNS error that persisted until timeout/max retries. Are your RaspberryPi synchronizing clock with NTP? You can check with timedatectl command. I suppose it is since this is default to Pi. Anyway, I am wondering if 3:35PM status was not sent at all (what should have some kind of error printed) or successfully sent but somehow ignored by pvoutput servers (by ignored I mean miss interpreted somehow due time span between Pi and server - this should not be the case because Pi is sending the time/date in the addstatus and it is NTP synchronized so no drifts are expected - at least not large enough to cause issues). I will add some more prints that will clutter logs a lot but will help on this, we can remove those later after debug.

Regarding the else clause issue I will test it more extensively. I confess I copy and paste that part from another script :blush: (but the other script works)

I keep you posted.

#43

Copy and paste almost always leads to some public shame and mea culpa… I forgot to paste a break after line 199 (raise_for_status()). The idea of the for loop is send the request, if it fails, try again, until mais tries (range(3)). The else in a for is executed after for iterate to the end. Hence, if the for reach the max retries it means no status was sent, so else will execute. But, if no error is found while sending status loop must be interrupt by a break statement and else would never be executed. The missing break plays double fault here: First it sends the package 3 times even if no errors, terrible! Second, the error message inside else statement is printed no matter what happened before (success or not). Will correct this too and sorry for that.

#44

So should the full line read print ‘Error: {}’.format(e)?

$ python -V
Python 2.7.13

$ python3 -V
Python 3.5.3

So as I understand it, as pvoutput.sh uses the comand “python” I’m using 2.7.13.

$ timedatectl
Local time: Wed 2018-09-12 10:15:20 BST
Universal time: Wed 2018-09-12 09:15:20 UTC
RTC time: n/a
Time zone: Europe/London (BST, +0100)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no

No worries, I appreciate your support on this :+1:

Bob.

#45

Yes, this is just a test but I am thinking that e might be empty. Anyway, it is an exception raised by OWM and as you said, we don´t care much since temperature is updated once an hour for free accounts.

jrbenito:

Yes, same as here. Just to know if we are at same page.

Again, as expected. NTP synchronized means your Pi has accurate time/date in sync with some reference server (probably one near you). Just for knowledge, Pi has no RTC (as shown by your output) and needs network to sync its time on boot. If you reboot with no internet your clock will set to sometime in past (time of power off). For our purposes this is no problem since, if we have internet to send status to PVOutput we can sync clocks as well. Anyway, if for some issue, it was not sync we may had issues but not the case here.

I am very happy to help and learning a lot on the way. I correct the else statement with the break I mentioned in the follow up post and put to production yesterday. Today I will see results… if you might be interested it was tagged 0.6.10 on my github.

Regarding the gaps, my logs had no DNS errors so I am left in the dark. I put a success message to the log to have baseline when it sends data correctly to pvoutput. Tomorrow I will compare logs with gaps on my system.

BRs,

Benito.

#46

Just to keep informed:

yesterday I logs were all ok but I also got no gaps. Let´s see today…

#47

Just implemented 0.6.10 myself so will let you know how it goes.

Bob.

#48

Hi,
I have three Growatt inverters linked with RS485 cable. I was using the webbox of the maker to monitor the inverters but this was damaged a month ago. Their implementation is a rather ancient friendlyarm board running kernel 2.6.xx.
Given the opportunity I am thinking of using your code to monitor the inverters and upload the data to pvoutput.
Now, I do have a spare RS485 to usb FTDI cable and an espressobin.
Two questions:

  1. Do I need something else (except from the FTDI cable) for RS485 communication? Any other cabling tip?
  2. In github I see that the number of inverters is not used so I am wondering if this will work with 3 inverters.
    Thank you
#49

This is the cable I’m using https://smile.amazon.co.uk/gp/product/B00QUZY4WO/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1 not sure if that helps.

I’ll leave jrbenito to answer the question about multiple inverters.

Bob.