Help Required Uploading Growatt Inverter To PVOutput

Hi Bob,

Sorry for late reply and thanks for detailed information.

First of all, Error: %s is a issue with srting formatting, please switch print "Error: %s" with print 'Error: {}' and let me know if it works. By the way, what python version are you using? I first used this with 2.7 but will migrate all code to 3.5+ soon due compatibility.

The missing 12:40 PM was clear an DNS error that persisted until timeout/max retries. Are your RaspberryPi synchronizing clock with NTP? You can check with timedatectl command. I suppose it is since this is default to Pi. Anyway, I am wondering if 3:35PM status was not sent at all (what should have some kind of error printed) or successfully sent but somehow ignored by pvoutput servers (by ignored I mean miss interpreted somehow due time span between Pi and server - this should not be the case because Pi is sending the time/date in the addstatus and it is NTP synchronized so no drifts are expected - at least not large enough to cause issues). I will add some more prints that will clutter logs a lot but will help on this, we can remove those later after debug.

Regarding the else clause issue I will test it more extensively. I confess I copy and paste that part from another script :blush: (but the other script works)

I keep you posted.

Copy and paste almost always leads to some public shame and mea culpa… I forgot to paste a break after line 199 (raise_for_status()). The idea of the for loop is send the request, if it fails, try again, until mais tries (range(3)). The else in a for is executed after for iterate to the end. Hence, if the for reach the max retries it means no status was sent, so else will execute. But, if no error is found while sending status loop must be interrupt by a break statement and else would never be executed. The missing break plays double fault here: First it sends the package 3 times even if no errors, terrible! Second, the error message inside else statement is printed no matter what happened before (success or not). Will correct this too and sorry for that.

So should the full line read print ‘Error: {}’.format(e)?

$ python -V
Python 2.7.13

$ python3 -V
Python 3.5.3

So as I understand it, as pvoutput.sh uses the comand “python” I’m using 2.7.13.

$ timedatectl
Local time: Wed 2018-09-12 10:15:20 BST
Universal time: Wed 2018-09-12 09:15:20 UTC
RTC time: n/a
Time zone: Europe/London (BST, +0100)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no

No worries, I appreciate your support on this :+1:

Bob.

Yes, this is just a test but I am thinking that e might be empty. Anyway, it is an exception raised by OWM and as you said, we don´t care much since temperature is updated once an hour for free accounts.

jrbenito:

Yes, same as here. Just to know if we are at same page.

Again, as expected. NTP synchronized means your Pi has accurate time/date in sync with some reference server (probably one near you). Just for knowledge, Pi has no RTC (as shown by your output) and needs network to sync its time on boot. If you reboot with no internet your clock will set to sometime in past (time of power off). For our purposes this is no problem since, if we have internet to send status to PVOutput we can sync clocks as well. Anyway, if for some issue, it was not sync we may had issues but not the case here.

I am very happy to help and learning a lot on the way. I correct the else statement with the break I mentioned in the follow up post and put to production yesterday. Today I will see results… if you might be interested it was tagged 0.6.10 on my github.

Regarding the gaps, my logs had no DNS errors so I am left in the dark. I put a success message to the log to have baseline when it sends data correctly to pvoutput. Tomorrow I will compare logs with gaps on my system.

BRs,

Benito.

Just to keep informed:

yesterday I logs were all ok but I also got no gaps. Let´s see today…

Just implemented 0.6.10 myself so will let you know how it goes.

Bob.

Hi,
I have three Growatt inverters linked with RS485 cable. I was using the webbox of the maker to monitor the inverters but this was damaged a month ago. Their implementation is a rather ancient friendlyarm board running kernel 2.6.xx.
Given the opportunity I am thinking of using your code to monitor the inverters and upload the data to pvoutput.
Now, I do have a spare RS485 to usb FTDI cable and an espressobin.
Two questions:

  1. Do I need something else (except from the FTDI cable) for RS485 communication? Any other cabling tip?
  2. In github I see that the number of inverters is not used so I am wondering if this will work with 3 inverters.
    Thank you

This is the cable I’m using https://smile.amazon.co.uk/gp/product/B00QUZY4WO/ref=oh_aui_detailpage_o02_s00?ie=UTF8&psc=1 not sure if that helps.

I’ll leave jrbenito to answer the question about multiple inverters.

Bob.

@jrbenito I was wondering how you were getting on with 0.6.10? I’ve been running it now for two full days, no errors on the upload to PVOutput but I’m still getting one missing 5 minute value each day.

I’m also continuing to get errors with OWM, a few each day. There are two types of error reported:

Error getting weather: Exception in calling OWM web API.
Reason: API call timeouted
Caused by: None

and

Error getting weather: Exception in calling OWM web API.
Reason: HTTPConnectionPool(host=‘api.openweathermap.org’, port=80): Max retries exceeded with url: /data/2.5/weather?APPID=966036f9c93216d1e19a44162c1886f1&lat=53.79&lang=en&lon=-1.75 (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x75ecc690>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’,))
Caused by: None

Strange that the query to OWM should be failing but not to PVOutput?

Bob.

Some more feedback on 0.6.10 - looks like the retries loop is working :slight_smile:

From my log yesterday:

(‘2018-09-17 07:36’, ‘Error Connecting:’, ConnectionError(MaxRetryError(“HTTPConnectionPool(host=‘pvoutput.org’, port=80): Max retries exceeded with url: /service/r2/addstatus.jsp (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x75f16f70>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’,))”,),))
(‘2018-09-17 07:36’, ‘Error Connecting:’, ConnectionError(MaxRetryError(“HTTPConnectionPool(host=‘pvoutput.org’, port=80): Max retries exceeded with url: /service/r2/addstatus.jsp (Caused by NewConnectionError(’<urllib3.connection.HTTPConnection object at 0x75eccb50>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution’,))”,),))

There was no missing value at this time so presumably the upload succeeded on the 3rd retry.

I’m consistently getting just one missing 5 minute value per day though with nothing in the log file.

Bob.

Hi @lampra,

  1. Could you please post a picture of your inverter ports where you do connect the cables? Mainly, how are your 3 inverters interconnected?
    Explanation: my inverter has only the RS-232 port and I use a USB/RS-232 cable. As far as I can tell, there is RS-485 on those inverters and the connection is a bit different. RS-485 is meant to communicate with many devices and cable connects those devices in a “daisy chain” manner (cable goes from one inverter to the other then to the other, than to the master -> PC/RPi/Controller/Whatever). Are your connection made this way? Or you have one cable going to each of your inverters?

  2. The code needs some changes (not complicated) to work with more than one inverter. I can change it as long as you can test because, again, I cannot test on my own since I do have only one Canadian Solar (Growatt rebranded) inverter.

Please let me know.

Just be aware that RS-232 and RS-485 are different hardware from electrical and logical perspective. Main difference (besides electrical wiring) is RS-232 is meant to Point-to-Point communication, hence only two devices (computer <-> inverter) while RS-485 is meant to Master - Slaves communication (Master (PC) -> many inverters or other devices). From electrical standpoint, RS-232 requires a very short cable and many wires (TX, RX, GND - at least - RTS, DTS… in a full implementation) while RS-485 can have a twisted pair cable up to 2km long and only requires 2 wires (at least - 3 at most in rare cases). RS-485 was developed for industrial environments.

TL;DR:
Modbus is logical protocol used to convey data from devices to a controller; widely used in industrial controls/monitoring. The python code you found uses external modbus program (modpoll) to query inverter while the code we developed here uses python native modbus library.

Long version:
Modpoll is a program that implements ModBus Protocol. While RS-232 or RS-485 are the ports used to communicate (electrical and logical), you can understand they as a “serial raw communication”, think it as data pipe. What you will pass through that pipe is another matter. In case of many devices, the protocol of choice over RS-485/RS-232 is ModBus because it is widely used on industrial controls. Growatt protocol is Modbus (just for counter example, ABB Aurora inverters uses proprietary Aurora protocol and ABB sells a hardware that converts Aurora into Modbus). Important to say that Modbus is not tied to RS-XXX, Modbus has variations to work over serial lines like RS-xxx or even TCP (Modbus RTU and Modbus TCP).

Python has its own modbus library that is very simple yet effective to read Growatt inverter. I used this library because, in my opinion, external programs do a great job when you don´t have native options. Originally this code used external “curl” program to send http requests to pvoutput. Again, while curl is great http client CLI app,I rather use python native “request” library. So I moved from calling “curl” to using request and, in my opinion, it is cleaner and more controllable this way (mainly regarding error handling and code readability). But this just my opinion. Both ways did same thing very well (same applies to external modpoll and modbus library).

Back to your inverters:

Modbus allows for communication with many devices hence, assuming your inverters are daisy chained in a RS-485 bus, each of then had a own address in the bus (and since your had it working I think all is already setup); what needs to be modified in the code is:

  1. stores address for each of the inverters in the RS-485 and iterate of those address to query each inverter independently
  2. stores values from each inverter separately
  3. send values to pvoutput for the other inverters

What I would do:

  1. modify the inverter classe to two classes:
    Class InverterComm would handle communication with any number of inverters, hence one object per serial port (in your case one object)
    Class Inverter would stores values read from inverter, hence one object per inverter

  2. pvoutput routine shall use each object inverter above to send data to Pvouput (since eache inverter has its own systemId for instance)

This is one way, there are many others. As very simple quick&dirty solution is add a loop to inverter class adding some local properties (wh_today2, wh_today3, wh_lifetime2, …) to the other inverters, and loop in pvouput function send all that too. This is tight coupled hence less maintainable.

Hi Bob,

I am also running 0.6.10 but I got no gaps in the first two days. I looked now and I have one gap on day 09/16 hence I will look into my logs. I slightly modified my code to also log success sending hence, since you have no errors reported on your report gaps I am hopping to get a “success” log on my report gap because this would mean that something went wrong at pvoutput side. Otherwise we need to figure out why our script is taking a break longer than 5 minutes :smile:

Later I report back with my logs.

1 Like

Hi @jrbenito
Thank you very much for the explanations regarding modbus and python. It’s all clear to me now as i was not aware that python has its own modbus library.

At the moment I have no picture to upload but I will do so the next few days (during the weekend). The inverters are connected to each other with one twisted pair cable starting from the old master (webbox) and ending at the last inverter.
Something like: webbox====INV1====INV2====INV3

I would appreciate if you could modify your code to support multiple inverters and I will definitely test and provide feedback.

For the moment I use espressobin with openwrt 18.06. I installed python and all the requirements of your code are already met. I will just need to connect the RS485 to usb cable during the weekend. (if needed I could change from openwrt to armbian)

I looked at the manual, here is the RS485 cabling of the inverters:


No2 cable at the inverters is the the shielding layer (if used)
The FTDI usb to RS485 cable that i will use provides the following signals:

Colour 	Name 		Type 	Description
Black 	GND 		GND 	Device ground supply pin.
Brown 	Terminator 1 	Input 	Pin 1 of 120R Terminating Resistor. Only Required if the 					USB-RS485-WE cable is the first or last device in a 					multi-drop RS485 System, to meet RS485 Termination 					Requirements.
Red 	POWER 		Output
Orange 	Data+(A) 	Bi-Direction Data + (A) Signal
Yellow 	Data-(B) 	Bi-Direction Data - (B) Signal
Green	Terminator 2	Input	Pin 2 of 120R Terminating Resistor. Only Required if the 					USB-RS485-WE cable is the first or last device in a 					multi-drop RS485 System, to meet RS485 Termination 					Requirements.

Some questions for the software and the connections:

  1. I expect that I will need to connect the two cables of the twisted pair from the first inverter to Data-(yellow) and Data+(orange) of the FTDI cable, is this correct? Does it matter which cable goes to eg Data+?
  2. I will need to connect green and brown cables of the FTDI as this will be the first device in the system, is this correct?
  3. How can I test the communication on site to ensure proper connections?
  4. I Installed the native openwrt driver for FTDI (kmod-usb-serial-ftdi). I am thinking of using minicom and test the connection. Any hint?
    Thank you

Hi @bobboulby,

I found gaps on days 13 and 14, I revisited and saw it since the first time I didn’t notice.

Here is Sep-13 pic and logs:

('2018-09-13 12:02', ' Enviado com sucesso')                                                                        
('2018-09-13 12:08', ' Enviado com sucesso')                                                                         
('2018-09-13 12:13', ' Enviado com sucesso')                                                                        
('2018-09-13 12:18', ' Enviado com sucesso')                                                                       
('2018-09-13 12:23', ' Enviado com sucesso')

Look the 12:30 success log, it sent the data for 12:10 as the 12:08 sent data for 12:05 so on so forth (sorry for Portuguese success log messages)

Now the Sep-14:

('2018-09-14 10:02', ' Enviado com sucesso')
('2018-09-14 10:08', ' Enviado com sucesso')
('2018-09-14 10:13', ' Enviado com sucesso')
('2018-09-14 10:18', ' Enviado com sucesso')

Again success on sending data to pvoutput server on 10:13 referring to 10:10 data.

Looking into code lines 172 and 173 is possible to see that we send date and time from moment inverter was read, hence no pvouput automatic guessing the date/time by the request date/time. @bankstownbloke do you know what could be happening? Seems some reports sent to pvoutput are missing in status, any clue?

BRs,
Benito.

1 Like

@lampra

Perfect! That’s what I was imagining. It shall work.

Regarding your questions:

Yes, theoretically is possible to invert Data+ and Data- on the software but it would save some debug if you could figure out who is Data+ and Data- coming from inverter and wire accordingly.

You shall put a resistor of 120R (ohms) between those two signals and your last inverter might already have one too. I don’t know what happens if you do not terminate it, guess that some reflections may occur but I am guessing wild. Is the webbox terminated with a 120R resistor?

Use a laptop with modbus program, if you have linux laptop use modpoll from the repository you suggested before. I use it here to test comm with my Eastron SDM630 (energy meter) and also used it when testing my mom’s inverter. Your USB/RS-485 will show up as /dev/ttyUSB0 or similar and modpoll would communicate with the inverter (as long you provide the correct address for each inverter). If you have windows laptop please find out some modbus test program since I really never used one on windows and can’t tell.

This would test the serial port but not the RS485 communication. Hence it is a partial test.

I will try coding some dev test for you this week. I keep you posted here.

BRs
Benito

@jrbenito, do you think it’s worth logging all the data sent to PVOutput for each upload? Maybe there is something in the upload data that is causing PVOutput to reject it.

Bob.

Sure! I am logging packets sent starting tomorrow. This may giver us some clue.

Regarding errors from weather service, I too had a lot of issues, most of than are “API call timeouted” but I do have couple of “HTTP 500” errors (server side error). At least one fail each day but sometime three or four errors. I strongly believe this is internet issues somewhere between my RPi and OWM server or even OWM server busy and not responding in time. “API call timeout” are clear a server responding in no time while HTTP was connected, hence, server side issue or connection issue somewhere in the Internet. HTTP 500 generally means server too busy to answer HTTP request or some internal issue preventing it from providing response (like a background service not responding - a database, a worker process, whatever); again, server side issue for sure. The error you got (“NewConnectionError” or “Temporary failure in name resolution”) is an issue with connection and/or DNS servers, I did experience this once in the past but it is not common on my logs. Something is preventing HTTP connection to the server, in this specific case, the DNS server did not answer well… We don´t know the DNS error, urllib3 did not provide enough logs to clarify it; it could be our internet provider DNS failing to answer (I don´t use internet provider DNS server, I use Google/OpenDNS as my primary and secondary DNS service - they could fail too but less likely as my small provider´s server). It also could be the OWM DNS provider failing (I did not check who is resolving for them) but I guess they might have DNS hosted by a major provider. Anyway, this is more or less out of our control (besides change our DNS servers there is not much we can do).

BRs,
Benito