Request for example python exception handling code (addstatus)

zonnehuukske6 · April 12, 2021, 12:30pm

An existing Python script that dumps serial smart meter data to InfluxDB every second is my starting point. The scripts stays running via a socat pipe. I managed to extend that script with a thread to gather metering data and post that information to the addstatus API every 5 minutes using requests.post.

A lot of exceptions can occur. The requests call may timeout, PVoutput API may return an error message.

PVoutput documentation contains nice and practical curl examples, that is nice for oneshot runs. Though I prefer to stay with Pythons requests.post and keep the script running in its loop.

PVoutput API documentation has loads of exceptions, ranging from Bad request 400 (in 11 subtypes for addstatus), 401’s, 403’s and 405’s.

Are there suggestions for timeout and retry values?
And are there suggestions for which errors to retry?

I think:
405 = retry
403. Exceeded number requests per hour = retry
403 other = don’t retry
401 = don’t retry
400 = don’t retry

Can someone point me to a good python pvoutput example that handles requests.post timeouts, error statuses and retries correctly?

bankstownbloke · April 14, 2021, 5:27am

Best practice is to retry any >= 500 responses and HTTP timeouts.

Those in the 400-499 range are data errors and require investigating/fixing and should not be retried.

Should be plenty of generic examples on stackoverflow

e.g. https://stackoverflow.com/questions/21965484/timeout-for-python-requests-get-entire-response

zonnehuukske6 · April 14, 2021, 9:14pm

Thank you. I have found some interesting python code to pvoutput examples on github:

github.com

pvl7/emu-to-pvoutput/blob/2edf7fff8d1af180fa203968b2d33acb80485dec/rainforest-to-pvoutput.py#L88


            rf_xml_block = rf_xml_block + str(line, 'utf-8').rstrip('\r\n')
        # once the last line detected, we stop recording the lines and return the XML block
        if b'</InstantaneousDemand>\r\n' in line:
            logging.debug ("XML data block:\n {}".format(rf_xml_block))
            return (rf_xml_block)

# HTTP post method
def http_post (url, xml_data):
    # HTTP post to pvoutput
    http = requests.Session()
    retry = Retry(connect=10, backoff_factor=0.5)
    adapter = HTTPAdapter(max_retries=retry)
    http.mount('http://', adapter)
    http.mount('https://', adapter)
    headers = {'Content-Type': 'application/xml'}
    res = http.post(url, data=xml_data, headers=headers).text
    logging.info(res)
    return res

# main routine
async def main():

github.com

jrbenito/eastron-pvoutput/blob/e55bfb7ac4df8b46a265e38045c4ab2f68f0e42e/sdm2pvoutput.py#L54


    print('Warnning: Missing system_id, doing nothing')
    return False

headers = {
    'X-Pvoutput-Apikey': self._API,
    'X-Pvoutput-SystemId': system_id,
    'X-Rate-Limit': '1'
}

# Make tree attempts
for i in range(3):
    try:
        r = requests.post(url, headers=headers, data=payload, timeout=10)
        reset = round(float(r.headers['X-Rate-Limit-Reset']) - time())
        if int(r.headers['X-Rate-Limit-Remaining']) < 10:
            print("Only {} requests left, reset after {} seconds".format(
                r.headers['X-Rate-Limit-Remaining'],
                reset))
        if r.status_code == 403:
            print("Forbidden: " + r.reason)
            sleep(reset + 1)

zonnehuukske6 · April 15, 2021, 3:14pm

Below is what I made of the threaded pvoutput poster with retries. The main thread posts meter readings to influxdb every second. Any suggestions?
A sleep(3) between retries?

def post_pvoutput():
    global arr_vals, pts, hivolt

    threading.Timer(300.0, post_pvoutput).start()
    if pts:
        d = 'd=20' + pts[:6]
        t = 't=' + pts[6:8] + ':' + pts[8:10]
        v1 = 'v1=' + str(int(arr_vals(['mr_byclient_t1'])[15:-1]) + int(arr_vals(['mr_byclient_t2'])[15:-1]))
        v3 = 'v3=' + str(int(arr_vals(['mr_toclient_t1'])[15:-1]) + int(arr_vals(['mr_toclient_t2'])[15:-1]))
        v6 = 'v6=' + str(hivolt)
        hivolt = 0.0
        postdata = "&".join([d, t, v1, v3, v6, 'c1=1'])

        for i in range(1, 9):
            try:
                r = requests.post(
                    urlpv, 
                    data=postdata, 
                    headers={
                      'Content-Type': 'application/x-www-form-urlencoded',
                        'X-Pvoutput-Apikey': pvoutput_token_6,
                        'X-Pvoutput-SystemId': pvoutput_sid_6,
                        'Accept': 'text/plain'
                    },
                   timeout=30)
                if print_output:
                    print(r.text)
                    #print(postdata)
                if r.status_code == 200:
                    return
                if 400 <= r.status_code < 500:
                    warningMsg = ("Unable to connect to pvoutput.org - Reason: " + r.reason)
                    logging.warning(warningMsg)
                    return
            except requests.exceptions.RequestException as e:
                print(str(e))
            else:
                if print_output:
                    print(str(r))