Mysterious 14 day gap in uploads. And now again the last 8 days…

wimleers · December 4, 2022, 4:56am

At G2Cv1 6.000kW, the output is missing between October 16 and October 29. And now again since November 12. But … sbfspot has been running all that time! And I even see that the *-Spot-20221016.csv etc files are logged, so I’m 100% certain the data is known.

Why is sbfspot not automatically uploading the data it did not yet upload?

I am a pvoutput.org donor.

EDIT: I restarted sbfspot, did not change a single setting, and now it is catching up on the Nov 12–now data. How is this possible? Literally the only thing I did was restart it…

EDIT 2: digging into the logs, I see it’s filled with lines like this, endlessly, including yesterday, whose data was missing until I rebooted sbfspot just now (as per my previous edit):

[00:13:31.440] ERROR: Uploading 100 datapoints, starting with 20221015,16:25,1236508,564,,,0,243.37 Bad request 400: Power value [2147483647] too high for system size [6000]
[00:14:31.004] ERROR: Uploading 100 datapoints, starting with 20221015,16:25,1236508,564,,,0,243.37 Bad request 400: Power value [2147483647] too high for system size [6000]
[00:15:30.616] ERROR: Uploading 100 datapoints, starting with 20221015,16:25,1236508,564,,,0,243.37 Bad request 400: Power value [2147483647] too high for system size [6000]
[00:16:31.231] ERROR: Uploading 100 datapoints, starting with 20221015,16:25,1236508,564,,,0,243.37 Bad request 400: Power value [2147483647] too high for system size [6000]

… but a simple reboot fixed that. How … is this possible?

wimleers · November 19, 2022, 8:28pm

… after digging through the database for a very long time, I spotted something suspicious:

sqlite> SELECT MIN(TotalYield) FROM DayData;
MIN(TotalYield)
429842
sqlite> SELECT datetime(TimeStamp, 'unixepoch', 'localtime'), TotalYield FROM DayData WHERE TotalYield=429842;
datetime(TimeStamp, 'unixepoch', 'localtime');TotalYield
2022-08-14 06:30:00;429842
2022-08-14 06:35:00;429842
2022-08-14 06:40:00;429842
2022-08-14 06:45:00;429842
2022-08-14 06:50:00;429842
2022-08-14 06:55:00;429842
2022-08-14 07:00:00;429842
2022-08-14 07:05:00;429842
2022-08-14 07:10:00;429842
2022-08-14 07:15:00;429842
2022-08-14 07:20:00;429842
2022-08-14 07:25:00;429842
2022-08-14 07:30:00;429842
2022-08-14 07:35:00;429842
2022-08-14 07:40:00;429842
2022-08-14 07:45:00;429842
2022-08-14 07:50:00;429842
2022-08-14 07:55:00;429842
2022-08-14 08:00:00;429842
2022-08-14 08:05:00;429842
2022-08-14 08:10:00;429842
2022-08-14 08:15:00;429842
2022-08-14 08:20:00;429842
2022-08-14 08:25:00;429842
2022-08-14 08:30:00;429842
2022-08-14 08:35:00;429842
2022-08-14 08:40:00;429842
2022-08-14 08:45:00;429842
2022-08-14 08:50:00;429842
2022-08-14 08:55:00;429842
2022-08-14 09:00:00;429842
2022-08-14 09:05:00;429842
2022-08-14 09:10:00;429842
2022-08-14 09:15:00;429842
2022-08-14 09:20:00;429842
2022-08-14 09:25:00;429842
2022-08-14 09:30:00;429842
2022-08-14 09:35:00;429842
2022-08-14 09:40:00;429842
2022-08-14 09:45:00;429842
2022-08-13 06:30:00;429842
2022-10-15 18:45:00;429842
2022-10-15 18:50:00;429842
2022-10-15 18:55:00;429842
2022-10-15 19:00:00;429842
2022-10-15 19:05:00;429842
2022-10-15 19:10:00;429842
2022-10-15 19:15:00;429842
2022-10-16 08:05:00;429842
2022-10-16 08:10:00;429842
2022-10-16 08:15:00;429842
2022-10-16 08:20:00;429842
2022-10-16 08:25:00;429842
2022-10-16 08:30:00;429842
2022-10-16 08:35:00;429842
2022-10-16 08:40:00;429842
2022-10-16 08:45:00;429842
2022-10-16 08:50:00;429842
2022-10-16 08:55:00;429842
2022-10-16 09:00:00;429842
2022-10-16 09:05:00;429842
2022-10-16 09:10:00;429842
2022-10-16 09:15:00;429842
2022-10-16 09:20:00;429842
2022-10-16 09:25:00;429842
2022-10-16 09:30:00;429842
2022-10-16 09:35:00;429842
2022-10-16 09:40:00;429842
2022-10-16 09:45:00;429842
2022-10-16 09:50:00;429842
2022-10-16 09:55:00;429842
2022-10-16 10:00:00;429842
2022-10-16 10:05:00;429842
2022-10-16 10:10:00;429842
2022-10-16 10:15:00;429842
2022-10-16 10:20:00;429842
2022-10-16 10:25:00;429842
2022-10-16 10:30:00;429842
2022-10-16 10:35:00;429842
2022-10-16 10:40:00;429842
2022-10-16 10:45:00;429842
2022-10-16 10:50:00;429842
2022-10-16 10:55:00;429842
2022-10-16 11:00:00;429842
2022-10-21 15:55:00;429842
2022-10-21 16:00:00;429842
2022-10-21 16:05:00;429842
2022-10-21 16:10:00;429842
2022-10-21 16:15:00;429842
2022-10-21 16:20:00;429842
2022-10-21 16:25:00;429842
2022-10-21 16:30:00;429842
2022-10-21 16:35:00;429842
2022-10-21 16:40:00;429842
2022-10-21 16:45:00;429842
2022-10-21 16:50:00;429842
2022-10-21 16:55:00;429842
2022-10-21 17:00:00;429842
2022-10-21 17:05:00;429842
2022-10-21 17:10:00;429842
2022-10-21 17:15:00;429842
2022-10-21 17:20:00;429842
2022-10-21 17:25:00;429842
2022-10-21 17:30:00;429842
2022-10-21 17:35:00;429842
2022-10-21 17:40:00;429842
2022-10-21 17:45:00;429842
2022-10-21 17:50:00;429842
2022-10-21 17:55:00;429842
2022-10-21 18:00:00;429842
2022-10-21 18:05:00;429842
2022-10-21 18:10:00;429842
2022-10-21 18:15:00;429842
2022-10-21 18:20:00;429842
2022-10-21 18:25:00;429842
2022-10-21 18:30:00;429842
2022-10-21 18:35:00;429842
2022-10-21 18:40:00;429842
2022-10-21 18:45:00;429842
2022-10-21 18:50:00;429842
2022-10-21 18:55:00;429842
2022-10-21 19:00:00;429842
2022-10-21 19:05:00;429842
2022-10-21 19:10:00;429842
2022-10-21 19:15:00;429842

wimleers · November 19, 2022, 9:29pm

Ah, looks like the first failing data point is much simpler:

[22:28:31.483] ERROR: Uploading 100 datapoints, starting with 20221015,16:25,1236508,564,,,0,243.37 Bad request 400: Power value [2147483647] too high for system size [6000]

wimleers · November 19, 2022, 9:39pm

Further digging revealed:

sqlite> select * from daydata ORDER by Power DESC LIMIT 13;
TimeStamp|Serial|TotalYield|Power|PVoutput
1666159800|3010933557|487703|61489146903183996|
1666246200|3010933557|493302|61489146903172536|
1666073100|3010933557|470800|61489146903055224|
1665986700|3010933557|449067|61489146902840124|
1666592100|3010933557|487703|61489146902830956|
1666505400|3010933557|470800|61489146902682240|
1665900300|3010933557|429842|61489146902679012|
1666419000|3010933557|449067|61489146902488788|
1666360500|3010933557|429842|61489146902346180|
1665852300|3010933557|429842|4392081921620023|
1656067500|3010933557|418354|4992|
1656068700|3010933557|419658|4992|
1660817400|3010933557|499461|4956|1

… based on the previous help I was given at Historical data import caused invalid data? - #5 by wimleers

For whatever reason there is a whole bunch of inaccurate entries

wimleers · November 19, 2022, 9:44pm

Fixed now:

sqlite> UPDATE daydata SET power=0 WHERE power > 4392081921620023;
sqlite> select * from daydata ORDER by Power DESC LIMIT 2;
TimeStamp|Serial|TotalYield|Power|PVoutput
1656067500|3010933557|418354|4992|
1656068700|3010933557|419658|4992|

→ data is now uploading to G2Cv1 6.000kW!

May my hours of digging and trying to connect dots help the next person!

grannos · November 20, 2022, 9:39am

I’m not sure if this helps but 2147483647 is the maximum value of a signed 32 bit integer → 2^31 - 1.

Could something have caused a ( calculation ) overflow / underflow or have signed and unsigned values been mixed?

wimleers · November 20, 2022, 4:44pm

I’m aware that it’s simply an integer overflow. So it’s likely -1 getting set somewhere in SBFSpot’s code or my inverter returning an invalid value. Either way, SBFSpot’s code is at fault, because it should protect against edge cases like this.

IOW: yes, something caused this, but SBFSpot does not log anything for this exceptional value, so … I don’t really have any way to determine the root cause. At least not without becoming an expert in SBFSpot’s code base and my inverter’s behavior.

SBF · November 23, 2022, 1:07pm

What SBFspot version are you running? The issue you have should have been fixed in V3.8.4 but the root cause can be different in your case.
Also take a look at this workaround
If you post a debug log (-d5 -v5) I might be able to pinpoint the problem.

wimleers · April 13, 2023, 8:21pm

Thanks again for the help here!

I just used the same trick again to fix another corruption. But this time it seems to have happened in a more mysterious way still, so I created an sbfspot issue: https://github.com/SBFspot/SBFspot/issues/635.

Also, I run SBFspot using Docker thanks to https://github.com/nakla/sbfspot. Which is using version 3.8.4.