Solis Cloud data naar PVOutput, Domoticz

[oldman2uk]

ZuinigeRijder schreef op zaterdag 7 december 2024 @ 14:05:
@oldman2uk It could be that the keyboard setting is in the wrong layout (on the Pi or SSH terminal), so some special characters are on a different place

Probably you can ask this question better in this topic, because you will get more (and probably better) help:

Raspberry Pi: Ervaringen en Discussie - deel 2

When that works, you can come back to this topic, if there are problems with SolisCloud2PVOutput.

I'm back for some assistance please.

I now have a folder in /home/pi/solis. In that folder are 4 files solis.sh, soliscloud_to_pvoutput.py, soliscloud_to_pvoutput.cfg and logging_config.ini.
I have added my solis cloud api detail and my pvoutput api too.
I used sudo chmod +x solis.sh at the command line pi@raspberrypi:~ $ and got no error message so assume that was ok.

The crontab -e: has me somewhat confused.
I want the script to run 24/7 to pvoutput, so will the command
2 5 * * * ~/solis/solis.sh > /dev/null
@reboot sleep 123 && ~/solis/solis.sh > /dev/null
do that or should I use some other?

Without just pulling the plug, how do I stop the script? No matter if I cannot, just curious.

I am using pi OS from May 2020 as any of the later ones were so very slugish on my old Pi.

[ZuinigeRijder]

oldman2uk schreef op vrijdag 13 december 2024 @ 13:12:
[...]


I'm back for some assistance please.

I now have a folder in /home/pi/solis. In that folder are 4 files solis.sh, soliscloud_to_pvoutput.py, soliscloud_to_pvoutput.cfg and logging_config.ini.
I have added my solis cloud api detail and my pvoutput api too.
I used sudo chmod +x solis.sh at the command line pi@raspberrypi:~ $ and got no error message so assume that was ok.

Great. looks Ok.

The crontab -e: has me somewhat confused.
I want the script to run 24/7 to pvoutput, so will the command
2 5 * * * ~/solis/solis.sh > /dev/null
@reboot sleep 123 && ~/solis/solis.sh > /dev/null
do that or should I use some other?

This means that the script will every day start at 5:02 each morning. Plus after reboot of the Raspberry Pi (123 seconds after reboot).

soliscloud_to_pvoutput.py will run in a loop, but will exit if it is before 5:00 or after 22:00. Crontab will start the script again the next day at 5:02.

So you can start the solis.sh script by rebooting your Pi

code:

sudo reboot

You can look at the log files created: solis.crontab.log and solis.log, e.g.

code:

cat solis.log
cat crontab.solis.log

Without just pulling the plug, how do I stop the script? No matter if I cannot, just curious.

I am using pi OS from May 2020 as any of the later ones were so very slugish on my old Pi.

You need to kill the script solis.sh. You can find the PID (process id) with the command ps -ef, example:

code:

rick@raspberrypi:~/solis $ ps -ef | grep solis.sh
rick      7138  7137  0 05:02 ?        00:00:00 /bin/sh -c ~/solis/solis.sh > /dev/null
rick      7139  7138  0 05:02 ?        00:00:00 /bin/bash /home/rick/solis/solis.sh
rick     11323 11175  0 13:26 pts/0    00:00:00 grep --color=auto solis.sh

rick@raspberrypi:~/solis $ kill -9 7138

[oldman2uk]

Thanks for the quick reply.

If I wish to have the pvoutput updated every 5 minutes 24/7 do I need to put this in the command line after typing crontab -e

*/5 * * * * ~/solis/solis.sh > /dev/null and hit return

followed by @reboot sleep 123 && ~/solis/solis.sh > /dev/null ? or something else on this 2nd line?

[ZuinigeRijder]

@oldman2uk No, this will not work, script will exit before 5:00 and after 23:00. This does also not make sense, because there is no solar generation at those hours. And otherwise it will run in a loop and send data to PVOutput whenever new SolisCloud data is available. So it is 18/7

[ Voor 15% gewijzigd door ZuinigeRijder op 13-12-2024 13:56 ]

[oldman2uk]

But I also have batteries as well as solar. I get this from solis.

[oldman2uk]

And this from Pvoutput updated every 5 minutes 24/7.

[ZuinigeRijder]

@oldman2uk Than you need to adapt soliscloud_to_pvoutput.py.

For example comment line 385 till 387:

code:

        # only check between 5 and 23 hours
        #if datetime_now.hour < 5 or datetime_now.hour > 22:
        #    logging.info("Outside solar generation hours (5..23)")
        #   sys.exit("Exiting program to start fresh tomorrow")

And change crontab in something like:

code:

0 0 * * * ~/solis/solis.sh > /dev/null
@reboot sleep 123 && ~/solis/solis.sh > /dev/null

[oldman2uk]

I did as suggested for comment lines 385 to 387 and changed crontab.

Rebooted Pi and no logs created: solis.crontab.log and solis.log

I have to guess the scripts not running?

No pvoutput change either.

Any easy way I can check if its running?

[ZuinigeRijder]

@oldman2uk in the crontab error messages are discarded. You can change the crontab such that also those messages (if any) are written to a separate log file. For example:

code:

0 0 * * * ~/solis/solis.sh >> ~/solis/crontab.log 2>&1
@reboot sleep 123 && ~/solis/solis.sh >> ~/solis/crontab.log 2>&1

I also have found a problem in my advice to comment the lines. The script computes compute hiRes energy_generation per day, so you need to reset the begin values.
So instead of commenting, probably something like:

Change in soliscloud_to_pvoutput.py line line 385 till 387:

code:

        # only check between 5 and 23 hours
        if datetime_now.hour < 5 or datetime_now.hour > 22:
            logging.info("Outside solar generation hours (5..23)")
            sys.exit("Exiting program to start fresh tomorrow")

Into:

code:

        # reset once a day
        if datetime_now.hour == 0 and datetime_now.minute < 4:
            timestamp_previous = "0"
            energy_generation = 0

I did not test if there are (other) side effects.

[oldman2uk]

I think this may be too difficult for me to understand.

Crontab doesn't show me anything that's been entered previously, in fact it doesn't show me anything at all full stop.
So I have no idea what I previously wrote there now.

The "# only check between 5 and 23 hours" lines to be changed for "# reset once a day" lines
I cannot find these to change in any of the 4 files in solis directory.

If they were entered in the crontab then that's just a black hole for me.

[ZuinigeRijder]

@oldman2uk My bad I was not clear, it is not crontab to be changed, but changes in soliscloud_to_pvoutput.py line line 385 till 387.

The only change you need to do in the crontab is these lines:

code:

0 0 * * * ~/solis/solis.sh >> ~/solis/crontab.log 2>&1
@reboot sleep 123 && ~/solis/solis.sh >> ~/solis/crontab.log 2>&1

I added ">> ~/solis/crontab.log 2>&1" instead of "> /dev/null". The latter will ignore any output.

[oldman2uk]

OK thanks again, I will return to this tomorrow, brain fog has descended for tonight

[oldman2uk]

Ive not got back to the Pi yet but I am dyslexic so some stuff is more difficult than it should be.

From your github code. solis.sh

Is the line 15 start an error fi instead of if ?

[ZuinigeRijder]

@oldman2uk No, fi is correct, it is the end of the if statement in a bash script. See here a tutorial.

[oldman2uk]

LOL they don't make it easy for a boy named Sue do they.

Back to my double checking everything and adding stuff.

Oh and I found my perfectly good in windows 11 Amazon basic wired keyboard does strange stuff on the Pi, like the @ sign is AWL and the ~ sign too which is not a great help. So today will feature another keyboard test run first.
Thanks once more.

[oldman2uk]

@ZuinigeRijder OK I found all went well with 4 files into solis directory at /home/pi/solis.

Now to write the crontab lines.

The changed wireless keyboard I used will not function in command mode, so I have to change keyboards again.

I type crontab -e
That gets me #green comments, I scroll down and using copy and paste I copy the 2 lines

0 0 * * * ~/solis/solis.sh >> ~/solis/crontab.log 2>&1 then enter key
then
@reboot sleep 123 && ~/solis/solis.sh >> ~/solis/crontab.log 2>&1 then enter key



I googled and found ctrl+c exits without changes
Also ctrl+k+x saves changes and exits


now I have many options and it tells me what it will save the lines as but I have no idea what to do other than save and exit.


I didn't get any of this happening yesterday after entering the crontab stuff so maybe it just didn't work at all yesterday.


Now it says, no crontab for pi - using an empty one.
Installing new crontab.



So after that I rebooted and I now have a crontab.log which says
/bin/sh: 1: /home/pi/solis/solis.sh: permission denied


The crontab command seems to be in place though as I used crontab -l to list the following.

[ZuinigeRijder]

@oldman2uk You can list the crontab with command:

code:

crontab -l

/bin/sh: 1: /home/pi/solis/solis.sh: permission denied

Looks like you did not make solis.sh executable. Did you do the command:

What are the permissions of solis/sh?

code:

ls -l /home/pi/solis/solis.sh

These are mine permissions:

code:

-rwxr-xr-x 1 rick rick   840 Aug 19  2022 solis.sh

code:

chmod +x /home/pi/solis/solis.sh

But you also need read permission, so maybe that is the problem. This command will set this:

code:

chmod u+r+x /home/pi/solis/solis.sh

[ Voor 17% gewijzigd door ZuinigeRijder op 14-12-2024 14:56 ]

[oldman2uk]

@ZuinigeRijder

Here are my permissions before and after.

I have rebooted.

I now have a crontab.log which says /bin/sh: 1: /home/pi/solis/solis.sh: not found

Here also is my solis.sh file

[ZuinigeRijder]

@oldman2uk I think you need to set execute permission to all users, so:

code:

chmod +r+x /home/pi/solis/solis.sh

[oldman2uk]

@ZuinigeRijder
I did this chmod +r+x /home/pi/solis/solis.sh

result in picture. I rebooted and crontab.log is still saying /home/pi/solis/solis.sh: not found (2nd entry)

Thanks once more.

[ZuinigeRijder]

@oldman2uk I think the problem is the permissions of the solis directory above.

code:

ls -l /home/pi
chmod +r+x /home/pi/solis
ls -l /home/pi

[oldman2uk]

@ZuinigeRijder

result in picture. I rebooted after and crontab.log is still saying /home/pi/solis/solis.sh: not found (3rd entry)

[ZuinigeRijder]

@oldman2uk something unexpected. On your OS bash is located elsewhere. What is the result of this command?

code:

which bash

That returned full path needs to be in the first line in solis.sh

[ Voor 3% gewijzigd door ZuinigeRijder op 14-12-2024 17:50 ]

[oldman2uk]

@ZuinigeRijder
which bash gave this photo.


The 2nd photo is how my solis.sh looked.


I have edited the file to remove the #! from the first line.

Now on reboot I get this in crontab.log.

[ZuinigeRijder]

@oldman2uk No, the #! needs to be in front. I do not have a clue now, so still it is a Raspberry Pi problem, so probably ask for help in rhe other forum I mentioned once.

[oldman2uk]

OK well many thanks for all the prompting and guidance you have given me. I will put the #! back and try to get further in the pi forum.
I might be back if I may, should I get a problem with the python you have written.

[ZuinigeRijder]

@oldman2uk Of course

[ZuinigeRijder]

@oldman2uk I have another thing to try, specifying full path in the crontab, e.g.

code:

0 0 * * * /home/pi/solis/solis.sh >> /home/pi/solis/crontab.log 2>&1
@reboot sleep 123 && /home/pi/solis/solis.sh >> /home/pi/solis/crontab.log 2>&1

Maybe this does not work, but worth a try.

[ZuinigeRijder]

@oldman2uk And another possible problem, your script solis.sh has Windows line endings, not Unix endings.

You can fix this with the command:

code:

dos2unix /home/pi/solis/solis.sh

My guess is that this is the real problem. Did you edit solis.sh on Windows?

[ Voor 16% gewijzigd door ZuinigeRijder op 14-12-2024 20:51 ]

[oldman2uk]

@ZuinigeRijder

As I didn't have a keyboard that would produce the right characters I got round this by copy and paste the 4 needed files from github into a folder on a memory stick called solis.
I then used the memory stick on a hub in the Pi to move the files and folder to home/Pi

I don't see any visible difference between the raw guthub file and one now in the Pi, but I take your word that they are not the same and I may have edited it in windows yes but I really don't remember now.

Should I use that dos2unix line on all four files I got from github (with the end part changed of course) ? in case I have contaminated them too with windows line ends, whatever they look like! Edit I have researched and now understand more about LF & CR's

I will delay the crontab adjustment until I have decontaminated my file and rebooted to test.

I will be tomorrow now, but again I thank you for the extra mile.

[ Voor 3% gewijzigd door oldman2uk op 15-12-2024 02:26 ]

[ZuinigeRijder]

@oldman2uk Yes, better to also use dos2unix on the other files too.

[ZuinigeRijder]

@oldman2uk You can configure the keyboard layout in Raspberry Pi raspi-config. See this post.

[oldman2uk]

ZuinigeRijder schreef op zondag 15 december 2024 @ 08:14:
@oldman2uk You can configure the keyboard layout in Raspberry Pi raspi-config. See this post.

Thanks, I did already find and set my 3rd keyboard so that's no longer an issue to add to the mix

[oldman2uk]

@ZuinigeRijder
That started badly, bash: dos2unix: command not found.

I used add & remove software to find and install dos2unix. At least the command line didn't complain when I typed dos2unix.
I then used it on all the files in solis & rebooted 3 times in all.
The result is consistent, crontab.log says....not found.






I've yet to try typing in the 2 lines into crontab. I'm not sure I remember how right now. Do I type one line then hit enter, then type the second line?
What happens to the other command we entered previously? does it need removing somehow?

In command line if I enter crontab -e that gives me the same 2 lines as your full path above ^ but without the ~ before the /
So do I attempt to edit the 2 lines there already to remove ~ or delete the lines (if thats possible) I didnt yet try.
Or will 2 new lines auto remove the existing? I have a row of edit choices at screen base.

[ Voor 28% gewijzigd door oldman2uk op 15-12-2024 10:47 ]

[ZuinigeRijder]

@oldman2uk With crontab -e existing entries keep there, unless you delete them in the editor. So you can change them or delete the lines. And yes, each command needs on a separate line, so enter after a command.

Note that you can also already try if soliscloud_to_pvoutput.py is working, without using crontab.

code:

cd /home/pi/solis
python soliscloud_to_pvoutput.py

At least you can see if sending to PVOutput is working. You can end the running with CTRL-C.

Thereafter you can try to solve the problem with the crontab and running solis.sh.

[oldman2uk]

@ZuinigeRijder I will go back to crontab and edit as needed, I think I have the way to do that now thanks.

I did try to run python soliscloud_to_pvoutput.pv and got this.

[ZuinigeRijder]

@oldman2uk Maybe you have an old version of python, you need at least python 3.9 or higher. Which version gives:

code:

python --version

It is also possible that python3 is installed and python is the default 2.x version:

code:

python3 --version

[ Voor 29% gewijzigd door ZuinigeRijder op 15-12-2024 11:21 ]

[oldman2uk]

@ZuinigeRijder I have version 2.7.16
This is Pi os from may 2020 so thats why python is too old.

Am I able to upgrade this OS python or is that not possible?

[ZuinigeRijder]

@oldman2uk I edited my previous comment, which version gives python3 --version?

[oldman2uk]

@ZuinigeRijder Python version 3.7.3 is inside already.

[ZuinigeRijder]

@oldman2uk Ok, it is too old, it needs at least 3.9 I think.

However, first check if 3.7.3 is also running, without the need of installing a newer version of python.

code:

cd /home/pi/solis
python3 soliscloud_to_pvoutput.py

If it is working, than you do not need to upgrade, but prepend the calls to soliscloud_to_pvoutput.py with python3. Otherwise you can upgrade to python 3.9.2 or later. Instructions here. Version 3.10.2 as specified in the example is also fine.

[ Voor 5% gewijzigd door ZuinigeRijder op 15-12-2024 11:35 ]

[oldman2uk]

@ZuinigeRijder error line 385

[ZuinigeRijder]

@oldman2uk There is an error in the line: "and and" should be one "and". I corrected my post where I did copy/paste it wrongly.

Good news is that it looks like your python3 version seems to work.

[ Voor 46% gewijzigd door ZuinigeRijder op 15-12-2024 11:55 ]

[oldman2uk]

@ZuinigeRijder

[ZuinigeRijder]

@oldman2uk Yeah, you need to install paho_mqtt. Probably something like:

code:

python3 -m pip install paho_mqtt

[oldman2uk]

@ZuinigeRijder

[ZuinigeRijder]

@oldman2uk Ah, the newer version of paho_mqtt is not compatible with python3. So you need to have an older version of paho_mqtt.

code:

python3 -m pip uninstall paho_mqtt
python3 -m pip install paho_mqt==1.6.1

[oldman2uk]

@ZuinigeRijder

[ZuinigeRijder]

@oldman2uk I made a typo, missing a t, should be:

code:

python3 -m pip install paho_mqtt==1.6.1

[oldman2uk]

@ZuinigeRijder that fixed it so its sending data but maybe an error on pvoutput? Ive blanked out my pv id and serial in this picture.

[ZuinigeRijder]

@oldman2uk Looks like your pvoutput_api_key and/or pvoutput_system_id is not correct in soliscloud_to_pvoutput.cfg. Did you configure/enable pvoutput.org (Settings -> API Access enabled)?

[oldman2uk]

@ZuinigeRijder Sorry for the delay, I was checking pvoutput. I made an error in the ID. Corrected now and running but not long enough yet to see if its collecting data
https://pvoutput.org/list.jsp?userid=124179

[oldman2uk]

@ZuinigeRijder https://pvoutput.org/intraday.jsp?id=124179&sid=107241

[ZuinigeRijder]

@oldman2uk Great. Now the crontab getting to work. Note that you need to edit solis.sh to change python intro python3.

I assume you also want to see battery information in the last column. Did you configure that?
pvoutput_fill_power_consumption_with_familyloadpower = False
pvoutput_fill_power_consumption_with_homeconsumption = False

[ Voor 47% gewijzigd door ZuinigeRijder op 15-12-2024 13:14 ]

[oldman2uk]

@ZuinigeRijder I have altered solis.sh to python3

So ctrl c to stop the script now and reboot to hopefully auto start it? Is that right?

Thank you for sticking at it with me

[ZuinigeRijder]

@oldman2uk Yes, still probably some problem in crontab though. But lets see.

[oldman2uk]

@ZuinigeRijder crontab.log reports
/bin/sh: 1: /home/pi/solis/solis.sh: not found

[oldman2uk]

@ZuinigeRijder
here is solis.sh

[ZuinigeRijder]

@oldman2uk Yeah, maybe you are looking at the error message the first time it was written. Because the command:

code:

0 0 * * * /home/pi/solis/solis.sh >> /home/pi/solis/crontab.log 2>&1
@reboot sleep 123 && /home/pi/solis/solis.sh >> /home/pi/solis/crontab.log 2>&1

">>" in these 2 commands mean that messages are appended to /home/pi/solis/crontab.log.

If you change it into one > it will rewrite the log file:

code:

0 0 * * * /home/pi/solis/solis.sh > /home/pi/solis/crontab.log 2>&1
@reboot sleep 123 && /home/pi/solis/solis.sh > /home/pi/solis/crontab.log 2>&1

So maybe it is now just running Ok. You can look into solis.log and (and remove prior the solis.log with command "rm solis.log").

[ Voor 9% gewijzigd door ZuinigeRijder op 15-12-2024 13:35 ]

[oldman2uk]

@ZuinigeRijder I edited the crontab but when I try to save it says I have errors?

[oldman2uk]

Hey, you just removed 2 lines

[oldman2uk]

@ZuinigeRijder

Just 1 entry in the new crontab log /bin/sh: 1: /home/pi/solis/solis.sh: not found

[oldman2uk]

@ZuinigeRijder

Perhaps name the crontab directly in solis.sh?

Its reporting /tmp/crontab.TDBblF/crontab when I use crontab -e

[ZuinigeRijder]

@oldman2uk No, the tmp is normal. The only I can think of now is to add bash to the crontab (although that would not be needed).

code:

0 0 * * * /bin/bash /home/pi/solis/solis.sh > /home/pi/solis/crontab.log 2>&1
@reboot sleep 123 && /bin/bash /home/pi/solis/solis.sh > /home/pi/solis/crontab.log 2>&1

And I assume you did the dos2unix solis.sh.

Otherwise I have no clue anymore, probably ask in the other forum for some help.

[oldman2uk]

@ZuinigeRijder You must be tired after nursing me all day, many thanks for the assistance.

I too will rest a while. How may I ask do I make a back up copy of my SD card in case it fails I wouldn't want to start again from scratch!
I would like to re asign voltage and perhaps temperature to battery soc and battery charge at some future date, I will look at the readme for your github scripts.

I did btw do dos2unix on all the files in the solis directory.

I did the bin/bash as you suggested and the result was.



Have a nice afternoon.

[ Voor 30% gewijzigd door oldman2uk op 15-12-2024 15:58 ]

[ZuinigeRijder]

@oldman2uk I think definitely something is wrong with the solis.sh file in mixing CRLF. Especially because of the '\r' in the error messages. Also the first line will be wrong (with mixing CRLF), therefor /bin/bash was not found. So you might copy the original solis.sh once again to the Raspberry Pi, do dos2unix and change python into python3.

You can also try to copy/paste the following lines directly in an editor of solis.sh on the Raspberry Pi and after saving do an additional dos2unix and make sure the execute bit is set (chmod +r+x solis.sh)

code:

#!/bin/bash
# ---------------------------------------------------------------
# A script to keep the soliscloud_to_pvoutput.py script running.
# Add to your crontab to run once a day, e.g. at 05:02 and at reboot
# 2 5 * * * ~/solis/solis.sh > /dev/null
# @reboot sleep 123 && ~/solis/solis.sh > /dev/null
# ---------------------------------------------------------------
script_name=$(basename -- "$0")
cd ~/solis

now=$(date)
if pidof -x "$script_name" -o $$ >/dev/null;then
   echo "$now: $script_name already running" >> solis.crontab.log 2>&1
   exit 1
fi
echo "WARNING: $now: $script_name NOT running" >> solis.crontab.log 2>&1
/usr/bin/python3 -u ~/solis/soliscloud_to_pvoutput.py >> solis.log 2>&1

Make sure the last line is an empty line.

[oldman2uk]

@ZuinigeRijder I did all the above, deleted the old solis.sh

Rebooted and crontab has just 1 error line now.
/home/pi/solis.sh: line 18: syntax error: unexpected end of file.

So I again opened solis.sh in a pi editor, set the cursor on a new line after the last text, saved the file then in command window used dos2unix again.
rebooted.
Same crontab error showing /home/pi/solis.sh: line 18: syntax error: unexpected end of file.

[ZuinigeRijder]

@oldman2uk I wonder if the python script is running in the background, despite the error message. You can check with

code:

ps -ef | grep python

[ Voor 3% gewijzigd door ZuinigeRijder op 15-12-2024 18:35 ]

[oldman2uk]

ZuinigeRijder schreef op zondag 15 december 2024 @ 18:34:
@oldman2uk I wonder if the python script is running in the background, despite the error message. You can check with

code:

ps -ef | grep python

I've not tried to check but the script can be started in the command line and returns an output to pvoutput.org every 5 minutes.
If I reboot and wait longer than 5mins and have no output to pvoutput.org my thinking is that the script is not running.
I have it running right now from the command line.

They tell me in another forum that cron was never designed to be used in this way. There are better ways I am told. https://forums.raspberrypi.com/viewtopic.php?t=314455

[ZuinigeRijder]

@oldman2uk Yes, but the script needs to be run in the background, when you are logged out. solis.sh should also run from the command line, running the python script in the background.

There is still something wrong with solis.sh.

Can you post the output of the command

cat solis.sh

[oldman2uk]

OK thanks for your help.

[ Voor 4% gewijzigd door oldman2uk op 15-12-2024 20:34 ]

[ZuinigeRijder]

@oldman2uk I think I spotted the error. On line 15 I miss the i from the fi (which is the endif).

[oldman2uk]

ZuinigeRijder schreef op zondag 15 december 2024 @ 21:27:
@oldman2uk I think I spotted the error. On line 15 I miss the i from the fi (which is the endif).

Well done that man! I now have a solis.crontab.log and a solis.log that were not there before and it rebooted and started its task without action from the command line by me.

I feel I must apologise to all who have helped me in this as I am dyslexic so things tend to look right when they are not. I found the if and fi statements and thought the fi was an error then I compounded that error when I corrected my earlier mistake.

So I just need to check it plays nice if the lan cable is disconnected and that it will run with no keyboard/mouse & monitor connected.
Then I need to hopefully add some data and move some as not all is showing in pvoutput right now.

Time for all to have a beer or two

[ZuinigeRijder]

@oldman2uk If I summarize what were the real problems:
- you must make sure that there are no Windows style CRLF, which was the case in solis.sh and probably in the other files too
- the read+execute permission should be set on solis.sh
- you have a older python version where python 3.x is not the default, so you have to adapt solis.sh to change python into python3
- you had to install the package paho_mqtt==1.6.1

The other steps was learning and trying to find the problem. Some of the steps taken were not necessary, but were helpful in tracking down the problems.

I still hope you enjoyed the journey

[oldman2uk]

@ZuinigeRijder

My journey continues

I removed the keyboard/mouse & hdmi connects and it worked fine still. Until midnight or 12:00AM then no more outputs.
By 12:30AM it was obvious (headless) that it wasn't going to start the new day so a reboot by power removal got it moving again, Its now 12:45AM and its sending output fine since the reboot to pvoutput.

The only data showing is the temperature of the inverter and the mains voltage (in the wrong slot). Power generated when the sun shines is correct and in the right slot.

Power being used is near 4kWh now but nothing showing on the graph, its slot is taken by mains voltage.

Battery SOC is an extra slot that I am happy to exchange inverter temperature for.

So once you are refreshed I will be back to pick your brain if I may?

https://pvoutput.org/intraday.jsp?id=124179&sid=107241

[ZuinigeRijder]

@oldman2uk You really must not power off the raspberry pi the hard way by power removal, this might corrupt your SD card.

You should try to start a ssh from your computer to your raspberry pi to open a terminal window. That way you can give commands and copy paste output. I use Putty for that. And also reboot the Raspberry Pi (sudo reboot) via a terminal. And at the end you can log out the terminal session (logout).

Tutorial how to setup Putty and SSH here.
If SSH is not enabled, see here how to do that.

can you post the content of solis.log?

[ Voor 56% gewijzigd door ZuinigeRijder op 16-12-2024 08:02 ]

[oldman2uk]

ZuinigeRijder schreef op maandag 16 december 2024 @ 07:20:
@oldman2uk can you post the content of solis.log.

@ZuinigeRijder

Its a very big log at a line every 5mins. Here is the part dealing with midnight. looks like the data didnt stop, just failed to make a new day start on pvoutput. I will look at putty later, thank you.

[oldman2uk]

@ZuinigeRijder

I have renamed the 3 old ssh files in my pc and set ssh up on my Pi. I have logged in on my pc to the pi via cmd. I have also changed the password from the default.
I still dont feel comfortable working in cmd line. I didnt download putty yet as I saw it asked I put ssh in the boot or root of my Pi sd card.

I am more concerned that my data to pvoutput will fail each night at midnight, I could put a digital timer to cut the Pi power at midnight and restart it at 12:02am but I would rather not as like you say, it may corrupt my sd card.
Also looking forward there is so far no usable data showing in pvoutput apart from the inverter temperature and system voltage, neither of which I really need. So unless I can get better usable data to work with then this exercise has been a little disappointing.
I value your help so far in this, but wonder if it will become usable without lots of time and effort?

[ZuinigeRijder]

oldman2uk schreef op maandag 16 december 2024 @ 20:40:
@ZuinigeRijder

I have renamed the 3 old ssh files in my pc and set ssh up on my Pi. I have logged in on my pc to the pi via cmd. I have also changed the password from the default. I still dont feel comfortable working in cmd line.

Ok, so you have another tool for remote logging in to the Raspberry Pi.

I didnt download putty yet as I saw it asked I put ssh in the boot or root of my Pi sd card.

I do not understand this, you do not need to install Putty on the Raspberry Pi, you can install this in your Windows PC. But apparently you do not need that, because you have another way to login on the Raspberry Pi. What did you use?

I am more concerned that my data to pvoutput will fail each night at midnight, I could put a digital timer to cut the Pi power at midnight and restart it at 12:02am but I would rather not as like you say, it may corrupt my sd card.

As far as I could see from the logging, there was no failure at midnight.

Also looking forward there is so far no usable data showing in pvoutput apart from the inverter temperature and system voltage, neither of which I really need. So unless I can get better usable data to work with then this exercise has been a little disappointing.
I value your help so far in this, but wonder if it will become usable without lots of time and effort?

I already mentioned the choice between 2 settings (one of them you can set to True):
pvoutput_fill_power_consumption_with_familyloadpower = False
pvoutput_fill_power_consumption_with_homeconsumption = False

Discussion here for the first one and here for the second one.

[oldman2uk]

@ZuinigeRijder
I was following a youtube video for installing an OS on the Pi, and after 3 installs I had a ssh file for each install as instructed in the video. I had to keep renaming them to save deleting.
Today I followed another tutorial for how to install ssh as I had forgotten how to do it in the Pi.
Then I changed the default password and wrote myself a note on how to get to the Pi from a windows pc cmd line.

As to putty it wasn't to install putty on the pi, it said to put a ssh file in the boot folder of the pi. Its here in red at the end of step 1 https://robu.in/putty-configuration-raspberry-pi/

My Pi did not update my pvoutput at or after midnight. I don't know why as it was running. My other pvoutput instance ran through midnight same as always here https://pvoutput.org/intr...6619&dt=20241216&gs=0&m=0
Click next day to see the rollover at midnight.

Thanks for the links to the 2 discussions, I will take a look later while I watch the midnight rollover

Are you able to tell me what data types are available from soliscloud api? I do need a minimum of Power used and power generated plus soc battery. I see power generated so far on pvoutput but it doesn't increase the total per day past 0.099kWh which is clearly incorrect, Its measuring the generated for the graph but I'm guessing solis doesnt total the day past 0.099kWh.

My brain is hurting again

[oldman2uk]

@ZuinigeRijder
Once more the end of day came and went, after 11:55pm no more data appears to flow to Pvoutput.
At 1:00am I rebooted after checking the solis log did have data after midnight. At 1:05am pvoutput again got data. Very strange.

[ZuinigeRijder]

@oldman2uk As I did not test my changes to run it for always, I see that my instructions for the local changes were NOT complete. As you see in your post at 00:04:53 and subsequent data still the old date is send.

There are more changes needed to fix this, so I decided to make a configuration item to run it unendless and implemented this in soliscloud_to_pvoutput.py. Also added some more information to the README for the problems encountered. See this release.

- Add "run_unendless = True" to the [api_secrets] section in soliscloud_to_pvoutput.cfg
- copy the updated soliscloud_to_pvoutput.py to your Raspbery Pi (and run dos2unix on this one)

[oldman2uk]

@ZuinigeRijder
Again I thank you for your perseverance on this task of ours.

Pretty sure I didn't stop the program when doing the file changes. I used control and C several times from the cmd line in the Pi but looking at the pvoutput later the time stamps don't stop until I did a reboot at 09:57am.

I swopped some stuff from false to true in the pvoutput section of the soliscloud_to_pvoutput.cfg to get more data viewable while I was there.

I ran dos2unix as instructed on the new file once installed.

I have a new file appeared in the solis folder since today, it may be because of my file changes "live" ?
Its called d2utmpUxXN65.log and looks to be empty.

[ZuinigeRijder]

@oldman2uk The python script runs in the background, if you want to stop that you have to kill that process. I have in one of my posts put that command ("ps -ef | grep soluis.sh" and "kill -9"). When you change settings, you need to restart the script.

The temporary log file can be removed.

PVOutput fields are limited, so I am curious which settings you did change. Because there are not much settings to choose from/overrule.

The information in the script available is:

code:

date
time
energy_generation
solar_power
battery_power
battery_soc
grid_power
family_load
home_consumption
inverter_temperature
dc_voltage
ac_voltage

Currently in PVOutput fields you can only override the power_consumption (default ac-voltage) with family_load or home_consumption for PVOutput. And voltage (default dc_voltage) with ac_voltage.

With Domoticz or MQTT (e.g. Home Assistant) all the settings above are available, but much much more complex and not suitable for your Raspberry Pi.

[oldman2uk]

@ZuinigeRijder
So to stop the script can I just type "kill -9" and must it be from cmd line or must I "cd solis" first? will that kill all scripts?

Or must I at the cmd line type "ps -ef | grep solis.sh" or do I need to cd to solis first?

Temp file has removed itself.


I put fill temp with inverter temp = true
fill voltage with ac voltage = true
fill power cons with homeconsumption = true
fill power cons with homeconsumption = false
fill power cons with ac voltage = false

date
time
energy_generation
solar_power
battery_power
battery_soc
grid_power
family_load
home_consumption
inverter_temperature
dc_voltage
ac_voltage

So although the above are available, they are not for the Pi so I cant use battery soc anyway or battery power.
Is that because they didn't yet get added to the script or because the Pi or pvoutput cant cope with the extra data?

Oh and the time seems to have gone out of sync. The pvoutput update is now near 2 mins behind real time updating. It used to be at worst 20seconds delayed.

[ZuinigeRijder]

oldman2uk schreef op dinsdag 17 december 2024 @ 15:13:
@ZuinigeRijder
So to stop the script can I just type "kill -9" and must it be from cmd line or must I "cd solis" first? will that kill all scripts?

Or must I at the cmd line type "ps -ef | grep solis.sh" or do I need to cd to solis first?

No, you have to find the process id first and put that number as parameter of kill -9 as described in this post earlier. But a reboot will of course kill also all running scripts.

Temp file has removed itself.

I put fill temp with inverter temp = true
fill voltage with ac voltage = true
fill power cons with homeconsumption = true
fill power cons with homeconsumption = false
fill power cons with ac voltage = false

date
time
energy_generation
solar_power
battery_power
battery_soc
grid_power
family_load
home_consumption
inverter_temperature
dc_voltage
ac_voltage

So although the above are available, they are not for the Pi so I cant use battery soc anyway or battery power.
Is that because they didn't yet get added to the script or because the Pi or pvoutput cant cope with the extra data?

The only way is to misuse one of the fields of PVOutput, e.g. inverter_temp. But then you must adapt soliscloud_to_pvoutput.py yourself by changing line 483 into something elese, e.g.:

code:

                pvoutput_string = f"data={today_yyymmdd},{current_time},{energy_generation},{solar_power},{energy_consumption},{power_consumption},{temperature},{voltage}"  # noqa

into

code:

                pvoutput_string = f"data={today_yyymmdd},{current_time},{energy_generation},{solar_power},{energy_consumption},{power_consumption},{battery_power},{voltage}"  # noqa

If you look closely you see I changed {temperature} into {battery_power}.
Because PVOutput is meant for solar panels, all the battery information cannot be send too. If you pay for PVOutput, you can have extended parameters.
However, then you also need to adapt the python script yourself.

Oh and the time seems to have gone out of sync. The pvoutput update is now near 2 mins behind real time updating. It used to be at worst 20seconds delayed.

That is probably more to when SolisCloud does the updates, the python script checks once per minute if there is new data. So in practice there can be a delay of 1 minute due to the python script. But that does not matter, because the timestamp of SolisCloud update is send with the data.

[oldman2uk]

@ZuinigeRijder pvoutput_string = f"data={today_yyymmdd},{current_time},{energy_generation},{solar_power},{energy_consumption},{power_consumption},{temperature},{voltage}" # noqa

So if I change the above to {battery_soc} instead of {temperature}

What exactly do I change the cfg line too?

[PVOutput]
send_to_pvoutput = True
pvoutput_fill_temperature_with_inverter_temperature = True

From above to below?

pvoutput_fill_temperature_with_battery_soc= True


Anywhere else I need to change ?

Finding the processs ID is confusing.

There are several numbers given, how do I know which is the right one to use? Is the ID always the same even after a reboot? Is there not a simple code if the ID constantly changes to use on the cmd line that tells a string to shut down the process. And maybe another to start it rather than have to go through a reboot?

Here is my list of ID's.

[ZuinigeRijder]

oldman2uk schreef op dinsdag 17 december 2024 @ 17:22:
@ZuinigeRijder pvoutput_string = f"data={today_yyymmdd},{current_time},{energy_generation},{solar_power},{energy_consumption},{power_consumption},{temperature},{voltage}" # noqa

So if I change the above to {battery_soc} instead of {temperature}

What exactly do I change the cfg line too?

[PVOutput]
send_to_pvoutput = True
pvoutput_fill_temperature_with_inverter_temperature = True

From above to below?

pvoutput_fill_temperature_with_battery_soc= True


Anywhere else I need to change ?

You do not need to change the cfg config file, because you programmatically overrule what is programmed in the settings for this item.

Finding the processs ID is confusing.

There are several numbers given, how do I know which is the right one to use? Is the ID always the same even after a reboot? Is there not a simple code if the ID constantly changes to use on the cmd line that tells a string to shut down the process. And maybe another to start it rather than have to go through a reboot?

Here is my list of ID's.

[Afbeelding]

It is the oldest number you need to kill.
Another approach is to use:

code:

pkill -9 solis.sh
nohup /home/pi/solis/solis.sh &

The first one kills the process with name solis.sh and the second one starts solis.sh in the background and it keeps running, also when logged out. I do not if pkill is installed on your Raspberry Pi, for me it is.

[oldman2uk]

@ZuinigeRijder Once more thank you for your continued assistance.

"It is the oldest number you need to kill."

There are 6 numbers in my picture which number is oldest?


pkill -9 solis.sh doesn't seem to do anything. pkill is installed though. help gets me a list of its options.

I assume that should I wish to shutdown the pi without maybe corrupting the script I can use shutdown on the gui desktop or sudo shutdown -r from the command line? I assume again that the os closes down any scripts before shutting down?
Reboot with sudo reboot?

I did use the file editor while the script was running it seems, I had no idea if it was stopped or not.
I now have my battery soc showing on the graph instead of inverter temperature, it will look worrying when the battery is fully charged (100 deg C inverter)

Oh and nohup solis.sh & gets errors no such file I think it said.

[ZuinigeRijder]

oldman2uk schreef op dinsdag 17 december 2024 @ 21:56:
@ZuinigeRijder Once more thank you for your continued assistance.

"It is the oldest number you need to kill."

There are 6 numbers in my picture which number is oldest?

270

pkill -9 solis.sh doesn't seem to do anything. pkill is installed though. help gets me a list of its options.

It kills the script silently, you can check with "ps -ef | grep solis.sh" if it is still running. Note that you should ignore the line with "grep" in it.

I assume that should I wish to shutdown the pi without maybe corrupting the script I can use shutdown on the gui desktop or sudo shutdown -r from the command line? I assume again that the os closes down any scripts before shutting down?
Reboot with sudo reboot?

Yes, that is correct.

I did use the file editor while the script was running it seems, I had no idea if it was stopped or not.
I now have my battery soc showing on the graph instead of inverter temperature, it will look worrying when the battery is fully charged (100 deg C inverter)

When the python script is running and you make changes, those will not be taken over yet. So you have to stop and start solis.sh or reboot.

Oh and nohup solis.sh & gets errors no such file I think it said.

You need to prepend the directory path, e.g.:

code:

nohup /home/pi/solis/solis.sh &

or when you are already in the solis directory:

code:

nohup ./solis.sh &

[oldman2uk]

@ZuinigeRijder

So tomorrow I will use Pi imager to make a backup of my SD card.

Then safe in the knowledge that I have a backup copy I will use it as is, if I need to make changes to the pvoutput lines I can either do that live and then reboot to save me from having to learn the 100's of commands you have shown me. Or make copies and substitute files though I cannot see a reason for needing to do that.

Oh and keep this page bookmarked so I can return for a refresher

Many thanks for the crash course.

Does the solis.log self empty older days data as its getting quite large after just 3 days.

[oldman2uk]

@ZuinigeRijder

ZuinigeRijder schreef op dinsdag 17 december 2024 @ 09:50:

- Add "run_unendless = True" to the [api_secrets] section in soliscloud_to_pvoutput.cfg
- copy the updated soliscloud_to_pvoutput.py to your Raspbery Pi (and run dos2unix on this one)

I did exactly that, remember i'm doing just what I am told

At 11pm the script stopped.



So I searched soliscloud_to_pvoutput.py for what could be wrong and of course found the False line.

69 RUN_UNENDLESS = get_bool(api_secrets, "run_unendless", False)

Changed to True and rebooted, now working again.

[ZuinigeRijder]

oldman2uk schreef op dinsdag 17 december 2024 @ 22:50:
@ZuinigeRijder

Does the solis.log self empty older days data as its getting quite large after just 3 days.

Does not matter how big solis.log gets. You can get e.g. the last 100 lines with the command:

tail -100 solis.log

And of course you can just delete solis.log if it gets too big, but is normally not needed:

rm solis.log

oldman2uk schreef op woensdag 18 december 2024 @ 04:06:
@ZuinigeRijder

[...]


I did exactly that, remember i'm doing just what I am told

At 11pm the script stopped.

[Afbeelding]

So I searched soliscloud_to_pvoutput.py for what could be wrong and of course found the False line.

69 RUN_UNENDLESS = get_bool(api_secrets, "run_unendless", False)

Changed to True and rebooted, now working again.

Did you not add "run_endless = True" to soliscloud_to_pvoutput.cfg See here example config.

Because you did not need to change soliscloud_to_pvoutput.py for that.

[ Voor 4% gewijzigd door ZuinigeRijder op 18-12-2024 08:13 ]

[oldman2uk]

@ZuinigeRijder Just this once you are wrong.

I did add run_endless = true at the time of the original post telling me to do so, picture below.

A reboot last night left me with the same "come back tomorrow" message. Only after changing soliscloud_to_pvoutput.py did the script run again, started at 2:40am

Oh and while I am awake nursing my script I tried changing the pvoutput string {voltage} to {grid_power} but the result wasn't what I expected. When the grid was supplying 3600watts I got 3.0v so even with a multiplier the number would have been wrong, I will try another substitution later today.

[ZuinigeRijder]

@oldman2uk I also make sometimes typo's, in the config the setting must be:

code:

run_unendless = True

You can also log all the values in the solis.log file, by changing lines 451 to 453:
From:

code:

if logging.DEBUG >= logging.root.level:
                debug_string = f"date={today_yyymmdd}, time={current_time}, energy_generation={energy_generation}, solar_power={solar_power}, battery_power={battery_power}, battery_soc={battery_soc}, grid_power={grid_power}, family_load={family_load}, home_consumption={home_consumption}, inverter_temperature={inverter_temperature}, dc_voltage={dc_voltage}, ac_voltage={ac_voltage}"  # noqa
                logging.debug(debug_string)

Into:

code:

            debug_string = f"date={today_yyymmdd}, time={current_time}, energy_generation={energy_generation}, solar_power={solar_power}, battery_power={battery_power}, battery_soc={battery_soc}, grid_power={grid_power}, family_load={family_load}, home_consumption={home_consumption}, inverter_temperature={inverter_temperature}, dc_voltage={dc_voltage}, ac_voltage={ac_voltage}"  # noqa
            print(debug_string)

Note that the "if" is removed and the indentation is 4 spaces less, logging.debug is changed into print and should align with the previous line:

code:

            current_time = datetime_current.strftime("%H:%M")

At least you can look at all the values got from SolisCloud, even afterwards, although not in PVOutput.

[oldman2uk]

ZuinigeRijder schreef op woensdag 18 december 2024 @ 09:16:

code:

            current_time = datetime_current.strftime("%H:%M")

At least you can look at all the values got from SolisCloud, even afterwards, although not in PVOutput.

Where does this last line go? in a file or?

I now have enough commands and notes from here to start a book

This task has become all consuming and with Christmas just days away I need to get stuff done as I expect do you, so I am going to thank you yet again and back up what I have, then put it to one side for the holiday before I go any deeper into this black hole

[ZuinigeRijder]

@oldman2uk Editing soliscloud_to_pvoutput.py, changing line 451 to 453. On line 450 you will find the line starting with current_time.

[Kroonkurk]

@ZuinigeRijder
Sinds kort verhuisd en mijn nieuwbouw appartement heeft zonnepanelen met een Solis omvormer met wifi stick. Connectie met Soliscloud is werkend en via jouw python script kan ik elke 5 minuten data naar PV Output sturen. Werkt allemaal netjes.
ik heb zelf lopen te stoeien om via een Bash script de data van de Solis API af te halen, maar heb het nog steeds niet voor elkaar gekregen. Wat ik nog graag wil, is de data die naar PV Output wordt geschreven in jouw script, naar een tekstbestand te schrijven. Totaal geen kaas gegeten van Python, maar een bestand openen om daar iets in weg te schrijven zal hopelijk nog wel lukken.
Maar op welke regel moet ik dit doen en welke variabele(n) moet ik gebruiken ?
Kan en wil je mij op weg helpen ?

[ZuinigeRijder]

@Kroonkurk Op regel 213 van soliscloud_to_pvoutput.py wordt de data die naar PVOutput gestuurd wordt, ook als info-logging gestuurd:

code:

logging.info(pvoutput_string)

Deze data kun je natuurlijk ook toevoegen aan een text-file. Zie hier een voorbeeld van hoe dat moet in Python.

[Kroonkurk]

@ZuinigeRijder
Bedankt, ga er mee aan de slag. Denk dat dit wel gaat lukken.