I have a stand alone python script using qgis that I wrote with the help of the SE question here
The script takes three arguments being; an input file path, an output file path, and another input file path.
As per advice from this question the script now happily runs in parallel using gnu parallel with the following command and gives the correct output files:
parallel myscript.py /tmp/tile_{1}.shp /tmp/output_{1}.shp /tmp/input.shp ::: {1..445} I want to do the work in parallel on a remote server (or servers). So I enabled a passwordless login to an available server as per this question and I installed QGIS on the remote server, copied over the script, and made it executable from bash. If I manually copy the required input files over to the server the same parallel command as above runs fine.
The problem I have is when trying to run the parallel work from the local machine. I can transfer the files over to the remote server using
SERVER1=192.168.88.222 parallel -S $SERVER1 --transfer cat ::: /tmp/tile_{1..445}.* > /dev/null And I can run my process in parallel on both the local and remote server using
parallel -S 192.168.88.222,: myscipt.py /tmp/tile_{1}.shp /tmp/output_{1}.shp /tmp/input.shp ::: {1..445} This results in a mix of output files spread over the two computers. So far so good?! Unfortunately I cant get my files back from the remote server. I tried
parallel -S 192.168.88.222 --return ::: /tmp/output_*.* > /dev/null But gnu parallel just tells me parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit. Which I assume is because I have no actual command. So now I'm at a loss.
How do I get the resulting output files back from my remote parallel qgis script operation? My best guess is that I should have done it all in a single line and used cat somehow but I cant even imagine how it that line should look.
أكثر...
The script takes three arguments being; an input file path, an output file path, and another input file path.
As per advice from this question the script now happily runs in parallel using gnu parallel with the following command and gives the correct output files:
parallel myscript.py /tmp/tile_{1}.shp /tmp/output_{1}.shp /tmp/input.shp ::: {1..445} I want to do the work in parallel on a remote server (or servers). So I enabled a passwordless login to an available server as per this question and I installed QGIS on the remote server, copied over the script, and made it executable from bash. If I manually copy the required input files over to the server the same parallel command as above runs fine.
The problem I have is when trying to run the parallel work from the local machine. I can transfer the files over to the remote server using
SERVER1=192.168.88.222 parallel -S $SERVER1 --transfer cat ::: /tmp/tile_{1..445}.* > /dev/null And I can run my process in parallel on both the local and remote server using
parallel -S 192.168.88.222,: myscipt.py /tmp/tile_{1}.shp /tmp/output_{1}.shp /tmp/input.shp ::: {1..445} This results in a mix of output files spread over the two computers. So far so good?! Unfortunately I cant get my files back from the remote server. I tried
parallel -S 192.168.88.222 --return ::: /tmp/output_*.* > /dev/null But gnu parallel just tells me parallel: Warning: Input is read from the terminal. Only experts do this on purpose. Press CTRL-D to exit. Which I assume is because I have no actual command. So now I'm at a loss.
How do I get the resulting output files back from my remote parallel qgis script operation? My best guess is that I should have done it all in a single line and used cat somehow but I cant even imagine how it that line should look.
أكثر...