This site is now 100% read-only, and retired.

Writing shell scripts which execute locally or remotely

Posted by Steve on Mon 12 Mar 2007 at 07:12

Tags: ,

There are a lot of times when it is useful to have a single shell script run both upon the local host, and also upon remote hosts. Here we'll show a simple trick which allows you to accomplish this easily.

To execute shell scripts remotely the most obvious approach is to copy it there, with scp, and then use ssh to actually execute it. This is similar to running simple commands remotely using ssh directly:

skx@mine:~$ ssh yours uptime
 07:12:25 up 3 days, 18:15,  0 users,  load average: 0.00, 0.00, 0.08
skx@mine:~$

With that in mind the solution becomes:

  • Write a simple shell script which will be useful.
  • Determine whether it should run remotely, and if so:
    • Copy itself there.
    • Execute itself there.

As an example we'll look at a simple script which will report upon the uptime of the system it is executed upon:

Here is the script:

#!/bin/sh

#
#  Are we installing locally?  Or remotely?
#
if [ ! -z $1 ]; then

    #  Hostname
    host=$1

    #  Create a secure temporary file.
    file=`mktemp`

    #  Create a temporary file, and copy the contents of ourself into
    # it.  Making sure it has a shebang.
    echo "#!/bin/sh"                > "${file}"
    grep -A2000 '^#-=-MARKER-=' $0 >> "${file}"
    chmod 755 "${file}"

    #  Copy the file to the remote host, and invoke it
    scp "${file}" ${host}:
    ssh "${host}" ./`basename ${file}`

    #  Cleanup remotely and locally.
    ssh "${host}" /bin/rm `basename ${file}`
    rm ${file}

    #  All done - the rest of the script will occur remotely.
    exit
fi


## THE NEXT LINE IS IMPORTANT     - DO NOT EDIT.  DO NOT REMOVE.
#-=-MARKER-=-
## THE PREVIOUS LINE IS IMPORTANT - DO NOT EDIT.  DO NOT REMOVE.

uptime

Here you can see that the script detects whether to run remotely or not based upon the presence of a command line argument, so this is local execution:

skx@mine:~$ ./uptime.sh
 14:05:10 up  4:59,  4 users,  load average: 0.05, 0.05, 0.07

Whereas this is remote:

skx@mine:~$ ./uptime.sh cfmaster.my.flat
tmp.RRjRSx9137                                100%   98     0.1KB/s   00:00
 14:05:28 up 430 days, 20:02,  0 users,  load average: 9.72, 6.58, 4.26

Neat huh?

The key to this script is that it can separate out the "real" work of the script so that only the end of the script is copied to the remote host - the part after the argument processing. This is achieved with the following command:

grep -A2000 '^#-=-MARKER-=' $0

This uses the "-A" option of GNU grep to cause it to print out a number of line after the line beginning "#-=-MARKER-=" - this is the part of the script that actually reports on the system uptime, and this is the part you'd replace with your own code.

The relevant lines are then placed into a temporary file and copied to the host upon which it should execute them. (If you didn't have key-based authentication setup you'd be prompted for your password three times; the first time for the copy, the second time for the execution, and the final time to cleanup the file which was copied.)

Using a simple system like this you could easily write scripts that would preform tasks like installing CFEngine locally or remotely.

 

 


Re: Writing shell scripts which execute locally or remotely
Posted by alfadir (141.58.xx.xx) on Mon 12 Mar 2007 at 10:38
[ View Weblogs ]
There are also the c3 tools, it is made for clusters, so I am not sure how
well it would work for you. It has to be set up once etc. but is then pretty easy to use

http://www.csm.ornl.gov/torc/C3/

There is no debian package that I know of.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by linulin (212.119.xx.xx) on Mon 12 Mar 2007 at 12:47
Use the following to avoid double ssh invocation:
ssh "${host}" "./`basename ${file}`; /bin/rm `basename ${file}`".

--
...Bye..Dmitry.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by Anonymous (163.192.xx.xx) on Mon 12 Mar 2007 at 16:38
I used the following technique to do remote monitoring of local parameters such as disk space. I called it 'poor man's .NET'. To make a long story short.

1. You write a script in your interpreted language of choice as if it was supposed to run locally.
2. You setup your SSH keys so a password is not needed to connect.
3. You run your script as follows
cat script.pl | ssh $host perl

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by randallb (68.45.xx.xx) on Mon 12 Mar 2007 at 18:58
Although I find Steve's script very clever, I still prefer the simplicity of just piping a script over SSH. However, I'm not sure how foolproof the piping method is.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by Steve (62.30.xx.xx) on Mon 12 Mar 2007 at 20:51
[ View Weblogs ]

The only comment I'd make here is that I've found that the explicit temporary file is very useful for debugging problems - which is probably why I had the "/bin/rm" as a distinct step.

It is very useful to be able to copy the scripts over with predictable names and leave them in situ for debugging - especially when trying to do the same thing on N hosts.

Otherwise I think that piping would be simple enough; the only failure cases I can imagine are ones that my script suffers from too - lack of disk space being the most obvious.

Steve

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by Steve (62.30.xx.xx) on Mon 12 Mar 2007 at 20:50
[ View Weblogs ]

I have no idea why that didn't occur to me; great idea.

Steve

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by windo (194.126.xx.xx) on Thu 15 Mar 2007 at 10:27
It's pretty convenient to run any shell scripts when you have only one hop to take and can use key auth. I was striving for something more general, so I've been working on something like this:
windo@dididijero:~/bin$ cat remote_execute.sh
#!/bin/sh

function escape_to () {
        dest=$1
        command=$2
        echo ssh&nbs p;-t ${dest} ${command} | sed -e "s /\( [\$()'\] \)/\\\\\1/g"
}

function make_binary () {
        executable=$1
        string=$(hexdump& nbsp;-vC < ${executable} | tr -d  ;" " | cut -f 1 -d '|' |& nbsp;\
        sed -e  's/........//;s/\(..\)/\\x\1/g' | awk '{printf&nbs p;$1}')
        echo eval&nb sp;\$\(printf \'${string}\'\)
}
used something like this:
source /home/windo/bin/remote_execute.sh

file=$(mktemp)

cat > $file << EOF
some commands that need to be execu ted
EOF

binary=$(make_binary $file)

from_intermediary=$(escape_to root@final-destination "$ {binary}")
from_us=$(escape_to me@intermediary-host "${from_interm ediary}")
eval "$from_us" | egrep -v 'Password|Con nection' > output
This way i can skip scp and therefore only have to insert the password once for each step.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by windo (194.126.xx.xx) on Fri 16 Mar 2007 at 09:16
OR

i could have just done

cat script | ssh host1 ssh host2 /bin/bash > output

a bit easier, wouldn't you say?

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by matej (212.5.xx.xx) on Wed 21 Mar 2007 at 10:36
if there are many host2's, you make as many ssh connections to host1. getting the script to host1 and then piping it to many host2's in a loop from host1 is hell faster, even for tens of hosts... think about hundreds.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by Anonymous (212.91.xx.xx) on Mon 26 Mar 2007 at 23:21
I don't get your point here.

If you want to execute say, 'uptime' on host "cfmaster.your.flat",
just invoke "ssh cfmaster.your.flat uptime".

To take care of the password, enable SSH keys as others suggested.

And if the command you want to run is longer, more like a script
then a command line, then install NFS, Samba or any similar thing and
mount the scripts/ directory on all clients. Hell, you can even
keep scripts/ directory in sync with rsync over ssh for which you
have keys anyway. And then run say,

ssh cfmaster.your.flat /scripts/uptime.sh

And if you need anything more elaborate, then also as others
suggested, there are scripts that automate mass invocation on
multiple hosts and have other features you'd expect..

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by Anonymous (80.135.xx.xx) on Thu 19 Apr 2007 at 19:39
It may be that I'm totally wrong with this comment, but what about this script:
#!/bin/sh
#

#
# rx <host> <shellscript> [<arg> ...]
#


escape() {
        awk '
                BEGIN {
                        while (getline > 0)
                                script = script $0 "\n";

                        sub(/^#[^\n]*\n/, "", script);

                        bs = sprintf ("%c", 92);
                        sq = sprintf ("%c", 39);
                        rp = sq bs sq sq;
                        gsub(sq, rp, script);
                        script = sq script sq;
                        print script;
                        }'

        return 0
        }


host=$1
scriptfile=$2
shift 2

IFS='' script=`escape <$scriptfile`
ssh $host /bin/sh -c $script $scriptfile $*

exit

Having the script test.sh
#!/bin/sh
#

echo Hello World!
echo This is a 'test'.
echo arguments: $#: $0 $*

exit
invoking this like
wzk@qe:~$ rx linux-box test.sh 'Neuer Test.' 12345
gives
Hello World!
This is a test.
arguments: 3: test.sh Neuer Test. 12345
Right? Of course, test.sh can also be run on the local machine. But as the example shows the script is not perfect. The number of arguments (to the rx script) is 2, not 3 as the script says when run on linux-box.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by Anonymous (75.177.xx.xx) on Wed 6 Feb 2008 at 06:11
your script is [perfect] :)

the argument count starts at zero, with the "zero argument" being the command itself. in your example, the first argument is "test.sh".

0 == test.sh
1 == Neuer Test.
2 == 12345

giving a total of 3.

[ Parent ]

Re: Writing shell scripts which execute locally or remotely
Posted by eddyAlessandro (66.0.xx.xx) on Tue 27 Oct 2009 at 23:51
Hi Steve,
Thanks for the article. I did the script and works fine.
But I have one question, I am trying to do this:
#!/bin/sh
...
Your script
...
## THE NEXT LINE IS IMPORTANT - DO NOT EDIT. DO NOT REMOVE.
#-=-MARKER-=-
## THE PREVIOUS LINE IS IMPORTANT - DO NOT EDIT. DO NOT REMOVE.

sudo su - user
#end script

But my problem is that after run the line "sudo su - user" appears a prompt in my local machine and stop reading the next lines.
But if I enter the "pwd" command in the prompt that works in the "user2" directory.
I appreciate any help with this, I need to execute the command below the line "sudo su - user2" into the script.

Thanks and Regards.

[ Parent ]