This site is now 100% read-only, and retired.

uploading with an ftp macro script

Posted by PJ_at_Belzabar_Software on Thu 2 Feb 2006 at 09:15

Gather round the hearth, young nerdlings and I will tell you a tale...just let me settle my creaking bones into my rocking chair, let me wipe my rheumy eyes and nose - there, that's better. Now pass me my ear trumpet. Do give me a little prod if I nod off or my voice wavers too much. Are we all settled in now? Yes? Marvellous! Now let me tell you about ftp upload.

People don't respect the ftp (the file transfer protocol) that much anymore. Partly because it is a plain text protocol, which means that passwords used are snoopable. But mainly because it isn't as visual and well-known as web-based file transfers for the non-geek, nor is it as intelligent and powerful as rsync for the geek. Nonetheless, ftp is often handily there - even that cesspit win95 had an ftp client.

Now, I have used FTP for 15 years now, and like all of you, I knew you could mget, get, mput, put, hash, bin, prompt etc. However, it was news to me that ftp had a macro facility hidden inside its depths.

Uploading with an FTP script

I wanted to do an automated upload of files and directories. Previously I had been looking at scripts that would upload files and directories via bash feeds. After playing around with those, I was somewhat dissatisfied because such scripts just spew out all the commands in one go. On large transfers or slow networks I worried that this might overwhelm the ftp server input buffer, and the server might lose commands.

A way around it is to use expect, which brings wait-for-responses interactivity to scripts. Or you can of course just throw up your hands in disgust and use that bazooka called perl to pound and pulverize the problem into submission with net::ftp.

But that is getting overly complicated, and I am a lazy and simple-minded fellow.

Enter the macro

It turns out that in most cases FTP macros are a splendidly lazy and simple way to do such things.

In a nutshell:

  • You use macdef to define the macro. See man ftp for details
  • Combine this with netrc. .netrc is the file that has the default settings for the ftp client - see man (5) netrc for details.
  • Use macdef init in the .netrc file, and you have a quick way to automatically script the upload.

Example: you have some files in your htdocs directory that you want to upload to a remote public_html directory. You also have files under an image subdirectory and under a css subdirectory that you want to upload. You want to keep the directory structure and files the same on the remote site as on your local box.

The .netrc file for this would be something like:

login mrexample
password mysecretpassword
macdef init
#this is an embarrassing way to do a comment
lcd /home/mrexample/htdocs
cd public_html
mput *
mkdir css
#already exists? no problem
lcd css
cd css
mput *
mkdir ../images
lcd ../images
cd ../images
mput *
#newline terminates macro

I think you get the idea.

Now, if you do a ftp the script will automatically run with no further input, (because of the init keyword).

How it works

By default glob is on, so the * lists all the files c-shell style, ignoring dot files. The files under htdocs will get mputted into Any directories won't mput (there will be a great wailing from the ftp client), but nothing evil will happen, so we just let the script run on. It runs on and does the copying of the directories css and images separately by explicitly dropping into the subdirectories.

The comment lines don't make sense (there will be much gnashing of teeth from the server), but nothing evil will happen, so we just keep them in for our own reference.


So, there you are. A simple, lazy way to script ftp to upload and leave a smug smile on your face.

PJ, Belzabar Software Design



Re: uploading with an ftp macro script
Posted by Anonymous (80.126.xx.xx) on Thu 2 Feb 2006 at 09:49
great article!
Just forgot to chmod 600 your .netrc file to prevent people on your system from reading your password (but then, if there is anyone sniffing your network, they will get it anyway).

[ Parent ]

Re: uploading with an ftp macro script
Posted by PJ_at_Belzabar_Software (61.11.xx.xx) on Thu 2 Feb 2006 at 11:01
[ View Weblogs ]
Thanks for the feedback.

Actually, these days debian's ftp won't let you auto-login if your netrc is readable by others if your password is in the script (like it was in the example script). The netrc man page mentions this. I guess you're an old fogey like me then, worrying about that sort of historical problem ;-)


[ Parent ]

Re: uploading with an ftp macro script
Posted by Anonymous (220.245.xx.xx) on Thu 2 Feb 2006 at 10:03
you can also script lftp with the mirror command.

[ Parent ]

Posted by Anonymous (84.133.xx.xx) on Thu 2 Feb 2006 at 12:07
I really see no advantage over piping some ftp commands into the ftp-client.
Using a here document in my scripts I have all pieces in front of me, using .netrc forces me to look up in two places if something fails.

Enlighten me... why should I use .netrc?

[ Parent ]

Re: Advantage?
Posted by PJ_at_Belzabar_Software (59.176.xx.xx) on Fri 3 Feb 2006 at 07:50
[ View Weblogs ]
Yup, there is always more than one way to do it.

My personal worry was overwhelming the ftp client with a large number of file names, and maybe having to use expect to get around it. The built in macro system of ftp would presumably not fail that way.

The macro method also worked out to be cleaner, simpler, and quicker to build compared with bash here docs, at least for the kind of stuff I did. But then I am the kind of guy who has to look up how bash loops are done.


[ Parent ]

Re: Advantage?
Posted by Anonymous (84.133.xx.xx) on Sat 4 Feb 2006 at 11:33

Kinda matter of taste... perfectly ok for me...

[ Parent ]

Re: uploading with an ftp macro script
Posted by alfadir (141.58.xx.xx) on Thu 2 Feb 2006 at 14:56
[ View Weblogs ]
That reminds me of when the school got email back in 1994.
No direct internet access but a dialup connection.

My first emailaddress. It did not take long until we figured out we that
we could surf the ftp archives with email!

By sending a mail to a gateway one could list and cd into a ftp archive.
Also getting files.

I remeber the joy when getting that first picture from a ftp server.
In the morning before class send a mail with
ls /pub
get the result back in time for 10:00 break to do
cd /pub/pictures
at lunchbreak the mail was waiting in the inbox.
a list of pictures. New mail.
cd /pub/pictures
get apache.jpg
14:00 break
10 emails with the picture neatly packaged in UUEncoded parts.
copy paste them together and if one did it correcly
the picture of a apache attack helicopter at sunset...

Here is a list of servers that provide this service still :

Not sure what configurations are needed or if the tools are availible under Debian.

[ Parent ]

Re: uploading with an ftp macro script
Posted by simonw (84.45.xx.xx) on Thu 2 Feb 2006 at 18:17
[ View Weblogs ]
You missed out the worst sin of many FTP clients which is error handling. Many of the Unix clients were quite capable of putting half a file, losing the TCP connection, and returing no error code.

I seem to remember ncftp is sound in this regard, but such experiences in the past have put me off trying to do anything too clever with FTP.

Always worth checking what happens when things go wrong.

[ Parent ]

Re: uploading with an ftp macro script
Posted by PJ_at_Belzabar_Software (59.176.xx.xx) on Fri 3 Feb 2006 at 07:40
[ View Weblogs ]
Which reminds me, I should really have put in a bin in the netrc script just after the first comment line.

Sure, in these bandwidth-rich days most ftp connections will default to binary mode. ftp will however spank you soundly for being unwary if you rely on it: You could leave your customer's website in a eye-wateringly technicoloured garbled state for a few days.

DAMHIKIJKOK. (Don't ask me how I know, I just know, OK?)


[ Parent ]

Re: uploading with an ftp macro script
Posted by Anonymous (68.209.xx.xx) on Mon 13 Feb 2006 at 03:08
This is a terrible way to do it.

You can do it in FEWER lines using PERL's Net::FTP. And you can do lots of error checking, traverse directories, add extra logic etc.

Ever since I learned how to do it in Perl, I would never try and automate and ftp like this anymore.


use Net::FTP;
print "retrieving file from myhost\n";
$ftpobj = Net::FTP -> new ("myhost");
$ftpobj -> login("p1","abc123");
$ftpobj -> cwd ("/export/home/www/cosite/about");
$ftpobj -> get ("melksham.html");
$ftpobj -> quit;
print "file size ",-s "melksham.html"," bytes\n";

Good article here: -050426/page1.html

[ Parent ]

Re: uploading with an ftp macro script
Posted by PJ_at_Belzabar_Software (61.11.xx.xx) on Mon 13 Feb 2006 at 04:41
[ View Weblogs ]
<mutter>...young whippersnapper...teaching your grandfather to chew tobacco...</mutter>

"FEWER lines"?


It's the same steps in the cases here, just covered in a layer of perl module to complicate matters.

I like using and abusing perl for everything. Perl is like bringing in the heavy artillery - you can tackle scarier stuff with it that you couldn't hope to do easily with ftp macros. Yes, go ahead and use Net::FTP for heavy-duty ftp transfers with multiple servers and loops that twist and turn like a twisty turny thing, and for when you want to handle timeouts gracefully.

But for simple stuff like we have here, you may as well keep things light, stay closer to the bare metal, and just use an ftp macro. Since you can build the macro very directly with a cut and paste of the ftp commands from a manual run, I say (with the wisdom that comes with beardy age) that doing the same thing in perl for
simple stuff like this is overkill and will take longer. Still, YMMV.

thanks for the feedback,

[ Parent ]