This site is now 100% read-only, and retired.

Splitting files into pieces and rejoining them

Posted by Steve on Thu 4 Aug 2005 at 14:47

Tags: none.

There are times when large files need to be chopped into small pieces, for easier movement, or for transfer via email with size limits in place. With the use of standard shell commands you can accomplish this easily.

The most natural way to do it is to use the aptly named split command which is contained in the coreutils package - and installed upon all Debian systems.

As its name suggests split will allow you to convert one large file into multiple smaller ones. To control it you simply specify the maximum size of each output piece.

Assuming you have a 2Mb AVI file which you need to email to somebody who cannot accept files larger than 1Mb you could run:

 split --bytes=1000000 1.avi output

This will split the input file, 1.avi, into separate files each containing not more than 1000000 bytes. The files will be named output??.

Once this has been done you'll two files, outputaa, and outputab.

If you wish to use numbers instead of letters in the output filename you can use the -d falg instead. (Or the longer version --numeric-suffixes. Debian's bash package with tab completion makes the longer form easier to use for me.):

split --numeric-suffixes --bytes=1000000 1.avi output

This will result in the files output00, output01, etc.

Joining files

Rejoining files is the very simple using the cat command. Simply invoke it with a list of all the files you wish to be joined - and redirect it to your desired output file.

Assuming you wish to place the contents of the files split1, split2, and split3 into the file output you could run:

cat split1 split2 split3 > output

Or if you have no other files beginning with split you can take advantage of your shells alphabetical ordering when performing glob operations to run:

cat split* > output

 

 


Re: Splitting files into pieces and rejoining them
Posted by andrewmc (195.218.xx.xx) on Thu 4 Aug 2005 at 14:31
While we're on the topic of splitting, another very useful command is csplit, which allows you to split text files based on patterns. For example if you had a python script and wanted to break it up by class you could enter:
csplit -f output test.py /class/ {*}
You'll then have a list of files called output00, output01, etc. (In reality you'll probably need a better pattern than just "class", but you get the idea).

[ Parent ]

Re: Splitting files into pieces and rejoining them
Posted by whizse (62.209.xx.xx) on Thu 4 Aug 2005 at 20:54
For newbies, it's easier to use a a multiplier suffix, eg:
split --bytes=1m 1.avi output

[ Parent ]

Re: Splitting files into pieces and rejoining them
Posted by Steve (82.41.xx.xx) on Thu 4 Aug 2005 at 21:02
[ View Weblogs ]

Indeed. Good catch ..

I'm more used to splitting at "non-regular" byte limits, so I'm pretty used to typing out the actual size.

Steve
-- Steve.org.uk

[ Parent ]

Re: Splitting files into pieces and rejoining them
Posted by debstar (193.251.xx.xx) on Mon 8 Aug 2005 at 07:26
[ View Weblogs ]
Humm!! Splitting avi file?

In the same line of sight, if it could be MPEG a/v file you can use mpgtx to manipulate it : split and join with the same command. The bonus? each splitted part can be processed as MPEG a/v file unlike with split.


NMC - Nirina Michel RATOANDROMANANA

[ Parent ]

Re: Splitting files into pieces and rejoining them
Posted by Steve (82.41.xx.xx) on Mon 8 Aug 2005 at 07:35
[ View Weblogs ]

Good tip, thanks.

In this case it was a job I had to do when a friend's home mail server refused to accept mail attachments over a given size. Actually playing the seperate chunks wasn't really anything that was necessary, the hard part was sharing the file at all.

(Had it been much bigger I'd have placed it on a HTTP server, or FTP server, instead.)

Steve
-- Steve.org.uk

[ Parent ]

splitting mp3 files (with cue sheet)
Posted by simms (69.157.xx.xx) on Wed 17 Aug 2005 at 22:33
speaking of splitting files, i'm not sure if this is the place to mention it, but today i came across mp3splt, which does a really nice job cutting big MP3 files (and other sound formats as well i think) into predefined small chunks (i.e. via CDDB or a CUE sheet file). it's neat.

[ Parent ]

Re: Splitting files into pieces and rejoining them
Posted by Anonymous (84.9.xx.xx) on Wed 14 Sep 2005 at 15:29
i gotta say steve ive been gooling for this and many other solutions and seem to end up here a lot!
and usually have 90% of my probs sorted here..cheers for a great site bud :D

[ Parent ]

Re: Splitting files into pieces and rejoining them
Posted by Steve (82.41.xx.xx) on Wed 14 Sep 2005 at 15:35
[ View Weblogs ]

You're welcome :)

Steve
--

[ Parent ]