Posted by Steve on Sat 5 Nov 2005 at 16:04
We've previously covered setting up your own repository for the Debian's apt-get system, but we didn't cover managing automatic uploads. Thankfully this is a simple task with the reprepro, and dupload tools and a small amount of scripting.
The reprepro package is tool for creating an APT repository with a pool structure, the same type of structure the official Debian mirrors use.
The repository may:
Removing PackagesInstalling the package is straightforward if you're using unstable, or etch, simply install it as you would install any other package:
apt-get install repreproUnfortunately the package contained within Debian's stable release, sarge, is a little outdated. For that reason I'd recommend that you install a back ported version. Thankfully this is readily available either from the projects homepage or elsewhere:
Once you've installed the software then we're ready to begin creating a repository. A new repository can be created in just a couple of steps:
- Decide where you wish the archive to be located.
- Create the configuration file.
- Import your first package
Almost certainly you will wish to serve your repository via a webserver, so the location should be beneath your webservers root directory at a simple path.
I use http://steve.org.uk/apt as my repository root location so my archive will be located beneath /home/www/www.steve.org.uk/htdocs/apt - your location will obviously differ.
First of all create the directory, a subdirectory conf/ to contain the configuration file, and a directory incoming/ to which we'll later setup automatic upload processing:
mkdir -p /home/www/www.steve.org.uk/htdocs/apt mkdir -p /home/www/www.steve.org.uk/htdocs/apt/conf mkdir -p /home/www/www.steve.org.uk/htdocs/apt/incomingNow that we have a directory to contain our repository we can look at creating the configuration file. The configuration file will specify which releases the repository will contain (sid, stable, etc) as well as the architectures. A sample configuration file will look like this:
Origin: Your Name Label: Your own label Suite: stable Codename: sarge Version: 3.1 Architectures: i386 all source Components: main non-free contrib Description: Your descriptionHere we've defined a repository which only contains packages for stable/sarge, which contains packages targeted to x86, or all. If you wish to contain both stable and unstable packages your configuration file will look like this:
Origin: Your Name Label: Your own label Suite: stable Codename: sarge Version: 3.1 Architectures: i386 all source Components: main non-free contrib Description: Your description Origin: Your Name Label: Your own label Suite: unstable Codename: sid Architectures: i386 all source Components: main non-free contrib Description: Your descriptionSave your configuration to the file conf/distributions and you should now be ready to import a package.
You can either import a .deb file into the repository, or a .changes file which is produced by building a package from source.
From the main directory run:
reprepro -Vb . include sarge name_of_fileFor example:
skx@lappy:~/apt$ reprepro -Vb . include sarge \ /home/skx/debian/sarge/reprepro/reprepro_0.6-1sarge0_i386.changes Created directory "./db" Created directory "./pool" Created directory "./pool/main" Created directory "./pool/main/r" Created directory "./pool/main/r/reprepro" db: 'reprepro' added to 'sarge|main|i386'. db: 'reprepro' added to 'sarge|main|source'. Created directory "./dists" Created directory "./dists/sarge" Created directory "./dists/sarge/main" Created directory "./dists/sarge/main/binary-i386" writing to './dists/sarge/main/binary-i386/Packages.new'... writing to './dists/sarge/main/binary-i386/Packages.gz.new'... Created directory "./dists/sarge/main/binary-all" writing to './dists/sarge/main/binary-all/Packages.new'... writing to './dists/sarge/main/binary-all/Packages.gz.new'... Created directory "./dists/sarge/main/source" writing to './dists/sarge/main/source/Sources.gz.new'... Created directory "./dists/sarge/non-free" Created directory "./dists/sarge/non-free/binary-i386" writing to './dists/sarge/non-free/binary-i386/Packages.new'... writing to './dists/sarge/non-free/binary-i386/Packages.gz.new'... Created directory "./dists/sarge/non-free/binary-all" writing to './dists/sarge/non-free/binary-all/Packages.new'... writing to './dists/sarge/non-free/binary-all/Packages.gz.new'... Created directory "./dists/sarge/non-free/source" writing to './dists/sarge/non-free/source/Sources.gz.new'... Created directory "./dists/sarge/contrib" Created directory "./dists/sarge/contrib/binary-i386" writing to './dists/sarge/contrib/binary-i386/Packages.new'... writing to './dists/sarge/contrib/binary-i386/Packages.gz.new'... Created directory "./dists/sarge/contrib/binary-all" writing to './dists/sarge/contrib/binary-all/Packages.new'... writing to './dists/sarge/contrib/binary-all/Packages.gz.new'... Created directory "./dists/sarge/contrib/source" writing to './dists/sarge/contrib/source/Sources.gz.new'...This has imported the package described in the file reprepro_0.6-1sarge0_i386.changes to the archive, creating the appropriate directories as required.
To have a less verbose output simply omit the -V flag, thusly:
skx@lappy:~/apt$ reprepro -b . include sarge \ /home/skx/debian/sarge/reprepro/reprepro_0.6-1sarge0_i386.changes skx@lappy:~/apt$
Using Your RepositoryIf you wish to remove a package from your repository you can do so with the remove command:
skx@lappy:~/apt$ reprepro -b . remove sarge reprepro Deleting files no longer referenced... deleting and forgetting pool/main/r/reprepro/reprepro_0.6-1sarge0_i386.deb deleting and forgetting pool/main/r/reprepro/reprepro_0.6-1sarge0.dsc deleting and forgetting pool/main/r/reprepro/reprepro_0.6.orig.tar.gz deleting and forgetting pool/main/r/reprepro/reprepro_0.6-1sarge0.diff.gzNote: you don't need to remove a package if you simply wish to replace an existing package with a newer version - this will be handled for you. When you import a newer version of a package contained in the archive already the older version will be removed.
Configuring Package Uploads With duploadOnce your packages has been added to the archive they may be downloaded via apt-get, or aptitude, with the appropriate lines in your /etc/apt/sources.list file:
deb http://example.com/apt sarge main contrib non-free deb-src http://example.com/apt sarge main contrib non-freeIf you're using two distributions simply repeat for each:
deb http://example.com/apt sarge main contrib non-free deb-src http://example.com/apt sarge main contrib non-free deb http://example.com/apt sid main contrib non-free deb-src http://example.com/apt sid main contrib non-freeYou should find that packages can be downloaded and installed as expected.
Importing Packages from an Incoming queue with repreprodupload is a tool which is designed to allow you to upload packages to different repositories.
Once installed (via "apt-get install dupload") it may be configured either via the file /etc/dupload.conf or ~/.dupload.conf.
To configure uploads for your remote host then simply add a section such as this:
$cfg{'example'} = { fqdn => "example.com", login => "steve", method => "scpb", incoming => "/incoming/", # The dinstall on ftp-master sends emails itself dinstall_runs => 1, };This configuration :
- Names the upload target 'example'.
- Specifies that file uploads should be conducted using scp.
- With the login name steve.
- That packages should be uploaded into the directory /incoming.
You will certainly need to change the hostname, login name, and incoming directory. You might also wish to specify an alternative means of uploading, such as anonymous FTP. For more details of the available options please see "man dupload.conf".
Once you've configured the upload settings you should be able to upload a Debian package by executing:
dupload --to example \ /home/skx/debian/sarge/reprepro/reprepro_0.6-1sarge0_i386.changes(Note that you should upload the .changes file, not the .deb file)
Once this command completes you'll find that you've successfully transferred the package to your incoming directory, beneath your apt root.
The next step is to allow reprepro to automatically add that package to the repository.
As a result of any dupload commands we'll expect to process the incoming files from the incoming/ directory. We've already seen that reprepro can handle the importing of packages via the .changes file.
So we simply need to write a small shell script which will locate each .changes file, import the package, then clean the directory afterwards.
(We must remove, or move, the files from the incoming directory to make sure we don't repeatedly try to re-add the same package.)
This is a sample script which does the job. It is based upon the one I use myself and should be easily usable elsewhere - simply change the incoming directory at the top of the script:
#!/bin/sh INCOMING=/home/www/www.steve.org.uk/htdocs/apt/incoming # # Make sure we're in the apt/ directory # cd $INCOMING cd .. # # See if we found any new packages # found=0 for i in $INCOMING/*.changes; do if [ -e $i ]; then found=`expr $found + 1` fi done # # If we found none then exit # if [ "$found" -lt 1 ]; then exit fi # # Now import each new package that we *did* find # for i in $INCOMING/*.changes; do # Import package to 'sarge' distribution. reprepro -Vb . include sarge $i # Delete the referenced files sed '1,/Files:/d' $i | sed '/BEGIN PGP SIGNATURE/,$d' \ | while read MD SIZE SECTION PRIORITY NAME; do if [ -z "$NAME" ]; then continue fi # # Delete the referenced file # if [ -f "$INCOMING/$NAME" ]; then rm "$INCOMING/$NAME" || exit 1 fi done # Finally delete the .changes file itself. rm $i doneTo use this simply setup a cronjob running every few minutes to process any new files in the incoming/ directory - something like this:
*/5 * * * * /usr/local/bin/import-new-packages.sh
Good catch on the rm problem - in my script rather than deleting the files I move them into an accepted/ directory - which is why there was a second argument left in there.
The idea of getting the distribution from the .changes file is a good one too - I honestly didn't think of that, but I guess you'd need to do that if you had more than one distribution available.
I'll leave that out of the script, since it might confuse people - but it will be available as a comment :)
Steve
--
[ Parent ]
Oops I forgot about your question!
gnupg-agent appears to be the only option, yes. I've not explored anything else, other than making .bz2 indexes available.
Steve
--
[ Parent ]
I'm not sure. Right now I just have this in conf/distributions:
SignWith: apt@steve.org.uk
And it all works because my key is explicitly for that, and has no passphrase.
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
Steve, your loop that increments (and checks for the value of) $found is not necessary.
for f in *some_file_filter*; do # this will only be run if $f actually holds a value now do the actual processing... done
You can simply remove this part:
#
# See if we found any new packages
#
found=0
for i in $INCOMING/*.changes; do
if [ -e $i ]; then
found=`expr $found + 1`
fi
done
#
# If we found none then exit
#
if [ "$found" -lt 1 ]; then
exit
fi
/peter
[ Parent ]
for f in *some_file_filter*; do # this will only be run if $f actually holds a value now do the actual processing... doneyou should do this:
for f in *some_file_filter*; do # if nothing matches the filter, the shell will return the filter string test -e "$f" || continue # this will only be run if $f actually holds a value now do the actual processing... doneTo witness this, issue this command:
echo *there_is_no_match*
[ Parent ]
[ Parent ]
Hmmm that is an interesting question.
I suspect you could build the package twice using the same orig.tar.gz, and a different .deb + changes for each and upload both. Since the .deb files would have different archs and the same version I think it would work out OK.
Here I think it would work because you can either import the .changes file or the .deb file. So import the .changes file for the x86 package - then afterwards import the x64 .deb file
Failing that hacking the .changes files manually to include both binary packages would presumably work, but would be more of a pain to apply since you'd need to resign the .changes file.
I guess the best thing to do is to try it and see!
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
Remove it from its old location. Rebuild with the correct section in the control file and re-upload?
[ Parent ]
[ Parent ]
[ Parent ]
It'll work just fine if you've uploaded the first revision of a package built with "-sa". (Since in that case the .orig.tar.gz file will be mentioned in the .changes file.)
For subsequent uploads don't include it as it will be already present.
[ Parent ]
Well there is always Debconf next year which will be held here!
[ Parent ]
[ Parent ]
[ Parent ]
When there's a new release of Debian, you may want to do a couple of things:
* edit conf/distributions, and
** add a set of lines for the new distribution
** update the Suite: lines, eg "stable" changes to "oldstable"
* reprepro -Vb . export # to update Packages and Release files
* reprepro -Vb . createsymlinks # to update symlinks, e.g. oldstable->etch
[ Parent ]
[ Parent ]
[ Parent ]
[ Parent ]
[ View Weblogs ]
How do you operate for unattended gpg signing? Is gnupg-agent the only option?
Regarding the script, I would parse the changes file to catch the right distribution (if you have one incoming directory for several distributions):
However, to use this, you have to invert "codename" and "suite" fields in the conf/distributions file.
I think the last bit is a typo from an adaptation:
Cheers,
Julien
[ Parent ]