Building DVD Images Of Ubuntu Repositories

Want to support HowtoForge? Become a subscriber!
 
Submitted by ginocic (Contact Author) (Forums) on Thu, 2007-03-29 18:02. :: Debian | Ubuntu

Building DVD Images Of Ubuntu Repositories


1 Preliminary Note

This tutorial was inspired by an articles I read at http://cargol.net/~ramon/ubuntu-dvd-en. So many thanks to Ramon Acedo (the one who made this HowTo, originally)

The pages are not reachable from some weeks, now. I saved the page to read it off-line. So...

I found it useful. I hope it will be the same for you.


2 Introduction

This howto offers a simple way of creating DVD images of Debian or Ubuntu http/ftp repositories.

Ubuntu doesn't offer DVDs ready to download with its main, universe, multiverse and/or restricted repositories. With the contents of this howto you can do it yourself.

Having the Ubuntu or Debian repositories on DVD can be useful for those users who don't have access to the Internet where they have their Ubuntu installed but have access somewhere else to download the repository and build and burn the DVDs.

3 Building a local mirror

We have to install debmirror:

sudo apt-get install debmirror

Now we get the Ubuntu repositories in a local directory. In the example below we get main, universe and multiverse sections of the repository in the i386 architecture.

debmirror --nosource -m --passive --host=archive.ubuntulinux.org --root=ubuntu/ --method=ftp --progress --dist=dapper --section=main,multiverse,universe --arch=i386 ubuntu/ --ignore-release-gpg

You could change the options below as you prefer:

  • --host - the URL of the repository.
  • --dist - the distro of your OS (dapper, edgy, sarge, ... ).
  • --section - the section you want to mirror locally.
  • --arch - the architecture of your box.

4 Separating the archive into DVD-sized directories

The repositories we got are too big (about 30Gb) to burn them to a DVD so we have to separate them into volumes.

The tool debpartial will do it for us.

sudo apt-get install debpartial

We make the directory where the volumes will reside.

mkdir ubuntu-dvd

and we make it to construct the package descriptors to every volume.

debpartial --nosource --dirprefix=ubuntu --section=main,universe,multiverse --dist=dapper --size=DVD ubuntu/ ubuntu-dvd/

Now we have to put the packages into the directories debpartial has just created. The script debcopy which also comes with the debpartial package will do it. The script needs ruby.

sudo apt-get install ruby

If everything is ok...

ruby debcopy ubuntu/ ubuntu-dvd/ubuntu0
ruby debcopy ubuntu/ ubuntu-dvd/ubuntu1
ruby debcopy ubuntu/ ubuntu-dvd/ubuntu2

Where ubuntu/ is the directory with the complete repository created with debmirror and ubuntu-dvd/* are the directories ready to host the new DVD-ready repository.
If we want to make soft links from the complete repository instead of copying the packages we can call debcopy with the option -l:

ruby debcopy -l ubuntu/ ubuntu-dvd/ubuntu0
ruby debcopy -l ubuntu/ ubuntu-dvd/ubuntu1
ruby debcopy -l ubuntu/ ubuntu-dvd/ubuntu2

Now every directory (ubuntu0, ubuntu1 and ubuntu2) fits on one DVD.


5 Making iso images

To get the directories ubuntu0, ubuntu1, ubuntu2 into an iso image ready to burn we can use mkisofs:

mkisofs -f -J -r -o ubuntu-dvd-0.iso ubuntu-dvd/ubuntu0
mkisofs -f -J -r -o ubuntu-dvd-1.iso ubuntu-dvd/ubuntu1
mkisofs -f -J -r -o ubuntu-dvd-2.iso ubuntu-dvd/ubuntu2

Now you can burn the iso images or mount them. Add them to the /etc/apt/source.list with the command:

sudo apt-cdrom add

Now we can verify the new repositories...

sudo apt-get update
sudo apt-get upgrade

... and, if I explain in the right way, you should have your box upgraded.


6 About the script 'debcopy'

I heard about someone who can not find the script debcopy, above described.
In that case, create a new file called debcopy where you want:

gedit /your_path_to/debcopy

and copy the lines below inside it:

#!/usr/bin/ruby
#
# debcopy - Debian Packages/Sources partial copy tool
#
# Usage: debcopy [-l]  
#
#  where  is a top directory of a debian archive,
#  and  is a top directory of a new debian partial archive.
#
#  debcopy searches all Packages.gz and Sources.gz under /dists
#  and copies all files listed in the Packages.gz and Sources.gz
#  files into  from . -l creates symbolic links
#  instead of copying files.
#
# Copyright (C) 2002  Masato Taruishi 
#
#  This program is free software; you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation; either version 2 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License with
#  the Debian GNU/Linux distribution in file /usr/share/common-licenses/GPL;
#  if not, write to the Free Software Foundation, Inc., 59 Temple Place,
#  Suite 330, Boston, MA  02111-1307  USA
#
require 'getoptlong'
require 'zlib'
require 'ftools'
$link = false
def usage
  $stderr.puts "Usage: #{__FILE__} [-l]  "
   exit 1
end
def each (file, &block)
  fin = Zlib::GzipReader.open(file)
  fin.each do |line|
    yield line
  end
  fin.close
end
def each_file (file, &block)
  each(file) do |line|
    if /Filename: (.*)/ =~ line
      yield $1
    end
  end
end
def each_sourcefile (file, &block)
  dir = nil
  each(file) do |line|
    case line
    when /^Directory: (.*)$/
      dir = $1
    when /^ \S+ \d+ (\S+)$/
      yield dir + "/" + $1
    end
  end
end
def calc_relpath (source, dest)
  pwd = Dir::pwd
  Dir::chdir source
  source = Dir::pwd
  Dir::chdir pwd
  Dir::chdir dest
  dest = Dir::pwd
  Dir::chdir pwd
  src_ary = source.split("/")
  src_ary.shift
  dest_ary = dest.split("/")
  dest_ary.shift
  return dest if src_ary[0] != dest_ary[0]
  src_ary.clone.each_index do |i|
    break if src_ary[0] != dest_ary[0]
    src_ary.shift
    dest_ary.shift
  end
  src_ary.size.times do |i|
    dest_ary.unshift("..")
  end
  dest_ary.join("/")
end
def do_copy(path)
  if $link
    pwd=calc_relpath(File.dirname($dest_dir + "/" + path), $source_dir)
    File.symlink(pwd + "/" + path, $dest_dir + "/" + path)
  else
    File.copy($source_dir + "/" + path, $dest_dir + "/" + path)
  end
end
def copy(path)
  s=$source_dir + "/" + path
  d=$dest_dir + "/" + path
  if FileTest.exist?(d)
    $stats["ignore"] += 1
    return
  end
  if FileTest.exist?(s)
    File.mkpath(File.dirname(d))
    do_copy(path)
    $stats["copy"] += 1
  else
    $stats["notfound"] += 1
    $stderr.puts s + " not found."
  end
end
opts = GetoptLong.new(["--symlink", "-l", GetoptLong::NO_ARGUMENT],
		      ["--help", "-h", GetoptLong::NO_ARGUMENT])
opts.each do |opt,arg|
  case opt
  when "--symlink"
    $link = true
  when "--help"
    usage
  end
end
usage if ARGV.size != 2
$source_dir = ARGV.shift
$dest_dir = ARGV.shift
if $link
  $source_dir = Dir::pwd + "/" + $source_dir unless $source_dir =~ /\A\//
  $dest_dir = Dir::pwd + "/" + $dest_dir unless $dest_dir =~ /\A\//
end
$stats = {}
$stats["ignore"] = 0
$stats["copy"] = 0
$stats["notfound"] = 0
open("|find #{$dest_dir}/dists -name Packages.gz") do |o|
  o.each_line do |file|
    file.chomp!
    print "Processing #{file}... "
    $stdout.flush
    each_file(file) do |path|
      copy(path)
    end
    puts "done"
  end
end
open("|find #{$dest_dir}/dists -name Sources.gz") do |o|
  o.each_line do |file|
    file.chomp!
    print "Processing #{file}... "
    $stdout.flush
    each_sourcefile(file.chomp) do |path|
      copy(path)
    end
    puts "done"
  end
end
puts "Number of Copied Files: " + $stats["copy"].to_s
puts "Number of Ignored Files: " + $stats["ignore"].to_s
puts "Number of Non-existence File: " + $stats["notfound"].to_s

Please do not use the comment function to ask for help! If you need help, please use our forum.
Comments will be published after administrator approval.
Submitted by Mozgoed (not registered) on Wed, 2011-09-07 09:52.

My new portable programm for Windows/Linux can download repository components, calculate total loading size and remove old packages from existing repository.

Ubuntu-repository programm: http://mozgoed-mgoy.narod.ru/ubuntu/ubuntu-repository.zip

Required:

Windows 2000, Windows XP - .Net Framework 2.0

Windows Vista, Windows 7 - Nothing

Ubuntu 10.10 - Mono project installed. It can be executed from terminal "mono ubuntu-repository.exe"Welcome to Visual tools for Linux.

.Net by Mozgoed

Submitted by Mozgoed (not registered) on Sun, 2012-09-30 13:24.
You can download the latest version of Ubuntu-repository programm from page http://mozgoed-mgoy.narod.ru/?mod=page&page=7 .
Submitted by BobSongs (registered user) on Sat, 2008-03-08 04:18.

If I may add this second correction:
The tutorial says: 

--dist - the distro of your OS (dapper, edgy, sarge, ... ).
Suggested text:

--dist - the distro of your OS (dapper, edgy, feisty, gutsy or hardy)

Submitted by BobSongs (registered user) on Wed, 2008-03-05 03:37.

Greetings;

Just a note that a similar and somewhat more detailed version of this tutorial exists over at the Ubuntu Forums.


CORRECTION:
An important difference is that this version states the necessity to prepare for five (5) DVDs, and not three (3). It also shows how to make CD-Rs if that is the only alternative. You may wish to update this tutorial.


It details how to use the local files as repositories right off the hard drive.

There's an F.A.Q., a welcome addition, that covers how to create CDs for other important repositories such as Medibuntu and Canonical Commercial.

Also covered is how to change version (Dapper, Edgy, Feisty, etc.) as well as available versions such as i386, amd64, and so on.

Feel free to drop by and compare notes. All comments and suggestions are welcome.

Link: http://ubuntuforums.org/showthread.php?t=352460 

Submitted by vj777 (registered user) on Sun, 2007-07-29 04:32.

Use APTonCD.  Simple, easy to use GUI based. 

It can create DVDs (or CD sets) of complete repositories and also will let you create local repositories of all packages you have already installed, for use on the same machine or another off-line. 

Submitted by maxxer (registered user) on Wed, 2007-05-23 16:53.
There is a bug in the debcopy part: if you want to use symlinks, the -l must be put AFTER debcopy and not before.

The original howto can be found at http://cargol.net/~ramon/ubuntu-dvd-en.html


Submitted by ginocic (registered user) on Wed, 2007-06-20 16:48.

Sorry!

Thanks for noting that. I correct the command right now ;)

Submitted by asashnov (registered user) on Sat, 2007-05-12 09:17.

Before making ISO images it may usefull to check files in pool with this utility:

http://sashnov.fanstvo.com/apt_repository_check.py

 Usage:

~/bin/apt_repository_check.py  ubuntu-dvd/ubuntu0

It check for file presense, sile size and md5sum (optionnaly).

Submitted by asashnov (registered user) on Mon, 2007-05-07 09:10.

I guess use popularity contest for put most popular packages to first DVD:

 wget --output-document - http://popcon.ubuntu.com/all-popcon-results.txt.gz | zcat \
 | ruby -ne '/(\S+)\s+(\d+)\s+\d+\s+(\d+)/ =~ $_; puts "#{$2.to_i+ $3.to_i} #{$1}"' \
 | sort -nr | cut -d' ' -f2  > ~/ubuntu_popularity_list.txt

$ debpartial ....  --include-from=~/ubuntu_popularity_list.txt .......

Submitted by snifer (registered user) on Sun, 2007-04-01 20:26.
Submitted by cwhamsun (registered user) on Fri, 2007-03-30 06:28.
I noticed that Kubuntu Beta just released has a link to DVD iso files, could not find similar link for Ubuntu, and did not look at Xubuntu yet. Not sure what is on the DVD, just extra packages, all install versions (alternative,server,desktop) or whatever else. Just wanted to point that out, I hope they make this a regular thing in the future. Also, thanks for the walkthrough, I don't need to use this info at the moment but may in the future.
Submitted by ginocic (registered user) on Fri, 2007-03-30 12:59.

Maybe, as in the past, those isos are a simple rebuild of desktop and server together (...sort of 'all-in-one') image version with more packages (above all, for full localization purpose) in it.

BTW, thank for the note. I'll check'em out... 

Submitted by iang (registered user) on Fri, 2007-03-30 00:59.

If you need the unofficial builds of DVD repositories, you can visit ftp://tuma.vlsm.org/Ubuntu_contrib/Edgy or ftp://tuma.vlsm.org/Ubuntu_contrib/Dapper. The Jigdo files are provided too.

Submitted by subitulibro (registered user) on Fri, 2007-07-20 16:43.
Hello !!

I want to ask you Iang  if you have another link, because I can´t open that links.

Thank´s a lot...
_____________________________________________
Submitted by iang (registered user) on Wed, 2007-09-26 10:16.

Sorry, I changed the directory structure. The new URL is ftp://kambing.ui.edu/pub/ubuntu-repository In that link, you can find the DVD ISOs for many versions of Ubuntu and not-so-regularly-updated extra repositories (update, security, and backports) as well. PS: You need to use FTP to download big file PS2: tuma.ui.edu now is just an alias for kambing.ui.edu. I changed the domain name also :)

Submitted by ginocic (registered user) on Fri, 2007-03-30 13:03.

Thanks for the links.

I will add a new point on my howto with your links.

It could usefull for the ones who can not have space on hd or time to build the isos.

Thanks again.

:)