Monday 27 July 2009

IPS repository mirror script

ips-mirror.py is a succesor to my IPS repository mirror script I have put together last year (indiana-discuss thread). I wrote the script in Python this time. It can mirror whole repositories or just parts of them now. It is possible to limit the architecture of the mirrored files now as well. Few interesting use cases:

You can mirror the whole OpenSolaris dev repository if you like. I don't recommend it as there are over 30GB of files to download :). It is doable though:

mkdir $HOME/dev; cd $HOME/dev
ips-mirror.py http://pkg.opensolaris.org/dev/


After the script has done its job, which can take few days in this case, you want to run pkg.depotd with the --rebuild parameter.

/usr/lib/pkg.depotd -d `pwd` -p 10000 --rebuild

pkg.depotd will generate a catalog from the manifests in the pkg directory and start serving the packages. It will also generate search indexes which can take a while.

Mirroring the OpenSolaris 2009.06 release packages for i386:

ips-mirror.py -r 5.11-0.111 -s -a i386 http://pkg.opensolaris.org/release/

Mirroring the latest OpenSolaris packages for i386 from the dev repository:

ips-mirror.py -l -a i386 http://pkg.opensolaris.org/dev/

It is also possible to mirror repositories over https with connection specific certificate and the key:

ips-mirror.py -c /var/pkg/ssl/OpenSolaris_extras.certificate.pem -k /var/pkg/ssl/OpenSolaris_extras.key.pem https://pkg.sun.com/opensolaris/extra/

4 comments:

  1. I used your script to download the lastest dev release, which appears to have work well. I have set up local site as a mirror and updates are downloaded locally after the "creating paln" has finished. I was hoping to use it to perform upgrades without requiring a connection back to pkg.opensolaris.org , have you been able to achieve this type of set up? I keep getting these errors.

    Creating Plan /
    pkg: 'SUNWipkg' matches multiple packages
    pkg://opensolarisdev/SUNWipkg
    pkg://opensolaris.org/SUNWipkg


    thanks

    alan

    ReplyDelete
  2. I quite often use local repo to perform the upgrades. I typically overwrite the 'opensolaris.org' publisher URL when I want to use the local mirror:

    pkg set-publisher -O http://localhost:10000/ opensolaris.org

    you probably want to run the command below on your machine as well:

    pkg unset-publisher opensolarisdev

    Your two defined repos have exactly same packages available and pkg can't decide which one to use. Undefining opensolarisdev repo should help here.

    Andrzej

    ReplyDelete
  3. i used ips-mirror.py and due to unstable connection, i had to restart it several times.

    I end up with corrupted files (incomplete).

    So i created a ip_check.py to delete incorrect files (incorrect size and/or checksum)

    http://github.com/seisen/opensolaris-scripts/blob/master/ips/ips_check.py

    ReplyDelete
  4. Thank you very much.
    The both scripts helped me (ips-mirror and ips_check)!

    ReplyDelete