OpenBSD Journal

ftp mirror changes

Contributed by mk/reverse on from the distribution bottleneck dept.

Bob Beck writes to misc@ and tech@:

I just committed a change to ftp.html to identify second level mirror sites. This is deliberate. We are going to be moving with this release towards a two level mirror distribution from the fanout site in canada.

This sounds like a very good idea since ftp.openbsd.org is quite unusable every six months and thus delays the distribution to the other mirrors. It's time for you to poke your favourite mirror operators and have them update their source.

(Comments are closed)


Comments
  1. By RC (4.16.255.217) on

    I think it's often easiest for someone to just pick a number, and use whatever mirror it happens to be... eg ftp6.usa.openbsd.org

    It would be a good idea to put all the lowest-level mirrors in their own group, and give the higher-up servers (that are not supposed to be users directly) a different DNS naming scheme, so they won't show-up when I pick a number and try it out.

    And even after going through the list, and finding the best mirror for yourself, it's easier to remeber "6", rather than "anonopenbsd.cs.colorado.edu".

    Comments
    1. By Anonymous Coward (69.156.206.199) on

      What about round robin DNS? or round robin with ftp.ca.openbsd.org, ftp.us.openbsd.org, ftp.mx.openbsd.org, etc?

      Comments
      1. By tedu (66.93.171.98) on

        that hoses any geographical locality. if you know that one mirror is close and fast, you won't want to get a different one 3 continents away.

        Comments
        1. By Anonymous Coward (69.156.206.199) on

          Doh! Good point, didn't think of the obvious. ;)

        2. By Semi-Anonymous Coward (216.238.113.174) on

          Call me odd, but I tend to us the mirrors far away becuase the time zone differences help to reduce the load on the mirror site.

  2. By Matt Van Mater (65.205.28.104) on

    Has anyone thought of allowing or setting up architecture specific mirrors? More people might mirror openbsd if they only had to sync with their preferred arch (ie: i386, sparc). It requires a lot less disk space if you're only mirroring the arch you want. I'm sure a lot of people already do this privately, but it might be a way to share the load. If a large percentage of obsd users are running i386, why not make sure we have plenty of i386 mirrors in place? My old university mirrors i386 obenbsd, but not the other arches so it wouldn't qualify as a mirror, so most people don't know about it. Those fat pipes could go a long way towards sharing the load.

    Also, I disagree with some people's suggestions to obfuscate the mirror names. Personally I really appreciate the fact that they list the organization and location where the mirror is at. Why would I want to sync with a mirror in colorado or california if I can get one 10 miles away in here in virginia? If you want to have alaises for neatness sake, thats cool, but I don't think the aliases alone will solve the bandwidth problems, and might even make them worse.

    Comments
    1. By James Carter (66.218.244.40) on http://www.opentorrent.org

      One/three of the secondary's could change over from ftp to bittorrent sharing a common tracker and really speed things up. Leech the source tree and help someone else at the same time. Anything to help the propagation of a quality OS.

  3. By Chas (147.154.235.53) on

    The ctorrent binary is only 55k under OpenBSD 3.5. Here is the source.

    ctorrent is not perfect. It is not firewall friendly, but it has the potential of vastly reducing the bandwidth consumed at release time.

    Comments
    1. By Anthony (68.145.111.152) on

      What's wrong with the original Python client? The probability of a security problem is higher with a C program, and computers that are slow enough to care about the speed are rare.

      Also... firewall friendliness can be improved with a little care.

      First, create a user and group called torrent, and don't give him permissions to anything else. All your downloads go to torrent's home directory.

      Second, tweak PF to allow torrent to listen for connections (in the same way that the FTP proxy is set up).

      Third, tweak PF to give the torrent user the lowest priority you can. You want torrents to run only when you're not using the bandwidth, otherwise your connection will become unusable.

      Fourth, install "screen" from ports. Run torrents (btdownloadheadless or btdownloadcurses) from within screen so you can log out.

      Comments
      1. By Chas (147.154.235.53) on

        ...and integrate it into pkg_add.

        While static linking against the C++ library may increase its size (from 50k), setting ctorrent as the default remote install method would take most of the load off ftp.openbsd.org.

        The python version would not be appropriate (binaries too large).

        Really, ftp is an ugly old protocol. Yes, ctorrent may not be the answer, but why not try something rather than just complain about bandwidth problems at release time?

        Yes, a code review would obviously be required for ctorrent integration into the base distribution.

        Comments
        1. By Anonymous Coward (69.156.206.199) on

          Really, ftp is an ugly old protocol. Yes, ctorrent may not be the answer, but why not try something rather than just complain about bandwidth problems at release time?

          An even better idea... Buy a CD!

          I'm finally doing it again, and I've even managed to expense another copy to my Company (cheap a** company) but still, buying 1 CD through them and one from me is better than not helping to support OpenBSD by just ftp installaing and not giving back. Don't forget, OpenBSD isn't commercially funded (at least not like RH, SuSe or such...) so every little bit helps, IMHO.

          Sincerily,

          A proud supporter of OpenBSD...

        2. By Anonymous Coward (131.130.1.143) on

          Admittedly I do not think this would really work. Torrent (as most p2p networks) depends on its users sharing. But when it's (only) used for and active while the installation there won't be many users sharing. (Or do you think many OpenBSD firewalls will have a running torrent client?;)

          Comments
          1. By Chas (147.154.235.53) on

            Let's just say that the i386 boot floppy and the .tgz files were all on torrents.

            For the duration of the installer download, yes, clients would share data with one another, which might be of negligable benefit.

            However, the clients would automatically seek out the best download sites, which would spread the load.

            Even without the client upload, distributed bandwidth consumption solves a lot of problems. However, at this stage, it may also create some significant problems.

      2. By Anonymous Coward (69.197.92.181) on

        Um, python performs terribly (maybe just on OpenBSD?). 4 torrents running uses 50-80% of my CPU, and its a 2 GHz p4.

        Comments
        1. By mirabile (212.185.103.56) tg@66h.42h.de on http://mirbsd.de/

          Are you using btdownloadcurses.py or btlaunchmanycurses.py ?

          Even with the former, I already managed to get 6-8 torrents
          running simultaneously on a Pentium 120 with 128 MiB RAM,
          running OpenBSD 3.5-current based MirBSD.

          With the latter, it's even less ressource intensive because
          the python core is loaded only once.

          Hint: if you're on PPPoE ADSL, try pppoe(8) from -current,
          it fixes a severe problem.

          As for FTP: it deserves to die anyway, but since httpd(8) comes
          with the base system, why not install via http? There are even
          some public OpenBSD http mirrors available.

          You could of course tape them up, or use NFS, or whatever, but
          CDs or HTTP seem quite appropriate for me.

          //mirabile

          Comments
          1. By Anonymous Coward (67.71.76.239) on

            Which PPPoE severe fix is it? Is that fix in 3.6-RELEASE also? Just curious. thx

            Comments
      3. By RC (4.16.254.193) on

        Bad info all around!

        > What's wrong with the original Python client?

        Maybe because you have to have python installed? Maybe because it's very limited? Maybe because it's very slow? Maybe because people don't want to learn python?

        > The probability of a security problem is higher with a C program,

        Not true.

        > and computers that are slow enough to care about the speed are rare.

        Also not true.

        > Second, tweak PF to allow torrent to listen for connections
        > (in the same way that the FTP proxy is set up).

        What? bittorrent isn't a proxy. You don't set up rules for it in any way remotely similar.

        > Run torrents (btdownloadheadless or btdownloadcurses) from
        > within screen so you can log out.

        Lowsy advice. There are a number of ways you could keep either program running after you've logged-out. 'nohup' would work well, and it comes with every Unix system I've come across yet.

    2. By Michael Knudsen (217.157.199.114) on

      The best reason I see for using Bittorrent is that file integrity will be checked via checksums. This is not possible when doing installs directly via ftp.

      Still, a secure way of distributing .torrents would be needed.

      Comments
      1. By Anonymous Coward (68.123.254.186) on

        Checksums don't exist anywhere but the official CD.

        Comments
        1. By jose nazario (204.181.64.2) on http://monkey.org/~jose/

          BT uses an internal checksum method to ensure that the pieces you downloaded are the ones you expect. these don't match the official MD5 checksums, but if someone you trust lines up the torrent and the officual MD5 checksums, then you're as set as you can be.

          Comments
          1. By Michael Knudsen (217.157.199.114) on

            Exactly. So, if the project distributed .torrents via https, this would actually be a good way of fetching the installation tarballs.

            The best way is of course to buy the CDs.

Latest Articles

Credits

Copyright © - Daniel Hartmeier. All rights reserved. Articles and comments are copyright their respective authors, submission implies license to publish on this web site. Contents of the archive prior to as well as images and HTML templates were copied from the fabulous original deadly.org with Jose's and Jim's kind permission. This journal runs as CGI with httpd(8) on OpenBSD, the source code is BSD licensed. undeadly \Un*dead"ly\, a. Not subject to death; immortal. [Obs.]