Translations of this page

Advice for Running Mirrors

Table of Contents

• Mirroring the GNU Web Server
• Mirroring the GNU FTP Server

Mirroring the GNU Web Server

Mirroring the GNU Web server requires roughly 2.5GB of disk space (as of 25 June 2003).

All the cross-references between our web pages use absolute file names (starting with /). We chose this convention because some browsers get confused by relative file names in links; in addition, counting the number of times to use ../ in a relative file name is not reliable.

The absolute file names assume that the root web directory is the top-level GNU directory. If your mirror server is not dedicated to the GNU project alone, that will not be true. Here are ways you can compensate for that:

• Set up a separate virtual server whose root web directory is the top-level GNU directory. That virtual server will be the GNU mirror.
• It should not be very hard to write a shell script to modify the contents of the pages to go in a subdirectory. The script should insert an additional directory name and a slash after each occurrence of HREF="/ inside an <A tag and of SRC="/ inside an <IMG tag. It should not modify other occurrences of HREF or SRC.

  If you'd like to write this script and verify that it works properly for all our pages, that would be useful.

Once your mirror site is operating, you will need to update the contents regularly. Due to resource and security concerns, we no longer offer direct rsync or FTP service to update mirrors.

If you would like to use rsync (which is advisable) or FTP, there is a reliable third party site, ibiblio.org, that is offering access to the www.gnu.org contents via both of these methods. They update directly from us on a regular basis and you may access them with:

• rsync://ftp.ibiblio.org/pub/gnu/www
• ftp://ftp.ibiblio.org/pub/mirrors/gnu/www

[Thanks to John Reuning and ibiblio for providing this service for us. Please see the section below for some tips on updating your mirror from ibiblio]

We currently do not have an adequate service to assist mirrors in updating directly from us. If you would like to try, the following methods may suit your needs:

• Using the GPLed program GNU Wget, a non-interactive web-mirroring utility.

  You can download Wget, via anonymous FTP, from http://ftp.gnu.org/pub/gnu/wget/, or from a local mirror site. This option (or mirror.pl) is not preferable, as the pages are transferred as interpretted by the webserver, not as given to the webserver. This method can also be inefficient and taxing on mirror resources.

• Using anonymous CVS. The following command will checkout the entire website, with a couple of exceptions (namely /prep, parts of /software, /directory, /brave-gnu-world and order.gnu.org, and a couple other pages that are autogenerated elsewhere. In these cases, you will have to use wget, or another method):
  
  export CVS_RSH="ssh"
  cvs -z3 -d:ext:anoncvs@savannah.gnu.org:/webcvs/www co www

  To keep the site up to date, use cron with "cvs update -P" and wget for those parts that are not in CVS.

• Using Apache's mod_proxy. The following example for daily mirroring may help:
  
  ProxyPass / http://www.gnu.org/
  CacheDefaultExpire 24

[Note]: If anyone is succesfully using other methods for mirroring www.gnu.org, and can give us input as to what methods you are using, please contact webmasters@gnu.org.

Mirroring the GNU FTP Server

Mirroring the GNU FTP server requires roughly 20GB disk space (as of 29 June 2003).

Due to resource and security concerns, we no longer offer direct rsync or FTP service to update mirrors. If you would like to use rsync (which is advisable) or FTP, there is a reliable third party site, ibiblio.org, that is offering access to the ftp.gnu.org contents via both of these methods. They update directly from us on a regular basis and you may access them with:

• rsync://ftp.ibiblio.org/pub/gnu/ftp
• ftp://ftp.ibiblio.org/pub/mirrors/gnu/ftp

[Thanks to John Reuning and ibiblio for providing this service for us. Please see the section below for some tips on updating your mirror from ibiblio]

ftp.funet.fi also offers rsync (and ftp) access to the GNU FTP server. rsync://ftp.funet.fi/ftp/pub/mirrors/ftp.gnu.org/pub/gnu/.

We currently do not have an adequate service to assist mirrors in updating directly from us. If you would like to try, the following methods may suit your needs:

• Using GNU Wget, or another free software ftp fetching utility.
  [Note] If Wget has issues with files.tar, or if you simply don't want to download an out-of-date 7.7GB file, you can exclude it with 'wget --reject files.tar'

Tips on Updating Your Mirror from Ibiblio

• They recommend using -rltp for mirroring content from ftp.ibiblio.org via rsync. Using -x tends to cause client-side problems because they disable compression on their end.
• For mirror admins, ibiblio-mirrors is a low-traffic email list for sites that mirror content from ibiblio servers. Here's a link to the mailman list page: http://lists.ibiblio.org/mailman/listinfo/ibiblio-mirrors
• Specific questions and problem reports can be sent to: help-at-ibiblio-dot-org.