diff --git a/en_US.ISO8859-1/articles/dialup-firewall/article.sgml b/en_US.ISO8859-1/articles/dialup-firewall/article.sgml index a31da47e90..c9a1d72c18 100644 --- a/en_US.ISO8859-1/articles/dialup-firewall/article.sgml +++ b/en_US.ISO8859-1/articles/dialup-firewall/article.sgml @@ -1,316 +1,316 @@ %articles.ent; ]>
Dialup firewalling with FreeBSD Marc Silver
marcs@draenor.org
 $FreeBSD$ &tm-attrib.freebsd; &tm-attrib.general; This article documents how to set up a firewall using a PPP dialup with FreeBSD and IPFW, and specifically with firewalling over a dialup with a dynamically assigned IP address. This document does not include information on setting up an initial PPP connection. For more information on setting up a PPP connection, consult the &man.ppp.8; manual page. Preface This document outlines the steps required to set up firewalling with FreeBSD when an IP address is assigned dynamically by your ISP. While every effort has been made to make this document as informative and correct as possible, you are welcome to mail any corrections, comments or suggestions to the author at marcs@draenor.org. Kernel Options In order to use IPFW, support for it must be compiled into the kernel. For more information on how to recompile the kernel, please see the kernel configuration section in the Handbook. The following options must be added into your kernel configuration file for IPFW support: options IPFIREWALL Enables the kernel firewall code. This document assumes that you are running &os; 5.X. Users running &os; 4.X will need to recompile their kernels with IPFW2 support. &os; 4.X users should consult the &man.ipfw.8; manual page for more information on using IPFW2 on their systems, and should pay particular attention to the USING IPFW2 IN FreeBSD-STABLE section. options IPFIREWALL_VERBOSE Sends logged packets to the system logger. options IPFIREWALL_VERBOSE_LIMIT=500 Limits the number of times a matching entry may be logged. This allows you to log firewall activity without the risk of syslog flooding in the event of a denial of service attack. 500 is a reasonable number to use, but may be adjusted based on your requirements. Once the kernel recompile has been completed, do not reboot your system. Doing so may result in you being locked out of your own system. You must only reboot once the ruleset is in place and all the relevant configuration files have been updated. Changing <filename>/etc/rc.conf</filename> to load the firewall /etc/rc.conf needs to be slightly modified in order to tell the system about the firewall and to specify the location for our rules file. Add the following lines to /etc/rc.conf: firewall_enable="YES" firewall_script="/etc/firewall/fwrules" For more information on the functions of these statements take a look at /etc/defaults/rc.conf and read &man.rc.conf.5; Enable PPP's network address translation In order to allow clients on your network to connect via your gateway, you will need to enable PPP's network address translation (NAT). In order to use PPP's NAT functions, add the following lines to /etc/rc.conf: ppp_enable="YES" ppp_mode="auto" ppp_nat="YES" ppp_profile="your_profile" Take care to change your_profile to the name of your own dialup profile. The rule set for the firewall This is the point where we define the firewall rules for your - system. The ruleset that we're about to describe is a generic + system. The ruleset that we are about to describe is a generic template for most dialup users. While it will not suit the exact needs of every user, it provides you with a basic idea of how IPFW works and should be fairly easy to customize. First, let's start with the basics of closed firewalling. Closed firewalling is based on the idea that everything is denied by default. The system administrator may then explicitly add rules for traffic that he or she would like to allow. Rules should be in the order of allow first, and then deny. The premise is that you add the rules for everything you would like to allow, and then everything else is automatically denied. Following that, let's create the directory where we will store our - firewall rules. In this example, we'll use /etc/firewall. Change into the directory and edit the file fwrules as we specified in rc.conf. Please note that you can change this filename to anything you wish. This guide merely gives an example of a filename you may want to use. Now, let's look at a nicely commented sample firewall file. # Define the firewall command (as in /etc/rc.firewall) for easy # reference. Helps to make it easier to read. fwcmd="/sbin/ipfw" # Define our outside interface. With userland-ppp this # defaults to tun0. oif="tun0" # Define our inside interface. This is usually your network # card. Be sure to change this to match your own network # interface. iif="fxp0" # Force a flushing of the current rules before we reload. $fwcmd -f flush # Check the state of all packets. $fwcmd add check-state # Stop spoofing on the outside interface. $fwcmd add deny ip from any to any in via $oif not verrevpath # Allow all connections that we initiate, and keep their state. # but deny established connections that don't have a dynamic rule. $fwcmd add allow ip from me to any out via $oif keep-state $fwcmd add deny tcp from any to any established in via $oif # Allow all connections within our network. $fwcmd add allow ip from any to any via $iif # Allow all local traffic. $fwcmd add allow all from any to any via lo0 $fwcmd add deny all from any to 127.0.0.0/8 $fwcmd add deny ip from 127.0.0.0/8 to any # Allow internet users to connect to the port 22 and 80. # This example specifically allows connections to the sshd and a # webserver. $fwcmd add allow tcp from any to me dst-port 22,80 in via $oif setup keep-state # Allow ICMP packets: remove type 8 if you don't want your host # to be pingable. $fwcmd add allow icmp from any to any via $oif icmptypes 0,3,8,11,12 # Deny and log all the rest. $fwcmd add deny log ip from any to any You now have a fully functional firewall that only allows connections to ports 22 and 80 and will log any other connection attempts. You may now safely reboot and the firewall should be automatically started and the ruleset loaded. If you find this incorrect in any way or experience any problems, or have any suggestions to improve this page, please email me. Questions I get messages like limit 500 reached on entry 2800 and after that my machine stops logging denied packets that match that rule number. Is my firewall still working? This merely means that the maximum logging count for the rule has been reached. The rule itself is still working, but it will no longer log until such time as you reset the logging counters. An example of how to clear your counters can be found below: &prompt.root; ipfw resetlog Alternatively, you may increase the log limit in your kernel configuration with the option as described above. You may also change this limit (without recompiling your kernel and having to reboot) by using the net.inet.ip.fw.verbose_limit &man.sysctl.8; value. There must be something wrong. I followed your instructions to the letter and now I am locked out. This tutorial assumes that you are running userland-ppp, therefore the supplied rule set operates on the tun0 interface, which corresponds to the first connection made with &man.ppp.8; (a.k.a. user-ppp). Additional connections would use tun1, tun2 and so on. You should also note that &man.pppd.8; uses the ppp0 interface instead, so if you start the connection with &man.pppd.8; you must substitute tun0 for ppp0. A quick way to edit the firewall rules to reflect this change is shown below. The original rule set is backed up as fwrules_tun0. &prompt.user; cd /etc/firewall /etc/firewall&prompt.user; su Password: /etc/firewall&prompt.root; mv fwrules fwrules_tun0 /etc/firewall&prompt.root; cat fwrules_tun0 | sed s/tun0/ppp0/g > fwrules To know whether you are currently using &man.ppp.8; or &man.pppd.8; you can examine the output of &man.ifconfig.8; once the connection is up. E.g., for a connection made with &man.pppd.8; you would see something like this (showing only the relevant lines): &prompt.user; ifconfig (skipped...) ppp0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1524 inet xxx.xxx.xxx.xxx --> xxx.xxx.xxx.xxx netmask 0xff000000 (skipped...) On the other hand, for a connection made with &man.ppp.8; (user-ppp) you should see something similar to this: &prompt.user; ifconfig (skipped...) ppp0: flags=8010<POINTOPOINT,MULTICAST> mtu 1500 (skipped...) tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1524 (IPv6 stuff skipped...) inet xxx.xxx.xxx.xxx --> xxx.xxx.xxx.xxx netmask 0xffffff00 Opened by PID xxxxx (skipped...)
diff --git a/en_US.ISO8859-1/articles/hubs/article.sgml b/en_US.ISO8859-1/articles/hubs/article.sgml index 7156aec574..b7b165f4db 100644 --- a/en_US.ISO8859-1/articles/hubs/article.sgml +++ b/en_US.ISO8859-1/articles/hubs/article.sgml @@ -1,1118 +1,1118 @@ %articles.ent; ]>
Mirroring FreeBSD $FreeBSD$ Jun Kuriyama
kuriyama@FreeBSD.org
 Valentino Vaschetto
logo@FreeBSD.org
 Daniel Lang
dl@leo.org
 Ken Smith
kensmith@FreeBSD.org
 &tm-attrib.freebsd; &tm-attrib.cvsup; &tm-attrib.general; An in-progress article on how to mirror FreeBSD, aimed at hub administrators. Contact Information The Mirror System Coordinators can be reached through email at mirror-admin@FreeBSD.org. There is also a &a.hubs;. Requirements for FreeBSD mirrors Disk Space Disk space is one of the most important requirements. Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind that official mirrors are probably required to be complete. The CVS repository and the web pages should always be mirrored completely. Also note that the numbers stated here are reflecting the current state (at &rel2.current;-RELEASE/&rel.current;-RELEASE). Further development and releases will only increase the required amount. Also make sure to keep some (ca. 10-20%) extra space around just to be sure. Here are some approximate figures: Full FTP Distribution: 126 GB CVS repository: 2.7 GB CTM deltas: 1.8 GB Web pages: 300 MB Network Connection/Bandwidth Of course, you need to be connected to the Internet. The required bandwidth depends on your intended use of the mirror. If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller than if you want to make the files publicly available. If you intend to become an official mirror, the bandwidth required will be even higher. We can only give rough estimates here: Local site, no public access: basically no minimum, but < 2 Mbps could make syncing too slow. Unofficial public site: 34 Mbps is probably a good start. Official site: > 100 Mbps is recommended, and your host should be connected as close as possible to your border router. System Requirements, CPU, RAM One thing this depends on the expected number of clients, which is determined by the server's policy. It is also affected by the types of services you want to offer. Plain FTP or HTTP services may not require a huge amount of resources. Watch out if you provide CVSup, rsync or even AnonCVS. This can have a huge impact on CPU and memory requirements. Especially rsync is considered a memory hog, and CVSup does indeed consume some CPU. For AnonCVS it might be a nice idea to set up a memory resident file system (MFS) of at least 300 MB, so you need to take this into account for your memory requirements. The following are just examples to give you a very rough hint. For a moderately visited site that offers rsync, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an official site. For a frequently used site you definitely need more RAM (consider 2GB as a good start) and possibly more CPU, which could also mean that you need to go for a SMP system. You also want to consider a fast disk subsystem. Operations on the CVS repository require a fast disk subsystem (RAID is highly advised). A SCSI controller that has a cache of its own can also speed up things since most of these services incur a large number of small modifications to the disk. Services to offer Every mirror site is required to have a set of core services available. In addition to these required services, there are a number of optional services that server administrators may choose to offer. This section explains which services you can provide and how to go about implementing them. FTP (required for FTP fileset) This is one of the most basic services, and it is required for each mirror offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and must never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path /pub/FreeBSD. There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order). /usr/libexec/ftpd: FreeBSD's own ftpd can be used. Be sure to read &man.ftpd.8;. ftp/ncftpd: A commercial package, free for educational use. ftp/oftpd: An ftpd designed with security as a main focus. ftp/proftpd: A modular and very flexible ftpd. ftp/pure-ftpd: Another ftpd developed with security in mind. ftp/twoftpd: As above. ftp/vsftpd: The very secure ftpd. ftp/wu-ftpd: The ftpd from Washington University. It has become infamous, because of the huge amount of security issues that have been found in it. If you do choose to use this software be sure to keep it up to date. FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the most commonly ones. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed. RSYNC (optional for FTP fileset) rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The protocol is different from FTP in many ways. It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change. rsync does require a significant amount of memory for each instance. The size depends on the size of the synced module in terms of the number of directories and files. rsync can use rsh and ssh (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available: net/rsync HTTP (required for web pages, optional for FTP fileset) If you want to offer the FreeBSD web pages, you need to install a web server a.k.a. httpd. You may optionally offer the FTP fileset via HTTP. The choice of web server software is left up to the mirror administrator. Some of the most popular choices are: www/apache13: Apache is the most widely deployed web server on the Internet. It is used extensively by the FreeBSD Project. You may also wish to use the next generation of the Apache web server, available in the ports collection as www/apache22. www/thttpd: If you are going to be serving a large amount of static content you may find that using an application such as thttpd is more efficient than Apache. It is optimized for excellent performance on FreeBSD. www/boa: Boa is another alternative to thttpd and Apache. It should provide considerably better performance than Apache for purely static content. It does not, at the time of writing, contain the same set of optimizations for FreeBSD that are found in thttpd. CVSup (desired for CVS repository) CVSup is a very efficient way of distributing files. It works similar to rsync, but was specially designed for use with CVS repositories. If you want to offer the FreeBSD CVS repository, you really want to consider offering it via CVSup. It is possible to offer the CVS repository via AnonCVS, FTP, Rsync or HTTP, but people would benefit much more from CVSup access. CVSup was developed by &a.jdp;. It is a bit tricky to install on non-FreeBSD platforms, since it is written in Modula-3 and therefore requires a Modula-3 environment. John Polstra has built a stripped down version of M3 that is sufficient to run CVSup, and can be installed much easier. See Ezm3 for details. Related ports are: net/cvsup: The native CVSup port (client and server) which requires lang/ezm3 now. net/cvsup-mirror: The CVSup mirror kit, which requires net/cvsup, and configures it mirror-ready. Some site administrators may want a different setup though. There are a few more like net/cvsup-without-gui you might want to have a look at. If you prefer a static binary package, take a look here. This page still refers to the S1G bug that was present in CVSup. Maybe John will set up a generic download-site to get static binaries for various platforms. It is possible to use CVSup to offer any kind of fileset, not just CVS repositories, but configuration can be complex. CVSup is known to eat some CPU on both the server and the client, since it needs to compare lots of files. AnonCVS (optional for CVS repository) If you have the CVS repository, you may want to offer anonymous CVS access. A short warning first: There is not much demand for it, it requires some experience, and you need to know what you are doing. Generally there are two ways to access a CVS repository remotely: via pserver or via ssh (we do not consider rsh). For anonymous access, pserver is very well suited, but some still offer ssh access as well. There is a custom crafted wrapper in the CVS repository, to be used as a login-shell for the anonymous ssh account. It does a chroot, and therefore requires the CVS repository to be available under the anonymous user's home-directory. This may not be possible for all sites. If you just offer pserver this restriction does not apply, but you may run with more security risks. You do not need to install any special software, since &man.cvs.1; comes with FreeBSD. You need to enable access via inetd, so add an entry into your /etc/inetd.conf like this: cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver See the manpage for details of the options. Also see the CVS info page about additional ways to make sure access is read-only. It is advised that you create an unprivileged account, preferably called anoncvs. Also you need to create a file passwd in your /home/ncvs/CVSROOT and assign a CVS password (empty or anoncvs) to that user. The directory /anoncvstmp is a special purpose memory based file system. It is not required but advised since &man.cvs.1; creates a shadow directory structure in your /tmp which is not used after the operation but slows things dramatically if real disk operations are required. Here is an excerpt from /etc/fstab, how to set up such a MFS: /dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid,nodev 0 0 This is (of course) tuned a lot, and was suggested by &a.jdp;. How to Mirror FreeBSD Ok, now you know the requirements and how to offer the services, but not how to get it. :-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from. FTP The FTP area is the largest amount of data that needs to be mirrored. It includes the distribution sets required for network installation, the branches which are actually snapshots of checked-out source trees, the ISO Images to write CD-ROMs with the installation distribution, a live file system, lots of packages, the ports tree, distfiles, and a huge amount of packages. All of course for various FreeBSD versions, and various architectures. With FTP mirror You can use a FTP mirror program to get the files. There are a lot around and widely used, like: ftp/mirror ftp/ftpmirror ftp/emirror ftp/spegla ftp/omi some even use ftp/wget ftp/mirror was very popular, but seemed to have some drawbacks, as it is written in &man.perl.1;, and had real problems with mirroring large directories like a FreeBSD site. There are rumors that the current version has fixed this by allowing a different algorithm for comparing the directory structure to be specified. In general FTP is not really good for mirroring. It transfers the whole file if it has changed, and does not create a single data stream which would benefit from a large TCP congestion window. With RSYNC A better way to mirror the FTP area is rsync. You can install the port net/rsync and then use rsync to sync with your upstream host. rsync is already mentioned in . Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access. Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror). A command line to mirror FreeBSD could look like that: &prompt.user; rsync -vaz --delete ftp4.de.FreeBSD.org::FreeBSD/ /pub/FreeBSD/ Consult the documentation for rsync, which is also available at http://rsync.samba.org/ about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via &man.cron.8;. With CVSup A few sites, including the one-and-only ftp-master.FreeBSD.org even offer CVSup to mirror the contents of the FTP space. You need to install a cvsup client, preferably from the port net/cvsup. (Also reread .) A sample supfile suitable for ftp-master.FreeBSD.org looks like this: # # FreeBSD archive supfile from master server # *default host=ftp-master.FreeBSD.org *default base=/usr *default prefix=/pub #*default release=all *default delete use-rel-suffix *default umask=002 # If your network link is a T1 or faster, comment out the following line. #*default compress FreeBSD-archive release=all preserve It seems CVSup would be the best way to mirror the archive in terms of efficiency, but it is only available from few sites. Please have look at the CVSup documentation like &man.cvsup.1; and consider using the option, as it can reduce the amount of work to be done a lot. Mirroring the CVS repository Again you have various possibilities, but the most recommended one is to use CVSup. Using CVSup CVSup was already described to some detail in and . Here we just describe an example to set up the supfile: # # FreeBSD CVS supfile from master server # *default host=cvsup-master.FreeBSD.org *default base=/usr *default prefix=/pub/FreeBSD/development/FreeBSD-CVS *default release=cvs *default delete use-rel-suffix *default umask=002 # If your network link is a T1 or faster, comment out the following line. #*default compress cvs-all You should also have a look at /usr/share/examples/cvsup Please do not forget to consider the hint mentioned in this note above. Using other methods Using other methods than CVSup is generally not recommended. We describe them in short here anyway. Since most sites offer the CVS repository as part of the FTP fileset under the path /pub/FreeBSD/development/FreeBSD-CVS, the following methods could be used. FTP RSYNC maybe even HTTP If you find a site that supports it, you could use net/sup. But it is inferior to CVSup and its deficiencies caused John Polstra to develop CVSup in the first place, so it is clearly not recommended. You can NOT use AnonCVS to mirror the CVS repository since CVS does not allow you to access the repository itself, but only checked out versions of the modules. Mirroring the WWW pages The best way is to check out the www distribution from CVS. If you have a local mirror of the CVS repository, it is probably as easy as: &prompt.user; cvs -d /home/ncvs co www and a cronjob, that calls cvs up -d -P on a regular basis, maybe just after your repository was updated. Of course, the files need to remain in a directory available for public WWW access. The installation and configuration of a web server is not discussed here. For the website to be visible, users must execute the &man.make.1; command in the main www directory. This command will create the standard *.html files for web viewing. For this to work however, the textproc/docproj port must be installed. If you do not have a local repository, you can use CVSup to maintain an up to date copy of the www pages. A sample supfile can be found in /usr/share/examples/cvsup/www-supfile and could look like this: # # WWW module supfile for FreeBSD # *default host=cvsup3.de.FreeBSD.org *default base=/usr *default prefix=/usr/local *default release=cvs tag=. *default delete use-rel-suffix # If your network link is a T1 or faster, comment out the following line. *default compress # This collection retrieves the www/ tree of the FreeBSD repository www Using ftp/wget or other web-mirror tools is probably not recommended. Mirroring the FreeBSD documentation Since the documentation is referenced a lot from the web pages, it is recommended that you mirror the FreeBSD documentation as well. However, this is not as trivial as the www-pages alone. First of all, you should get the doc sources, again preferably via CVSup. Here is a corresponding sample supfile: # # FreeBSD documentation supfile # *default host=cvsup3.de.FreeBSD.org *default base=/usr *default prefix=/usr/share *default release=cvs tag=. *default delete use-rel-suffix # If your network link is a T1 or faster, comment out the following line. #*default compress # This will retrieve the entire doc branch of the FreeBSD repository. # This includes the handbook, FAQ, and translations thereof. doc-all Then you need to install a couple of ports. You are lucky, there is a meta-port: textproc/docproj to do the work for you. You need to set up some environment variables, like SGML_CATALOG_FILES. Also have a look at your /etc/make.conf (copy /usr/share/examples/etc/make.conf if you do not have one), and look at the DOC_LANG variable. Now you are probably ready to run make in your doc directory (/usr/share/doc by default) and build the documentation. Again you need to make it accessible for your web server and make sure the links point to the right location. The building of the documentation, as well as lots of side issues, is documented itself in the fdp-primer. Please read this piece of documentation, especially if you have problems building the documentation. XXX MAYBE THIS CAN BE LINKED FROM WITHIN - NOT USING AN ABSOLUTE URL XXX How often should I mirror? Every mirror should be updated on a regular basis. You will certainly need some script framework for it that will be called by &man.cron.8;. Since nearly every admin does this his own way, we cannot give specific instructions. It could work like this: Put the command to run your mirroring application in a script. Use of a plain /bin/sh script is recommended. Add some output redirections so diagnostic messages are logged to a file. Test if your script works. Check the logs. Use &man.crontab.1; to add the script to the appropriate user's &man.crontab.5;. This should be a different user than what your FTP daemon runs as so that if file permissions inside your FTP area are not world-readable those files can not be accessed by anonymous FTP. This is used to stage releases — making sure all of the official mirror sites have all of the necessary release files on release day. Here are some recommended schedules: FTP fileset: daily CVS repository: daily to hourly WWW pages: daily Where to mirror from This is an important issue. So this section will spend some effort to explain the backgrounds. We will say this several times: under no circumstances should you mirror from ftp.FreeBSD.org. A few words about the organization Mirrors are organized by country. All official mirrors have a DNS entry of the form ftpN.CC.FreeBSD.org. CC (i.e. country code) is the top level domain (TLD) of the country where this mirror is located. N is a number, telling that the host would be the Nth mirror in that country. (Same applies to cvsupN.CC.FreeBSD.org, wwwN.CC.FreeBSD.org, etc.) There are mirrors with no CC part. These are the mirror sites that are very well connected and allow a large number of concurrent users. ftp.FreeBSD.org is actually two machines, one currently located in Denmark and the other in the United States. It is NOT a master site and should never be used to mirror from. Lots of online documentation leads interactiveusers to ftp.FreeBSD.org so automated mirroring systems should find a different machine to mirror from. Additionally there exists a hierarchy of mirrors, which is described in terms of tiers. The master sites are not referred to but can be described as Tier-0. Mirrors that mirror from these sites can be considered Tier-1, mirrors of Tier-1-mirrors, are Tier-2, etc. Official sites are encouraged to be of a low tier, but the lower the tier the higher the requirements in terms as described in . Also access to low-tier-mirrors may be restricted, and access to master sites is definitely restricted. The tier-hierarchy is not reflected by DNS and generally not documented anywhere except for the master sites. However, official mirrors with low numbers like 1-4, are usually Tier-1 (this is just a rough hint, and there is no rule). Ok, but where should I get the stuff now? Under no circumstances should you mirror from ftp.FreeBSD.org. The short answer is: from the site that is closest to you in Internet terms, or gives you the fastest access. I just want to mirror from somewhere! If you have no special intentions or requirements, the statement in applies. This means: Look at available mirrors in your country. The FreeBSD Mirror Database can help you with this. Check for those which provide fastest access (number of hops, round-trip-times) and offer the services you intend to use (like rsync or CVSup). Contact the administrators of your chosen site stating your request, and asking about their terms and policies. Set up your mirror as described above. I am an official mirror, what is the right site for me? In general the description in still applies. Of course you may want to put some weight on the fact that your upstream should be of a low tier. There are some other considerations about official mirrors that are described in . I want to access the master sites! If you have good reasons and good prerequisites, you may want and get access to one of the master sites. Access to these sites is generally restricted, and there are special policies for access. If you are already an official mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. If it already has three or more, ask the zone administrator (hostmaster@CC.FreeBSD.org) or &a.hubs; first. Whoever helped you become, an official should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site. If not, you can send email to mirror-admin@FreeBSD.org to request help with that. There are three master sites for the FTP fileset and one for the CVS repository (the web pages and docs are obtained from CVS, so there is no need for master). ftp-master.FreeBSD.org This is the master site for the FTP fileset. ftp-master.FreeBSD.org provides rsync and CVSup access, rather in addition to ftp protocol. Refer to and how to access via these protocols. Mirrors should be encouraged to also allow rsync access for the FTP contents, since they are Tier-1-mirrors. cvsup-master.FreeBSD.org This is the master site for the CVS repository. cvsup-master.FreeBSD.org provides CVSup access only. See for details. To get access, you need to contact &a.cvsup-master;. Make sure you read FreeBSD CVSup Access Policy first! Set up the required authentication by following these instructions. Make sure you specify the server as freefall.FreeBSD.org on the cvpasswd command line, as described in this document, even when you are contacting cvsup-master.FreeBSD.org Official Mirrors Official mirrors are mirrors that a) have a FreeBSD.org DNS entry (usually a CNAME). b) are listed as an official mirror in the FreeBSD documentation (like handbook). So far to distinguish official mirrors. Official mirrors are not necessarily Tier-1-mirrors. - However you probably won't find a Tier-1-mirror, + However you probably will not find a Tier-1-mirror, that is not also official. Special Requirements for official (tier-1) mirrors It is not so easy to state requirements for all official mirrors, since the project is sort of tolerant here. It is more easy to say, what official tier-1 mirrors are required to. All other official mirrors can consider this a big should. The following applies mainly to the FTP fileset, since a CVS repository should always be mirrored completely, and the web pages are a case of its own. Tier-1 mirrors are required to: carry the complete fileset allow access to other mirror sites provide FTP and RSYNC access Furthermore, admins should be subscribed to the &a.hubs;. See this link for details, how to subscribe. It is very important for a hub administrator, especially Tier-1 hub admins, to check the release schedule for the next FreeBSD release. This is important because it will tell you when the next release is scheduled to come out, and thus giving you time to prepare for the big spike of traffic which follows it. It is also important that hub administrators try to keep their mirrors as up-to-date as possible (again, even more crucial for Tier-1 mirrors). If Mirror1 does not update for a while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins a downward spiral... Keep your mirrors up to date! How to become official then? An interesting question, especially, since the state of being official comes with some benefits, like a much higher bill from your ISP as more people will be using your site. Also it may be a key requirement to get access to a master site. Before applying, please consider (again) if another official mirror is really needed for your region. Check first with your zone administrator (hostmaster@CC.FreeBSD.org) or, if that fails, ask on the &a.hubs;. Ok, here is how to do it: Get the mirror running in first place (maybe not using a master site, yet). Subscribe to the &a.hubs;. If everything works so far, contact the DNS administrator responsible for your region/country, and ask for a DNS entry for your site. The admin should able to be contacted via hostmaster@CC.FreeBSD.org, where CC is your country code/TLD. Your DNS entry will be as described in . If there is no subdomain set up for your country yet, you should contact mirror-admin@FreeBSD.org, or you can try the &a.hubs; first. Whoever helps you get an official name should send email to mirror-admin@FreeBSD.org so your site will be added to the mirror list in the FreeBSD Handbook. That is it. Some statistics from mirror sites Here are links to the stat pages of your favorite mirrors (a.k.a. the only ones who feel like providing stats). FTP site statistics ftp2.FreeBSD.org - grisha@ispol.com - (Bandwidth) ftp.is.FreeBSD.org - hostmaster@is.FreeBSD.org - (Bandwidth) (FTP processes) (HTTP processes) ftp.cz.FreeBSD.org - cejkar@fit.vutbr.cz - (Bandwidth) (FTP processes) (Rsync processes) ftp2.ru.FreeBSD.org - mirror@macomnet.ru - (Bandwidth) (HTTP and FTP users) CVSup site stats cvsup[23456].jp.FreeBSD.org - kuriyama@FreeBSD.org - (CVSup processes) cvsup.cz.FreeBSD.org - cejkar@fit.vutbr.cz - (CVSup processes)
diff --git a/en_US.ISO8859-1/articles/pam/article.sgml b/en_US.ISO8859-1/articles/pam/article.sgml index f5d4cb58ab..9398f7bc1b 100644 --- a/en_US.ISO8859-1/articles/pam/article.sgml +++ b/en_US.ISO8859-1/articles/pam/article.sgml @@ -1,1362 +1,1362 @@ %articles.ent; ]>
Pluggable Authentication Modules $FreeBSD$ This article describes the underlying principles and mechanisms of the Pluggable Authentication Modules (PAM) library, and explains how to configure PAM, how to integrate PAM into applications, and how to write PAM modules. 2001 2002 2003 Networks Associates Technology, Inc. Dag-Erling Smørgrav Contributed by This article was written for the FreeBSD Project by ThinkSec AS and Network Associates Laboratories, the Security Research Division of Network Associates, Inc. under DARPA/SPAWAR contract N66001-01-C-8035 (CBOSS), as part of the DARPA CHATS research program. &tm-attrib.freebsd; &tm-attrib.linux; &tm-attrib.opengroup; &tm-attrib.sun; &tm-attrib.general;
Introduction The Pluggable Authentication Modules (PAM) library is a generalized API for authentication-related services which allows a system administrator to add new authentication methods simply by installing new PAM modules, and to modify authentication policies by editing configuration files. PAM was defined and developed in 1995 by Vipin Samar and Charlie Lai of Sun Microsystems, and has not changed much since. In 1997, the Open Group published the X/Open Single Sign-on (XSSO) preliminary specification, which standardized the PAM API and added extensions for single (or rather integrated) sign-on. At the time of this writing, this specification has not yet been adopted as a standard. Although this article focuses primarily on FreeBSD 5.x, which uses OpenPAM, it should be equally applicable to FreeBSD 4.x, which uses Linux-PAM, and other operating systems such as Linux and &solaris;.
Terms and conventions
Definitions The terminology surrounding PAM is rather confused. Neither Samar and Lai's original paper nor the XSSO specification made any attempt at formally defining terms for the various actors and entities involved in PAM, and the terms that they do use (but do not define) are sometimes misleading and ambiguous. The first attempt at establishing a consistent and unambiguous terminology was a whitepaper written by Andrew G. Morgan (author of Linux-PAM) in 1999. While Morgan's choice of terminology was a huge leap forward, it is in this author's opinion by no means perfect. What follows is an attempt, heavily inspired by Morgan, to define precise and unambiguous terms for all actors and entities involved in PAM. account The set of credentials the applicant is requesting from the arbitrator. applicant The user or entity requesting authentication. arbitrator The user or entity who has the privileges necessary to verify the applicant's credentials and the authority to grant or deny the request. chain A sequence of modules that will be invoked in response to a PAM request. The chain includes information about the order in which to invoke the modules, what arguments to pass to them, and how to interpret the results. client The application responsible for initiating an authentication request on behalf of the applicant and for obtaining the necessary authentication information from him. facility One of the four basic groups of functionality provided by PAM: authentication, account management, session management and authentication token update. module A collection of one or more related functions implementing a particular authentication facility, gathered into a single (normally dynamically loadable) binary file and identified by a single name. policy The complete set of configuration statements describing how to handle PAM requests for a particular service. A policy normally consists of four chains, one for each facility, though some services do not use all four facilities. server The application acting on behalf of the arbitrator to converse with the client, retrieve authentication information, verify the applicant's credentials and grant or deny requests. service A class of servers providing similar or related functionality and requiring similar authentication. PAM policies are defined on a per-service basis, so all servers that claim the same service name will be subject to the same policy. session The context within which service is rendered to the applicant by the server. One of PAM's four facilities, session management, is concerned exclusively with setting up and tearing down this context. token A chunk of information associated with the account, such as a password or passphrase, which the applicant must provide to prove his identity. transaction A sequence of requests from the same applicant to the same instance of the same server, beginning with authentication and session set-up and ending with session tear-down.
Usage examples This section aims to illustrate the meanings of some of the terms defined above by way of a handful of simple examples.
Client and server are one This simple example shows alice &man.su.1;'ing to root. &prompt.user; whoami alice &prompt.user; ls -l `which su` -r-sr-xr-x 1 root wheel 10744 Dec 6 19:06 /usr/bin/su &prompt.user; su - Password: xi3kiune &prompt.root; whoami root The applicant is alice. The account is root. The &man.su.1; process is both client and server. The authentication token is xi3kiune. The arbitrator is root, which is why &man.su.1; is setuid root.
Client and server are separate The example below shows eve try to initiate an &man.ssh.1; connection to login.example.com, ask to log in as bob, and succeed. Bob should have chosen a better password! &prompt.user; whoami eve &prompt.user; ssh bob@login.example.com bob@login.example.com's password: god Last login: Thu Oct 11 09:52:57 2001 from 192.168.0.1 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.4-STABLE (LOGIN) #4: Tue Nov 27 18:10:34 PST 2001 Welcome to FreeBSD! &prompt.user; The applicant is eve. The client is Eve's &man.ssh.1; process. The server is the &man.sshd.8; process on login.example.com The account is bob. The authentication token is god. Although this is not shown in this example, the arbitrator is root.
Sample policy The following is FreeBSD's default policy for sshd: sshd auth required pam_nologin.so no_warn sshd auth required pam_unix.so no_warn try_first_pass sshd account required pam_login_access.so sshd account required pam_unix.so sshd session required pam_lastlog.so no_fail sshd password required pam_permit.so This policy applies to the sshd service (which is not necessarily restricted to the &man.sshd.8; server.) auth, account, session and password are facilities. pam_nologin.so, pam_unix.so, pam_login_access.so, pam_lastlog.so and pam_permit.so are modules. It is clear from this example that pam_unix.so provides at least two facilities (authentication and account management.)
PAM Essentials
Facilities and primitives The PAM API offers six different authentication primitives grouped in four facilities, which are described below. auth Authentication. This facility concerns itself with authenticating the applicant and establishing the account credentials. It provides two primitives: &man.pam.authenticate.3; authenticates the applicant, usually by requesting an authentication token and comparing it with a value stored in a database or obtained from an authentication server. &man.pam.setcred.3; establishes account credentials such as user ID, group membership and resource limits. account Account management. This facility handles non-authentication-related issues of account availability, such as access restrictions based on the time of day or the server's work load. It provides a single primitive: &man.pam.acct.mgmt.3; verifies that the requested account is available. session Session management. This facility handles tasks associated with session set-up and tear-down, such as login accounting. It provides two primitives: &man.pam.open.session.3; performs tasks associated with session set-up: add an entry in the utmp and wtmp databases, start an SSH agent, etc. &man.pam.close.session.3; performs tasks associated with session tear-down: add an entry in the utmp and wtmp databases, stop the SSH agent, etc. password Password management. This facility is used to change the authentication token associated with an account, either because it has expired or because the user wishes to change it. It provides a single primitive: &man.pam.chauthtok.3; changes the authentication token, optionally verifying that it is sufficiently hard to guess, has not been used previously, etc.
Modules Modules are a very central concept in PAM; after all, they are the M in PAM. A PAM module is a self-contained piece of program code that implements the primitives in one or more facilities for one particular mechanism; possible mechanisms for the authentication facility, for instance, include the &unix; password database, NIS, LDAP and Radius.
Module Naming FreeBSD implements each mechanism in a single module, named pam_mechanism.so (for instance, pam_unix.so for the &unix; mechanism.) Other implementations sometimes have separate modules for separate facilities, and include the facility name as well as the mechanism name in the module name. To name one example, &solaris; has a pam_dial_auth.so.1 module which is commonly used to authenticate dialup users.
Module Versioning FreeBSD's original PAM implementation, based on Linux-PAM, did not use version numbers for PAM modules. This would commonly cause problems with legacy applications, which might be linked against older versions of the system libraries, as there was no way to load a matching version of the required modules. OpenPAM, on the other hand, looks for modules that have the same version number as the PAM library (currently 2), and only falls back to an unversioned module if no versioned module could be loaded. Thus legacy modules can be provided for legacy applications, while allowing new (or newly built) applications to take advantage of the most recent modules. Although &solaris; PAM modules commonly have a version - number, they're not truly versioned, because the number is a + number, they are not truly versioned, because the number is a part of the module name and must be included in the configuration.
Chains and policies When a server initiates a PAM transaction, the PAM library tries to load a policy for the service specified in the &man.pam.start.3; call. The policy specifies how authentication requests should be processed, and is defined in a configuration file. This is the other central concept in PAM: the possibility for the admin to tune the system security policy (in the wider sense of the word) simply by editing a text file. A policy consists of four chains, one for each of the four PAM facilities. Each chain is a sequence of configuration statements, each specifying a module to invoke, some (optional) parameters to pass to the module, and a control flag that describes how to interpret the return code from the module. Understanding the control flags is essential to understanding PAM configuration files. There are four different control flags: binding If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the rest of the chain is executed, but the request is ultimately denied. This control flag was introduced by Sun in &solaris; 9 (&sunos; 5.9), and is also supported by OpenPAM. required If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the rest of the chain is also executed, but the request is ultimately denied. requisite If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the chain is immediately terminated and the request is denied. sufficient If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the module is ignored and the rest of the chain is executed. As the semantics of this flag may be somewhat confusing, especially when it is used for the last module in a chain, it is recommended that the binding control flag be used instead if the implementation supports it. optional The module is executed, but its result is ignored. If all modules in a chain are marked optional, all requests will always be granted. When a server invokes one of the six PAM primitives, PAM retrieves the chain for the facility the primitive belongs to, and invokes each of the modules listed in the chain, in the order they are listed, until it reaches the end, or determines that no further processing is necessary (either because a binding or sufficient module succeeded, or because a requisite module failed.) The request is granted if and only if at least one module was invoked, and all non-optional modules succeeded. Note that it is possible, though not very common, to have the same module listed several times in the same chain. For instance, a module that looks up user names and passwords in a directory server could be invoked multiple times with different parameters specifying different directory servers to contact. PAM treat different occurrences of the same module in the same chain as different, unrelated modules.
Transactions The lifecycle of a typical PAM transaction is described below. Note that if any of these steps fails, the server should report a suitable error message to the client and abort the transaction. If necessary, the server obtains arbitrator credentials through a mechanism independent of PAM—most commonly by virtue of having been started by root, or of being setuid root. The server calls &man.pam.start.3; to initialize the PAM library and specify its service name and the target account, and register a suitable conversation function. The server obtains various information relating to the transaction (such as the applicant's user name and the name of the host the client runs on) and submits it to PAM using &man.pam.set.item.3;. The server calls &man.pam.authenticate.3; to authenticate the applicant. The server calls &man.pam.acct.mgmt.3; to verify that the requested account is available and valid. If the password is correct but has expired, &man.pam.acct.mgmt.3; will return PAM_NEW_AUTHTOK_REQD instead of PAM_SUCCESS. If the previous step returned PAM_NEW_AUTHTOK_REQD, the server now calls &man.pam.chauthtok.3; to force the client to change the authentication token for the requested account. Now that the applicant has been properly authenticated, the server calls &man.pam.setcred.3; to establish the credentials of the requested account. It is able to do this because it acts on behalf of the arbitrator, and holds the arbitrator's credentials. Once the correct credentials have been established, the server calls &man.pam.open.session.3; to set up the session. The server now performs whatever service the client requested—for instance, provide the applicant with a shell. Once the server is done serving the client, it calls &man.pam.close.session.3; to tear down the session. Finally, the server calls &man.pam.end.3; to notify the PAM library that it is done and that it can release whatever resources it has allocated in the course of the transaction.
PAM Configuration
PAM policy files
The <filename>/etc/pam.conf</filename> file The traditional PAM policy file is /etc/pam.conf. This file contains all the PAM policies for your system. Each line of the file describes one step in a chain, as shown below: login auth required pam_nologin.so no_warn The fields are, in order: service name, facility name, control flag, module name, and module arguments. Any additional fields are interpreted as additional module arguments. A separate chain is constructed for each service / facility pair, so while the order in which lines for the same service and facility appear is significant, the order in which the individual services and facilities are listed is not. The examples in the original PAM paper grouped configuration lines by facility, and the &solaris; stock pam.conf still does that, but FreeBSD's stock configuration groups configuration lines by service. Either way is fine; either way makes equal sense.
The <filename>/etc/pam.d</filename> directory OpenPAM and Linux-PAM support an alternate configuration mechanism, which is the preferred mechanism in FreeBSD. In this scheme, each policy is contained in a separate file bearing the name of the service it applies to. These files are stored in /etc/pam.d/. These per-service policy files have only four fields instead of pam.conf's five: the service name field is omitted. Thus, instead of the sample pam.conf line from the previous section, one would have the following line in /etc/pam.d/login: auth required pam_nologin.so no_warn As a consequence of this simplified syntax, it is possible to use the same policy for multiple services by linking each service name to a same policy file. For instance, to use the same policy for the su and sudo services, one could do as follows: &prompt.root; cd /etc/pam.d &prompt.root; ln -s su sudo This works because the service name is determined from the file name rather than specified in the policy file, so the same file can be used for multiple differently-named services. Since each service's policy is stored in a separate file, the pam.d mechanism also makes it very easy to install additional policies for third-party software packages.
The policy search order As we have seen above, PAM policies can be found in a number of places. What happens if policies for the same service exist in multiple places? It is essential to understand that PAM's configuration system is centered on chains.
Breakdown of a configuration line As explained in the section, each line in /etc/pam.conf consists of four or more fields: the service name, the facility name, the control flag, the module name, and zero or more module arguments. The service name is generally (though not always) the name of the application the statement applies to. If you are unsure, refer to the individual application's documentation to determine what service name it uses. Note that if you use /etc/pam.d/ instead of /etc/pam.conf, the service name is specified by the name of the policy file, and omitted from the actual configuration lines, which then start with the facility name. The facility is one of the four facility keywords described in the section. Likewise, the control flag is one of the four keywords described in the section, describing how to interpret the return code from the module. Linux-PAM supports an alternate syntax that lets you specify the action to associate with each possible return code, but this should be avoided as it is non-standard and closely tied in with the way Linux-PAM dispatches service calls (which differs greatly from the way &solaris; and OpenPAM do it.) Unsurprisingly, OpenPAM does not support this syntax.
Policies To configure PAM correctly, it is essential to understand how policies are interpreted. When an application calls &man.pam.start.3;, the PAM library loads the policy for the specified service and constructs four module chains (one for each facility.) If one or more of these chains are empty, the corresponding chains from the policy for the other service are substituted. When the application later calls one of the six PAM primitives, the PAM library retrieves the chain for the corresponding facility and calls the appropriate service function in each module listed in the chain, in the order in which they were listed in the configuration. After each call to a service function, the module type and the error code returned by the service function are used to determine what happens next. With a few exceptions, which we discuss below, the following table applies: PAM chain execution summary PAM_SUCCESS PAM_IGNORE other binding if (!fail) break; - fail = true; required - - fail = true; requisite - - fail = true; break; sufficient if (!fail) break; - - optional - - -
If fail is true at the end of a chain, or when a break is reached, the dispatcher returns the error code returned by the first module that failed. Otherwise, it returns PAM_SUCCESS. The first exception of note is that the error code PAM_NEW_AUTHTOK_REQD is treated like a success, except that if no module failed, and at least one module returned PAM_NEW_AUTHTOK_REQD, the dispatcher will return PAM_NEW_AUTHTOK_REQD. The second exception is that &man.pam.setcred.3; treats binding and sufficient modules as if they were required. The third and final exception is that &man.pam.chauthtok.3; runs the entire chain twice (once for preliminary checks and once to actually set the password), and in the preliminary phase it treats binding and sufficient modules as if they were required.
FreeBSD PAM Modules
&man.pam.deny.8; The &man.pam.deny.8; module is one of the simplest modules available; it responds to any request with PAM_AUTH_ERR. It is useful for quickly disabling a service (add it to the top of every chain), or for terminating chains of sufficient modules.
&man.pam.echo.8; The &man.pam.echo.8; module simply passes its arguments to the conversation function as a PAM_TEXT_INFO message. It is mostly useful for debugging, but can also serve to display messages such as Unauthorized access will be prosecuted before starting the authentication procedure.
&man.pam.exec.8; The &man.pam.exec.8; module takes its first argument to be the name of a program to execute, and the remaining arguments are passed to that program as command-line arguments. One possible application is to use it to run a program at login time which mounts the user's home directory.
&man.pam.ftpusers.8; The &man.pam.ftpusers.8; module
&man.pam.group.8; The &man.pam.group.8; module accepts or rejects applicants on the basis of their membership in a particular file group (normally wheel for &man.su.1;). It is primarily intended for maintaining the traditional behaviour of BSD &man.su.1;, but has many other uses, such as excluding certain groups of users from a particular service.
&man.pam.guest.8; The &man.pam.guest.8; module allows guest logins using fixed login names. Various requirements can be placed on the password, but the default behaviour is to allow any password as long as the login name is that of a guest account. The &man.pam.guest.8; module can easily be used to implement anonymous FTP logins.
&man.pam.krb5.8; The &man.pam.krb5.8; module
&man.pam.ksu.8; The &man.pam.ksu.8; module
&man.pam.lastlog.8; The &man.pam.lastlog.8; module
&man.pam.login.access.8; The &man.pam.login.access.8; module provides an implementation of the account management primitive which enforces the login restrictions specified in the &man.login.access.5; table.
&man.pam.nologin.8; The &man.pam.nologin.8; module refuses non-root logins when /var/run/nologin exists. This file is normally created by &man.shutdown.8; when less than five minutes remain until the scheduled shutdown time.
&man.pam.opie.8; The &man.pam.opie.8; module implements the &man.opie.4; authentication method. The &man.opie.4; system is a challenge-response mechanism where the response to each challenge is a direct function of the challenge and a passphrase, so the response can be easily computed just in time by anyone possessing the passphrase, eliminating the need for password lists. Moreover, since &man.opie.4; never reuses a challenge that has been correctly answered, it is not vulnerable to replay attacks.
&man.pam.opieaccess.8; The &man.pam.opieaccess.8; module is a companion module to &man.pam.opie.8;. Its purpose is to enforce the restrictions codified in &man.opieaccess.5;, which regulate the conditions under which a user who would normally authenticate herself using &man.opie.4; is allowed to use alternate methods. This is most often used to prohibit the use of password authentication from untrusted hosts. In order to be effective, the &man.pam.opieaccess.8; module must be listed as requisite immediately after a sufficient entry for &man.pam.opie.8;, and before any other modules, in the auth chain.
&man.pam.passwdqc.8; The &man.pam.passwdqc.8; module
&man.pam.permit.8; The &man.pam.permit.8; module is one of the simplest modules available; it responds to any request with PAM_SUCCESS. It is useful as a placeholder for services where one or more chains would otherwise be empty.
&man.pam.radius.8; The &man.pam.radius.8; module
&man.pam.rhosts.8; The &man.pam.rhosts.8; module
&man.pam.rootok.8; The &man.pam.rootok.8; module reports success if and only if the real user id of the process calling it (which is assumed to be run by the applicant) is 0. This is useful for non-networked services such as &man.su.1; or &man.passwd.1;, to which the root should have automatic access.
&man.pam.securetty.8; The &man.pam.securetty.8; module
&man.pam.self.8; The &man.pam.self.8; module reports success if and only if the names of the applicant matches that of the target account. It is most useful for non-networked services such as &man.su.1;, where the identity of the applicant can be easily verified.
&man.pam.ssh.8; The &man.pam.ssh.8; module provides both authentication and session services. The authentication service allows users who have passphrase-protected SSH secret keys in their ~/.ssh directory to authenticate themselves by typing their passphrase. The session service starts &man.ssh-agent.1; and preloads it with the keys that were decrypted in the authentication phase. This feature is particularly useful for local logins, whether in X (using &man.xdm.1; or another PAM-aware X login manager) or at the console.
&man.pam.tacplus.8; The &man.pam.tacplus.8; module
&man.pam.unix.8; The &man.pam.unix.8; module implements traditional &unix; password authentication, using &man.getpwnam.3; to obtain the target account's password and compare it with the one provided by the applicant. It also provides account management services (enforcing account and password expiration times) and password-changing services. This is probably the single most useful module, as the great majority of admins will want to maintain historical behaviour for at least some services.
PAM Application Programming This section has not yet been written.
PAM Module Programming This section has not yet been written.
Sample PAM Application The following is a minimal implementation of &man.su.1; using PAM. Note that it uses the OpenPAM-specific &man.openpam.ttyconv.3; conversation function, which is prototyped in security/openpam.h. If you wish build this application on a system with a different PAM library, you will have to provide your own conversation function. A robust conversation function is surprisingly difficult to implement; the one presented in the appendix is a good starting point, but should not be used in real-world applications. Sample PAM Module The following is a minimal implementation of &man.pam.unix.8;, offering only authentication services. It should build and run with most PAM implementations, but takes advantage of OpenPAM extensions if available: note the use of &man.pam.get.authtok.3;, which enormously simplifies prompting the user for a password. Sample PAM Conversation Function The conversation function presented below is a greatly simplified version of OpenPAM's &man.openpam.ttyconv.3;. It is fully functional, and should give the reader a good idea of how a conversation function should behave, but it is far too simple - for real-world use. Even if you're not using OpenPAM, feel free + for real-world use. Even if you are not using OpenPAM, feel free to download the source code and adapt &man.openpam.ttyconv.3; to your uses; we believe it to be as robust as a tty-oriented conversation function can reasonably get. Further Reading This is a list of documents relevant to PAM and related issues. It is by no means complete. Papers <ulink url="http://www.sun.com/software/solaris/pam/pam.external.pdf"> Making Login Services Independent of Authentication Technologies</ulink> Samar Vipin Lai Charlie Sun Microsystems <ulink url="http://www.opengroup.org/pubs/catalog/p702.htm">X/Open Single Sign-on Preliminary Specification</ulink> The Open Group 1-85912-144-6 June 1997 <ulink url="http://www.kernel.org/pub/linux/libs/pam/pre/doc/current-draft.txt"> Pluggable Authentication Modules</ulink> Morgan Andrew G. October 6, 1999 User Manuals <ulink url="http://www.sun.com/software/solaris/pam/pam.admin.pdf">PAM Administration</ulink> Sun Microsystems Related Web pages <ulink url="http://openpam.sourceforge.net/">OpenPAM homepage</ulink> Smørgrav Dag-Erling ThinkSec AS <ulink url="http://www.kernel.org/pub/linux/libs/pam/">Linux-PAM homepage</ulink> Morgan Andrew G. <ulink url="http://wwws.sun.com/software/solaris/pam/">Solaris PAM homepage</ulink> Sun Microsystems
diff --git a/en_US.ISO8859-1/articles/problem-reports/article.sgml b/en_US.ISO8859-1/articles/problem-reports/article.sgml index 6427d9224a..03affef18e 100644 --- a/en_US.ISO8859-1/articles/problem-reports/article.sgml +++ b/en_US.ISO8859-1/articles/problem-reports/article.sgml @@ -1,1113 +1,1113 @@ %articles.ent; ]>
Writing &os; Problem Reports $FreeBSD$ &tm-attrib.freebsd; &tm-attrib.cvsup; &tm-attrib.ibm; &tm-attrib.intel; &tm-attrib.sparc; &tm-attrib.sun; &tm-attrib.general; This article describes how to best formulate and submit a problem report to the &os; Project. Dag-Erling Smørgrav Contributed by Mark Linimon problem reports
Introduction One of the most frustrating experiences one can have as a software user is to submit a problem report only to have it summarily closed with a terse and unhelpful explanation like not a bug or bogus PR. Similarly, one of the most frustrating experiences as a software developer is to be flooded with problem reports that are not really problem reports but requests for support, or that contain little or no information about what the problem is and how to reproduce it. This document attempts to describe how to write good problem reports. What, you ask, is a good problem report? Well, to go straight to the bottom line, a good problem report is one that can be analyzed and dealt with swiftly, to the mutual satisfaction of both user and developer. Although the primary focus of this article is on &os; problem reports, most of it should apply quite well to other software projects. Note that this article is organized thematically, not chronologically, so you should read through the entire document before submitting a problem report, rather than treat it as a step-by-step tutorial.
When to submit a problem report There are many types of problems, and not all of them should engender a problem report. Of course, nobody is perfect, and there will be times when you are convinced you have found a bug in a program when in fact you have misunderstood the syntax for a command or made a typographical error in a configuration file (though that in itself may sometimes be indicative of poor documentation or poor error handling in the application). There are still many cases where submitting a problem report is clearly not the right course of action, and will only serve to frustrate you and the developers. Conversely, there are cases where it might be appropriate to submit a problem report about something else than a bug—an enhancement or a feature request, for instance. So how do you determine what is a bug and what is not? As a simple rule of thumb your problem is not a bug if it can be expressed as a question (usually of the form How do I do X? or Where can I find Y?). It is not always quite so black and white, but the question rule covers a large majority of cases. If you are looking for an answer, consider posing your question to the &a.questions;. Some cases where it may be appropriate to submit a problem report about something that is not a bug are: Requests for feature enhancements. It is generally a good idea to air these on the mailing lists before submitting a problem report. Notification of updates to externally maintained software (mainly ports, but also externally maintained base system components such as BIND or various GNU utilities). For unmaintained ports (MAINTAINER contains ports@FreeBSD.org), such update notifications might get picked up by an interested committer, or you might be asked to provide a patch to update the port; providing it upfront will greatly improve your chances that the port will get updated in a timely manner. If the port is maintained, PRs announcing new upstream releases are usually not very useful since they generate supplementary work for the committers, and the maintainer likely knows already there is a new version, they have probably worked with the developers on it, they are probably testing to see there is no regression, etc. In either case, following the process described in Porter's Handbook will yield the best results. A bug that can not be reproduced can rarely be fixed. If the bug only occurred once and you can not reproduce it, and it does not seem to happen to anybody else, chances are none of the developers will be able to reproduce it or figure out what is wrong. That does not mean it did not happen, but it does mean that the chances of your problem report ever leading to a bug fix are very slim. To make matters worse, often these kinds of bugs are actually caused by failing hard drives or overheating processors — you should always try to rule out these causes, whenever possible, before submitting a PR. Next, to decide to whom you should file your problem report, you need to understand that the software that makes up &os; is composed of several different elements: Code in the base system that is written and maintained by &os; contributors, such as the kernel, the C library, and the device drivers (categorized as kern); the binary utilities (bin); the manual pages and documentation (docs); and the web pages (www). All bugs in these areas should be reported to the &os; developers. Code in the base system that is written and maintained by others, and imported into &os; and adapted. Examples include bind, &man.gcc.1;, and &man.sendmail.8;. Most bugs in these areas should be reported to the &os; developers; but in some cases they may need to be reported to the original authors instead if the problems are not &os;-specific. Usually these bugs will fall under either the bin or gnu categories. Individual applications that are not in the base system but are instead part of the &os; Ports Collection (category ports). Most of these applications are not written by &os; developers; what &os; provides is merely a framework for installing the application. Therefore, you should only report a problem to the &os; developers when you believe the problem is &os;-specific; otherwise, you should report it to the authors of the software. Then you should ascertain whether or not the problem is timely. There are few things that will annoy a developer more than receiving a problem report about a bug she has already fixed. If the problem is in the base system, you should first read the FAQ section on &os; versions, if you are not already familiar with the topic. It is not possible for &os; to fix problems in anything other than certain recent branches of the base system, so filing a bug report about an older version will probably only result in a developer advising you to upgrade to a supported version to see if the problem still recurs. The Security Officer team maintains the list of supported versions. If the problem is in a port, note that you must first upgrade to the latest version of the Ports Collection and see if the problem still applies. Due to the rapid pace of changes in these applications, it is infeasible for &os; to support anything other than the absolute latest versions, and problems with older version of applications simply cannot be fixed.
Preparations A good rule to follow is to always do a background search before submitting a problem report. Maybe your problem has already been reported; maybe it is being discussed on the mailing lists, or recently was; it may even already be fixed in a newer version than what you are running. You should therefore check all the obvious places before submitting your problem report. For &os;, this means: The &os; Frequently Asked Questions (FAQ) list. The FAQ attempts to provide answers for a wide range of questions, such as those concerning hardware compatibility, user applications, and kernel configuration. The mailing lists—if you are not subscribed, use the searchable archives on the &os; web site. If your problem has not been discussed on the lists, you might try posting a message about it and waiting a few days to see if someone can spot something you have overlooked. Optionally, the entire web—use your favorite search engine to locate any references to your problem. You may even get hits from archived mailing lists or newsgroups you did not know of or had not thought to search through. Next, the searchable &os; PR database (GNATS). Unless your problem is recent or obscure, there is a fair chance it has already been reported. Most importantly, you should attempt to see if existing documentation in the source base addresses your problem. For the base &os; code, you should carefully study the contents of the /usr/src/UPDATING file on your system or its latest version at . (This is vital information if you are upgrading from one version to another—especially if you are upgrading to the &os.current; branch). However, if the problem is in something that was installed as a part of the &os; Ports Collection, you should refer to /usr/ports/UPDATING (for individual ports) or /usr/ports/CHANGES (for changes that affect the entire Ports Collection). and are also available via CVSweb.
Writing the problem report Now that you have decided that your issue merits a problem report, and that it is a &os; problem, it is time to write the actual problem report. Before we get into the mechanics of the program used to generate and submit PRs, here are some tips and tricks to help make sure that your PR will be most effective.
Tips and tricks for writing a good problem report Do not leave the Synopsis line empty. The PRs go both onto a mailing list that goes all over the world (where the Synopsis is used for the Subject: line), but also into a database. Anyone who comes along later and browses the database by synopsis, and finds a PR with a blank subject line, tends just to skip over it. Remember that PRs stay in this database until they are closed by someone; an anonymous one will usually just disappear in the noise. Avoid using a weak Synopsis line. You should not assume that anyone reading your PR has any context for your submission, so the more you provide, the better. For instance, what part of the system does the problem apply to? Do you only see the problem while installing, or while running? To illustrate, instead of Synopsis: portupgrade is broken, see how much more informative this seems: Synopsis: port sysutils/portupgrade coredumps on -current. (In the case of ports, it is especially helpful to have both the category and portname in the Synopsis line.) If you have a patch, say so. A PR with a patch included is much more likely to be looked at than one without. If you are including one, put the string [patch] at the beginning of the Synopsis. (Although it is not mandatory to use that exact string, by convention, that is the one that is used.) If you are a maintainer, say so. If you are maintaining a part of the source code (for instance, a port), you might consider adding the string [maintainer update] at the beginning of your synopsis line, and you definitely should set the Class of your PR to maintainer-update. This way any committer that handles your PR will not have to check. Be specific. The more information you supply about what problem you are having, the better your chance of getting a response. Include the version of &os; you are running (there is a place to put that, see below) and on which architecture. You should include whether you are running from a release (e.g. from a CDROM or download), or from a system maintained by &man.cvsup.1; (and, if so, how recently you updated). If you are tracking the &os.current; branch, that is the very first thing someone will ask, because fixes (especially for high-profile problems) tend to get committed very quickly, and &os.current; users are expected to keep up. Include which global options you have specified in your make.conf. Note: specifying -O2 and above to &man.gcc.1; is known to be buggy in many situations. While the &os; developers will accept patches, they are generally unwilling to investigate such issues due to simple lack of time and volunteers, and may instead respond that this just is not supported. If this is a kernel problem, then be prepared to supply the following information. (You do not have to include these by default, which only tends to fill up the database, but you should include excerpts that you think might be relevant): your kernel configuration (including which hardware devices you have installed) whether or not you have debugging options enabled (such as WITNESS), and if so, whether the problem persists when you change the sense of that option a backtrace, if one was generated the fact that you have read src/UPDATING and that your problem is not listed there (someone is guaranteed to ask) whether or not you can run any other kernel as a fallback (this is to rule out hardware-related issues such as failing disks and overheating CPUs, which can masquerade as kernel problems) If this is a ports problem, then be prepared to supply the following information. (You do not have to include these by default, which only tends to fill up the database, but you should include excerpts that you think might be relevant): which ports you have installed any environment variables that override the defaults in bsd.port.mk, such as PORTSDIR the fact that you have read ports/UPDATING and that your problem is not listed there (someone is guaranteed to ask) Avoid vague requests for features. PRs of the form someone should really implement something that does so-and-so are less likely to get results than very specific requests. Remember, the source is available to everyone, so if you want a feature, the best way to ensure it being included is to get to work! Also consider the fact that many things like this would make a better topic for discussion on freebsd-questions than an entry in the PR database, as discussed above. Make sure no one else has already submitted a similar PR. Although this has already been mentioned above, it bears repeating here. It only take a minute or two to use the web-based search engine at . (Of course, everyone is guilty of forgetting to do this now and then.) Avoid controversial requests. If your PR addresses an area that has been controversial in the past, you should probably be prepared to not only offer patches, but also justification for why the patches are The Right Thing To Do. As noted above, a careful search of the mailing lists using the archives at is always good preparation. Be polite. Almost anyone who would potentially work on your PR is a volunteer. No one likes to be told that they have to do something when they are already doing it for some motivation other than monetary gain. This is a good thing to keep in mind at all times on Open Source projects.
Before you begin If you are using the &man.send-pr.1; program, make sure your VISUAL (or EDITOR if VISUAL is not set) environment variable is set to something sensible. You should also make sure that mail delivery works fine. &man.send-pr.1; uses mail messages for the submission and tracking of problem reports. If you cannot post mail messages - from the machine you're running &man.send-pr.1; on, your + from the machine you are running &man.send-pr.1; on, your problem report will not reach the GNATS database. For details on the setup of mail on &os;, see the Electronic Mail chapter of the &os; Handbook at . Make sure that your mailer will not mangle the message on its way to GNATS. In particular, if your mailer automatically breaks lines, changes tabs to spaces, or escapes newline characters, any patch that you submit will be rendered unusable. For the text sections, however, we request that you insert manual linebreaks somewhere around 70 characters, so that the web display of the PR will be readable. Similar considerations apply if you are using the web-based PR submittal form instead of &man.send-pr.1;. Note that cut-and-paste operations can have their own side-effects on text formatting. In certain cases it may be necessary to use &man.uuencode.1; to ensure that patches arrive unmodified. Finally, if your submission will be lengthy, you should to prepare your work offline so that nothing will be lost in case there is a problem submitting it. This can be an especial problem with the web form.
Attaching patches or files The following applies to submitting PRs via email: The &man.send-pr.1; program has provisions for attaching files to a problem report. You can attach as many files as you want provided that each has a unique base name (i.e. the name of the file proper, without the path). Just use the command-line option to specify the names of the files you wish to attach: &prompt.user; send-pr -a /var/run/dmesg -a /tmp/errors Do not worry about binary files, they will be automatically encoded so as not to upset your mail agent. If you attach a patch, make sure you use the or option to &man.diff.1; to create a context or unified diff (unified is preferred), and make sure to specify the exact CVS revision numbers of the files you modified so the developers who read your report will be able to apply them easily. For problems with the kernel or the base utilities, a patch against &os.current; (the HEAD CVS branch) is preferred since all new code should be applied and tested there first. After appropriate or substantial testing has been done, the code will be merged/migrated to the &os.stable; branch. If you attach a patch inline, instead of as an attachment, note that the most common problem by far is the tendency of some email programs to render tabs as spaces, which will completely ruin anything intended to be part of a Makefile. Do not send patches as attachments using Content-Transfer-Encoding: quoted-printable. These will perform character escaping and the entire patch will be useless. Also note that while including small patches in a PR is generally all right—particularly when they fix the problem described in the PR—large patches and especially new code which may require substantial review before committing should be placed on a web or ftp server, and the URL should be included in the PR instead of the patch. Patches in email tend to get mangled, especially when GNATS is involved, and the larger the patch, the harder it will be for interested parties to unmangle it. Also, posting a patch on the web allows you to modify it without having to resubmit the entire patch in a followup to the original PR. Finally, large patches simply increase the size of the database, since closed PRs are not actually deleted but instead kept and simply marked as closed. You should also take note that unless you explicitly specify otherwise in your PR or in the patch itself, any patches you submit will be assumed to be licensed under the same terms as the original file you modified.
Filling out the template The next section applies to the email method only: When you run &man.send-pr.1;, you are presented with a template. The template consists of a list of fields, some of which are pre-filled, and some of which have comments explaining their purpose or listing acceptable values. Do not worry about the comments; they will be removed automatically if you do not modify them or remove them yourself. At the top of the template, below the SEND-PR: lines, are the email headers. You do not normally need to modify these, unless you are sending the problem report from a machine or account that can send but not receive mail, in which case you will want to set the From: and Reply-To: to your real email address. You may also want to send yourself (or someone else) a carbon copy of the problem report by adding one or more email addresses to the Cc: header. In the email template you will find the following two single-line fields: Submitter-Id: Do not change this. The default value of current-users is correct, even if you run &os.stable;. Confidential: This is prefilled to no. Changing it makes no sense as there is no such thing as a confidential &os; problem report—the PR database is distributed worldwide by CVSup. The next section describes fields that are common to both the email interface and the web interface: Originator: Please specify your real name, optionally followed by your email address in angle brackets. In the email interface, this is normally prefilled with the gecos field of the currently logged-in user. The email address you use will become public information and may become available to spammers. You should either have spam handling procedures in place, or use a temporary email account. However, please note that if you do not use a valid email account at all, we will not be able to ask you questions about your PR. Organization: Whatever you feel like. This field is not used for anything significant. Synopsis: Fill this out with a short and accurate description of the problem. The synopsis is used as the subject of the problem report email, and is used in problem report listings and summaries; problem reports with obscure synopses tend to get ignored. As noted above, if your problem report includes a patch, please have the synopsis start with [patch]; if this is a ports PR and you are the maintainer, you may consider adding [maintainer update] and set the Class of your PR to maintainer-update. Severity: One of non-critical, serious or critical. Do not overreact; refrain from labeling your problem critical unless it really is (e.g. data corruption issues, serious regression from previous functionality in -CURRENT) or serious unless it is something that will affect many users (kernel panics or freezes; problems with particular device drivers or system utilities). &os; developers will not necessarily work on your problem faster if you inflate its importance since there are so many other people who have done exactly that — in fact, some developers pay little attention to this field because of this. Major security problems should not be filed in GNATS, because all GNATS information is public knowledge. Please send such problems in private email to &a.security-officer;. Priority: One of low, medium or high. high should be reserved for problems that will affect practically every user of &os; and medium for something that will affect many users. This field has become so widely abused that it is almost completely meaningless. Category: Choose an appropriate category. There are a number of "platform" categories into which bugs in the base system that are specific to one particular hardware architecture should be filed. Problems that are generic all across versions of &os; should probably be filed as kern or bin; see discussion of those categories below. Example: you have a common PC-based machine, and think you have encountered a problem specific to a particular chipset or a particular motherboard: i386 is the right category. Example: You are having a problem with an add-in peripheral card on a commonly seen bus, or a problem with a particular type of hard disk drive: in this case, it probably applies to more than one architecture, and kern is the right category. Here is the current list of categories (taken from ): advocacy: problems relating to &os;'s public image. Rarely used. alpha: problems specific to the Alpha platform. amd64: problems specific to the AMD64 platform. bin: problems with userland programs in the base system. If running &man.whereis.1; shows /bin, /usr/sbin, or something similar, then this is probably the right category. (A few contributed programs might instead need to be in gnu; see below.) conf: problems with configuration files, default values, and so forth. Things that affect /usr/share or /etc/rc* belong here. docs: problems with manual pages or on-line documentation. gnu: problems with imported GNU software such as &man.gcc.1; or &man.grep.1;. i386: problems specific to the &i386; platform. ia64: problems specific to the ia64 platform. java: problems related to the &java; Virtual Machine. (Ports that merely depend on &java; to run should be filed under ports.) kern: problems with the kernel, (non-platform-specific) device drivers, or the base libraries. misc: anything that does not fit in any of the other categories. (Note that there is almost nothing that truly belongs in this category, except for problems with the release and build infrastructure. Temporary build failures on HEAD do not belong here. Also note that it is easy for things to get lost in this category). ports: problems relating to the ports tree. powerpc: problems specific to the &powerpc; platform. sparc64: problems specific to the &sparc64; platform. standards: Standards conformance issues. threads: problems related to the &os; threads implementation (especially on &os.current;). usb: problems related to the &os; USB implementation. www: Changes or enhancements to the &os; website. Problems with code found in /usr/ports/www do not belong here, they belong in ports instead. Class: Choose one of the following: sw-bug: software bugs. doc-bug: errors in documentation. change-request: requests for additional features or changes in existing features. update: updates to ports or other contributed software. maintainer-update: updates to ports for which you are the maintainer. Release: The version of &os; that you are running. This is filled out automatically if you are using &man.send-pr.1; and need only be changed if you are sending a problem report from a different system than the one that exhibits the problem. Finally, there is a series of multi-line fields: Environment: This should describe, as accurately as possible, the environment in which the problem has been observed. This includes the operating system version, the version of the specific program or file that contains the problem, and any other relevant items such as system configuration, other installed software that influences the problem, etc.—quite simply everything a developer needs to know to reconstruct the environment in which the problem occurs. Description: A complete and accurate description of the problem you are experiencing. Try to avoid speculating about the causes of the problem unless you are certain that you are on the right track, as it may mislead a developer into making incorrect assumptions about the problem. How-To-Repeat: A summary of the actions you need to take to reproduce the problem. Fix: Preferably a patch, or at least a workaround (which not only helps other people with the same problem work around it, but may also help a developer understand the cause for the problem), but if you do not have any firm ideas for either, it is better to leave this field blank than to speculate.
Sending off the problem report If you are using &man.send-pr.1;: Once you are done filling out the template, have saved it, and exit your editor, &man.send-pr.1; will prompt you with s)end, e)dit or a)bort?. You can then hit s to go ahead and submit the problem report, e to restart the editor and make further modifications, or a to abort. If you choose the latter, your problem report will remain on disk (&man.send-pr.1; will tell you the filename before it terminates), so you can edit it at your leisure, or maybe transfer it to a system with better net connectivity, before sending it with the to &man.send-pr.1;: &prompt.user; send-pr -f ~/my-problem-report This will read the specified file, validate the contents, strip comments and send it off. If you are using the web form: Before you hit submit, you will need to fill in a field containing text that is represented in image form on the page. This unfortunate measure has had to be adopted due to misuse by automated systems and a few misguided individuals. It is a necessary evil that no one likes; please do not ask us to remove it. Note that you are strongly advised to save your work somewhere before hitting submit. A common problem for users is to have their web browser displaying a stale image from its cache. If this happens to you, your submission will be rejected and you may lose your work. If you are unable to view images for any reason, and are also unable to use &man.send-pr.1;, please accept our apologies for the inconvenience and email your problem report to the bugbuster team at freebsd-bugbusters@FreeBSD.org.
Follow-up Once your problem report has been filed, you will receive a confirmation by email which will include the tracking number that was assigned to your problem report and a URL you can use to check its status. With a little luck, someone will take an interest in your problem and try to address it, or, as the case may be, explain why it is not a problem. You will be automatically notified of any change of status, and you will receive copies of any comments or patches someone may attach to your problem report's audit trail. If someone requests additional information from you, or you remember or discover something you did not mention in the initial report, please use one of two methods to submit your followup: The easiest way is to use the followup link on the individual PR's web page, which you can reach from the PR search page. Clicking on this link will bring up an an email window with the correct To: and Subject: lines filled in (if your browser is configured to do this). Alternatively, you can just mail it to &a.bugfollowup;, making sure that the tracking number is included in the subject so the bug tracking system will know what problem report to attach it to. If you do not include the tracking number, GNATS will become confused and create an entirely new PR which it then assigns to the GNATS administrator, and then your followup will become lost until someone comes in to clean up the mess, which could be days or weeks afterwards. Wrong way: Subject: that PR I sent Right way: Subject: Re: ports/12345: compilation problem with foo/bar If the problem report remains open after the problem has gone away, just send a follow-up (in the manner prescribed above) saying that the problem report can be closed, and, if possible, explaining how or when the problem was fixed.
If you are having problems Most PRs go through the system and are accepted quickly; however, at times GNATS runs behind and you may not get your email confirmation for 10 minutes or even longer. Please try to be patient. In addition, because GNATS receives all its input via email, it is absolutely vital that &os; runs all its submissions through spam filters. If you do not get a response within an hour or two, you may have fallen afoul of them; if so, please contact the GNATS administrators at bugmeister@FreeBSD.org and ask for help. Among the anti-spam measures is one that weighs against many common abuses seen HTML-based email (although not necessarily the mere inclusion of HTML in a PR). We strongly recommend against the use of HTML-based email when sending PRs: not only is it more likely to fall afoul of the filters, it also tends to merely clutter up the database. Plain old email is strongly preferred. On rare occasions you will encounter a GNATS bug where a PR is accepted and assigned a tracking number but it does not show up on the list of PRs on any of the web query pages. What may have happened is that the database index has gotten out of synchronization with the database itself. The way that you can test whether this has happened is to pull up the view a single PR page and see whether the PR shows up. If it does, please notify the GNATS administrators at bugmeister@FreeBSD.org. Note that there is a cron job that periodically rebuilds the database, so unless you are in a hurry, no action needs to be taken.
Further Reading This is a list of resources relevant to the proper writing and processing of problem reports. It is by no means complete. How to Report Bugs Effectively—an excellent essay by Simon G. Tatham on composing useful (non-&os;-specific) problem reports. Problem Report Handling Guidelines—valuable insight into how problem reports are handled by the &os; developers.
diff --git a/en_US.ISO8859-1/articles/releng/article.sgml b/en_US.ISO8859-1/articles/releng/article.sgml index 23234fe132..3566413169 100644 --- a/en_US.ISO8859-1/articles/releng/article.sgml +++ b/en_US.ISO8859-1/articles/releng/article.sgml @@ -1,1110 +1,1110 @@ %articles.ent; The Release Engineering of Third Party Packages'> ]>
FreeBSD Release Engineering November 2001 BSDCon Europe Murray Stokely I've been involved in the development of FreeBSD based products since 1997 at Walnut Creek CDROM, BSDi, and now Wind River Systems. FreeBSD 4.4 was the first official release of FreeBSD that I played a significant part in.
murray@FreeBSD.org
$FreeBSD$ &tm-attrib.freebsd; &tm-attrib.cvsup; &tm-attrib.intel; &tm-attrib.xfree86; &tm-attrib.general; This paper describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It details the methodology used for the official FreeBSD releases and describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization. Introduction The development of FreeBSD is a very open process. FreeBSD is comprised of contributions from thousands of people around the world. The FreeBSD Project provides anonymous CVS[1] access to the general public so that others can have access to log messages, diffs (patches) between development branches, and other productivity enhancements that formal source code management provides. This has been a huge help in attracting more talented developers to FreeBSD. However, I think everyone would agree that chaos would soon manifest if write access was opened up to everyone on the Internet. Therefore only a select group of nearly 300 people are given write access to the CVS repository. These committers[5] are responsible for the bulk of FreeBSD development. An elected core-team[6] of very senior developers provides some level of direction over the project. The rapid pace of FreeBSD development leaves little time for polishing the development system into a production quality release. To solve this dilemma, development continues on two parallel tracks. The main development branch is the HEAD or trunk of our CVS tree, known as FreeBSD-CURRENT or -CURRENT for short. A more stable branch is maintained, known as FreeBSD-STABLE or -STABLE for short. Both branches live in a master CVS repository in California and are replicated via CVSup[2] to mirrors all over the world. FreeBSD-CURRENT[7] is the bleeding-edge of FreeBSD development where all new changes first enter the system. FreeBSD-STABLE is the development branch from which major releases are made. Changes go into this branch at a different pace, and with general assumption that they have first gone into FreeBSD-CURRENT and have been thoroughly tested by our user community. In the interim period between releases, nightly snapshots are built automatically by the FreeBSD Project build machines and made available for download from ftp://stable.FreeBSD.org/. The widespread availability of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with CVSup and make world[7] helps to keep FreeBSD-STABLE in a very reliable condition even before the quality assurance activities ramp up pending a major release. Bug reports and feature requests are continuously submitted by users throughout the release cycle. Problems reports are entered into our GNATS[8] database through email, the &man.send-pr.1; application, or via the web interface provided at . In addition to the multitude of different technical mailing lists about FreeBSD, the &a.qa; provides a forum for discussing the finer points of release-polishing. To service our most conservative users, individual release branches were introduced with FreeBSD 4.3. These release branches are created shortly before a final release is made. After the release goes out, only the most critical security fixes and additions are merged onto the release branch. In addition to source updates via CVS, binary patchkits are available to keep systems on the RELENG_X_Y branches updated. discusses the different phases of the release engineering process leading up to the actual system build and describes the actual build process. describes how the base release may be extended by third parties and details some of the lessons learned through the release of FreeBSD 4.4. Finally, presents future directions of development. Release Process New releases of FreeBSD are released from the -STABLE branch at approximately four month intervals. The FreeBSD release process begins to ramp up 45 days before the anticipated release date when the release engineer sends an email to the development mailing lists to remind developers that they only have 15 days to integrate new changes before the code freeze. During this time, many developers perform what have become known as MFC sweeps. MFC stands for Merge From CURRENT and it describes the process of merging a tested change from our -CURRENT development branch to our -STABLE branch. Code Review Thirty days before the anticipated release, the source repository enters a code slush. During this time, all commits to the -STABLE branch must be approved by the &a.re;. The kinds of changes that are allowed during this 15 day period include: Bug fixes. Documentation updates. Security-related fixes of any kind. Minor changes to device drivers, such as adding new Device IDs. Any additional change that the release engineering team feels is justified, given the potential risk. After the first 15 days of the code slush, a release candidate is released for widespread testing and the code enters a code freeze where it becomes much harder to justify new changes to the system unless a serious bug-fix or security issue is involved. During the code freeze, at least one release candidate is released per week, until the final release is ready. During the days leading to the final release, the release engineering team is in constant communication with the security-officer team, the documentation maintainers, and the port maintainers, to ensure that all of the different components required for a successful release are available. Final Release Checklist When several release candidates have been made available for widespread testing and all major issues have been resolved, the final release polishing can begin. Creating the Release Branch As described in the introduction, the RELENG_X_Y release branch is a relatively new addition to our release engineering methodology. The first step in creating this branch is to ensure that you are working with the newest version of the RELENG_X sources that you want to branch from. /usr/src&prompt.root; cvs update -rRELENG_4 -P -d The next step is to create a branch point tag, so that diffs against the start of the branch are easier with CVS: /usr/src&prompt.root; cvs rtag -rRELENG_4 RELENG_4_8_BP src And then a new branch tag is created with: /usr/src&prompt.root; cvs rtag -b -rRELENG_4_8_BP RELENG_4_8 src The RELENG_* tags are restricted for use by the CVS-meisters and release engineers. A tag is CVS vernacular for a label that identifies the source at a specific point in time. By tagging the tree, we ensure that future release builders will always be able to use the same source we used to create the official FreeBSD Project releases. FreeBSD Development Branch FreeBSD 3.x STABLE Branch FreeBSD 4.x STABLE Branch FreeBSD 5.x STABLE Branch FreeBSD 6.x STABLE Branch Bumping up the Version Number Before the final release can be tagged, built, and released, the following files need to be modified to reflect the correct version of FreeBSD: doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.sgml doc/en_US.ISO8859-1/books/porters-handbook/book.sgml doc/share/sgml/freebsd.ent src/Makefile.inc1 src/UPDATING src/gnu/usr.bin/groff/tmac/mdoc.local src/release/Makefile src/release/doc/en_US.ISO8859-1/share/sgml/release.dsl src/release/doc/share/examples/Makefile.relnotesng src/release/doc/share/sgml/release.ent src/share/examples/cvsup/standard-supfile src/sys/conf/newvers.sh src/sys/sys/param.h src/usr.sbin/pkg_install/add/main.c www/en/docs.sgml www/en/cgi/ports.cgi ports/Tools/scripts/release/config The release notes and errata files also need to be adjusted for the new release (on the release branch) and truncated appropriately (on the stable/current branch): src/release/doc/en_US.ISO8859-1/relnotes/common/new.sgml src/release/doc/en_US.ISO8859-1/errata/article.sgml Sysinstall should be updated to note the number of available ports and the amount of disk space required for the Ports Collection[4]. This information is currently kept in src/usr.sbin/sysinstall/dist.c. After the release has been built, a number of file should be updated to announce the release to the world. doc/share/images/articles/releng/branches-relengX.pic www/share/sgml/advisories.xml www/share/sgml/includes.release.sgml www/share/sgml/includes.release.xsl www/en/releases/* www/en/releng/index.sgml www/en/news/news.xml www/en/search/web.atoz src/share/misc/bsd-family-tree Preparing a new major release branch (RELENG_<replaceable>X</replaceable>) When a new major release branch, such as RELENG_6 is branched from HEAD, some additional files must be updated before releases can be made from this new branch. src/share/examples/cvsup/stable-supfile - must be updated to point to the new -STABLE branch, when applicable. Creating Release Tags When the final release is ready, the following command will create the RELENG_4_8_0_RELEASE tag. /usr/src&prompt.root; cvs rtag -rRELENG_4_8 RELENG_4_8_0_RELEASE src The Documentation and Ports managers are responsible for tagging the respective trees with the RELEASE_4_8_0 tag. Occasionally, a last minute fix may be required after the final tags have been created. - In practice this isn't a problem, since CVS + In practice this is not a problem, since CVS allows tags to be manipulated with cvs tag -d tagname filename. It is very important that any last minute changes be tagged appropriately as part of the release. FreeBSD releases must always be reproduceable. Local hacks in the release engineer's environment are not acceptable. Release Building FreeBSD releases can be built by anyone with a fast machine and access to a source repository. (That should be everyone, since we offer anonymous CVS! See The Handbook for details.) The only special requirement is that the &man.vn.4; device must be available. (On -CURRENT, this device has been replaced by the new &man.md.4; memory disk driver.) If the device is not loaded into your kernel, then the kernel module should be automatically loaded when &man.vnconfig.8; is executed during the boot media creation phase. All of the tools necessary to build a release are available from the CVS repository in src/release. These tools aim to provide a consistent way to build FreeBSD releases. A complete release can actually be built with only a single command, including the creation of ISO images suitable for burning to CDROM, installation floppies, and an FTP install directory. This command is aptly named make release. <command>make release</command> To successfully build a release, you must first populate /usr/obj by running make world or simply make buildworld. The release target requires several variables be set properly to build a release: CHROOTDIR - The directory to be used as the chroot environment for the entire release build. BUILDNAME - The name of the release to be built. CVSROOT - The location of a CVS Repository. RELEASETAG - The CVS tag corresponding to the release you would like to build. If you do not already have access to a local CVS repository, then you may mirror one with CVSup. The supplied supfile, /usr/share/examples/cvsup/cvs-supfile, is a useful starting point for mirroring the CVS repository. If RELEASETAG is omitted, then the release will be built from the HEAD (a.k.a. -CURRENT) branch. Releases built from this branch are normally referred to as -CURRENT snapshots. There are many other variables available to customize the release build. Most of these variables are documented at the top of src/release/Makefile. The exact command used to build the official FreeBSD 4.7 (x86) release was: make release CHROOTDIR=/local3/release \ BUILDNAME=4.7-RELEASE \ CVSROOT=/host/cvs/usr/home/ncvs \ RELEASETAG=RELENG_4_7_0_RELEASE The release Makefile can be broken down into several distinct steps. Creation of a sanitized system environment in a separate directory hierarchy with make installworld. Checkout from CVS of a clean version of the system source, documentation, and ports into the release build hierarchy. Population of /etc and /dev in the chrooted environment. chroot into the release build hierarchy, to make it harder for the outside environment to taint this build. make world in the chrooted environment. Build of Kerberos-related binaries. Build GENERIC kernel. Creation of a staging directory tree where the binary distributions will be built and packaged. Build and installation of the documentation toolchain needed to convert the documentation source (SGML) into HTML and text documents that will accompany the release. Build and installation of the actual documentation (user manuals, tutorials, release notes, hardware compatibility lists, and so on.) Build of the crunched binaries used for installation floppies. Package up distribution tarballs of the binaries and sources. Create the boot media and a fixit floppy. Create FTP installation hierarchy. (optionally) Create ISO images for CDROM/DVD media. For more information about the release build infrastructure, please see &man.release.7;. Building <application>&xfree86;</application> &xfree86; is an important component for many desktop users. Prior to FreeBSD 4.6-RELEASE, releases used &xfree86; 3.X by default. The easiest way to build these versions is to use the src/release/scripts/X11/build_x.sh script. This script requires that &xfree86; and Tcl/Tk already be installed on the build host. After compiling the necessary X servers, the script will package all of the files into tarballs that &man.sysinstall.8; expects to find in the XF86336 directory of the installation media. Beginning with FreeBSD 4.6-RELEASE, &man.sysinstall.8; installs &xfree86; 4.X by default, as a set of normal packages. These can either be the packages generated by the package-building cluster or packages built from an appropriately tagged ports tree. Beginning with FreeBSD 5.3-RELEASE, &man.sysinstall.8; installs &xorg; packages instead of &xfree86; packages by default. It is important to remove any site-specific settings from /etc/make.conf. For example, it would be unwise to distribute binaries that were built on a system with CPUTYPE set to a specific processor. Contributed Software (<quote>ports</quote>) The FreeBSD Ports collection is a collection of over &os.numports; third-party software packages available for FreeBSD. The &a.portmgr; is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany official FreeBSD releases. The release engineering activities for our collection of third-party packages is beyond the scope of this document. A separate article, &art.re.pkgs;, covers this topic in depth. Release ISOs Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the BSDi/Wind River Systems/FreeBSD Mall official CDROM distributions. Each of the four discs must contain a README.TXT file that explains the contents of the disc, a CDROM.INF file that provides meta-data for the disc so that &man.sysinstall.8; can validate and use the contents, and a filename.txt file that provides a manifest for the disc. This manifest can be created with a simple command: /stage/cdrom&prompt.root; find . -type f | sed -e 's/^\.\///' | sort > filename.txt The specific requirements of each CD are outlined below. Disc 1 The first disc is almost completely created by make release. The only changes that should be made to the disc1 directory are the addition of a tools directory, &xfree86;, and as many popular third party software packages as will fit on the disc. The tools directory contains software that allow users to create installation floppies from other operating systems. This disc should be made bootable so that users of modern PCs do not need to create installation floppy disks. If an alternate version of &xfree86; is to be provided, then &man.sysinstall.8; must be updated to reflect the new location and installation instructions. The relevant code is contained in src/usr.sbin/sysinstall. Specifically, the files dist.c, menus.c, and config.c will need to be updated. Disc 2 The second disc is also largely created by make release. This disc contains a live filesystem that can be used from &man.sysinstall.8; to troubleshoot a FreeBSD installation. This disc should be bootable and should also contain a compressed copy of the CVS repository in the CVSROOT directory and commercial software demos in the commerce directory. Discs 3 and 4 The remaining two discs contain additional software packages for FreeBSD. The packages should be clustered so that a package and all of its dependencies are included on the same disc. More information about the creation of these discs is provided in the &art.re.pkgs; article. Multi-volume support Sysinstall supports multiple volume package installations. This requires that each disc have an INDEX file containing all of the packages on all volumes of a set, along with an extra field that indicates which volume that particular package is on. Each volume in the set must also have the CD_VOLUME variable set in the cdrom.inf file so that sysinstall can tell which volume is which. When a user attempts to install a package that is not on the current disc, sysinstall will prompt the user to insert the appropriate one. Distribution FTP Sites When the release has been thoroughly tested and packaged for distribution, the master FTP site must be updated. The official FreeBSD public FTP sites are all mirrors of a master server that is open only to other FTP sites. This site is known as ftp-master. When the release is ready, the following files must be modified on ftp-master: /pub/FreeBSD/releases/arch/X.Y-RELEASE/ The installable FTP directory as output from make release. /pub/FreeBSD/ports/arch/packages-X.Y-release/ The complete package build for this release. /pub/FreeBSD/releases/arch/X.Y-RELEASE/tools A symlink to ../../../tools. /pub/FreeBSD/releases/arch/X.Y-RELEASE/packages A symlink to ../../../ports/arch/packages-X.Y-release. /pub/FreeBSD/releases/arch/ISO-IMAGES/X.Y/X.Y-RELEASE-arch-*.iso The ISO images. The * is disc1, disc2, etc. Only if there is a disc1 and there is an alternative first installation CD (for example a stripped-down install with no windowing system) there may be a mini as well. For more information about the distribution mirror architecture of the FreeBSD FTP sites, please see the Mirroring FreeBSD article. It may take many hours to two days after updating ftp-master before a majority of the Tier-1 FTP sites have the new software depending on whether or not a package set got loaded at the same time. It is imperative that the release engineers coordinate with the &a.mirror-announce; before announcing the general availability of new software on the FTP sites. Ideally the release package set should be loaded at least four days prior to release day. The release bits should be loaded between 24 and 48 hours before the planned release time with other file permissions turned off. This will allow the mirror sites to download it but the general public will not be able to download it from the mirror sites. Mail should be sent to &a.mirror-announce; at the time the release bits get posted saying the release has been staged and giving the time that the mirror sites should begin allowing access. Be sure to include a time zone with the time, for example make it relative to GMT. CD-ROM Replication Coming soon: Tips for sending FreeBSD ISOs to a replicator and quality assurance measures to be taken. Extensibility Although FreeBSD forms a complete operating system, there is nothing that forces you to use the system exactly as we have packaged it up for distribution. We have tried to design the system to be as extensible as possible so that it can serve as a platform that other commercial products can be built on top of. The only rule we have about this is that if you are going to distribute FreeBSD with non-trivial changes, we encourage you to document your enhancements! The FreeBSD community can only help support users of the software we provide. We certainly encourage innovation in the form of advanced - installation and administration tools, for example, but we can't + installation and administration tools, for example, but we cannot be expected to answer questions about it. Creating Customized Boot floppies Many sites have complex requirements that may require additional kernel modules or userland tools be added to the installation floppies. The quick and dirty way to accomplish this would be to modify the staging directory of an existing make release build hierarchy: Apply patches or add additional files inside the chroot release build directory. rm ${CHROOTDIR}/usr/obj/usr/src/release/release.[59] rebuild &man.sysinstall.8;, the kernel, or whatever parts of the system your change affected. chroot ${CHROOTDIR} ./mk floppies New release floppies will be located in ${CHROOTDIR}/R/stage/floppies. Alternatively, the boot.flp make target can be called, or the filesystem creating script, src/release/scripts/doFS.sh, may be invoked directly. Local patches may also be supplied to the release build by defining the LOCAL_PATCH variable in make release. Scripting <command>sysinstall</command> The FreeBSD system installation and configuration tool, &man.sysinstall.8;, can be scripted to provide automated installs for large sites. This functionality can be used in conjunction with &intel; PXE[12] to bootstrap systems from the network, or via custom boot floppies with a sysinstall script. An example sysinstall script is available in the CVS tree as src/usr.sbin/sysinstall/install.cfg. Lessons Learned from FreeBSD 4.4 The release engineering process for 4.4 formally began on August 1st, 2001. After that date all commits to the RELENG_4 branch of FreeBSD had to be explicitly approved by the &a.re;. The first release candidate for the x86 architecture was released on August 16, followed by 4 more release candidates leading up to the final release on September 18th. The security officer was very involved in the last week of the process as several security issues were found in the earlier release candidates. A total of over 500 emails were sent to the &a.re; in little over a month. Our user community has made it very clear that the security and stability of a FreeBSD release should not be sacrificed for any self-imposed deadlines or target release dates. The FreeBSD Project has grown tremendously over its lifetime and the need for standardized release engineering procedures has never been more apparent. This will become even more important as FreeBSD is ported to new platforms. Future Directions It is imperative for our release engineering activities to scale with our growing userbase. Along these lines we are working very hard to document the procedures involved in producing FreeBSD releases. Parallelism - Certain portions of the release build are actually embarrassingly parallel. Most of the tasks are very I/O intensive, so having multiple high-speed disk drives is actually more important than using multiple processors in speeding up the make release process. If multiple disks are used for different hierarchies in the &man.chroot.2; environment, then the CVS checkout of the ports and doc trees can be happening simultaneously as the make world on another disk. Using a RAID solution (hardware or software) can significantly decrease the overall build time. Cross-building releases - Building IA-64 or Alpha release on x86 hardware? make TARGET=ia64 release. Regression Testing - We need better automated correctness testing for FreeBSD. Installation Tools - Our installation program has long since outlived its intended life span. Several projects are under development to provide a more advanced installation mechanism. The libh project was one such project that aimed to provide an intelligent new package framework and GUI installation program. Acknowledgements I would like to thank Jordan Hubbard for giving me the opportunity to take on some of the release engineering responsibilities for FreeBSD 4.4 and also for all of his work throughout the years making FreeBSD what it is today. Of course - the release wouldn't have been possible without all of the + the release would not have been possible without all of the release-related work done by &a.asami;, &a.steve;, &a.bmah;, &a.nik;, &a.obrien;, &a.kris;, &a.jhb; and the rest of the FreeBSD development community. I would also like to thank &a.rgrimes;, &a.phk;, and others who worked on the release engineering tools in the very early days of FreeBSD. This article was influenced by release engineering documents from the CSRG[13], the NetBSD Project[10], and John Baldwin's proposed release engineering process notes[11]. References [1] CVS - Concurrent Versions System [2] CVSup - The CVS-Optimized General Purpose Network File Distribution System [3] [4] FreeBSD Ports Collection [5] FreeBSD Committers [6] FreeBSD Core-Team [7] FreeBSD Handbook [8] GNATS: The GNU Bug Tracking System [9] FreeBSD PR Statistics [10] NetBSD Developer Documentation: Release Engineering [11] John Baldwin's FreeBSD Release Engineering Proposal [12] PXE Jumpstart Guide [13] Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: The Release Engineering of 4.3BSD