Committer Guide

Committer Guide The FreeBSD Documentation Project $FreeBSD$ 1999 2000 2001 The FreeBSD Documentation Project This document provides information for the FreeBSD committer community. All new committers should read this document before they start, and existing committers are strongly encouraged to review it from time to time. Administrative Details Main Repository Host freefall.FreeBSD.org Login Methods &man.ssh.1; Main CVSROOT /home/ncvs Main CVS Repository Meisters &a.jdp; and &a.peter; as well as &a.asami; for ports/ Mailing List developers@FreeBSD.org, cvs-committers@FreeBSD.org Noteworthy CVS Tags RELENG_3 (3.x-STABLE), RELENG_4 (4.x-STABLE), HEAD (-CURRENT) It is required that you use &man.ssh.1; or &man.telnet.1; with Kerberos 5 to connect to the repository hosts. These are generally more secure than plain &man.telnet.1; or &man.rlogin.1; since credential negotiation will always be encrypted. All traffic is encrypted by default with &man.ssh.1;. With utilities like &man.ssh-agent.1; and &man.scp.1; also available, &man.ssh.1; is also far more convenient. If you do not know anything about &man.ssh.1;, please see . CVS Operations It is assumed that you are already familiar with the basic operation of CVS. The CVS Repository Meisters (Peter Wemm and John Polstra) are the owners of the CVS repository and are responsible for any and all direct modification of it for the purposes of cleanup or fixing some grievous abuse of CVS by a committer. No one else should attempt to touch the repository directly. Should you cause some repository accident, say a bad cvs import or tag operation, do not attempt to fix it yourself! Mail or call John or Peter immediately and report the problem to one of them instead. The only ones allowed to directly fiddle the repository bits are the repomeisters. Satoshi Asami is also a repomeister for the ports/ portion of the tree. CVS operations are usually done by logging into freefall, making sure the CVSROOT environment variable is set to /home/ncvs, and then doing the appropriate check-out/check-in operations. If you wish to add something which is wholly new (like contrib-ified sources, etc), a script called easy-import is also provided for making the process easier. It automatically adds the new module entry, does the appropriate thing with cvs import, etc. – just run it without arguments and it will prompt you for everything it needs to know. Note that when you use CVS on freefall, you should set your umask to 2, as well as setting the CVSUMASK environment variable to 2. This ensures that any new files created by cvs add will have the correct permissions. If you add a file or directory and discover that the file in the repository has incorrect permissions (specifically, all files in the repository should be group writable by group ncvs), contact one of the repository meisters as described below. If you are familiar with remote CVS and consider yourself pretty studly with CVS in general, you can also do CVS operations directly from your own machine and local working sources. Just remember to set CVS_RSH to ssh so that you are using a relatively secure and reliable transport. If you have no idea what any of the above even means, on the other hand, then please stick with logging into freefall and applying your diffs with &man.patch.1;. If you need to use CVS add and delete operations in a manner that is effectively a mv operation, then a repository copy is in order rather than your CVS add and delete. In a repository copy, a CVS Meister will copy the file(s) to their new name and/or location and let you know when it is done. The purpose of a repository copy is to preserve file change history, or logs. We in the FreeBSD Project greatly value the change history CVS gives to the project. CVS reference information, tutorials, and FAQs can also be found at: http://www.cvshome.org/docs/, and the information in Karl Fogel's chapters from Open Source Development with CVS are also very useful. &a.des; also supplied the following mini primer for CVS. Check out a module with the co or checkout command. &prompt.user; cvs checkout shazam This checks out a copy of the shazam module. If there is no shazam module in the modules file, it looks for a top-level directory named shazam instead. Useful <command>cvs checkout</command> options Do not create empty directories Check out a single level, no subdirectories Check out revision, branch or tag rev Check out the sources as they were on date date

Practical FreeBSD examples: Check out the miscfs module, which corresponds to src/sys/miscfs: &prompt.user; cvs co miscfs You now have a directory named miscfs with subdirectories CVS, deadfs, devfs, and so on. One of these (linprocfs) is empty. Check out the same files, but with full path: &prompt.user; cvs co src/sys/miscfs You now have a directory named src, with subdirectories CVS and sys. src/sys has subdirectories CVS and miscfs, etc. Check out the same files, but prunes empty directories: &prompt.user; cvs co -P miscfs You now have a directory named miscfs with subdirectories CVS, deadfs, devfs... but note that there is no linprocfs subdirectory, because there are no files in it. Check out the directory miscfs, but none of the subdirectories: &prompt.root; cvs co -l miscfs You now have a directory named miscfs with just one subdirectory named CVS. Check out the miscfs module as it is in the 4.x branch: &prompt.user; cvs co -rRELENG_4 miscfs You can modify the sources and commit along this branch. Check out the miscfs module as it was in 3.4-RELEASE. &prompt.user; cvs co -rRELENG_3_4_0_RELEASE miscfs You will not be able to commit modifications, since RELENG_3_4_0_RELEASE is a point in time, not a branch. Check out the miscfs module as it was on Jan 15 2000. &prompt.user; cvs co -D'01/15/2000' miscfs You will not be able to commit modifications. Check out the miscfs module as it was one week ago. &prompt.user; cvs co -D'last week' miscfs You will not be able to commit modifications. Note that cvs stores metadata in subdirectories named CVS. Arguments to and are sticky, which means cvs will remember them later, e.g. when you do a cvs update. Check the status of checked-out files with the status command. &prompt.user; cvs status shazam This displays the status of the shazam file or of every file in the shazam directory. For every file, the status is given as one of: Up-to-date File is up-to-date and unmodified. Needs Patch File is unmodified, but there is a newer revision in the repository. Locally Modified File is up-to-date, but modified. Needs Merge File is modified, and there is a newer revision in the repository. File had conflicts on merge There were conflicts the last time this file was updated, and they have not been resolved yet. You will also see the local revision and date, the revision number of the newest applicable version (newest applicable because if you have a sticky date, tag or branch, it may not be the actual newest revision), and any sticky tags, dates or options. Once you have checked something out, update it with the update command. &prompt.user; cvs update shazam This updates the shazam file or the contents of the shazam directory to the latest version along the branch you checked out. If you checked out a point in time, does nothing unless the tags have moved in the repository or some other weird stuff is going on. Useful options, in addition to those listed above for checkout: Check out any additional missing directories. Update to head of main branch. More magic (see below). If you checked out a module with or , running cvs update with a different or argument or with will select a new branch, revision or date. The option clears all sticky tags, dates or revisions whereas and set new ones. Theoretically, specifying HEAD as argument to will give you the same result as , but that is just theory. The option is useful if: somebody has added subdirectories to the module you have checked out after you checked it out. you checked out with , and later change your mind and want to check out the subdirectories as well. you deleted some subdirectories and want to check them all back out. Watch the output of the cvs update with care. The letter in front of each file name indicates what was done with it: U The file was updated with no trouble. P The file was updated with no trouble (you will only see this when working against a remote repo). M The file had been modified, and was merged with no conflicts. C The file had been modified, and was merged with conflicts. Merging is what happens if you check out a copy of some source code, modify it, then someone else commits a change, and you run cvs update. CVS notices that you have made local changes, and tries to merge your changes with the changes between the version you originally checked out and the one you updated to. If the changes are to separate portions of the file, it will almost always work fine (though the result might not be syntactically or semantically correct). CVS will print an M in front of every locally modified file even if there is no newer version in the repository, so cvs update is handy for getting a summary of what you have changed locally. If you get a C, then your changes conflicted with the changes in the repository (the changes were to the same lines, or neighboring lines, or you changed the local file so much that cvs can not figure out how to apply the repository's changes). You will have to go through the file manually and resolve the conflicts; they will be marked with rows of <, = and > signs. For every conflict, there will be a marker line with seven < signs and the name of the file, followed by a chunk of what your local file contained, followed by a separator line with seven = signs, followed by the corresponding chunk in the repository version, followed by a marker line with seven > signs and the revision number you updated to. The option is slightly voodoo. It updates the local file to the specified revision as if you used , but it does not change the recorded revision number or branch of the local file. It is not really useful except when used twice, in which case it will merge the changes between the two specified versions into the working copy. For instance, say you commit a change to shazam/shazam.c in &os.current; and later want to MFC it. The change you want to MFC was revision 1.15: Check out the &os.stable; version of the shazam module: &prompt.user; cvs co -rRELENG_4 shazam Apply the changes between rev 1.14 and 1.15: &prompt.user; cvs update -j1.14 -j1.15 shazam/shazam.c You will almost certainly get a conflict because - of the $Id: article.sgml,v 1.87 2001-10-23 21:19:23 murray Exp $ (or in FreeBSD's case, + of the $Id: article.sgml,v 1.88 2001-10-31 23:10:30 chern Exp $ (or in FreeBSD's case, $FreeBSD$) lines, so you will have to edit the file to resolve the conflict (remove the marker lines and - the second $Id: article.sgml,v 1.87 2001-10-23 21:19:23 murray Exp $ line, leaving the original - $Id: article.sgml,v 1.87 2001-10-23 21:19:23 murray Exp $ line intact). + the second $Id: article.sgml,v 1.88 2001-10-31 23:10:30 chern Exp $ line, leaving the original + $Id: article.sgml,v 1.88 2001-10-31 23:10:30 chern Exp $ line intact). View differences between the local version and the repository version with the diff command. &prompt.user; cvs diff shazam shows you every modification you have made to the shazam file or module. Useful <command>cvs diff</command> options Uses the unified diff format. Shows missing or added files.

You always want to use , since unified diffs are much easier to read than almost any other diff format (in some circumstances, context diffs may be better, but they are much bulkier). A unified diff consists of a series of hunks. Each hunk begins with a line that starts with two @ signs and specifies where in the file the differences are and how many lines they span. This is followed by a number of lines; some (preceded by a blank) are context; some (preceded by a - sign) - are outtakes and some (preceded by a +) are + are out takes and some (preceded by a +) are additions. You can also diff against a different version than the one you checked out by specifying a version with or as in checkout or update, or even view the diffs between two arbitrary versions (with no regard for what you have locally) by specifying two versions with or . View log entries with the log command. &prompt.user; cvs log shazam If shazam is a file, this will print a header with information about this file, such as where in the repository this file is stored, which revision is the HEAD for this file, what branches this file is in, and any tags that are valid for this file. Then, for each revision of this file, a log message is printed. This includes the date and time of the commit, who did the commit, how many lines were added and/or deleted, and finally the log message that the committer who did the change wrote. If shazam is a directory, then the log information described above is printed for each file in the directory in turn. Unless you give the to log, the log for all subdirectories of shazam is printed too, in a recursive manner. Use the log command to view the history of one or more files, as it is stored in the CVS repository. You can even use it to view the log message of a specific revision, if you add the to the log command: &prompt.user; cvs log -r1.2 shazam This will print only the log message for revision 1.2 of file shazam if it is a file, or the log message for revision 1.2 of each file under shazam if it is a directory. See who did what with the annotate command. This command shows you each line of the specified file or files, along with which user most recently changed that line. &prompt.user; cvs annotate shazam Add new files with the add command. Create the file, cvs add it, then cvs commit it. Similarly, you can add new directories by creating them and then cvs adding them. Note that you do not need to commit directories. Remove obsolete files with the remove command. Remove the file, then cvs rm it, then cvs commit it. Commit with the commit or checkin command. Useful <command>cvs commit</command> options Force a commit of an unmodified file. Specify a commit message on the command line rather than invoking an editor.

Use the option if you realize that you left out important information from the commit message. Good commit messages are important. They tell others why you did the changes you did, not just right here and now, but months or years from now when someone wonders why some seemingly illogical or inefficient piece of code snuck into your source file. It is also an invaluable aid to deciding which changes to MFC and which not to MFC. Do not waste space in the commit messages explaining what you did. That is what cvs diff is for. Instead, tell us why you did it. Avoid committing several unrelated changes in one go. It makes merging difficult, and also makes it harder to determine which change is the culprit if a bug crops up. Avoid committing style or whitespace fixes and functionality fixes in one go. It makes merging difficult, and also makes it harder to understand just what functional changes were made. Avoid committing changes to multiple files in one go with a generic, vague message. Instead, commit each file (or small groups of files) with tailored commit messages. Before committing, always: verify which branch you are committing to, using cvs status. review your diffs, using cvs diff Also, ALWAYS specify which files to commit explicitly on the command line, so you do not accidentally commit other files than the ones you intended - cvs commit with no arguments will commit every modification in your current working directory and every subdirectory. Additional tips and tricks: You can place commonly used options in your ~/.cvsrc, like this: cvs -z3 diff -Nu update -Pd checkout -P This example says: always use compression level 3 when talking to a remote server. This is a life-saver when working over a slow connection. always use the (show added or removed files) and (unified diff format) options to &man.diff.1;. always use the (prune empty directories) and (check out new directories) options when updating. always use the (prune empty directories) option when checking out. Use Eivind Eklund's cdiff script to view unidiffs. It is a wrapper for &man.less.1; that adds ANSI color codes to make hunk headers, outtakes and additions stand out; context and garbage are unmodified. It also expands tabs properly (tabs often look wrong in diffs because of the extra character in front of each line). http://people.FreeBSD.org/~eivind/cdiff Simply use it instead of &man.more.1; or &man.less.1;: &prompt.user; cvs diff -Nu shazam | cdiff Alternatively some editors like &man.vim.1; (ports/editors/vim5) have color support and when used as a pager with color syntax highlighting switched on will highlight many types of file, including diffs, patches, and cvs/rcs logs. &prompt.user; echo "syn on" >> ~/.vimrc &prompt.user; cvs diff -Nu shazam | vim - &prompt.user; cvs log shazam | vim - CVS is old, arcane, crufty and buggy, and sometimes exhibits non-deterministic behavior which some claim as proof that it is actually merely the Newtonian manifestation of a sentient transdimensional entity. It is not humanly possible to know its every quirk inside out, so do not be afraid to ask the resident AI (cvs@FreeBSD.org) for help when you screw up. Do not leave the cvs commit command in commit message editing mode for too long (more than 2-3 minutes). It locks the directory you are working with and will prevent other developers from committing into the same directory. If you have to type a long commit message, type it before executing cvs commit, and insert it into the commit message. Conventions and Traditions As a new committer there are a number of things you should do first. Add yourself to the Developers section of the Contributors List and remove yourself from the Additional Contributors section. This is a relatively easy task, but remains a good first test of your CVS skills. Add an entry for yourself to www/en/news/news.xml. Look for the other entries that look like A new committer and follow the format. If you have a PGP or GnuPG key, you may want to add it to doc/en_US.ISO8859-1/books/handbook/pgpkeys. Some people add an entry for themselves to ports/astro/xearth/files/freebsd.committers.markers. Some people add an entry for themselves to src/usr.bin/calendar/calendars/calendar.freebsd. Introduce yourself to the other committers, otherwise no one will have any idea who you are or what you are working on. You do not have to write a comprehensive biography, just write a paragraph or two about who you are and what you plan to be working on as a committer in FreeBSD. Email this to developers@FreeBSD.org and you will be on your way! Log into hub.FreeBSD.org and create a /var/forward/user (where user is your username) file containing the e-mail address where you want mail addressed to yourusername@FreeBSD.org to be forwarded. This includes all of the commit messages as well as any other mail addressed to cvs-committers@FreeBSD.org and developers@FreeBSD.org. Really large mailboxes which have taken up permanent residence on hub often get accidently truncated without warning, so forward it or read it and you will not lose it. All new committers also have a mentor assigned to them for the first few months. Your mentor is more or less responsible for explaining anything which is confusing to you and is also responsible for your actions during this initial period. If you make a bogus commit, it is only going to embarrass your mentor and you should probably make it a policy to pass at least your first few commits by your mentor before committing it to the repository. All commits should go to &os.current; first before being merged to &os.stable;. No major new features or high-risk modifications should be made to the &os.stable; branch. Developer Relations If you are working directly on your own code or on code which is already well established as your responsibility, then there is probably little need to check with other committers before jumping in with a commit. If you see a bug in an area of the system which is clearly orphaned (and there are a few such areas, to our shame), the same applies. If, however, you are about to modify something which is clearly being actively maintained by someone else (and it is only by watching the cvs-committers mailing list that you can really get a feel for just what is and is not) then consider sending the change to them instead, just as you would have before becoming a committer. For ports, you should contact the listed MAINTAINER in the Makefile. For other parts of the repository, if you are unsure who the active maintainer might be, it may help to scan the output of cvs log to see who has committed changes in the past. &a.fenner; has written a nice shell script that can help determine who the active maintainer might be. It lists each person who has committed to a given file along with the number of commits each person has made. It can be found on freefall at ~fenner/bin/whodid. If your queries go unanswered or the committer otherwise indicates a lack of proprietary interest in the area affected, go ahead and commit it. If you are unsure about a commit for any reason at all, have it reviewed by -hackers before committing. Better to have it flamed then and there rather than when it is part of the CVS repository. If you do happen to commit something which results in controversy erupting, you may also wish to consider backing the change out again until the matter is settled. Remember – with CVS we can always change it back. GNATS The FreeBSD Project utilizes GNATS for tracking bugs and change requests. Be sure that if you commit a fix or suggestion found in a GNATS PR, you use edit-pr pr-number on freefall to close it. It is also considered nice if you take time to close any PRs associated with your commits, if appropriate. You can also make use of &man.send-pr.1; yourself for proposing any change which you feel should probably be made, pending a more extensive peer-review first. You can find out more about GNATS at: http://www.cs.utah.edu/csinfo/texinfo/gnats/gnats.html http://www.FreeBSD.org/support.html http://www.FreeBSD.org/send-pr.html &man.send-pr.1; You can run a local copy of GNATS, and then integrate the FreeBSD GNATS tree in to it using CVSup. Then you can run GNATS commands locally, or use other interfaces, such as tkgnats. This lets you query the PR database without needing to be connected to the Internet. Using a local GNATS tree If you are not already downloading the GNATS tree, add this line to your supfile, and re-sup. Note that since GNATS is not under CVS control it has no tag, so if you are adding it to your existing supfile it should appear before any tag= entry as these remain active once set. gnats release=current prefix=/usr This will place the FreeBSD GNATS tree in /usr/gnats. You can use a refuse file to control which categories to receive. For example, to only receive docs PRs, put this line in /usr/local/etc/cvsup/sup/refuse The precise path depends on the *default base setting in your supfile. . gnats/[a-ce-z]* The rest of these examples assume you have only supped the docs category. Adjust them as necessary, depending on the categories you are synching. Install the GNATS port from ports/databases/gnats. This will place the various GNATS directories under $PREFIX/share/gnats. Symlink the GNATS directories you are supping under the version of GNATS you have installed. &prompt.root; cd /usr/local/share/gnats/gnats-db &prompt.root; ln -s /usr/gnats/docs Repeat as necessary, depending on how many GNATS categories you are synching. Update the GNATS categories file with these categories. The file is $PREFIX/share/gnats/gnats-db/gnats-adm/categories. # This category is mandatory pending:Category for faulty PRs:gnats-admin: # # FreeBSD categories # docs:Documentation Bug:nik: Run $PREFIX/libexec/gnats/gen-index to recreate the GNATS index. The output has to be redirected to $PREFIX/share/gnats/gnats-db/gnats-adm/index. You can do this periodically from &man.cron.8;, or run &man.cvsup.1; from a shell script that does this as well. &prompt.root; /usr/local/libexec/gnats/gen-index \ > /usr/local/share/gnats/gnats-db/gnats-adm/index Test the configuration by querying the PR database. This command shows open docs PRs. &prompt.root; query-pr -c docs -s open Other interfaces, like ports/databases/tkgnats should also work nicely. Pick a PR and close it. This procedure only works to allow you to view and query the PRs locally. To edit or close them you will still have to log in to freefall and do it from there. Who's Who Besides Peter Wemm and John Polstra, the repository meisters, there are other FreeBSD project members whom you will probably get to know in your role as a committer. Briefly, and by no means all-inclusively, these are: &a.asami; Satoshi is the Ports Wraith, meaning that he has ultimate authority over any modifications to the ports collection or the ports skeleton makefiles. He is also the one responsible for administering ports freezes before the releases. &a.nik; Nik oversees the Doc. Project. As well as writing documentation he put together the infrastructure under doc/share/mk and the stylesheets and related code under doc/share/sgml. If you have got questions about these you are encouraged to send them via the doc@FreeBSD.org mailing list. Committers interested in contributing to the documentation should familiarise - themself with the + themselves with the Documentation Project Primer. &a.ru; Ruslan is Mister &man.mdoc.7;. If you are writing a man page and need some advice on the structure, or the markup, ask Ruslan. &a.bde; Bruce is the Style Police-Meister. When you do a commit that could have been done better, Bruce will be there to tell you. Be thankful that someone is. Bruce is also very knowledgeable on the various standards applicable to FreeBSD. &a.gallatin; &a.mjacob; &a.dfr; &a.obrien; These are the primary developers and overseers of the DEC Alpha AXP platform. &a.dg; David is the overseer of the VM system. If you have a VM system change in mind, coordinate it with David. &a.jkh; Jordan is the release engineer. He is responsible for setting release deadlines and controlling the release process. During code freezes, he also has final authority on all changes to the system for whichever branch is pending release status. If there is something you want merged from &os.current; to &os.stable; (whatever values those may have at any given time), he is also the one to talk to about it. &a.bmah; Bruce is keeper of the release notes (src/release/texts/* or src/release/doc/*, as appropriate). If you commit a change that you think is worthy of mention in the release notes, please make sure Bruce knows about it. Better still, send him a patch with your suggested commentary for the release notes. &a.obrien; David is the unofficial src/contrib-Meister. If you have something significant you would like to do there, you should probably coordinate it with David first. Please consult him before importing into src/contrib if you have never done this before in the FreeBSD CVS repository. Also if you need to commit to something you do not maintain in src/contrib and it is unclear who the maintainer / point of contact is. (It is also not a bad idea to consult David if you need to make a non-import commit to something you maintain in src/contrib and you are new to how FreeBSD does things.) &a.brian; Official maintainer of /usr/sbin/ppp. &a.wollman; If you need advice on obscure network internals or are not sure of some potential change to the networking subsystem you have in mind, Garrett is someone to talk to. Garrett is also very knowledgeable on the various standards applicable to FreeBSD. &a.committers; cvs-committers is the entity that CVS uses to send you all your commit messages. You should never send email directly to this list. You should only send replies to this list when they are short and are directly related to a commit. &a.developers; developers is all committers. This list was created to be a forum for the committers community issues. Examples are Core voting, announcements, etc... developers@FreeBSD.org is not intended as a place for code reviews or a replacement for arch@FreeBSD.org or audit@FreeBSD.org. In fact using it as such hurts the FreeBSD Project as it gives a sense of a closed list where general decisions affecting all of the FreeBSD using community are made with out being open. SSH Quick-Start Guide If you are using FreeBSD 4.0 or later, OpenSSH is included in the base system. If you are using an earlier release, update and install one of the SSH ports. In general, you will probably want to get OpenSSH from the port in /usr/ports/security/openssh. You may also wish to check out the original ssh1 in /usr/ports/security/ssh, but make certain you pay attention to its license. Note that both of these ports cannot be installed at the same time. If you do not wish to type your password in every time you use &man.ssh.1;, and you use RSA keys to authenticate, &man.ssh-agent.1; is there for your convenience. If you want to use &man.ssh-agent.1;, make sure that you run it before running other applications. X users, for example, usually do this from their .xsession or .xinitrc file. See &man.ssh-agent.1; for details. Generate a key pair using &man.ssh-keygen.1;. The key pair will wind up in the $HOME/.ssh directory. Send your public key ($HOME/.ssh/identity.pub) to the person setting you up as a committer so it can be put into your authorized_keys file in your home directory on freefall (i.e. $HOME/.ssh/authorized_keys). Now you should be able to use &man.ssh-add.1; for authentication once per session. This will prompt you for your private key's pass phrase, and then store it in your authentication agent (&man.ssh-agent.1;). If you no longer wish to have your key stored in the agent, issuing ssh-add -d will remove it. Test by doing something such as ssh freefall.FreeBSD.org ls /usr. For more information, see /usr/ports/security/openssh, &man.ssh.1;, &man.ssh-add.1;, &man.ssh-agent.1;, &man.ssh-keygen.1;, and &man.scp.1;. The FreeBSD Committers' Big List of Rules Respect other committers. Respect other contributors. Discuss any significant change before committing. Respect existing maintainers (if listed in the MAINTAINER field in Makefile or in the MAINTAINER file in the top-level directory). Never touch the repository directly. Ask a Repomeister. Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. Changes go to &os.current; before &os.stable; unless specifically permitted by the release engineer or unless they are not applicable to &os.current;. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in &os.current; for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the &os.stable; branch as outlined for the maintainer in rule #6. Do not fight in public with other committers; it looks bad. If you must strongly disagree about something, do so only in private. Respect all code freezes and read the committers mailing list in a timely manner so you know when a code freeze is in effect. When in doubt on any procedure, ask first! Test your changes before committing them. Do not commit to anything under the src/contrib, src/crypto, and src/sys/contrib trees without explicit approval from the respective maintainer(s). As noted, breaking some of these rules can be grounds for suspension or, upon repeated offense, permanent removal of commit privileges. Three or more members of core acting in unison, have the power to temporarily suspend commit privileges until -core as a whole has the chance to review the issue. In case of an emergency (a committer doing damage to the repository), a temporary suspension may also be done by the repository meisters or any other member of core who may happen to be awake at the time. Only core as a whole has the authority to suspend commit privileges for any significant length of time or to remove them permanently, the latter generally only being done after consultation with committers. This rule does not exist to set core up as a bunch of cruel dictators who can dispose of committers as casually as empty soda cans, but to give the project a kind of safety fuse. If someone is seriously out of control, it is important to be able to deal with this immediately rather than be paralyzed by debate. In all cases, a committer whose privileges are suspended or revoked is entitled to a hearing, the total duration of the suspension being determined at that time. A committer whose privileges are suspended may also request a review of the decision after 30 days and every 30 days thereafter (unless the total suspension period is less than 30 days). A committer whose privileges have been revoked entirely may request a review after a period of 6 months have elapsed. This review policy is strictly informal and, in all cases, core reserves the right to either act on or disregard requests for review if they feel their original decision to be the right one. In all other aspects of project operation, core is a subset of committers and is bound by the same rules. Just because someone is in core does not mean that they have special dispensation to step outside of any of the lines painted here; core's special powers only kick in when it acts as a group, not on an individual basis. As individuals, we are all committers first and core second. Details Respect other committers. This means that you need to treat other committers as the peer-group developers that they are. Despite our occasional attempts to prove the contrary, one does not get into committers by being stupid and nothing rankles more than being treated that way by one of your peers. Whether we always feel respect for one another or not (and everyone has off days), we still have to treat other committers with respect at all times or the whole team structure rapidly breaks down. Being able to work together long term is this project's greatest asset, one far more important than any set of changes to the code, and turning arguments about code into issues that affect our long-term ability to work harmoniously together is just not worth the trade-off by any conceivable stretch of the imagination. To comply with this rule, do not send email when you are angry or otherwise behave in a manner which is likely to strike others as needlessly confrontational. First calm down, then think about how to communicate in the most effective fashion for convincing the other person(s) that your side of the argument is correct, do not just blow off some steam so you can feel better in the short term at the cost of a long-term flame war. Not only is this very bad energy economics, but repeated displays of public aggression which impair our ability to work well together will be dealt with severely by the project leadership and may result in suspension or termination of your commit privileges. That is never an option which the project's leadership enjoys in the slightest, but unity comes first. No amount of code or good advice is worth trading that away. Respect other contributors. You were not always a committer. At one time you were a contributor. Remember that at all times. Remember what it was like trying to get help and attention. Do not forget that your work as a contributor time was very important to you. Remember what it was like. Do not discourage, belittle, or demean contributors. Treat them with respect. They are our committers in waiting. They are every bit as important to the project as committers. Their contributions are as valid and as important as your own. After all, you made many contributions before you became a committer. Always remember that. Consider the points raised under and apply them also to contributors. Discuss any significant change before committing. The CVS repository is not where changes should be initially submitted for correctness or argued over, that should happen first in the mailing lists and then committed only once something resembling consensus has been reached. This does not mean that you have to ask permission before correcting every obvious syntax error or man page misspelling, simply that you should try to develop a feel for when a proposed change is not quite such a no-brainer and requires some feedback first. People really do not mind sweeping changes if the result is something clearly better than what they had before, they just do not like being surprised by those changes. The very best way of making sure that you are on the right track is to have your code reviewed by one or more other committers. When in doubt, ask for review! Respect existing maintainers if listed. Many parts of FreeBSD are not owned in the sense that any specific individual will jump up and yell if you commit a change to their area, but it still pays to check first. One convention we use is to put a maintainer line in the Makefile for any package or subtree which is being actively maintained by one or more people; see http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/developers-handbook/policies.html for documentation on this. Where sections of code have several maintainers, commits to affected areas by one maintainer need to be reviewed by at least one other maintainer. In cases where the maintainer-ship of something is not clear, you can also look at the CVS logs for the file(s) in question and see if someone has been working recently or predominantly in that area. Other areas of FreeBSD fall under the control of someone who manages an overall category of FreeBSD evolution, such as internationalization or networking. See http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributors/staff-who.html for more information on this. Never touch the repository directly. Ask a Repomeister. This is pretty clear - you are not allowed to make direct modifications to the CVS repository, period. In case of difficulty, ask one of the repository meisters by sending mail to cvs@FreeBSD.org and simply wait for them to fix the problem and get back to you. Do not attempt to fix the problem yourself! If you are thinking about putting down a tag or doing a new import of code on a vendor branch, you might also find it useful to ask for advice first. A lot of people get this wrong the first few times and the consequences are expensive in terms of files touched and angry CVSup/CTM folks who are suddenly getting a lot of changes sent over unnecessarily. Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. This may be hard to swallow in times of conflict (when each side is convinced that they are in the right, of course) but CVS makes it unnecessary to have an ongoing dispute raging when it is far easier to simply reverse the disputed change, get everyone calmed down again and then try and figure out how best to proceed. If the change turns out to be the best thing after all, it can be easily brought back. If it turns out not to be, then the users did not have to live with the bogus change in the tree while everyone was busily debating its merits. People very very rarely call for back-outs in the repository since discussion generally exposes bad or controversial changes before the commit even happens, but on such rare occasions the back-out should be done without argument so that we can get immediately on to the topic of figuring out whether it was bogus or not. Changes go to &os.current; before &os.stable; unless specifically permitted by the release engineer or unless they are not applicable to &os.current;. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in &os.current; for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the &os.stable; branch as outlined in rule #6. This is another do not argue about it issue since it is the release engineer who is ultimately responsible (and gets beaten up) if a change turns out to be bad. Please respect this and give the release engineer your full cooperation when it comes to the &os.stable; branch. The management of &os.stable; may frequently seem to be overly conservative to the casual observer, but also bear in mind the fact that conservatism is supposed to be the hallmark of &os.stable; and different rules apply there than in &os.current;. There is also really no point in having &os.current; be a testing ground if changes are merged over to &os.stable; immediately. Changes need a chance to be tested by the &os.current; developers, so allow some time to elapse before merging unless the &os.stable; fix is critical, time sensitive or so obvious as to make further testing - unnecessary (spelling fixes to manpages, obvious bug/typo + unnecessary (spelling fixes to man pages, obvious bug/typo fixes, etc.) In other words, apply common sense. Do not fight in public with other committers; it looks bad. If you must strongly disagree about something, do so only in private. This project has a public image to uphold and that image is very important to all of us, especially if we are to continue to attract new members. There will be occasions when, despite everyone's very best attempts at self-control, tempers are lost and angry words are exchanged, and the best we can do is try and minimize the effects of this until everyone has cooled back down. That means that you should not air your angry words in public and you should not forward private correspondence to public mailing lists or aliases. What people say one-to-one is often much less sugar-coated than what they would say in public, and such communications therefore have no place there - they only serve to inflame an already bad situation. If the person sending you a flame-o-gram at least had the grace to send it privately, then have the grace to keep it private yourself. If you feel you are being unfairly treated by another developer, and it is causing you anguish, bring the matter up with core rather than taking it public. We will do our best to play peace makers and get things back to sanity. In cases where the dispute involves a change to the codebase and the participants do not appear to be reaching an amicable agreement, core may appoint a mutually-agreeable 3rd party to resolve the dispute. All parties involved must then agree to be bound by the decision reached by this 3rd party. Respect all code freezes and read the committers mailing list on a timely basis so you know when they are. Committing changes during a code freeze is a really big mistake and committers are expected to keep up-to-date on what is going on before jumping in after a long absence and committing 10 megabytes worth of accumulated stuff. People who abuse this on a regular basis will have their commit privileges suspended until they get back from the FreeBSD Happy Reeducation Camp we run in Greenland. When in doubt on any procedure, ask first! Many mistakes are made because someone is in a hurry and just assumes they know the right way of doing something. If you have not done it before, chances are good that you do not actually know the way we do things and really need to ask first or you are going to completely embarrass yourself in public. There is no shame in asking how in the heck do I do this? We already know you are an intelligent person; otherwise, you would not be a committer. Test your changes before committing them. This may sound obvious, but if it really were so obvious then we probably would not see so many cases of people clearly not doing this. If your changes are to the kernel, make sure you can still compile both GENERIC and LINT. If your changes are anywhere else, make sure you can still make world. If your changes are to a branch, make sure your testing occurs with a machine which is running that code. If you have a change which also may break another architecture, be sure and test on all supported architectures. Currently, this is only the x86 and the Alpha so it is pretty easy to do. If you need to test on the AXP, your account on beast.FreeBSD.org will let you compile and test Alpha binaries/kernels/etc. As other architectures are added to the FreeBSD supported platforms list, the appropriate shared testing resources will be made available. Do not commit to anything under the src/contrib, src/crypto, and src/sys/contrib trees without explicit approval from the respective maintainer(s). The trees mentioned above are for contributed software usually imported onto a vendor branch. Committing something there, even if it does not take the file off the vendor branch, may cause unnecessary headaches for those responsible for maintaining that particular piece of software. Thus, unless you have explicit approval from the maintainer (or you are the maintainer), do not commit there! Please note that this does not mean you should not try to improve the software in question; you are still more than welcome to do so. Ideally, you should submit your patches to the vendor. If your changes are FreeBSD-specific, talk to the maintainer; they may be willing to apply them locally. But whatever you do, do not commit there by yourself! Contact the &a.core; if you wish to take up maintainership of an unmaintained part of the tree. Other Suggestions When committing documentation changes, use a spell checker before committing. :) For all SGML docs, you should also verify that your formatting directives are correct by running make lint. For all on-line manual pages, run manck (from ports) over the man page to verify all of the cross references and file references are correct and that the man page has all of the appropriate MLINKs installed. Do not mix style fixes with new functionality. A style fix is any change which does not modify the functionality of the code. Mixing the changes obfuscates the functionality change when using cvs diff, which can hide any new bugs. Do not include whitespace changes with content changes in commits to doc/ or www/. The extra clutter in the diffs makes the translators' job much more difficult. Instead, make any style or whitespace changes in separate commits that are clearly labeled as such in the commit message. Deprecating Features When it is necessary to remove functionality from software in the base system the following guidelines should be followed whenever possible: Mention is made in the manual page and possibly the release notes that the option, utility, or interface is deprecated. Use of the deprecated feature generates a warning. The option, utility, or interface is preserved until the next major (point zero) release. The option, utility, or interface is removed and no longer documented. It is now obsolete. It is also generally a good idea to note its removal in the release notes. Ports Specific FAQ Adding a New Port How do I add a new port? First, please read the section about repository copy. The easiest way to add a new port is to use the addport script on freefall. It will add a port from the directory you specify, determining the category automatically from the port Makefile. It will also add an entry to the CVSROOT/modules file and the port's category Makefile. It was written by &a.mharo; and &a.will;, but Will is the current maintainer so please send questions/patches about addport to him. Any other things I need to know when I add a new port? Check the port, preferably to make sure it compiles and packages correctly. This is the recommended sequence: &prompt.root; make install &prompt.root; make package &prompt.root; make deinstall &prompt.root; pkg_add package you built above &prompt.root; make deinstall &prompt.root; make reinstall &prompt.root; make package The Porters Handbook contains more detailed instructions. Use &man.portlint.1; to check the syntax of the port. You do not necessarily have to eliminate all warnings but make sure you have fixed the simple ones. If the port came from a submitter who has not contributed to the project before, add that person's name to the Additional Contributors section of the FreeBSD Contributors List. Close the PR if the port came in as a PR. To close a PR, just do edit-pr PR# on freefall and change the state from open to closed. You will be asked to enter a log message and then you are done. Repository Copies When do we need a repository copy? When you want to add a port that is related to any port that is already in the tree in a separate directory, please send mail to the ports manager asking about it. Here related means it is a different version or a slightly modified version. Examples are print/ghostscript* (different versions) and x11-wm/windowmaker* (English-only and internationalized version). Another example is when a port is moved from one subdirectory to another, or when you want to change the name of a directory because the author(s) renamed their software even though it is a descendant of a port already in a tree. When do we not need a repository copy? When there is no history to preserve. If a port is added into a wrong category and is moved immediately, it suffices to simply cvs remove the old one and addport the new one. What do I need to do? Send mail to the ports manager, who will do a copy from the old location/name to the new location/name. You will then get a notice, at which point you are expected to perform the following: cvs remove the old port (if necessary) Adjust the parent (category) Makefile Update CVSROOT/modules If other ports depend on the updated port, change their Makefiles' dependency lines If the port changed categories, modify the CATEGORIES line of the port's Makefile accordingly Ports Freeze What is a ports freeze? Before a release, it is necessary to restrict commits to the ports tree for a short period of time while the packages and the release itself are being built. This is to ensure consistency among the various parts of the release, and is called the ports freeze. How long is a ports freeze? Usually an hour or two. What does it mean to me? During the ports freeze, you are not allowed to commit anything to the tree without explicit approval from the ports manager. Explicit approval here means either of the following: You asked the ports manager and got a reply saying, Go ahead and commit it. The ports manager sent a mail to you or the mailing lists during the ports freeze pointing out that the port is broken and has to be fixed. Note that you do not have implicit permission to fix a port during the freeze just because it is broken. How do I know when the ports freeze starts? The ports manager will send out warning messages to the freebsd-ports@FreeBSD.org and cvs-committers@FreeBSD.org mailing lists announcing the start of the impending release, usually two or three weeks in advance. The exact starting time will not be determined until a few days before the actual release. This is because the ports freeze has to be synchronized with the release, and it is usually not known until then when exactly the release will be rolled. When the freeze starts, there will be another announcement to the cvs-committers@FreeBSD.org list, of course. How do I know when the ports freeze ends? A few hours after the release, the ports manager will send out a mail to the freebsd-ports@FreeBSD.org and cvs-committers@FreeBSD.org mailing lists announcing the end of the ports freeze. Note that the release being cut does not automatically end the freeze. We have to make sure there will not be any last minute snafus that result in an immediate re-rolling of the release. Miscellaneous Questions How do I know if my port is building correctly or not? First, go check http://bento.FreeBSD.org/~asami/errorlogs/. There you will find error logs from the latest package building runs on 3-stable, 4-stable and 5-current. However, just because the port does not show up there does not mean it is building correctly. (One of the dependencies may have failed, for instance.) Here are the relevant directories on bento, so feel free to dig around. /a/asami/portbuild/3/errors error logs from latest 3-stable run /logs all logs from latest 3-stable run /packages packages from latest 3-stable run /bak/errors error logs from last complete 3-stable run /bak/logs all logs from last complete 3-stable run /bak/packages packages from last complete 3-stable run /4/errors error logs from latest 4-stable run /logs all logs from latest 4-stable run /packages packages from latest 4-stable run /bak/errors error logs from last complete 4-stable run /bak/logs all logs from last complete 4-stable run /bak/packages packages from last complete 4-stable run /5/errors error logs from latest 5-current run /logs all logs from latest 5-current run /packages packages from latest 5-current run /bak/errors error logs from last complete 5-current run /bak/logs all logs from last complete 5-current run /bak/packages packages from last complete 5-current run Basically, if the port shows up in packages, or it is in logs but not in errors, it built fine. (The errors directories are what you get from the web page.) I added a new port. Do I need to add it to the INDEX? No. The ports manager will regenerate the INDEX and commit it every few days. Are there any other files I am not allowed to touch? Any file directly under ports/, or any file under a subdirectory that starts with an uppercase letter (Mk/, Tools/, etc.). In particular, the ports manager is very protective of ports/Mk/bsd.port*.mk so do not commit changes to those files unless you want to face his wra(i)th. Miscellaneous Questions Why are trivial or cosmetic changes to files on a vendor branch a bad idea? From now on, every new vendor release of that file will need to have patches merged in by hand. From now on, every new vendor release of that file will need to have patches verified by hand. The option does not work very well. Ask &a.obrien; for horror stories. How do I add a new file to a CVS branch? To add a file onto a branch, simply checkout or update to the branch you want to add to and then add the file using cvs add as you normally would. For example, if you wanted to MFC the file src/sys/alpha/include/smp.h from HEAD to RELENG_4 and it does not exist in RELENG_4 yet, you would use the following steps: MFC'ing a New File &prompt.user; cd sys/alpha/include &prompt.user; cvs update -rRELENG_4 cvs update: Updating . U clockvar.h U console.h ... &prompt.user; cvs update -kk -Ap smp.h > smp.h =================================================================== Checking out smp.h RCS: /usr/cvs/src/sys/alpha/include/smp.h,v VERS: 1.1 *************** &prompt.user; cvs add smp.h cvs add: scheduling file `smp.h' for addition on branch `RELENG_4' cvs add: use 'cvs commit' to add this file permanently &prompt.user; cvs commit What meta information should I include in a commit message? As well as including an informative message with each commit you may need to include some additional information as well. This information consists of one or more lines containing the the key word or phrase, a colon, tabs for formatting, and then the additional information. The key words or phrases are: PR: The problem report (if any) which is affected (typically, by being closed) by this commit. Submitted by: The name and e-mail address of the person that submitted the fix. Reviewed by: The name and e-mail address of the person or people that reviewed the change. If a patch was submitted to a mailing list for review, and the review was favourable, then just include the list name. Approved by: The name and e-mail address of the person or people that approved the change. It is customary to get prior approval for a commit if it is to an area of the tree to which you do not usually commit. In addition, during the run up to a new release all commits must be approved by the release engineer. If these are your first commits then you should have passed them past your mentor first for approval, and you should list your mentor. Obtained from: The name of the project (if any) from which the code was obtained. MFC after: If you wish to receive an e-mail reminder to MFC at a later date, specify the number of days, weeks, or months after which an MFC is planned. Commit log for a commit based on a PR You want to commit a change based on a PR submitted by John Smith containing a patch. The end of the commit message should look something like this. ... PR: foo/12345 Submitted by: John Smith <John.Smith@example.com> Commit log for a commit needing review You want to change the virtual memory system. You have posted patches to the appropriate mailing list (in this case, freebsd-arch) and the changes have been approved. ... Reviewed by: -arch Commit log for a commit needing approval You want to commit a change to a section of the tree with a MAINTAINER assigned. You have collaborated with the listed MAINTAINER, who has told you to go ahead and commit. ... Approved by: abc Where abc is the account name of the person who approved. Commit log for a commit bringing in code from OpenBSD You want to commit some code based on work done in the OpenBSD project. ... Obtained from: OpenBSD Commit log for a change to &os.current; with a planned commit to &os.stable; to follow at a later date. You want to commit some code which will be merged from &os.current; into the &os.stable; branch after two weeks. ... MFC after: 2 weeks Where 2 is the number of days, weeks, or months after which an MFC is planned. The weeks option may be day, days, week, weeks, month, months, or may be left off (in which case, days will be assumed). In some cases you may need to combine some of these. Consider the situation where a user has submitted a PR containing code from the NetBSD project. You are looking at the PR, but it is not an area of the tree you normally work in, so you have decided to get the change reviewed by the arch mailing list. Since the change is complex, you opt to MFC after one month to allow adequate testing. The extra information to include in the commit would look something like PR: foo/54321 Submitted by: John Smith <John.Smith@example.com> Reviewed by: -arch Obtained from: NetBSD MFC after: 1 month

Console Server

The Problem

You have a computer room with lots of Unix server machines and lots of comms hardware. Each of these machines needs a serial console. But serial terminals are hard to find and quite expensive (especially compared to a much more capable PC). And they take up a lot of precious space in the computer room.

You need access to the console because when things break, that is where error messages go. And some tasks have to be done on the console (e.g. boot problems or OS installs/upgrades). Some Unix systems allow the console to break out to the ROM monitor which can sometimes be the only way to unstick a hung machine. This is often done with a LINE BREAK sent on the console serial port.

If we are going to play about with consoles, then there are a couple of other things that would be great:

Possible Solutions

If you use PC hardware for your servers, then a so-called "KVM switch" is one possible solution. This allows the use of a single Keyboard, Video screen and Mouse for multiple boxes. This cuts down on the space problem, but only works for PC hardware (not any comms gear you might have), and is not accessible from outside the computer room. Nor does it have much scroll-back or logging, and you have to handle alerting some other way. The big downside is that it will not work for serial-only devices, such as communications hardware. This means that even with a room full of PC-based servers, you are probably still going to need some sort of serial console solution. [Actually, Doug Schache has pointed out that you can get KVM switches that also do serial consoles or Sun-compatible KVM switching as well as PCs, but they are expensive. See Avocent for example.]

You might be tempted to do without a console terminal, but when things go pear-shaped you really need to see what is on the console. And you have to use the console to boot the machine and do things like OS upgrades or installs.

You might try having a single console terminal and switching from server to server as needed, either with a serial switch or just by patching it into the required machine. Serial switches are also hard to come by and not cheap, and may cause problems with sending BREAK when they switch. And (if your computer room is anything like ours) you never seem to have the right combination of patch leads to connect to the machine you need to, and even if the leads are there you can never work out exactly which combination of DTE/DCE headshells goes with which lead goes with which hardware. So you spend the first 10 minutes fooling around with breakout boxes and a box of leads, all while the server is down and the users are screaming. Of course this does not deal with the logging or remote access requirements. And inevitably the console is not switched to the machine you need so you lose all the console messages that might tell you what is going on.

One popular solution is to use terminal server hardware. Typically, the serial ports are connected to the various machine consoles, and set up for "reverse telnet" access. This means a user can telnet to a given IP/port and be connected to the appropriate console. This can be very cost-effective, as suitable old terminal servers can be picked up fairly cheaply (assuming you do not have a couple lying around). And it is of course network-accessible so suitable for remote access. But it suffers from one major drawback: if the network is down, then you have no access to any console, even if you are standing right next to the machine. (This may be partially alleviated by having a suitable terminal connected to one of the terminal server ports and connecting from there, but the terminal server software may not support that.) Also there is no logging or replay of console messages. But with a bit of work, and the addition of some software such as conserver (described below), this can be made to work pretty well.

A possibility suggested by Bron Gondwana is similar to the above solution. If you use servers with multiple serial ports, you can connect each spare serial port to the console port of the "next" server, creating a ring of console connections (in some sort of order). This can be made to work reasonably well with the aid of the conserver software, but can be a bit confusing otherwise (i.e. remembering which port is connected to which console). And you are stuck if you need to use serial ports for other things (such as modems) or you have machines without spare ports.

Or, if your budget exceeds your willingness to hack, you can buy an off-the-shelf solution. These vary in price and capability. See, for example, Lightwave, Perle, Avocent or Black Box. These solutions can be quite expensive - typically $USD100 - $USD400 per port.

Our Solution

In light of the above requirements, we chose a solution based on a dedicated PC running Unix with a multiport serial card, and some software designed to handle serial consoles.

Setting Up The Server

Patching the Stallion driver

The only hitch with setting up the server PC is getting the device drivers for the Stallion card. FreeBSD has supported Stallion ISA cards for many years, but unfortunately the driver has not been actively maintained for some years, and does not support the newer Stallion cards (such as the EasyIO-PCI card or some of the other ISA cards with newer UART chips). I have put together a patch file for FreeBSD 4.3 (based on the work of many other people) that will update the system to support these newer cards. Hopefully, this patch will be committed to the FreeBSD tree in time for the 4.4 release. If you are running FreeBSD 4 from prior to this time, you will need to download and apply this patch (instructions are included at the top of the patch file). If you are running FreeBSD 3, then you had probably best upgrade.

Configuring a new kernel

The Stallion driver is not included in the default GENERIC kernel, so you will need to create a kernel config file with the appropriate entries. See the stl(4) man page and the appropriate section of the FreeBSD Handbook.

Building World

After applying the patch you will certainly need to update the whole system, including all the user programs and the kernel. See the documentation in /usr/src/UPDATING and the FreeBSD Handbook.

Making The Devices

You will need to make the device notes for the Stallion card (which are not made by default). A new version of /dev/MAKEDEV with Stallion support will have been created by the mergemaster run during the above procedure. If you have a Stallion card with more than 8 ports, then you will need to edit /dev/MAKEDEV and change the definition of maxport at about line 250. By default, MAKEDEV only makes device nodes for 8 ports to keep the size of the /dev directory down.

to create dial-out devices for the first Stallion card. See the comments in MAKEDEV and the stl(4) man page for more details.

Compiling conserver

(See the section on Conserver versions below; the version I use is not the one available in the FreeBSD ports collection.)

There are two ways to install conserver. You can either compile from the source or use the FreeBSD ports framework.

Using the port framework

Using the ports framework is a bit cleaner, as the package system can then keep track of installed software and cleanly delete them when not being used. Download the port framework from here, unpack it to create a directory called conserver-port. CD to that directory and (as root) type

where consolehost is the name of the machine running the console server. Specifying this when the binary is compiled will avoid having to either specify it each time the program is run on remote hosts or having to maintain a conserver.cf file on every host. This command will fetch, patch, configure, compile and install the conserver application.

You can then run make package to create a binary package that can be installed on all the other FreeBSD hosts with the pkg_add command. For extra style points, you can make a two versions of the package: one for the console server machine without a DEFAULTHOST argument, and one for all the other hosts with a DEFAULTHOST argument. This will mean the console client program on the console server machine will default to localhost, which will work in the absence of name servers when the network is busted, and also allow "trusted" (i.e. no password required) connections via the localhost IP address for users logged into the console server machine (either via the console screen or the emergency backup modem). The version for the other machines with a DEFAULTHOST argument means users can just use the console client without specifying a hostname every time, and without needing to configure the conserver.cf file on every machine.

From the source tarball

If you prefer, you can download conserver and compile it yourself. You might need to do this if you want to install the console client on non-FreeBSD systems. We run the client on our Solaris hosts and it inter-operates with the FreeBSD-hosted server with no problems. This allows anyone in the whole company (many of whom have PCs and no FreeBSD host access on their desk) to access the console server.

Download the file from the conserver.com FTP site. Extract it into a handy directory then configure it by running

The --with-master argument avoids having to specify the master server every time the client is run remotely (or keeping up-to-date config files on all remote hosts). The --with-port argument avoids having to update /etc/services on every machine.

Configuring conserver

The conserver program is configured via a file called conserver.cf. This file usually lives in /usr/local/etc and is documented in the conserver.cf(5) man page.

The first line means all the console log files by default go into the directory /var/log/consoles/. The & in each line says the log file for that machine will be /var/log/consoles/machine.

The next three lines show three machines to which we need to connect. We use the cuaEx devices rather than the ttyEx devices because console ports typically do not show carrier. This means that opening ttyEx would hang and conserver would never connect. Using the cuaEx device avoids this problem. Another solution would be to use the ttyEx devices and enable soft carrier on these ports, perhaps by setting this using the ttyiEx device in the /etc/rc.serial file. See the comments in this file for more details. Also see the sio(4) man page for information on the initial-state and locked-state devices. (The Stallion driver also supports these conventions). And see the stty(1) man page for details on setting device modes.

The last section shows that any user logged into the server machine has passwordless access to all consoles. We do this because there are no user accounts on this machine and it is safely isolated from the wide world behind our firewall. The allow line allows anyone on a machine inside our organisation to access the console server if they provide their password, which is recorded in the conserver.passwd file (see next section).

Setting conserver passwords

The conserver.passwd file contains the encrypted version of the password that each user. The file is documented in the conserver.cf(5) man page.

The only tricky bit is loading the file with encoded passwords. It appeared in FreeBSD that was is no obvious way to generate an encrypted password for inclusion in another file (but see below). So I put together a quick hack perl script to do this:

(Note that this uses the FreeBSD MD5-style encrypted passwords. Running this on other Unix variants, or on FreeBSD with DES passwords, will likely need a different style of salt.)

Kris Kennaway has since pointed out you can get the same effect using the openssl passwd command:

Starting conserver at system boot time

There are two ways this can be done. Firstly, you could start up conserver from init by including an entry in /etc/ttys that looks a bit like this:

This has two advantages: init will restart the master console server if it ever crashes for any reason (but we have not noticed any crashes so far), and it arranges for STDOUT of the conserver process to be directed to the named tty (in this case cuaE0). This is useful because you can plug a terminal into this port, and the conserver program will show all console output not otherwise captured by a client console connection. This is useful as a general monitoring tool to see if anything is going on. We set this terminal up in the computer room but visible from the main office. It is a very handy feature. The downside of running conserver from the ttys file is that it cannot run in daemon mode (else init would continually restart it). This means conserver will not write a PID file, which makes it hard to rotate the log files.

So we start conserver from an rc.d script. If you installed conserver via the port, there will be a conserver.sh.sample file installed in /usr/local/etc/rc.d. Copy and/or rename this to conserver.sh to enable conserver to start at boot time.

In fact we use a modified version of this script which also connects conserver to a terminal via a tty device so we can monitor unwatched console output. Our conserver.sh script looks like this:

(Note the use of cuaE0 device and the need to set TTY modes for proper NL->CR handling).

Keeping the log files trimmed

FreeBSD has a program called newsyslog that will automatically handle log file trimming. Just add some lines to the configuration file /etc/newsyslog.conf for the console logs:

This tells newsyslog (which is run from cron every hour on the hour) that the console log files should be archived and compressed once they reach 1Mb, that we should keep 10 of them, and that to signal the server program you send a SIGHUP to the process whose PID is in the conserver.pid file. This is the master server, and it will arrange to signal all the child processes. Yes, this will send a HUP to all clients whenever a single log file needs rotating, but that is quite cheap. See the newsysylog(8) man page for details.

Cabling

This is always the hardest part of this kind of problem!! We had only a dozen or so cables/headshells to build, and we already had a collection of the appropriate crimping tools and hardware, so we did it ourselves. But if you are not set up for this, or you have a large number of cables to make, then you might consider getting some cables custom made. Look in the - yellow pages, there are a suprising number of places that do + yellow pages, there are a surprising number of places that do this! Getting custom-made cabling is good, and you can get much more professional results, but can be expensive. For example, the RJ-45 to DB-25 adapter kits described below are about $10 each; custom-made headshells are about twice that (and take a couple of weeks to arrive). Similarly, crimping custom RJ-45 to RJ-45 leads is quite cheap (say, $5 each) but it takes a fair amount of time. Custom made RJ-45 socket to RJ-45 plug converters cost about $25 each.

We have settled on RJ-45 Cat-V cabling for all our office and computer room cabling needs. This included patching between racks in the computer room. For serial connections, we use patchable headshells that have RJ-45 sockets on the back. This allows us to patch whatever RJ-45 - DB-25 connections we need.

Which is just as well, because there are many incompatible ways to represent serial connections on the RJ-45 plug. So the cabling has to be very careful to use the right mapping.

RJ-45 colors

RJ-45 cables and plugs have 8 pins/conductors. These are used as 4 matched pairs. There are a couple of conventions about how the pairs are mapped onto pins, but 100baseT uses the most common (known as EIA 586B). There are three common color-coding conventions for the individual conductors in RJ-45 cables. They are:

Note EIA 468A and EIA 568B are very similar, simply swapping the colors assigned to pair 2 and pair 3.

The pins in the RJ-45 plug are numbered from 1 to 8. Holding a patch lead with the cable pointing down and the clip away from you, pin 1 is at the left. Or, looking into an RJ-45 socket with the clip to the top, pin 1 is on the right. The following illustration (shamelessly lifted from the Cabletron web site above) shows it pretty well:

Pin	Scheme 1	Scheme 2 (EIA 568B)	Scheme 3 (EIA 568A)	Pair
1	Blue	White+Green	White+Orange	2+
2	Orange	Green	Orange	2-
3	Black	White+Orange	White+Green	3+
4	Red	Blue	Blue	1+
5	Green	White+Blue	White+Blue	1-
6	Yellow	Orange	Green	3-
7	Brown	White+Brown	White+Brown	4+
8	White or Grey	Brown	Brown	4-

Stallion RJ-45 Pin	Colour	Signal	Sun DB-25 Male Pin	RS232 Signal
1	Blue	DCD	20	DTR
2	Orange	RTS	5	CTS
3	Black	Chassis Gnd	1	Chassis Gnd
4	Red	TxD	3	RxD
5	Green	RxD	2	TxD
6	Yellow	Signal Gnd	7	Signal Gnd
7	Brown	CTS	4	RTS
8	White	RTS	8	DCD

Cisco RJ-45 Pin	Colour	Cisco Signal	Stallion RJ-45 Pin	Stallion Signal
1	White/Green	RTS	N/C
2	Green	DTR	N/C
3	White/Orange	TxD	5	RxD
4	Blue	Gnd	3	Gnd
5	White/Blue	Gnd	6	Gnd
6	Orange	RxD	4	TxD
7	White/Brown	DSR	N/C
8	Brown	CTS	N/C

Stallion RJ-45 Pin	Colour	Signal	PC DB-9 Female Pin	RS232 Signal
1	Blue	DCD	4	DTR
2	Orange	RTS	8	CTS
3	Black	Chassis Gnd	N/C
4	Red	TxD	2	RxD
5	Green	RxD	3	TxD
6	Yellow	Signal Gnd	5	Signal Gnd
7	Brown	CTS	7	RTS
8	White	RTS	1	DCD

On Sun Systems And Break

Anyone who has turned off a terminal used as a console for a Sun system will know what happens and why this is a problem. Sun hardware recognises a serial BREAK as a command to halt the OS and return to the ROM monitor prompt. A serial BREAK is an out-of-band signal on an RS-232 serial port that involves making the TX DATA line active (i.e. pulled down to less than -5v) for more than two whole character times (or about 2ms on a 9600bps line). Alas, this BREAK signal is all to easily generated by serial hardware during power-on or power-off. And the Stallion card does, in fact, generate breaks when the power to the PC fails. Unless fixed, this problem would mean that every Sun box connected to the console server would be halted whenever the power failed (due to dead power supplies, or fat-fingered operators unplugging it, or whatever). This is clearly not an acceptable situation.

Fortunately, Sun have come up with a set of fixes for this. For Solaris 2.6 and later (and perhaps earlier, but who was crazy enough to run 2.5??), the kbd(1) command can be used to disable the ROM-on-BREAK behaviour. This is a good start, but leaves you out of luck in the situation where a break is needed to get into a broken machine.

Starting with Solaris 8, the kbd command can also be used to enable an alternate break sequence using the kbd -a alternate command. When this is set, the key sequence <Return><Tilda><control-B> (within 5 seconds) will drop to the ROM. You can enable this permanently by editing the /etc/default/kbd file; see the kbd(1) man page. Note that this alternate break sequence is only active once the kernel has started running multiuser and processed the default file. While the ROM is active (during power-on and during the boot process) and while running single-user, you still need to use a BREAK to get to the ROM prompt. The console client can cause the server to send a BREAK using the escape sequence "^Ecl1" (i.e. escape, c, ell, one).

If you have a Sun software support contract, there are patches available for Solaris 2.6 and 2.7 that add the"alternate break" capability integrated into Solaris 2.8. Solaris 2.6 requires patch 105924-10 or higher. Solaris 2.7 requires patch 107589-02 or higher.

We have added this patch to all our Solaris 2.6 servers, and added it (and the entry in the /etc/default/kbd file) to our jumpstart configuration so it will automatically be added to every new install.

We have confirmed by direct testing that neither the Cisco 16xx, 26xx, or Catalyst hardware suffers from the BREAK sent when the Stallion card loses power.

Using a Serial Console on FreeBSD

The procedure for doing this is described in detail in the FreeBSD Handbook. This is a quick summary.

Check the kernel configuration

Check that the kernel configuration file has flags 0x10 in the config line for the sio0 device. This signals this device (known as COM1 in DOS/Windows or /dev/ttyd0 in FreeBSD) can be used as a console. This flag is set on the GENERIC and LINT sample configs, so is likely to be set in your kernel.

Create the /boot.conf file

This file should be created containing a single line containing just "-h". This tells the FreeBSD boot blocks to use the serial console.

Edit /etc/ttys

If you are not going to have any keyboard/video screen on this server at all, you should find all the lines for ttyv0 through ttyv8:

Change the on to off. This will stop login screens being run on the useless video consoles.

(replacing vt100 with the term type of your console. xterms might be a good choice). This allows you to log in to the console port once the system is running multi-user.

Security Implications

The client-server protocol for conserver requires the user of the console client to enter a password. This password is passed across the net in cleartext!!! This means conserver is not really suitable for use across untrusted networks (such as the Internet). Use of conserver-only passwords (in the conserver.passwd file) slightly mitigate this problem, but anyone sniffing a conserver connection can easily get console access, and from there prang your machine using the console break sequence. For operating across the Internet, use something secure like SSH to log into to the server machine, and run the console client there.

On Conserver Versions

The conserver program has fractured into a number of versions. The home page referenced above seems to be the latest and most featureful version around, and carries a version number of "7.0.2". This is maintained by Bryan Stansell (bryan@conserver.com), who has brought together the work of many people (listed on his webpage).

The FreeBSD ports collection contains a port for version 8.5 of conserver. This seems to be older and less featureful than the 7.0.2 version (in particular, it does not support consoles connected to terminal server ports and does not support a conserver.passwd file), and is written in a fairly idiosyncratic manner (using a preprocessor to generate C code). Version 8.5 is maintained by Kevin S Braunsdorf (ksb+conserver@sa.fedex.com) who did most of the original work on conserver, and whose work Bryan Stansell is building on. The 8.5 version does support one feature not in the 7.0.2 version (controlling power to remote machines via a specific serial-interfaced power controller hardware).

As shipped with FreeBSD, the 8.5 version does not authenticate against FreeBSD servers using MD5 passwords. There is a patch in the FreeBSD GNATS system (ports/28432) that fixes this problem. Kevin has indicated this patch will be included in future versions of conserver from him. There are a couple of other problems with the 8.5 version as well.

Links

His Greater Scroll of Console Knowledge contains evern more specific info on connecting various devices to various other devices. Oh the joys of standards!

The Real Weasel company makes a ISA or PCI video card that looks like a PC video card but actually talks to a serial port. This can be used to implement serial consoles on PC hardware for operating systems that can not be forced to use serial console ports early enough.

Files

Man Pages

Change Log

Filtering Bridges Nick Sayer

nsayer@FreeBSD.org

$FreeBSD$ For those of you who do not know, DSL differs from more traditional connectivity methods in that the "connectivity spigot" that comes out of the wall has no possibility for packet filtering. If you get a T1 line or some such it will come with a router that can generally include a packet filter. If you get ISDN or a dialup link, you also either have a software routing component (a PPP daemon, specifically) that can do some filtering or can be combined with a filter on the machine running the link. But with DSL you only get a little white - box with some Blinkenlights on it and an Ethernet port that takes + box with some blinking lights on it and an Ethernet port that takes your traffic back and forth from the Internet and nothing else (to some extent the same can be said of other mass-market high speed connectivity methods, like cable modems or high speed wireless links as well. The same technique I plan to describe works just as well for them, or for any other technology that provides an Ethernet port with no filtering). Why use a filtering bridge? Bridging is not the only conceivable option. It is possible to set up a two Ethernet machine as a router instead of a bridge. Where it is possible to do so, it is actually a better idea. Bridges run their interfaces in promiscuous mode, meaning they must process every packet presented to them. The problem is that routers can only route traffic between different subnets. Also, subnets can only be made by by cutting an existing space in half or defining a new space that is typically unroutable (see RFC 1918). This wastes half of the useful addresses (or at least puts them on the "wrong" side of the router -- the thing that is doing the packet filtering that makes the inside network safe). Using a bridge costs some CPU cycles, but makes all of the problems of adding a 2nd router go away. Configuring a Kernel After configuring and installing a kernel as shown here, you should carry out the other final preperation tasks before booting into your new kernel. Adding bridging to a FreeBSD machine is not hard to do. It means having 2 (or more, but we will just use 2 here) Ethernet cards and adding a couple of lines to the kernel configuration. Since May of 2000, RELENG_4 and -current have had bridging support for all Ethernet interfaces. This does not mean that any Ethernet interface will work. For them to work, they have to support a working promiscuous mode for - both reception and transmition -- that is, they have to be able to + both reception and transmission -- that is, they have to be able to transmit Ethernet packets with any source address, not just their own. In order to get good throughput, the cards should also be PCI bus mastering cards. The best choices still are the Intel EtherExpress Pro 100 cards, with 3com 3c9xx cards being second. So you will want to add the following to your kernel configuration file: device fxp (or whatever is appropriate for the cards you are using) options BRIDGE options IPFIREWALL options IPFIREWALL_VERBOSE Note that recent versions of FreeBSD support dynamically loading the IP Firewall code into the kernel. You can not do this, however, with bridging, as the bridge code itself needs to interact with IPFIREWALL in a special way. It is also a good idea at this point to see if Luigi has updated versions of the bridge code available that are more recent than what is in the distribution. As an example, 3.3-RELEASE comes with 981214, but as of this writing, the most up-to-date bridge code is 990810. You can fetch the latest version from http://www.iet.unipi.it/~luigi/. You will want to fetch bridge.c and bridge.h and drop them into sys/net/. For instructions on how to build and install a new kernel, refer to the Building and Installing a Custom Kernel section of the handbook Final Preperation Before you boot the new kernel, you must make some preparations in rc.boot and rc.firewall. The default rule for the firewall is to drop all packets on the floor. You will want to override this by setting up the 'open' firewall in /etc/rc.conf. Put these lines in /etc/rc.conf to achieve this: firewall_enable="YES" firewall_type="open" There is one more thing that is necessary. When running IP over Ethernet, there are actually two Ethernet protocols in use. One is IP, the other is ARP. ARP is used when a machine must figure out - what Ethernet address coresponds to a given IP address. ARP is not + what Ethernet address corresponds to a given IP address. ARP is not a part of the IP layer, since it only applies to IP when run over Ethernet. The standard ipfirewall rule for the open firewall is pass ip from any to any but what about ARP? If ARP is not passed, no IP traffic can flow at all. But IPFIREWALL has no provisions for dealing with non-IP protocols, and that includes ARP. Fortunately, a hackish extension was made to the ipfirewall code to assist filtering bridges. If you set up a special rule for UDP packets from IP address 0.0.0.0, the UDP port number will be used to match the Ethernet protocol number for bridged packets. In this way your bridge can be configured to pass or reject non IP protocols. So add this line just below the two lines near the top of /etc/rc.firewall that deal with lo0 (the ones that say that you should almost never change those two rules). ${fwcmd} add allow udp from 0.0.0.0 2054 to 0.0.0.0 This rule makes almost no sense at all from a normal perspective on IPFIREWALL, but the bridge code will use it to pass ARP packets without restriction (which you almost certainly want to do). Now you should be able to reboot your machine and have it act no differently than it did before. There will be some new boot messages about bridging, but the bridging will not be enabled. If there are any problems, you should try and sort them out at this point before proceeding. Enabling The Bridge Next, you should do this: &prompt.root; sysctl -w net.link.ether.bridge_ipfw=1 &prompt.root; sysctl -w net.link.ether.bridge=1 At this point, the bridge should be enabled, and because of the previous changes to /etc/rc.conf, the firewall should be wide open. At this point, you should be able to insert the machine between two sets of hosts and go back and forth without difficulty. If so, the next step is to add those two sysctl lines to either /etc/rc.local or add the net.link.[blah blah]=1 portions of the lines to /etc/sysctl.conf (which path you take depends on what version of FreeBSD you have). Now before we started all of this, you should have had a machine with two Ethernet interfaces, but with only one of them configured. That is, there should only be one ifconfig line /etc/rc.conf. With the bridge in place, that is still true. But there is a detail that deserves some thought. The bridge is not in place by default. That means that until the sysctls are run that turn the bridge on, rather late in the startup, it is still an ordinary machine with two interfaces, only one of which is configured by /etc/rc.conf. This becomes important for those portions of the startup that require network access, say for DNS resolution. Some care must be made in picking which interface is going to be the configured one. In most cases, you are best to pick the "outside" one (that is, the interface connected to the Internet). Let's presume for the sake of the examples to come, that fxp0 is the "outside" interface, and fxp1 is the "inside" one. That means that fxp0 should be mentioned in /etc/rc.conf's ifconfig sections, but fxp1 should not be. The sysctl that turns the bridge on will make fxp1 start working automagically. Configuring The Firewall Now it is time to start adding ipfirewall rules to secure the inside network. There are some complications in doing this because not all of the ipfirewall functionality is available on bridged packets. Also, there is a difference between packets that are in the process of being bridged and packets that are being received by the local machine. In general, packets being bridged are only run through ipfirewall once, not twice as is usually the case. Bridged packets are filtered while they are being received, so rules that use 'out' or 'xmit' will never match. I usually use 'in via' which is an older syntax, but one that makes sense as you read it. Another limitation is that you are restricted only to 'pass' or 'drop' for filtering bridged packets. Sophisticated things like 'divert' or 'forward' or 'reject' are not available. Such options can still be used, but only on traffic to or from the bridge machine itself. New in FreeBSD 4.0 is the concept of stateful filtering. This is a big boost for UDP traffic, which typically is a request going out, followed shortly thereafter by a response with the exact same set of IP addresses and port numbers (but with source and dest reversed, of course). For firewalls that have no statekeeping, there is almost no way to deal with this sort of traffic short of setting up proxies. But a firewall that can "remember" an outgoing UDP packet and for the next few minutes allow a response, handling UDP services is trivial. The example to follow shows how to do this. The truly paranoid can also set up rules like this to handle TCP. This allows you to avoid some sorts of denial of service attacks or other nasty tricks, but it also typically makes your state table mushroom in size. Let's look at an example setup. Note first that at the top of /etc/rc.firewall we should already have taken care of the loopback interface and the special hack for ARP should still be in place. So we will not worry about them any further. us_ip=192.168.1.1 oif=fxp0 iif=fxp1 # Things that we've kept state on before get to go through in a hurry. ${ipfw} add check-state # Throw away RFC 1918 networks ${ipfw} add deny log ip from 10.0.0.0/8 to any in via ${oif} ${ipfw} add deny log ip from 172.16.0.0/12 to any in via ${oif} ${ipfw} add deny log ip from 192.68.0.0/16 to any in via ${oif} # Allow the bridge machine to say anything it wants (keep state if UDP) ${ipfw} add pass udp from ${us_ip} to any keep-state ${ipfw} add pass ip from ${us_ip} to any # Allow the inside net to say anything it wants (keep state if UDP) ${ipfw} add pass udp from any to any in via ${iif} keep-state ${ipfw} add pass ip from any to any in via ${iif} # Allow all manner of ICMP ${ipfw} add pass icmp from any to any # TCP section # established TCP sessions are ok everywhere. ${ipfw} add pass tcp from any to any established # Pass the "quarantine" range. ${ipfw} add pass tcp from any to any 49152-65535 in via ${oif} # Pass ident probes. It's better than waiting for them to timeout ${ipfw} add pass tcp from any to any 113 in via ${oif} # Pass SSH. ${ipfw} add pass tcp from any to any 22 in via ${oif} # Pass DNS. Only if you have name servers inside. #${ipfw} add pass tcp from any to any 53 in via ${oif} # Pass SMTP to the mail server only ${ipfw} add pass tcp from any to mailhost 25 in via ${oif} # UDP section # Pass the "quarantine" range. ${ipfw} add pass udp from any to any 49152-65535 in via ${oif} # Pass DNS. Only if you have name servers inside. #${ipfw} add pass udp from any to any 53 in via ${oif} # Everything else is suspect ${ipfw} add deny log ip from any to any Those of you who have set up firewalls before may notice some things missing. In particular, there are no anti-spoofing rules. That is, we did not add: ${ipfw} add deny ip from ${us_ip}/24 to any in via ${oif} That is, drop packets claiming to be from our network that are coming in from the outside. This is something that you would commonly do to make sure that someone does not try and evade the packet filter by generating nefarious packets that look like they are from the inside. The problem with that is that there is at least one host on the outside interface that you do not want to ignore -- your router. In my particular case, I have some machines on the outside and some on the inside, but I do not necessarily want the outside machines to have routine access to the inside. At the same time, I do not want to throw their traffic away. In my own case, my ISP anti-spoofs at their router, so I do not need to bother. And in general, the fewer rules the better, since it will take time and CPU to process each one. Note also that the last rule is almost an exact duplicate of the default rule 65536. There are two major differences when it comes to bridging, however. Our rule logs what it drops, of course, but our rule will only apply to IP traffic. Apart from the UDP 0.0.0.0 trick there is no way to deal with non IP traffic, so the default rule at 65536 will drop ALL traffic, not merely all non-IP traffic. So the net effect is that unmatched IP traffic will be logged, but not non-IP traffic. If you want, you can add option IPFIREWALL_DEFAULT_TO_ACCEPT to your kernel configuration and non-IP traffic will be passed instead of dropped. But in the case of a filtering bridge between you and the Internet, it is unlikely that you would want to do this (if you are sufficiently paranoid). There is a rule for passing SMTP to a mailhost if you have one. Obviously the whole ruleset above should be flavored to taste, and that is an example of a specific service exemption. Note that in order for 'mailhost' to work, name service lookups must work BEFORE the bridge is enabled. This is an example of making sure that you enable the correct interface. Another item to note is that the DNS rules are set up only to allow DNS servers to work. This means that if do not set up a DNS server, you do not need them. Folks used to setting up IP firewalls also probably are used to either having a 'reset' or a 'forward' rule for ident packets (TCP port 113). Unfortunately, this is not an option with the bridging code, so the path of least resistance is to simply pass them to their destination. As long as that destination machine is not running an ident daemon, this is relatively harmless. The alternative is dropping port 113 connections, which makes firing up things like IRC take forever (the ident probe must timeout). The only other thing that is a little weird that you may have noticed is that there is a rule to let ${us_ip} speak and a separate rule to allow the inside network to speak. Remember that this is because the two sets of traffic will be taking different paths through the kernel and into the packet filter. The inside net will be going through the bridge code. The local machine, however, will be using the normal IP stack to speak. Thus the two rules to handle the different cases. The in via ${oif} rules work for both paths. In general if you use in via rules throughout the filter, you will need to make an exception for locally generated packets, because they did not "come in" via anything. Contributors To some extent the material for this discussion is a combination of the items that were discussed by Luigi Rizzo in his Dummynet lecture at FreeBSDcon '99 and by Mark Murray during his Network Security lecture. In addition, for quite some time now I have been putting together filtering bridges for friends and colleagues who were getting DSL connections for their home.

Serial and UART Tutorial Frank Durda

uhclem@FreeBSD.org

$FreeBSD$ This article talks about using serial hardware with FreeBSD. The UART: What it is and how it works Copyright © 1996 &a.uhclem;, All Rights Reserved. 13 January 1996. The Universal Asynchronous Receiver/Transmitter (UART) controller is the key component of the serial communications subsystem of a computer. The UART takes bytes of data and transmits the individual bits in a sequential fashion. At the destination, a second UART re-assembles the bits into complete bytes. Serial transmission is commonly used with modems and for non-networked communication between computers, terminals and other devices. There are two primary forms of serial transmission: Synchronous and Asynchronous. Depending on the modes that are supported by the hardware, the name of the communication sub-system will usually include a A if it supports Asynchronous communications, and a S if it supports Synchronous communications. Both forms are described below. Some common acronyms are:

UART Universal Asynchronous Receiver/Transmitter

USART Universal Synchronous-Asynchronous Receiver/Transmitter

Synchronous Serial Transmission Synchronous serial transmission requires that the sender and receiver share a clock with one another, or that the sender provide a strobe or other timing signal so that the receiver knows when to read the next bit of the data. In most forms of serial Synchronous communication, if there is no data available at a given instant to transmit, a fill character must be sent instead so that data is always being transmitted. Synchronous communication is usually more efficient because only data bits are transmitted between sender and receiver, and synchronous communication can be more costly if extra wiring and circuits are required to share a clock signal between the sender and receiver. A form of Synchronous transmission is used with printers and fixed disk devices in that the data is sent on one set of wires while a clock or strobe is sent on a different wire. Printers and fixed disk devices are not normally serial devices because most fixed disk interface standards send an entire word of data for each clock or strobe signal by using a separate wire for each bit of the word. In the PC industry, these are known as Parallel devices. The standard serial communications hardware in the PC does not support Synchronous operations. This mode is described here for comparison purposes only. Asynchronous Serial Transmission Asynchronous transmission allows data to be transmitted without the sender having to send a clock signal to the receiver. Instead, the sender and receiver must agree on timing parameters in advance and special bits are added to each word which are used to synchronize the sending and receiving units. When a word is given to the UART for Asynchronous transmissions, a bit called the "Start Bit" is added to the beginning of each word that is to be transmitted. The Start Bit is used to alert the receiver that a word of data is about to be sent, and to force the clock in the receiver into synchronization with the clock in the transmitter. These two clocks must be accurate enough to not have the frequency drift by more than 10% during the transmission of the remaining bits in the word. (This requirement was set in the days of mechanical teleprinters and is easily met by modern electronic equipment.) After the Start Bit, the individual bits of the word of data are sent, with the Least Significant Bit (LSB) being sent first. Each bit in the transmission is transmitted for exactly the same amount of time as all of the other bits, and the receiver looks at the wire at approximately halfway through the period assigned to each bit to determine if the bit is a 1 or a 0. For example, if it takes two seconds to send each bit, the receiver will examine the signal to determine if it is a 1 or a 0 after one second has passed, then it will wait two seconds and then examine the value of the next bit, and so on. The sender does not know when the receiver has looked at the value of the bit. The sender only knows when the clock says to begin transmitting the next bit of the word. When the entire data word has been sent, the transmitter may add a Parity Bit that the transmitter generates. The Parity Bit may be used by the receiver to perform simple error checking. Then at least one Stop Bit is sent by the transmitter. When the receiver has received all of the bits in the data word, it may check for the Parity Bits (both sender and receiver must agree on whether a Parity Bit is to be used), and then the receiver looks for a Stop Bit. If the Stop Bit does not appear when it is supposed to, the UART considers the entire word to be garbled and will report a Framing Error to the host processor when the data word is read. The usual cause of a Framing Error is that the sender and receiver clocks were not running at the same speed, or that the signal was interrupted. Regardless of whether the data was received correctly or not, the UART automatically discards the Start, Parity and Stop bits. If the sender and receiver are configured identically, these bits are not passed to the host. If another word is ready for transmission, the Start Bit for the new word can be sent as soon as the Stop Bit for the previous word has been sent. Because asynchronous data is self synchronizing, if there is no data to transmit, the transmission line can be idle. Other UART Functions In addition to the basic job of converting data from parallel to serial for transmission and from serial to parallel on reception, a UART will usually provide additional circuits for signals that can be used to indicate the state of the transmission media, and to regulate the flow of data in the event that the remote device is not prepared to accept more data. For example, when the device connected to the UART is a modem, the modem may report the presence of a carrier on the phone line while the computer may be able to instruct the modem to reset itself or to not - take calls by asserting or disasserting one more of these + take calls by asserting or dis-asserting one more of these extra signals. The function of each of these additional signals is defined in the EIA RS232-C standard. The RS232-C and V.24 Standards In most computer systems, the UART is connected to circuitry that generates signals that comply with the EIA RS232-C specification. There is also a CCITT standard named V.24 that mirrors the specifications included in RS232-C. RS232-C Bit Assignments (Marks and Spaces) In RS232-C, a value of 1 is called a Mark and a value of 0 is called a Space. When a communication line is idle, the line is said to be Marking, or transmitting continuous 1 values. The Start bit always has a value of 0 (a Space). The Stop Bit always has a value of 1 (a Mark). This means that there will always be a Mark (1) to Space (0) transition on the line at the start of every word, even when multiple word are transmitted back to back. This guarantees that sender and receiver can resynchronize their clocks regardless of the content of the data bits that are being transmitted. The idle time between Stop and Start bits does not have to be an exact multiple (including zero) of the bit rate of the communication link, but most UARTs are designed this way for simplicity. In RS232-C, the "Marking" signal (a 1) is represented by a voltage between -2 VDC and -12 VDC, and a "Spacing" signal (a 0) is represented by a voltage between 0 and +12 VDC. The transmitter is supposed to send +12 VDC or -12 VDC, and the receiver is supposed to allow for some voltage loss in long cables. Some transmitters in low power devices (like portable computers) sometimes use only +5 VDC and -5 VDC, but these values are still acceptable to a RS232-C receiver, provided that the cable lengths are short. RS232-C Break Signal RS232-C also specifies a signal called a Break, which is caused by sending continuous Spacing values (no Start or Stop bits). When there is no electricity present on the data circuit, the line is considered to be sending Break. The Break signal must be of a duration longer than the time it takes to send a complete byte plus Start, Stop and Parity bits. Most UARTs can distinguish between a Framing Error and a Break, but if the UART cannot do this, the Framing Error detection can be used to identify Breaks. In the days of teleprinters, when numerous printers around the country were wired in series (such as news services), any unit could cause a Break by temporarily opening the entire circuit so that no current flowed. This was used to allow a location with urgent news to interrupt some other location that was currently sending information. In modern systems there are two types of Break signals. If the Break is longer than 1.6 seconds, it is considered a "Modem Break", and some modems can be programmed to terminate the conversation and go on-hook or enter the modems' command mode when the modem detects this signal. If the Break is smaller than 1.6 seconds, it signifies a Data Break and it is up to the remote computer to respond to this signal. Sometimes this form of Break is used as an Attention or Interrupt signal and sometimes is accepted as a substitute for the ASCII CONTROL-C character. Marks and Spaces are also equivalent to Holes and No Holes in paper tape systems. Breaks cannot be generated from paper tape or from any other byte value, since bytes are always sent with Start and Stop bit. The UART is usually capable of generating the continuous Spacing signal in response to a special command from the host processor. RS232-C DTE and DCE Devices The RS232-C specification defines two types of equipment: the Data Terminal Equipment (DTE) and the Data Carrier Equipment (DCE). Usually, the DTE device is the terminal (or computer), and the DCE is a modem. Across the phone line at the other end of a conversation, the receiving modem is also a DCE device and the computer that is connected to that modem is a DTE device. The DCE device receives signals on the pins that the DTE device transmits on, and vice versa. When two devices that are both DTE or both DCE must be connected together without a modem or a similar media translater between them, a NULL modem must be used. The NULL modem electrically re-arranges the cabling so that the transmitter output is connected to the receiver input on the other device, and vice versa. Similar translations are performed on all of the control signals so that each device will see what it thinks are DCE (or DTE) signals from the other device. The number of signals generated by the DTE and DCE devices are not symmetrical. The DTE device generates fewer signals for the DCE device than the DTE device receives from the DCE. RS232-C Pin Assignments The EIA RS232-C specification (and the ITU equivalent, V.24) calls for a twenty-five pin connector (usually a DB25) and defines the purpose of most of the pins in that connector. In the IBM Personal Computer and similar systems, a subset of RS232-C signals are provided via nine pin connectors (DB9). The signals that are not included on the PC connector deal mainly with synchronous operation, and this transmission mode is not supported by the UART that IBM selected for use in the IBM PC. Depending on the computer manufacturer, a DB25, a DB9, or both types of connector may be used for RS232-C communications. (The IBM PC also uses a DB25 connector for the parallel printer interface which causes some confusion.) Below is a table of the RS232-C signal assignments in the DB25 and DB9 connectors. DB25 RS232-C Pin DB9 IBM PC Pin EIA Circuit Symbol CCITT Circuit Symbol Common Name Signal Source Description 1 - AA 101 PG/FG - Frame/Protective Ground 2 3 BA 103 TD DTE Transmit Data 3 2 BB 104 RD DCE Receive Data 4 7 CA 105 RTS DTE Request to Send 5 8 CB 106 CTS DCE Clear to Send 6 6 CC 107 DSR DCE Data Set Ready 7 5 AV 102 SG/GND - Signal Ground 8 1 CF 109 DCD/CD DCE Data Carrier Detect 9 - - - - - Reserved for Test 10 - - - - - Reserved for Test 11 - - - - - Reserved for Test 12 - CI 122 SRLSD DCE Sec. Recv. Line Signal Detector 13 - SCB 121 SCTS DCE Secondary Clear to Send 14 - SBA 118 STD DTE Secondary Transmit Data 15 - DB 114 TSET DCE Trans. Sig. Element Timing 16 - SBB 119 SRD DCE Secondary Received Data 17 - DD 115 RSET DCE Receiver Signal Element Timing 18 - - 141 LOOP DTE Local Loopback 19 - SCA 120 SRS DTE Secondary Request to Send 20 4 CD 108.2 DTR DTE Data Terminal Ready 21 - - - RDL DTE Remote Digital Loopback 22 9 CE 125 RI DCE Ring Indicator 23 - CH 111 DSRS DTE Data Signal Rate Selector 24 - DA 113 TSET DTE Trans. Sig. Element Timing 25 - - 142 - DCE Test Mode Bits, Baud and Symbols Baud is a measurement of transmission speed in asynchronous communication. Because of advances in modem communication technology, this term is frequently misused when describing the data rates in newer devices. Traditionally, a Baud Rate represents the number of bits that are actually being sent over the media, not the amount of data that is actually moved from one DTE device to the other. The Baud count includes the overhead bits Start, Stop and Parity that are generated by the sending UART and removed by the receiving UART. This means that seven-bit words of data actually take 10 bits to be completely transmitted. Therefore, a modem capable of moving 300 bits per second from one place to another can normally only move 30 7-bit words if Parity is used and one Start and Stop bit are present. If 8-bit data words are used and Parity bits are also used, the data rate falls to 27.27 words per second, because it now takes 11 bits to send the eight-bit words, and the modem still only sends 300 bits per second. The formula for converting bytes per second into a baud rate and vice versa was simple until error-correcting modems came along. These modems receive the serial stream of bits from the UART in the host computer (even when internal modems are used the data is still frequently serialized) and converts the bits back into bytes. These bytes are then combined into packets and sent over the phone line using a Synchronous transmission method. This means that the Stop, Start, and Parity bits added by the UART in the DTE (the computer) were removed by the modem before transmission by the sending modem. When these bytes are received by the remote modem, the remote modem adds Start, Stop and Parity bits to the words, converts them to a serial format and then sends them to the receiving UART in the remote computer, who then strips the Start, Stop and Parity bits. The reason all these extra conversions are done is so that the two modems can perform error correction, which means that the receiving modem is able to ask the sending modem to resend a block of data that was not received with the correct checksum. This checking is handled by the modems, and the DTE devices are usually unaware that the process is occurring. By striping the Start, Stop and Parity bits, the additional bits of data that the two modems must share between themselves to perform error-correction are mostly concealed from the effective transmission rate seen by the sending and receiving DTE equipment. For example, if a modem sends ten 7-bit words to another modem without including the Start, Stop and Parity bits, the sending modem will be able to add 30 bits of its own information that the receiving modem can use to do error-correction without impacting the transmission speed of the real data. The use of the term Baud is further confused by modems that perform compression. A single 8-bit word passed over the telephone line might represent a dozen words that were transmitted to the sending modem. The receiving modem will expand the data back to its original content and pass that data to the receiving DTE. Modern modems also include buffers that allow the rate that bits move across the phone line (DCE to DCE) to be a different speed than the speed that the bits move between the DTE and DCE on both ends of the conversation. Normally the speed between the DTE and DCE is higher than the DCE to DCE speed because of the use of compression by the modems. Because the number of bits needed to describe a byte varied during the trip between the two machines plus the differing bits-per-seconds speeds that are used present on the DTE-DCE and DCE-DCE links, the usage of the term Baud to describe the overall communication speed causes problems and can misrepresent the true transmission speed. So Bits Per Second (bps) is the correct term to use to describe the transmission rate seen at the DCE to DCE interface and Baud or Bits Per Second are acceptable terms to use when a connection is made between two systems with a wired connection, or if a modem is in use that is not performing error-correction or compression. Modern high speed modems (2400, 9600, 14,400, and 19,200bps) in reality still operate at or below 2400 baud, or more accurately, 2400 Symbols per second. High speed modem are able to encode more bits of data into each Symbol using a technique called Constellation Stuffing, which is why the effective bits per second rate of the modem is higher, but the modem continues to operate within the limited audio bandwidth that the telephone system provides. Modems operating at 28,800 and higher speeds have variable Symbol rates, but the technique is the same. The IBM Personal Computer UART Starting with the original IBM Personal Computer, IBM selected the National Semiconductor INS8250 UART for use in the IBM PC Parallel/Serial Adapter. Subsequent generations of compatible computers from IBM and other vendors continued to use the INS8250 or improved versions of the National Semiconductor UART family. National Semiconductor UART Family Tree There have been several versions and subsequent generations of the INS8250 UART. Each major version is described below. INS8250 -> INS8250B \ \ \-> INS8250A -> INS82C50A \ \ \-> NS16450 -> NS16C450 \ \ \-> NS16550 -> NS16550A -> PC16550D INS8250 This part was used in the original IBM PC and IBM PC/XT. The original name for this part was the INS8250 ACE (Asynchronous Communications Element) and it is made from NMOS technology. The 8250 uses eight I/O ports and has a one-byte send and a one-byte receive buffer. This original UART has several race conditions and other flaws. The original IBM BIOS includes code to work around these flaws, but this made the BIOS dependent on the flaws being present, so subsequent parts like the 8250A, 16450 or 16550 could not be used in the original IBM PC or IBM PC/XT. INS8250-B This is the slower speed of the INS8250 made from NMOS technology. It contains the same problems as the original INS8250. INS8250A An improved version of the INS8250 using XMOS technology with various functional flaws corrected. The INS8250A was used initially in PC clone computers by vendors who used clean BIOS designs. Because of the corrections in the chip, this part could not be used with a BIOS compatible with the INS8250 or INS8250B. INS82C50A This is a CMOS version (low power consumption) of the INS8250A and has similar functional characteristics. NS16450 Same as NS8250A with improvements so it can be used with faster CPU bus designs. IBM used this part in the IBM AT and updated the IBM BIOS to no longer rely on the bugs in the INS8250. NS16C450 This is a CMOS version (low power consumption) of the NS16450. NS16550 Same as NS16450 with a 16-byte send and receive buffer but the buffer design was flawed and could not be reliably be used. NS16550A Same as NS16550 with the buffer flaws corrected. The 16550A and its successors have become the most popular UART design in the PC industry, mainly due it its ability to reliably handle higher data rates on operating systems with sluggish interrupt response times. NS16C552 This component consists of two NS16C550A CMOS UARTs in a single package. PC16550D Same as NS16550A with subtle flaws corrected. This is revision D of the 16550 family and is the latest design available from National Semiconductor. The NS16550AF and the PC16550D are the same thing National reorganized their part numbering system a few years ago, and the NS16550AFN no longer exists by that name. (If you have a NS16550AFN, look at the date code on the part, which is a four digit number that usually starts with a nine. The first two digits of the number are the year, and the last two digits are the week in that year when the part was packaged. If you have a NS16550AFN, it is probably a few years old.) The new numbers are like PC16550DV, with minor differences in the suffix letters depending on the package material and its shape. (A description of the numbering system can be found below.) It is important to understand that in some stores, you may pay $15(US) for a NS16550AFN made in 1990 and in the next bin are the new PC16550DN parts with minor fixes that National has made since the AFN part was in production, the PC16550DN was probably made in the past six months and it costs half (as low as $5(US) in volume) as much as the NS16550AFN because they are readily available. As the supply of NS16550AFN chips continues to shrink, the price will probably continue to increase until more people discover and accept that the PC16550DN really has the same function as the old part number. National Semiconductor Part Numbering System The older NSnnnnnrqp part numbers are now of the format PCnnnnnrgp. The r is the revision field. The current revision of the 16550 from National Semiconductor is D. The p is the package-type field. The types are: "F" QFP (quad flat pack) L lead type "N" DIP (dual inline package) through hole straight lead type "V" LPCC (lead plastic chip carrier) J lead type The g is the product grade field. If an I precedes the package-type letter, it indicates an industrial grade part, which has higher specs than a standard part but not as high as Military Specification (Milspec) component. This is an optional field. So what we used to call a NS16550AFN (DIP Package) is now called a PC16550DN or PC16550DIN. Other Vendors and Similar UARTs Over the years, the 8250, 8250A, 16450 and 16550 have been licensed or copied by other chip vendors. In the case of the 8250, 8250A and 16450, the exact circuit (the megacell) was licensed to many vendors, including Western Digital and Intel. Other vendors reverse-engineered the part or produced emulations that had similar behavior. In internal modems, the modem designer will frequently emulate the 8250A/16450 with the modem microprocessor, and the emulated UART will frequently have a hidden buffer consisting of several hundred bytes. Because of the size of the buffer, these emulations can be as reliable as a 16550A in their ability to handle high speed data. However, most operating systems will still report that the UART is only a 8250A or 16450, and may not make effective use of the extra buffering present in the emulated UART unless special drivers are used. Some modem makers are driven by market forces to abandon a design that has hundreds of bytes of buffer and instead use a 16550A UART so that the product will compare favorably in market comparisons even though the effective performance may be lowered by this action. A common misconception is that all parts with 16550A written on them are identical in performance. There are differences, and in some cases, outright flaws in most of these 16550A clones. When the NS16550 was developed, the National Semiconductor obtained several patents on the design and they also limited licensing, making it harder for other vendors to provide a chip with similar features. Because of the patents, reverse-engineered designs and emulations had to avoid infringing the claims covered by the patents. Subsequently, these copies almost never perform exactly the same as the NS16550A or PC16550D, which are the parts most computer and modem makers want to buy but are sometimes unwilling to pay the price required to get the genuine part. Some of the differences in the clone 16550A parts are unimportant, while others can prevent the device from being used at all with a given operating system or driver. These differences may show up when using other drivers, or when particular combinations of events occur that were not well tested or considered in the Windows driver. This is because most modem vendors and 16550-clone makers use the Microsoft drivers from Windows for Workgroups 3.11 and the Microsoft MS-DOS utility as the primary tests for compatibility with the NS16550A. This over-simplistic criteria means that if a different operating system is used, problems could appear due to subtle differences between the clones and genuine components. National Semiconductor has made available a program named COMTEST that performs compatibility tests independent of any OS drivers. It should be remembered that the purpose of this type of program is to demonstrate the flaws in the products of the competition, so the program will report major as well as extremely subtle differences in behavior in the part being tested. In a series of tests performed by the author of this document in 1994, components made by National Semiconductor, TI, StarTech, and CMD as well as megacells and emulations embedded in internal modems were tested with COMTEST. A difference count for some of these components is listed below. Because these tests were performed in 1994, they may not reflect the current performance of the given product from a vendor. It should be noted that COMTEST normally aborts when an excessive number or certain types of problems have been detected. As part of this testing, COMTEST was modified so that it would not abort no matter how many differences were encountered. Vendor Part Number Errors (aka "differences" reported) National (PC16550DV) 0 National (NS16550AFN) 0 National (NS16C552V) 0 TI (TL16550AFN) 3 CMD (16C550PE) 19 StarTech (ST16C550J) 23 Rockwell Reference modem with internal 16550 or an emulation (RC144DPi/C3000-25) 117 Sierra Modem with an internal 16550 (SC11951/SC11351) 91 To date, the author of this document has not found any non-National parts that report zero differences using the COMTEST program. It should also be noted that National has had five versions of the 16550 over the years and the newest parts behave a bit differently than the classic NS16550AFN that is considered the benchmark for functionality. COMTEST appears to turn a blind eye to the differences within the National product line and reports no errors on the National parts (except for the original 16550) even when there are official erratas that describe bugs in the A, B and C revisions of the parts, so this bias in COMTEST must be taken into account. It is important to understand that a simple count of differences from COMTEST does not reveal a lot about what differences are important and which are not. For example, about half of the differences reported in the two modems listed above that have internal UARTs were caused by the clone UARTs not supporting five- and six-bit character modes. The real 16550, 16450, and 8250 UARTs all support these modes and COMTEST checks the functionality of these modes so over fifty differences are reported. However, almost no modern modem supports five- or six-bit characters, particularly those with error-correction and compression capabilities. This means that the differences related to five- and six-bit character modes can be discounted. Many of the differences COMTEST reports have to do with timing. In many of the clone designs, when the host reads from one port, the status bits in some other port may not update in the same amount of time (some faster, some slower) as a real NS16550AFN and COMTEST looks for these differences. This means that the number of differences can be misleading in that one device may only have one or two differences but they are extremely serious, and some other device that updates the status registers faster or slower than the reference part (that would probably never affect the operation of a properly written driver) could have dozens of differences reported. COMTEST can be used as a screening tool to alert the administrator to the presence of potentially incompatible components that might cause problems or have to be handled as a special case. If you run COMTEST on a 16550 that is in a modem or a modem is attached to the serial port, you need to first issue a ATE0&W command to the modem so that the modem will not echo any of the test characters. If you forget to do this, COMTEST will report at least this one difference: Error (6)...Timeout interrupt failed: IIR = c1 LSR = 61 8250/16450/16550 Registers The 8250/16450/16550 UART occupies eight contiguous I/O port addresses. In the IBM PC, there are two defined locations for these eight ports and they are known collectively as COM1 and COM2. The makers of PC-clones and add-on cards have created two additional areas known as COM3 and COM4, but these extra COM ports conflict with other hardware on some systems. The most common conflict is with video adapters that provide IBM 8514 emulation. COM1 is located from 0x3f8 to 0x3ff and normally uses IRQ 4 COM2 is located from 0x2f8 to 0x2ff and normally uses IRQ 3 COM3 is located from 0x3e8 to 0x3ef and has no standardized IRQ COM4 is located from 0x2e8 to 0x2ef and has no standardized IRQ. A description of the I/O ports of the 8250/16450/16550 UART is provided below. I/O Port Access Allowed Description +0x00 write (DLAB==0) Transmit Holding Register (THR).Information written to this port are treated as data words and will be transmitted by the UART. +0x00 read (DLAB==0) Receive Buffer Register (RBR).Any data words received by the UART form the serial link are accessed by the host by reading this port. +0x00 write/read (DLAB==1) Divisor Latch LSB (DLL)This value will be divided from the master input clock (in the IBM PC, the master clock is 1.8432MHz) and the resulting clock will determine the baud rate of the UART. This register holds bits 0 thru 7 of the divisor. +0x01 write/read (DLAB==1) Divisor Latch MSB (DLH)This value will be divided from the master input clock (in the IBM PC, the master clock is 1.8432MHz) and the resulting clock will determine the baud rate of the UART. This register holds bits 8 thru 15 of the divisor. +0x01 write/read (DLAB==0) Interrupt Enable Register (IER)The 8250/16450/16550 UART classifies events into one of four categories. Each category can be configured to generate an interrupt when any of the events occurs. The 8250/16450/16550 UART generates a single external interrupt signal regardless of how many events in the enabled categories have occurred. It is up to the host processor to respond to the interrupt and then poll the enabled interrupt categories (usually all categories have interrupts enabled) to determine the true cause(s) of the interrupt. Bit 7 Reserved, always 0. Bit 6 Reserved, always 0. Bit 5 Reserved, always 0. Bit 4 Reserved, always 0. Bit 3 Enable Modem Status Interrupt (EDSSI). Setting this bit to "1" allows the UART to generate an interrupt when a change occurs on one or more of the status lines. Bit 2 Enable Receiver Line Status Interrupt (ELSI) Setting this bit to "1" causes the UART to generate an interrupt when the an error (or a BREAK signal) has been detected in the incoming data. Bit 1 Enable Transmitter Holding Register Empty Interrupt (ETBEI) Setting this bit to "1" causes the UART to generate an interrupt when the UART has room for one or more additional characters that are to be transmitted. Bit 0 Enable Received Data Available Interrupt (ERBFI) Setting this bit to "1" causes the UART to generate an interrupt when the UART has received enough characters to exceed the trigger level of the FIFO, or the FIFO timer has expired (stale data), or a single character has been received when the FIFO is disabled. +0x02 write FIFO Control Register (FCR) (This port does not exist on the 8250 and 16450 UART.) Bit 7 Receiver Trigger Bit #1 Bit 6 Receiver Trigger Bit #0These two bits control at what point the receiver is to generate an interrupt when the FIFO is active. 7 6 How many words are received before an interrupt is generated 0 0 1 0 1 4 1 0 8 1 1 14 Bit 5 Reserved, always 0. Bit 4 Reserved, always 0. Bit 3 DMA Mode Select. If Bit 0 is set to "1" (FIFOs enabled), setting this bit changes the operation of the -RXRDY and -TXRDY signals from Mode 0 to Mode 1. Bit 2 Transmit FIFO Reset. When a "1" is written to this bit, the contents of the FIFO are discarded. Any word currently being transmitted will be sent intact. This function is useful in aborting transfers. Bit 1 Receiver FIFO Reset. When a "1" is written to this bit, the contents of the FIFO are discarded. Any word currently being assembled in the shift register will be received intact. Bit 0 16550 FIFO Enable. When set, both the transmit and receive FIFOs are enabled. Any contents in the holding register, shift registers or FIFOs are lost when FIFOs are enabled or disabled. +0x02 read Interrupt Identification Register Bit 7 FIFOs enabled. On the 8250/16450 UART, this bit is zero. Bit 6 FIFOs enabled. On the 8250/16450 UART, this bit is zero. Bit 5 Reserved, always 0. Bit 4 Reserved, always 0. Bit 3 Interrupt ID Bit #2. On the 8250/16450 UART, this bit is zero. Bit 2 Interrupt ID Bit #1 Bit 1 Interrupt ID Bit #0.These three bits combine to report the category of event that caused the interrupt that is in progress. These categories have priorities, so if multiple categories of events occur at the same time, the UART will report the more important events first and the host must resolve the events in the order they are reported. All events that caused the current interrupt must be resolved before any new interrupts will be generated. (This is a limitation of the PC architecture.) 2 1 0 Priority Description 0 1 1 First Received Error (OE, PE, BI, or FE) 0 1 0 Second Received Data Available 1 1 0 Second Trigger level identification (Stale data in receive buffer) 0 0 1 Third Transmitter has room for more words (THRE) 0 0 0 Fourth Modem Status Change (-CTS, -DSR, -RI, or -DCD) Bit 0 Interrupt Pending Bit. If this bit is set to "0", then at least one interrupt is pending. +0x03 write/read Line Control Register (LCR) Bit 7 Divisor Latch Access Bit (DLAB). When set, access to the data transmit/receive register (THR/RBR) and the Interrupt Enable Register (IER) is disabled. Any access to these ports is now redirected to the Divisor Latch Registers. Setting this bit, loading the Divisor Registers, and clearing DLAB should be done with interrupts disabled. Bit 6 Set Break. When set to "1", the transmitter begins to transmit continuous Spacing until this bit is set to "0". This overrides any bits of characters that are being transmitted. Bit 5 Stick Parity. When parity is enabled, setting this bit causes parity to always be "1" or "0", based on the value of Bit 4. Bit 4 Even Parity Select (EPS). When parity is enabled and Bit 5 is "0", setting this bit causes even parity to be transmitted and expected. Otherwise, odd parity is used. Bit 3 Parity Enable (PEN). When set to "1", a parity bit is inserted between the last bit of the data and the Stop Bit. The UART will also expect parity to be present in the received data. Bit 2 Number of Stop Bits (STB). If set to "1" and using 5-bit data words, 1.5 Stop Bits are transmitted and expected in each data word. For 6, 7 and 8-bit data words, 2 Stop Bits are transmitted and expected. When this bit is set to "0", one Stop Bit is used on each data word. Bit 1 Word Length Select Bit #1 (WLSB1) Bit 0 Word Length Select Bit #0 (WLSB0) Together these bits specify the number of bits in each data word. 1 0 Word Length 0 0 5 Data Bits 0 1 6 Data Bits 1 0 7 Data Bits 1 1 8 Data Bits +0x04 write/read Modem Control Register (MCR) Bit 7 Reserved, always 0. Bit 6 Reserved, always 0. Bit 5 Reserved, always 0. Bit 4 Loop-Back Enable. When set to "1", the UART transmitter and receiver are internally connected together to allow diagnostic operations. In addition, the UART modem control outputs are connected to the UART modem control inputs. CTS is connected to RTS, DTR is connected to DSR, OUT1 is connected to RI, and OUT 2 is connected to DCD. Bit 3 OUT 2. An auxiliary output that the host processor may set high or low. In the IBM PC serial adapter (and most clones), OUT 2 is used to tri-state (disable) the interrupt signal from the 8250/16450/16550 UART. Bit 2 OUT 1. An auxiliary output that the host processor may set high or low. This output is not used on the IBM PC serial adapter. Bit 1 Request to Send (RTS). When set to "1", the output of the UART -RTS line is Low (Active). Bit 0 Data Terminal Ready (DTR). When set to "1", the output of the UART -DTR line is Low (Active). +0x05 write/read Line Status Register (LSR) Bit 7 Error in Receiver FIFO. On the 8250/16450 UART, this bit is zero. This bit is set to "1" when any of the bytes in the FIFO have one or more of the following error conditions: PE, FE, or BI. Bit 6 Transmitter Empty (TEMT). When set to "1", there are no words remaining in the transmit FIFO or the transmit shift register. The transmitter is completely idle. Bit 5 Transmitter Holding Register Empty (THRE). When set to "1", the FIFO (or holding register) now has room for at least one additional word to transmit. The transmitter may still be transmitting when this bit is set to "1". Bit 4 Break Interrupt (BI). The receiver has detected a Break signal. Bit 3 Framing Error (FE). A Start Bit was detected but the Stop Bit did not appear at the expected time. The received word is probably garbled. Bit 2 Parity Error (PE). The parity bit was incorrect for the word received. Bit 1 Overrun Error (OE). A new word was received and there was no room in the receive buffer. The newly-arrived word in the shift register is discarded. On 8250/16450 UARTs, the word in the holding register is discarded and the newly- arrived word is put in the holding register. Bit 0 Data Ready (DR) One or more words are in the receive FIFO that the host may read. A word must be completely received and moved from the shift register into the FIFO (or holding register for 8250/16450 designs) before this bit is set. +0x06 write/read Modem Status Register (MSR) Bit 7 Data Carrier Detect (DCD). Reflects the state of the DCD line on the UART. Bit 6 Ring Indicator (RI). Reflects the state of the RI line on the UART. Bit 5 Data Set Ready (DSR). Reflects the state of the DSR line on the UART. Bit 4 Clear To Send (CTS). Reflects the state of the CTS line on the UART. Bit 3 Delta Data Carrier Detect (DDCD). Set to "1" if the -DCD line has changed state one more time since the last time the MSR was read by the host. Bit 2 Trailing Edge Ring Indicator (TERI). Set to "1" if the -RI line has had a low to high transition since the last time the MSR was read by the host. Bit 1 Delta Data Set Ready (DDSR). Set to "1" if the -DSR line has changed state one more time since the last time the MSR was read by the host. Bit 0 Delta Clear To Send (DCTS). Set to "1" if the -CTS line has changed state one more time since the last time the MSR was read by the host. +0x07 write/read Scratch Register (SCR). This register performs no function in the UART. Any value can be written by the host to this location and read by the host later on. Beyond the 16550A UART Although National Semiconductor has not offered any components compatible with the 16550 that provide additional features, various other vendors have. Some of these components are described below. It should be understood that to effectively utilize these improvements, drivers may have to be provided by the chip vendor since most of the popular operating systems do not support features beyond those provided by the 16550. ST16650 By default this part is similar to the NS16550A, but an extended 32-byte send and receive buffer can be optionally enabled. Made by StarTech. TIL16660 By default this part behaves similar to the NS16550A, but an extended 64-byte send and receive buffer can be optionally enabled. Made by Texas Instruments. Hayes ESP This proprietary plug-in card contains a 2048-byte send and receive buffer, and supports data rates to 230.4Kbit/sec. Made by Hayes. In addition to these dumb UARTs, many vendors produce intelligent serial communication boards. This type of design usually provides a microprocessor that interfaces with several UARTs, processes and buffers the data, and then alerts the main PC processor when necessary. Because the UARTs are not directly accessed by the PC processor in this type of communication system, it is not necessary for the vendor to use UARTs that are compatible with the 8250, 16450, or the 16550 UART. This leaves the designer free to components that may have better performance characteristics. Configuring the <devicename>sio</devicename> driver The sio driver provides support for NS8250-, NS16450-, NS16550 and NS16550A-based EIA RS-232C (CCITT V.24) communications interfaces. Several multiport cards are supported as well. See the &man.sio.4; manual page for detailed technical documentation. Digi International (DigiBoard) PC/8 Contributed by &a.awebster;. 26 August 1995. Here is a config snippet from a machine with a Digi International PC/8 with 16550. It has 8 modems connected to these 8 lines, and they work just great. Do not forget to add options COM_MULTIPORT or it will not work very well! device sio4 at isa? port 0x100 tty flags 0xb05 device sio5 at isa? port 0x108 tty flags 0xb05 device sio6 at isa? port 0x110 tty flags 0xb05 device sio7 at isa? port 0x118 tty flags 0xb05 device sio8 at isa? port 0x120 tty flags 0xb05 device sio9 at isa? port 0x128 tty flags 0xb05 device sio10 at isa? port 0x130 tty flags 0xb05 device sio11 at isa? port 0x138 tty flags 0xb05 irq 9 vector siointr The trick in setting this up is that the MSB of the flags represent the last SIO port, in this case 11 so flags are 0xb05. Boca 16 Contributed by &a.whiteside;. 26 August 1995. The procedures to make a Boca 16 port board with FreeBSD are pretty straightforward, but you will need a couple things to make it work: You either need the kernel sources installed so you can recompile the necessary options or you will need someone else to compile it for you. The 2.0.5 default kernel does not come with multiport support enabled and you will need to add a device entry for each port anyways. Two, you will need to know the interrupt and IO setting for your Boca Board so you can set these options properly in the kernel. One important note — the actual UART chips for the Boca 16 are in the connector box, not on the internal board itself. So if you have it unplugged, probes of those ports will fail. I have never tested booting with the box unplugged and plugging it back in, and I suggest you do not either. If you do not already have a custom kernel configuration file set up, refer to Kernel Configuration chapter of the FreeBSD Handbook for general procedures. The following are the specifics for the Boca 16 board and assume you are using the kernel name MYKERNEL and editing with vi. Add the line options COM_MULTIPORT to the config file. Where the current device sion lines are, you will need to add 16 more devices. Only the last device includes the interrupt vector for the board. (See the &man.sio.4; manual page for detail as to why.) The following example is for a Boca Board with an interrupt of 3, and a base IO address 100h. The IO address for Each port is +8 hexadecimal from the previous port, thus the 100h, 108h, 110h... addresses. device sio1 at isa? port 0x100 tty flags 0x1005 device sio2 at isa? port 0x108 tty flags 0x1005 device sio3 at isa? port 0x110 tty flags 0x1005 device sio4 at isa? port 0x118 tty flags 0x1005 … device sio15 at isa? port 0x170 tty flags 0x1005 device sio16 at isa? port 0x178 tty flags 0x1005 irq 3 vector siointr The flags entry must be changed from this example unless you are using the exact same sio assignments. Flags are set according to 0xMYY where M indicates the minor number of the master port (the last port on a Boca 16) and YY indicates if FIFO is enabled or disabled(enabled), IRQ sharing is used(yes) and if there is an AST/4 compatible IRQ control register(no). In this example, flags 0x1005 indicates that the master port is sio16. If I added another board and assigned sio17 through sio28, the flags for all 16 ports on that board would be 0x1C05, where 1C indicates the minor number of the master port. Do not change the 05 setting. Save and complete the kernel configuration, recompile, install and reboot. Presuming you have successfully installed the recompiled kernel and have it set to the correct address and IRQ, your boot message should indicate the successful probe of the Boca ports as follows: (obviously the sio numbers, IO and IRQ could be different) sio1 at 0x100-0x107 flags 0x1005 on isa sio1: type 16550A (multiport) sio2 at 0x108-0x10f flags 0x1005 on isa sio2: type 16550A (multiport) sio3 at 0x110-0x117 flags 0x1005 on isa sio3: type 16550A (multiport) sio4 at 0x118-0x11f flags 0x1005 on isa sio4: type 16550A (multiport) sio5 at 0x120-0x127 flags 0x1005 on isa sio5: type 16550A (multiport) sio6 at 0x128-0x12f flags 0x1005 on isa sio6: type 16550A (multiport) sio7 at 0x130-0x137 flags 0x1005 on isa sio7: type 16550A (multiport) sio8 at 0x138-0x13f flags 0x1005 on isa sio8: type 16550A (multiport) sio9 at 0x140-0x147 flags 0x1005 on isa sio9: type 16550A (multiport) sio10 at 0x148-0x14f flags 0x1005 on isa sio10: type 16550A (multiport) sio11 at 0x150-0x157 flags 0x1005 on isa sio11: type 16550A (multiport) sio12 at 0x158-0x15f flags 0x1005 on isa sio12: type 16550A (multiport) sio13 at 0x160-0x167 flags 0x1005 on isa sio13: type 16550A (multiport) sio14 at 0x168-0x16f flags 0x1005 on isa sio14: type 16550A (multiport) sio15 at 0x170-0x177 flags 0x1005 on isa sio15: type 16550A (multiport) sio16 at 0x178-0x17f irq 3 flags 0x1005 on isa sio16: type 16550A (multiport master) If the messages go by too fast to see, &prompt.root; dmesg | more will show you the boot messages. Next, appropriate entries in /dev for the devices must be made using the /dev/MAKEDEV script. After becoming root: &prompt.root; cd /dev &prompt.root; ./MAKEDEV tty1 &prompt.root; ./MAKEDEV cua1 (everything in between) &prompt.root; ./MAKEDEV ttyg &prompt.root; ./MAKEDEV cuag If you do not want or need call-out devices for some reason, you can dispense with making the cua* devices. If you want a quick and sloppy way to make sure the devices are working, you can simply plug a modem into each port and (as root) &prompt.root; echo at > ttyd* for each device you have made. You should see the RX lights flash for each working port. Support for Cheap Multi-UART Cards Contributed by Helge Oldach hmo@sep.hamburg.com, September 1999 Ever wondered about FreeBSD support for your 20$ multi-I/O card with two (or more) COM ports, sharing IRQs? Here is how: Usually the only option to support these kind of boards is to use a distinct IRQ for each port. For example, if your CPU board has an on-board COM1 port (aka sio0–I/O address 0x3F8 and IRQ 4) and you have an extension board with two UARTs, you will commonly need to configure them as COM2 (aka sio1–I/O address 0x2F8 and IRQ 3), and the third port (aka sio2) as I/O 0x3E8 and IRQ 5. Obviously this is a waste of IRQ resources, as it should be basically possible to run both extension board ports using a single IRQ with the COM_MULTIPORT configuration described in the previous sections. Such cheap I/O boards commonly have a 4 by 3 jumper matrix for the COM ports, similar to the following: o o o * Port A | o * o * Port B | o * o o IRQ 2 3 4 5 Shown here is port A wired for IRQ 5 and port B wired for IRQ 3. The IRQ columns on your specific board may vary—other boards may supply jumpers for IRQs 3, 4, 5, and 7 instead. One could conclude that wiring both ports for IRQ 3 using a handcrafted wire-made jumper covering all three connection points in the IRQ 3 column would solve the issue, but no. You cannot duplicate IRQ 3 because the output drivers of each UART are wired in a totem pole fashion, so if one of the UARTs drives IRQ 3, the output signal will not be what you would expect. Depending on the implementation of the extension board or your motherboard, the IRQ 3 line will continuously stay up, or always stay low. You need to decouple the IRQ drivers for the two UARTs, so that the IRQ line of the board only goes up if (and only if) one of the UARTs asserts a IRQ, and stays low otherwise. The solution was proposed by Joerg Wunsch j@ida.interface-business.de: To solder up a wired-or consisting of two diodes (Germanium or Schottky-types strongly preferred) and a 1 kOhm resistor. Here is the schematic, starting from the 4 by 3 jumper field above: Diode +---------->|-------+ / | o * o o | 1 kOhm Port A +----|######|-------+ o * o o | | Port B `-------------------+ ==+== o * o o | Ground \ | +--------->|-------+ IRQ 2 3 4 5 Diode The cathodes of the diodes are connected to a common point, together with a 1 kOhm pull-down resistor. It is essential to connect the resistor to ground to avoid floating of the IRQ line on the bus. Now we are ready to configure a kernel. Staying with this example, we would configure: # standard on-board COM1 port device sio0 at isa? port "IO_COM1" tty flags 0x10 # patched-up multi-I/O extension board options COM_MULTIPORT device sio1 at isa? port "IO_COM2" tty flags 0x205 device sio2 at isa? port "IO_COM3" tty flags 0x205 irq 3 Note that the flags setting for sio1 and sio2 is truly essential; refer to &man.sio.4; for details. (Generally, the 2 in the "flags" attribute refers to sio2 which holds the IRQ, and you surely want a 5 low nibble.) With kernel verbose mode turned on this should yield something similar to this: sio0: irq maps: 0x1 0x11 0x1 0x1 sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A sio1: irq maps: 0x1 0x9 0x1 0x1 sio1 at 0x2f8-0x2ff flags 0x205 on isa sio1: type 16550A (multiport) sio2: irq maps: 0x1 0x9 0x1 0x1 sio2 at 0x3e8-0x3ef irq 3 flags 0x205 on isa sio2: type 16550A (multiport master) Though /sys/i386/isa/sio.c is somewhat cryptic with its use of the irq maps array above, the basic idea is that you observe 0x1 in the first, third, and fourth place. This means that the corresponding IRQ was set upon output and cleared after, which is just what we would expect. If your kernel does not display this behavior, most likely there is something wrong with your wiring. Configuring the <devicename>cy</devicename> driver Contributed by Alex Nash. 6 June 1996. The Cyclades multiport cards are based on the cy driver instead of the usual sio driver used by other multiport cards. Configuration is a simple matter of: Add the cy device to your kernel configuration (note that your irq and iomem settings may differ). device cy0 at isa? tty irq 10 iomem 0xd4000 iosiz 0x2000 vector cyintr Rebuild and install the new kernel. Make the device nodes by typing (the following example assumes an 8-port board): &prompt.root; cd /dev &prompt.root; for i in 0 1 2 3 4 5 6 7;do ./MAKEDEV cuac$i ttyc$i;done If appropriate, add dialup entries to /etc/ttys by duplicating serial device (ttyd) entries and using ttyc in place of ttyd. For example: ttyc0 "/usr/libexec/getty std.38400" unknown on insecure ttyc1 "/usr/libexec/getty std.38400" unknown on insecure ttyc2 "/usr/libexec/getty std.38400" unknown on insecure … ttyc7 "/usr/libexec/getty std.38400" unknown on insecure Reboot with the new kernel. Configuring the <devicename>si</devicename> driver Contributed by &a.nsayer;. 25 March 1998. The Specialix SI/XIO and SX multiport cards use the si driver. A single machine can have up to 4 host cards. The following host cards are supported: ISA SI/XIO host card (2 versions) EISA SI/XIO host card PCI SI/XIO host card ISA SX host card PCI SX host card Although the SX and SI/XIO host cards look markedly different, their functionality are basically the same. The host cards do not use I/O locations, but instead require a 32K chunk of memory. The factory configuration for ISA cards places this at 0xd0000-0xd7fff. They also require an IRQ. PCI cards will, of course, auto-configure themselves. You can attach up to 4 external modules to each host card. The external modules contain either 4 or 8 serial ports. They come in the following varieties: SI 4 or 8 port modules. Up to 57600 bps on each port supported. XIO 8 port modules. Up to 115200 bps on each port supported. One type of XIO module has 7 serial and 1 parallel port. SXDC 8 port modules. Up to 921600 bps on each port supported. Like XIO, a module is available with one parallel port as well. To configure an ISA host card, add the following line to your kernel configuration file, changing the numbers as appropriate: device si0 at isa? tty iomem 0xd0000 irq 11 Valid IRQ numbers are 9, 10, 11, 12 and 15 for SX ISA host cards and 11, 12 and 15 for SI/XIO ISA host cards. To configure an EISA or PCI host card, use this line: device si0 After adding the configuration entry, rebuild and install your new kernel. After rebooting with the new kernel, you need to make the device nodes in /dev. The MAKEDEV script will take care of this for you. Count how many total ports you have and type: &prompt.root; cd /dev &prompt.root; ./MAKEDEV ttyAnn cuaAnn (where nn is the number of ports) If you want login prompts to appear on these ports, you will need to add lines like this to /etc/ttys: ttyA01 "/usr/libexec/getty std.9600" vt100 on insecure Change the terminal type as appropriate. For modems, dialup or unknown is fine.

Storage Devices Wilko Bulte

wilko@FreeBSD.org

$FreeBSD$ This article talks about storage devices with FreeBSD. Using ESDI hard disks Copyright © 1995, &a.wilko;. 24 September 1995. ESDI is an acronym that means Enhanced Small Device Interface. It is loosely based on the good old ST506/412 interface originally devised by Seagate Technology, the makers of the first affordable 5.25" winchester disk. The acronym says Enhanced, and rightly so. In the first place the speed of the interface is higher, 10 or 15 Mbits/second instead of the 5 Mbits/second of ST412 interfaced drives. Secondly some higher level commands are added, making the ESDI interface somewhat 'smarter' to the operating system driver writers. It is by no means as smart as SCSI by the way. ESDI is standardized by ANSI. Capacities of the drives are boosted by putting more sectors on each track. Typical is 35 sectors per track, high capacity drives I have seen were up to 54 sectors/track. Although ESDI has been largely obsoleted by IDE and SCSI interfaces, the availability of free or cheap surplus drives makes them ideal for low (or now) budget systems. Concepts of ESDI Physical connections The ESDI interface uses two cables connected to each drive. One cable is a 34 pin flat cable edge connector that carries the command and status signals from the controller to the drive and vice-versa. The command cable is daisy chained between all the drives. So, it forms a bus onto which all drives are connected. The second cable is a 20 pin flat cable edge connector that carries the data to and from the drive. This cable is radially connected, so each drive has its own direct connection to the controller. To the best of my knowledge PC ESDI controllers are limited to using a maximum of 2 drives per controller. This is compatibility feature(?) left over from the WD1003 standard that reserves only a single bit for device addressing. Device addressing On each command cable a maximum of 7 devices and 1 controller can be present. To enable the controller to uniquely identify which drive it addresses, each ESDI device is equipped with jumpers or switches to select the devices address. On PC type controllers the first drive is set to address 0, the second disk to address 1. Always make sure you set each disk to an unique address! So, on a PC with its two drives/controller maximum the first drive is drive 0, the second is drive 1. Termination The daisy chained command cable (the 34 pin cable remember?) needs to be terminated at the last drive on the chain. For this purpose ESDI drives come with a termination resistor network that can be removed or disabled by a jumper when it is not used. So, one and only one drive, the one at the farthest end of the command cable has its terminator installed/enabled. The controller automatically terminates the other end of the cable. Please note that this implies that the controller must be at one end of the cable and not in the middle. Using ESDI disks with FreeBSD Why is ESDI such a pain to get working in the first place? People who tried ESDI disks with FreeBSD are known to have developed a profound sense of frustration. A combination of factors works against you to produce effects that are hard to understand when you have never seen them before. This has also led to the popular legend ESDI and FreeBSD is a plain NO-GO. The following sections try to list all the pitfalls and solutions. ESDI speed variants As briefly mentioned before, ESDI comes in two speed flavors. The older drives and controllers use a 10 Mbits/second data transfer rate. Newer stuff uses 15 Mbits/second. It is not hard to imagine that 15 Mbits/second drive cause problems on controllers laid out for 10 Mbits/second. As always, consult your controller and drive documentation to see if things match. Stay on track Mainstream ESDI drives use 34 to 36 sectors per track. Most (older) controllers cannot handle more than this number of sectors. Newer, higher capacity, drives use higher numbers of sectors per track. For instance, I own a 670 MB drive that has 54 sectors per track. In my case, the controller could not handle this number of sectors. It proved to work well except that it only used 35 sectors on each track. This meant losing a lot of disk space. Once again, check the documentation of your hardware for more info. Going out-of-spec like in the example might or might not work. Give it a try or get another more capable controller. Hard or soft sectoring Most ESDI drives allow hard or soft sectoring to be selected using a jumper. Hard sectoring means that the drive will produce a sector pulse on the start of each new sector. The controller uses this pulse to tell when it should start to write or read. Hard sectoring allows a selection of sector size (normally 256, 512 or 1024 bytes per formatted sector). FreeBSD uses 512 byte sectors. The number of sectors per track also varies while still using the same number of bytes per formatted sector. The number of unformatted bytes per sector varies, dependent on your controller it needs more or less overhead bytes to work correctly. Pushing more sectors on a track of course gives you more usable space, but might give problems if your controller needs more bytes than the drive offers. In case of soft sectoring, the controller itself determines where to start/stop reading or writing. For ESDI hard sectoring is the default (at least on everything I came across). I never felt the urge to try soft sectoring. In general, experiment with sector settings before you install FreeBSD because you need to re-run the low-level format after each change. Low level formatting ESDI drives need to be low level formatted before they are usable. A reformat is needed whenever you figgle with the number of sectors/track jumpers or the physical orientation of the drive (horizontal, vertical). So, first think, then format. The format time must not be underestimated, for big disks it can take hours. After a low level format, a surface scan is done to find and flag bad sectors. Most disks have a manufacturer bad block list listed on a piece of paper or adhesive sticker. In addition, on most disks the list is also written onto the disk. Please use the manufacturer's list. It is much easier to remap a defect now than after FreeBSD is installed. Stay away from low-level formatters that mark all sectors of a track as bad as soon as they find one bad sector. Not only does this waste space, it also and more importantly causes you grief with bad144 (see the section on bad144). Translations Translations, although not exclusively a ESDI-only problem, might give you real trouble. Translations come in multiple flavors. Most of them have in common that they attempt to work around the limitations posed upon disk geometries by the original IBM PC/AT design (thanks IBM!). First of all there is the (in)famous 1024 cylinder limit. For a system to be able to boot, the stuff (whatever operating system) must be in the first 1024 cylinders of a disk. Only 10 bits are available to encode the cylinder number. For the number of sectors the limit is 64 (0-63). When you combine the 1024 cylinder limit with the 16 head limit (also a design feature) you max out at fairly limited disk sizes. To work around this problem, the manufacturers of ESDI PC controllers added a BIOS prom extension on their boards. This BIOS extension handles disk I/O for booting (and for some operating systems all disk I/O) by using translation. For instance, a big drive might be presented to the system as having 32 heads and 64 sectors/track. The result is that the number of cylinders is reduced to something below 1024 and is therefore usable by the system without problems. It is noteworthy to know that FreeBSD does not use the BIOS after its kernel has started. More on this later. A second reason for translations is the fact that most older system BIOSes could only handle drives with 17 sectors per track (the old ST412 standard). Newer system BIOSes usually have a user-defined drive type (in most cases this is drive type 47). Whatever you do to translations after reading this document, keep in mind that if you have multiple operating systems on the same disk, all must use the same translation While on the subject of translations, I have seen one controller type (but there are probably more like this) offer the option to logically split a drive in multiple partitions as a BIOS option. I had select 1 drive == 1 partition because this controller wrote this info onto the disk. On power-up it read the info and presented itself to the system based on the info from the disk. Spare sectoring Most ESDI controllers offer the possibility to remap bad sectors. During/after the low-level format of the disk bad sectors are marked as such, and a replacement sector is put in place (logically of course) of the bad one. In most cases the remapping is done by using N-1 sectors on each track for actual data storage, and sector N itself is the spare sector. N is the total number of sectors physically available on the track. The idea behind this is that the operating system sees a 'perfect' disk without bad sectors. In the case of FreeBSD this concept is not usable. The problem is that the translation from bad to good is performed by the BIOS of the ESDI controller. FreeBSD, being a true 32 bit operating system, does not use the BIOS after it has been booted. Instead, it has device drivers that talk directly to the hardware. So: do not use spare sectoring, bad block remapping or whatever it may be called by the controller manufacturer when you want to use the disk for FreeBSD. Bad block handling The preceding section leaves us with a problem. The controller's bad block handling is not usable and still FreeBSD's filesystems assume perfect media without any flaws. To solve this problem, FreeBSD use the bad144 tool. Bad144 (named after a Digital Equipment standard for bad block handling) scans a FreeBSD slice for bad blocks. Having found these bad blocks, it writes a table with the offending block numbers to the end of the FreeBSD slice. When the disk is in operation, the disk accesses are checked against the table read from the disk. Whenever a block number is requested that is in the bad144 list, a replacement block (also from the end of the FreeBSD slice) is used. In this way, the bad144 replacement scheme presents 'perfect' media to the FreeBSD filesystems. There are a number of potential pitfalls associated with the use of bad144. First of all, the slice cannot have more than 126 bad sectors. If your drive has a high number of bad sectors, you might need to divide it into multiple FreeBSD slices each containing less than 126 bad sectors. Stay away from low-level format programs that mark every sector of a track as bad when they find a flaw on the track. As you can imagine, the 126 limit is quickly reached when the low-level format is done this way. Second, if the slice contains the root filesystem, the slice should be within the 1024 cylinder BIOS limit. During the boot process the bad144 list is read using the BIOS and this only succeeds when the list is within the 1024 cylinder limit. The restriction is not that only the root filesystem must be within the 1024 cylinder limit, but rather the entire slice that contains the root filesystem. Kernel configuration ESDI disks are handled by the same wddriver as IDE and ST412 MFM disks. The wd driver should work for all WD1003 compatible interfaces. Most hardware is jumperable for one of two different I/O address ranges and IRQ lines. This allows you to have two wd type controllers in one system. When your hardware allows non-standard strappings, you can use these with FreeBSD as long as you enter the correct info into the kernel config file. An example from the kernel config file (they live in /sys/i386/conf BTW). # First WD compatible controller controller wdc0 at isa? port "IO_WD1" bio irq 14 vector wdintr disk wd0 at wdc0 drive 0 disk wd1 at wdc0 drive 1 # Second WD compatible controller controller wdc1 at isa? port "IO_WD2" bio irq 15 vector wdintr disk wd2 at wdc1 drive 0 disk wd3 at wdc1 drive 1 Particulars on ESDI hardware Adaptec 2320 controllers I successfully installed FreeBSD onto a ESDI disk controlled by a ACB-2320. No other operating system was present on the disk. To do so I low level formatted the disk using NEFMT.EXE (ftpable from www.adaptec.com) and answered NO to the question whether the disk should be formatted with a spare sector on each track. The BIOS on the ACD-2320 was disabled. I used the free configurable option in the system BIOS to allow the BIOS to boot it. Before using NEFMT.EXE I tried to format the disk using the ACB-2320 BIOS built-in formatter. This proved to be a show stopper, because it did not give me an option to disable spare sectoring. With spare sectoring enabled the FreeBSD installation process broke down on the bad144 run. Please check carefully which ACB-232xy variant you have. The x is either 0 or 2, indicating a controller without or with a floppy controller on board. The y is more interesting. It can either be a blank, a A-8 or a D. A blank indicates a plain 10 Mbits/second controller. An A-8 indicates a 15 Mbits/second controller capable of handling 52 sectors/track. A D means a 15 Mbits/second controller that can also handle drives with > 36 sectors/track (also 52 ?). All variations should be capable of using 1:1 interleaving. Use 1:1, FreeBSD is fast enough to handle it. Western Digital WD1007 controllers I successfully installed FreeBSD onto a ESDI disk controlled by a WD1007 controller. To be precise, it was a WD1007-WA2. Other variations of the WD1007 do exist. To get it to work, I had to disable the sector translation and the WD1007's onboard BIOS. This implied I could not use the low-level formatter built into this BIOS. Instead, I grabbed WDFMT.EXE from www.wdc.com Running this formatted my drive just fine. Ultrastor U14F controllers According to multiple reports from the net, Ultrastor ESDI boards work OK with FreeBSD. I lack any further info on particular settings. Further reading If you intend to do some serious ESDI hacking, you might want to have the official standard at hand: The latest ANSI X3T10 committee document is: Enhanced Small Device Interface (ESDI) [X3.170-1990/X3.170a-1991] [X3T10/792D Rev 11] On Usenet the newsgroup comp.periphs is a noteworthy place to look for more info. The World Wide Web (WWW) also proves to be a very handy info source: For info on Adaptec ESDI controllers see http://www.adaptec.com/. For info on Western Digital controllers see http://www.wdc.com/. Thanks to... Andrew Gordon for sending me an Adaptec 2320 controller and ESDI disk for testing. What is SCSI? Copyright © 1995, &a.wilko;. July 6, 1996. SCSI is an acronym for Small Computer Systems Interface. It is an ANSI standard that has become one of the leading I/O buses in the computer industry. The foundation of the SCSI standard was laid by Shugart Associates (the same guys that gave the world the first mini floppy disks) when they introduced the SASI bus (Shugart Associates Standard Interface). After some time an industry effort was started to come to a more strict standard allowing devices from different vendors to work together. This effort was recognized in the ANSI SCSI-1 standard. The SCSI-1 standard (approximately 1985) is rapidly becoming obsolete. The current standard is SCSI-2 (see Further reading), with SCSI-3 on the drawing boards. In addition to a physical interconnection standard, SCSI defines a logical (command set) standard to which disk devices must adhere. This standard is called the Common Command Set (CCS) and was developed more or less in parallel with ANSI SCSI-1. SCSI-2 includes the (revised) CCS as part of the standard itself. The commands are dependent on the type of device at hand. It does not make much sense of course to define a Write command for a scanner. The SCSI bus is a parallel bus, which comes in a number of variants. The oldest and most used is an 8 bit wide bus, with single-ended signals, carried on 50 wires. (If you do not know what single-ended means, do not worry, that is what this document is all about.) Modern designs also use 16 bit wide buses, with differential signals. This allows transfer speeds of 20Mbytes/second, on cables lengths of up to 25 meters. SCSI-2 allows a maximum bus width of 32 bits, using an additional cable. Quickly emerging are Ultra SCSI (also called Fast-20) and Ultra2 (also called Fast-40). Fast-20 is 20 million transfers per second (20 Mbytes/sec on a 8 bit bus), Fast-40 is 40 million transfers per second (40 Mbytes/sec on a 8 bit bus). Most hard drives sold today are single-ended Ultra SCSI (8 or 16 bits). Of course the SCSI bus not only has data lines, but also a number of control signals. A very elaborate protocol is part of the standard to allow multiple devices to share the bus in an efficient manner. In SCSI-2, the data is always checked using a separate parity line. In pre-SCSI-2 designs parity was optional. In SCSI-3 even faster bus types are introduced, along with a serial SCSI busses that reduces the cabling overhead and allows a higher maximum bus length. You might see names like SSA and fibre channel in this context. None of the serial buses are currently in widespread use (especially not in the typical FreeBSD environment). For this reason the serial bus types are not discussed any further. As you could have guessed from the description above, SCSI devices are intelligent. They have to be to adhere to the SCSI standard (which is over 2 inches thick BTW). So, for a hard disk drive for instance you do not specify a head/cylinder/sector to address a particular block, but simply the number of the block you want. Elaborate caching schemes, automatic bad block replacement etc are all made possible by this 'intelligent device' approach. On a SCSI bus, each possible pair of devices can communicate. Whether their function allows this is another matter, but the standard does not restrict it. To avoid signal contention, the 2 devices have to arbitrate for the bus before using it. The philosophy of SCSI is to have a standard that allows older-standard devices to work with newer-standard ones. So, an old SCSI-1 device should normally work on a SCSI-2 bus. I say Normally, because it is not absolutely sure that the implementation of an old device follows the (old) standard closely enough to be acceptable on a new bus. Modern devices are usually more well-behaved, because the standardization has become more strict and is better adhered to by the device manufacturers. Generally speaking, the chances of getting a working set of devices on a single bus is better when all the devices are SCSI-2 or newer. This implies that you do not have to dump all your old stuff when you get that shiny 2GB disk: I own a system on which a pre-SCSI-1 disk, a SCSI-2 QIC tape unit, a SCSI-1 helical scan tape unit and 2 SCSI-1 disks work together quite happily. From a performance standpoint you might want to separate your older and newer (=faster) devices however. Components of SCSI As said before, SCSI devices are smart. The idea is to put the knowledge about intimate hardware details onto the SCSI device itself. In this way, the host system does not have to worry about things like how many heads a hard disks has, or how many tracks there are on a specific tape device. If you are curious, the standard specifies commands with which you can query your devices on their hardware particulars. FreeBSD uses this capability during boot to check out what devices are connected and whether they need any special treatment. The advantage of intelligent devices is obvious: the device drivers on the host can be made in a much more generic fashion, there is no longer a need to change (and qualify!) drivers for every odd new device that is introduced. For cabling and connectors there is a golden rule: get good stuff. With bus speeds going up all the time you will save yourself a lot of grief by using good material. So, gold plated connectors, shielded cabling, sturdy connector hoods with strain reliefs etc are the way to go. Second golden rule: do no use cables longer than necessary. I once spent 3 days hunting down a problem with a flaky machine only to discover that shortening the SCSI bus by 1 meter solved the problem. And the original bus length was well within the SCSI specification. SCSI bus types From an electrical point of view, there are two incompatible bus types: single-ended and differential. This means that there are two different main groups of SCSI devices and controllers, which cannot be mixed on the same bus. It is possible however to use special converter hardware to transform a single-ended bus into a differential one (and vice versa). The differences between the bus types are explained in the next sections. In lots of SCSI related documentation there is a sort of jargon in use to abbreviate the different bus types. A small list: FWD: Fast Wide Differential FND: Fast Narrow Differential SE: Single Ended FN: Fast Narrow etc. With a minor amount of imagination one can usually imagine what is meant. Wide is a bit ambiguous, it can indicate 16 or 32 bit buses. As far as I know, the 32 bit variant is not (yet) in use, so wide normally means 16 bit. Fast means that the timing on the bus is somewhat different, so that on a narrow (8 bit) bus 10 Mbytes/sec are possible instead of 5 Mbytes/sec for 'slow' SCSI. As discussed before, bus speeds of 20 and 40 million transfers/second are also emerging (Fast-20 == Ultra SCSI and Fast-40 == Ultra2 SCSI). The data lines > 8 are only used for data transfers and device addressing. The transfers of commands and status messages etc are only performed on the lowest 8 data lines. The standard allows narrow devices to operate on a wide bus. The usable bus width is negotiated between the devices. You have to watch your device addressing closely when mixing wide and narrow. Single ended buses A single-ended SCSI bus uses signals that are either 5 Volts or 0 Volts (indeed, TTL levels) and are relative to a COMMON ground reference. A singled ended 8 bit SCSI bus has approximately 25 ground lines, who are all tied to a single `rail' on all devices. A standard single ended bus has a maximum length of 6 meters. If the same bus is used with fast-SCSI devices, the maximum length allowed drops to 3 meters. Fast-SCSI means that instead of 5Mbytes/sec the bus allows 10Mbytes/sec transfers. Fast-20 (Ultra SCSI) and Fast-40 allow for 20 and 40 million transfers/second respectively. So, F20 is 20 Mbytes/second on a 8 bit bus, 40 Mbytes/second on a 16 bit bus etc. For F20 the max bus length is 1.5 meters, for F40 it becomes 0.75 meters. Be aware that F20 is pushing the limits quite a bit, so you will quickly find out if your SCSI bus is electrically sound. If some devices on your bus use 'fast' to communicate your bus must adhere to the length restrictions for fast buses! It is obvious that with the newer fast-SCSI devices the bus length can become a real bottleneck. This is why the differential SCSI bus was introduced in the SCSI-2 standard. For connector pinning and connector types please refer to the SCSI-2 standard (see Further reading) itself, connectors etc are listed there in painstaking detail. Beware of devices using non-standard cabling. For instance Apple uses a 25pin D-type connecter (like the one on serial ports and parallel printers). Considering that the official SCSI bus needs 50 pins you can imagine the use of this connector needs some 'creative cabling'. The reduction of the number of ground wires they used is a bad idea, you better stick to 50 pins cabling in accordance with the SCSI standard. For Fast-20 and 40 do not even think about buses like this. Differential buses A differential SCSI bus has a maximum length of 25 meters. Quite a difference from the 3 meters for a single-ended fast-SCSI bus. The idea behind differential signals is that each bus signal has its own return wire. So, each signal is carried on a (preferably twisted) pair of wires. The voltage difference between these two wires determines whether the signal is asserted or de-asserted. To a certain extent the voltage difference between ground and the signal wire pair is not relevant (do not try 10 kVolts though). It is beyond the scope of this document to explain why this differential idea is so much better. Just accept that electrically seen the use of differential signals gives a much better noise margin. You will normally find differential buses in use for inter-cabinet connections. Because of the lower cost single ended is mostly used for shorter buses like inside cabinets. There is nothing that stops you from using differential stuff with FreeBSD, as long as you use a controller that has device driver support in FreeBSD. As an example, Adaptec marketed the AHA1740 as a single ended board, whereas the AHA1744 was differential. The software interface to the host is identical for both. Terminators Terminators in SCSI terminology are resistor networks that are used to get a correct impedance matching. Impedance matching is important to get clean signals on the bus, without reflections or ringing. If you once made a long distance telephone call on a bad line you probably know what reflections are. With 20Mbytes/sec traveling over your SCSI bus, you do not want signals echoing back. Terminators come in various incarnations, with more or less sophisticated designs. Of course, there are internal and external variants. Many SCSI devices come with a number of sockets in which a number of resistor networks can (must be!) installed. If you remove terminators from a device, carefully store them. You will need them when you ever decide to reconfigure your SCSI bus. There is enough variation in even these simple tiny things to make finding the exact replacement a frustrating business. There are also SCSI devices that have a single jumper to enable or disable a built-in terminator. There are special terminators you can stick onto a flat cable bus. Others look like external connectors, or a connector hood without a cable. So, lots of choice as you can see. There is much debate going on if and when you should switch from simple resistor (passive) terminators to active terminators. Active terminators contain slightly more elaborate circuit to give cleaner bus signals. The general consensus seems to be that the usefulness of active termination increases when you have long buses and/or fast devices. If you ever have problems with your SCSI buses you might consider trying an active terminator. Try to borrow one first, they reputedly are quite expensive. Please keep in mind that terminators for differential and single-ended buses are not identical. You should not mix the two variants. OK, and now where should you install your terminators? This is by far the most misunderstood part of SCSI. And it is by far the simplest. The rule is: every single line on the SCSI bus has 2 (two) terminators, one at each end of the bus. So, two and not one or three or whatever. Do yourself a favor and stick to this rule. It will save you endless grief, because wrong termination has the potential to introduce highly mysterious bugs. (Note the potential here; the nastiest part is that it may or may not work.) A common pitfall is to have an internal (flat) cable in a machine and also an external cable attached to the controller. It seems almost everybody forgets to remove the terminators from the controller. The terminator must now be on the last external device, and not on the controller! In general, every reconfiguration of a SCSI bus must pay attention to this. Termination is to be done on a per-line basis. This means if you have both narrow and wide buses connected to the same host adapter, you need to enable termination on the higher 8 bits of the bus on the adapter (as well as the last devices on each bus, of course). What I did myself is remove all terminators from my SCSI devices and controllers. I own a couple of external terminators, for both the Centronics-type external cabling and for the internal flat cable connectors. This makes reconfiguration much easier. On modern devices, sometimes integrated terminators are used. These things are special purpose integrated circuits that can be enabled or disabled with a control pin. It is not necessary to physically remove them from a device. You may find them on newer host adapters, sometimes they are software configurable, using some sort of setup tool. Some will even auto-detect the cables attached to the connectors and automatically set up the termination as necessary. At any rate, consult your documentation! Terminator power The terminators discussed in the previous chapter need power to operate properly. On the SCSI bus, a line is dedicated to this purpose. So, simple huh? Not so. Each device can provide its own terminator power to the terminator sockets it has on-device. But if you have external terminators, or when the device supplying the terminator power to the SCSI bus line is switched off you are in trouble. The idea is that initiators (these are devices that initiate actions on the bus, a discussion follows) must supply terminator power. All SCSI devices are allowed (but not required) to supply terminator power. To allow for un-powered devices on a bus, the terminator power must be supplied to the bus via a diode. This prevents the backflow of current to un-powered devices. To prevent all kinds of nastiness, the terminator power is usually fused. As you can imagine, fuses might blow. This can, but does not have to, lead to a non functional bus. If multiple devices supply terminator power, a single blown fuse will not put you out of business. A single supplier with a blown fuse certainly will. Clever external terminators sometimes have a LED indication that shows whether terminator power is present. In newer designs auto-restoring fuses that 'reset' themselves after some time are sometimes used. Device addressing Because the SCSI bus is, ehh, a bus there must be a way to distinguish or address the different devices connected to it. This is done by means of the SCSI or target ID. Each device has a unique target ID. You can select the ID to which a device must respond using a set of jumpers, or a dip switch, or something similar. Some SCSI host adapters let you change the target ID from the boot menu. (Yet some others will not let you change the ID from 7.) Consult the documentation of your device for more information. Beware of multiple devices configured to use the same ID. Chaos normally reigns in this case. A pitfall is that one of the devices sharing the same ID sometimes even manages to answer to I/O requests! For an 8 bit bus, a maximum of 8 targets is possible. The maximum is 8 because the selection is done bitwise using the 8 data lines on the bus. For wide buses this increases to the number of data lines (usually 16). A narrow SCSI device can not communicate with a SCSI device with a target ID larger than 7. This means it is generally not a good idea to move your SCSI host adapter's target ID to something higher than 7 (or your CDROM will stop working). The higher the SCSI target ID, the higher the priority the devices has. When it comes to arbitration between devices that want to use the bus at the same time, the device that has the highest SCSI ID will win. This also means that the SCSI host adapter usually uses target ID 7. Note however that the lower 8 IDs have higher priorities than the higher 8 IDs on a wide-SCSI bus. Thus, the order of target IDs is: [7 6 .. 1 0 15 14 .. 9 8] on a wide-SCSI system. (If you are wondering why the lower 8 have higher priority, read the previous paragraph for a hint.) For a further subdivision, the standard allows for Logical Units or LUNs for short. A single target ID may have multiple LUNs. For example, a tape device including a tape changer may have LUN 0 for the tape device itself, and LUN 1 for the tape changer. In this way, the host system can address each of the functional units of the tape changer as desired. Bus layout SCSI buses are linear. So, not shaped like Y-junctions, star topologies, rings, cobwebs or whatever else people might want to invent. One of the most common mistakes is for people with wide-SCSI host adapters to connect devices on all three connecters (external connector, internal wide connector, internal narrow connector). Do not do that. It may appear to work if you are really lucky, but I can almost guarantee that your system will stop functioning at the most unfortunate moment (this is also known as Murphy's law). You might notice that the terminator issue discussed earlier becomes rather hairy if your bus is not linear. Also, if you have more connectors than devices on your internal SCSI cable, make sure you attach devices on connectors on both ends instead of using the connectors in the middle and let one or both ends dangle. This will screw up the termination of the bus. The electrical characteristics, its noise margins and ultimately the reliability of it all are tightly related to linear bus rule. Stick to the linear bus rule! Using SCSI with FreeBSD About translations, BIOSes and magic... As stated before, you should first make sure that you have a electrically sound bus. When you want to use a SCSI disk on your PC as boot disk, you must aware of some quirks related to PC BIOSes. The PC BIOS in its first incarnation used a low level physical interface to the hard disk. So, you had to tell the BIOS (using a setup tool or a BIOS built-in setup) how your disk physically looked like. This involved stating number of heads, number of cylinders, number of sectors per track, obscure things like precompensation and reduced write current cylinder etc. One might be inclined to think that since SCSI disks are smart you can forget about this. Alas, the arcane setup issue is still present today. The system BIOS needs to know how to access your SCSI disk with the head/cyl/sector method in order to load the FreeBSD kernel during boot. The SCSI host adapter or SCSI controller you have put in your AT/EISA/PCI/whatever bus to connect your disk therefore has its own on-board BIOS. During system startup, the SCSI BIOS takes over the hard disk interface routines from the system BIOS. To fool the system BIOS, the system setup is normally set to No hard disk present. Obvious, is it not? The SCSI BIOS itself presents to the system a so called translated drive. This means that a fake drive table is constructed that allows the PC to boot the drive. This translation is often (but not always) done using a pseudo drive with 64 heads and 32 sectors per track. By varying the number of cylinders, the SCSI BIOS adapts to the actual drive size. It is useful to note that 32 * 64 / 2 = the size of your drive in megabytes. The division by 2 is to get from disk blocks that are normally 512 bytes in size to Kbytes. Right. All is well now?! No, it is not. The system BIOS has another quirk you might run into. The number of cylinders of a bootable hard disk cannot be greater than 1024. Using the translation above, this is a show-stopper for disks greater than 1 GB. With disk capacities going up all the time this is causing problems. Fortunately, the solution is simple: just use another translation, e.g. with 128 heads instead of 32. In most cases new SCSI BIOS versions are available to upgrade older SCSI host adapters. Some newer adapters have an option, in the form of a jumper or software setup selection, to switch the translation the SCSI BIOS uses. It is very important that all operating systems on the disk use the same translation to get the right idea about where to find the relevant partitions. So, when installing FreeBSD you must answer any questions about heads/cylinders etc using the translated values your host adapter uses. Failing to observe the translation issue might lead to un-bootable systems or operating systems overwriting each others partitions. Using fdisk you should be able to see all partitions. You might have heard some talk of lying devices? Older FreeBSD kernels used to report the geometry of SCSI disks when booting. An example from one of my systems: aha0 targ 0 lun 0: <MICROP 1588-15MB1057404HSP4> sd0: 636MB (1303250 total sec), 1632 cyl, 15 head, 53 sec, bytes/sec 512 Newer kernels usually do not report this information. e.g. (bt0:0:0): "SEAGATE ST41651 7574" type 0 fixed SCSI 2 sd0(bt0:0:0): Direct-Access 1350MB (2766300 512 byte sectors) Why has this changed? This info is retrieved from the SCSI disk itself. Newer disks often use a technique called zone bit recording. The idea is that on the outer cylinders of the drive there is more space so more sectors per track can be put on them. This results in disks that have more tracks on outer cylinders than on the inner cylinders and, last but not least, have more capacity. You can imagine that the value reported by the drive when inquiring about the geometry now becomes suspect at best, and nearly always misleading. When asked for a geometry , it is nearly always better to supply the geometry used by the BIOS, or if the BIOS is never going to know about this disk, (e.g. it is not a booting disk) to supply a fictitious geometry that is convenient. SCSI subsystem design FreeBSD uses a layered SCSI subsystem. For each different controller card a device driver is written. This driver knows all the intimate details about the hardware it controls. The driver has a interface to the upper layers of the SCSI subsystem through which it receives its commands and reports back any status. On top of the card drivers there are a number of more generic drivers for a class of devices. More specific: a driver for tape devices (abbreviation: st), magnetic disks (sd), CDROMs (cd) etc. In case you are wondering where you can find this stuff, it all lives in /sys/scsi. See the man pages in section 4 for more details. The multi level design allows a decoupling of low-level bit banging and more high level stuff. Adding support for another piece of hardware is a much more manageable problem. Kernel configuration Dependent on your hardware, the kernel configuration file must contain one or more lines describing your host adapter(s). This includes I/O addresses, interrupts etc. Consult the man page for your adapter driver to get more info. Apart from that, check out /sys/i386/conf/LINT for an overview of a kernel config file. LINT contains every possible option you can dream of. It does not imply LINT will actually get you to a working kernel at all. Although it is probably stating the obvious: the kernel config file should reflect your actual hardware setup. So, interrupts, I/O addresses etc must match the kernel config file. During system boot messages will be displayed to indicate whether the configured hardware was actually found. Note that most of the EISA/PCI drivers (namely ahb, ahc, ncr and amd will automatically obtain the correct parameters from the host adapters themselves at boot time; thus, you just need to write, for instance, controller ahc0. An example loosely based on the FreeBSD 2.2.5-Release kernel config file LINT with some added comments (between []): # SCSI host adapters: `aha', `ahb', `aic', `bt', `nca' # # aha: Adaptec 154x # ahb: Adaptec 174x # ahc: Adaptec 274x/284x/294x # aic: Adaptec 152x and sound cards using the Adaptec AIC-6360 (slow!) # amd: AMD 53c974 based SCSI cards (e.g., Tekram DC-390 and 390T) # bt: Most Buslogic controllers # nca: ProAudioSpectrum cards using the NCR 5380 or Trantor T130 # ncr: NCR/Symbios 53c810/815/825/875 etc based SCSI cards # uha: UltraStore 14F and 34F # sea: Seagate ST01/02 8 bit controller (slow!) # wds: Western Digital WD7000 controller (no scatter/gather!). # [For an Adaptec AHA274x/284x/294x/394x etc controller] controller ahc0 [For an NCR/Symbios 53c875 based controller] controller ncr0 [For an Ultrastor adapter] controller uha0 at isa? port "IO_UHA0" bio irq ? drq 5 vector uhaintr # Map SCSI buses to specific SCSI adapters controller scbus0 at ahc0 controller scbus2 at ncr0 controller scbus1 at uha0 # The actual SCSI devices disk sd0 at scbus0 target 0 unit 0 [SCSI disk 0 is at scbus 0, LUN 0] disk sd1 at scbus0 target 1 [implicit LUN 0 if omitted] disk sd2 at scbus1 target 3 [SCSI disk on the uha0] disk sd3 at scbus2 target 4 [SCSI disk on the ncr0] tape st1 at scbus0 target 6 [SCSI tape at target 6] device cd0 at scbus? [the first ever CDROM found, no wiring] The example above tells the kernel to look for a ahc (Adaptec 274x) controller, then for an NCR/Symbios board, and so on. The lines following the controller specifications tell the kernel to configure specific devices but only attach them when they match the target ID and LUN specified on the corresponding bus. Wired down devices get first shot at the unit numbers so the first non wired down device, is allocated the unit number one greater than the highest wired down unit number for that kind of device. So, if you had a SCSI tape at target ID 2 it would be configured as st2, as the tape at target ID 6 is wired down to unit number 1. Wired down devices need not be found to get their unit number. The unit number for a wired down device is reserved for that device, even if it is turned off at boot time. This allows the device to be turned on and brought on-line at a later time, without rebooting. Notice that a device's unit number has no relationship with its target ID on the SCSI bus. Below is another example of a kernel config file as used by FreeBSD version < 2.0.5. The difference with the first example is that devices are not wired down. Wired down means that you specify which SCSI target belongs to which device. A kernel built to the config file below will attach the first SCSI disk it finds to sd0, the second disk to sd1 etc. If you ever removed or added a disk, all other devices of the same type (disk in this case) would 'move around'. This implies you have to change /etc/fstab each time. Although the old style still works, you are strongly recommended to use this new feature. It will save you a lot of grief whenever you shift your hardware around on the SCSI buses. So, when you re-use your old trusty config file after upgrading from a pre-FreeBSD2.0.5.R system check this out. [driver for Adaptec 174x] controller ahb0 at isa? bio irq 11 vector ahbintr [for Adaptec 154x] controller aha0 at isa? port "IO_AHA0" bio irq 11 drq 5 vector ahaintr [for Seagate ST01/02] controller sea0 at isa? bio irq 5 iomem 0xc8000 iosiz 0x2000 vector seaintr controller scbus0 device sd0 [support for 4 SCSI harddisks, sd0 up sd3] device st0 [support for 2 SCSI tapes] [for the CDROM] device cd0 #Only need one of these, the code dynamically grows Both examples support SCSI disks. If during boot more devices of a specific type (e.g. sd disks) are found than are configured in the booting kernel, the system will simply allocate more devices, incrementing the unit number starting at the last number wired down. If there are no wired down devices then counting starts at unit 0. Use man 4 scsi to check for the latest info on the SCSI subsystem. For more detailed info on host adapter drivers use e.g., man 4 ahc for info on the Adaptec 294x driver. Tuning your SCSI kernel setup Experience has shown that some devices are slow to respond to INQUIRY commands after a SCSI bus reset (which happens at boot time). An INQUIRY command is sent by the kernel on boot to see what kind of device (disk, tape, CDROM etc.) is connected to a specific target ID. This process is called device probing by the way. To work around the 'slow response' problem, FreeBSD allows a tunable delay time before the SCSI devices are probed following a SCSI bus reset. You can set this delay time in your kernel configuration file using a line like: options SCSI_DELAY=15 #Be pessimistic about Joe SCSI device This line sets the delay time to 15 seconds. On my own system I had to use 3 seconds minimum to get my trusty old CDROM drive to be recognized. Start with a high value (say 30 seconds or so) when you have problems with device recognition. If this helps, tune it back until it just stays working. Rogue SCSI devices Although the SCSI standard tries to be complete and concise, it is a complex standard and implementing things correctly is no easy task. Some vendors do a better job then others. This is exactly where the rogue devices come into view. Rogues are devices that are recognized by the FreeBSD kernel as behaving slightly (...) non-standard. Rogue devices are reported by the kernel when booting. An example for two of my cartridge tape units: Feb 25 21:03:34 yedi /kernel: ahb0 targ 5 lun 0: <TANDBERG TDC 3600 -06:> Feb 25 21:03:34 yedi /kernel: st0: Tandberg tdc3600 is a known rogue Mar 29 21:16:37 yedi /kernel: aha0 targ 5 lun 0: <ARCHIVE VIPER 150 21247-005> Mar 29 21:16:37 yedi /kernel: st1: Archive Viper 150 is a known rogue For instance, there are devices that respond to all LUNs on a certain target ID, even if they are actually only one device. It is easy to see that the kernel might be fooled into believing that there are 8 LUNs at that particular target ID. The confusion this causes is left as an exercise to the reader. The SCSI subsystem of FreeBSD recognizes devices with bad habits by looking at the INQUIRY response they send when probed. Because the INQUIRY response also includes the version number of the device firmware, it is even possible that for different firmware versions different workarounds are used. See e.g. /sys/scsi/st.c and /sys/scsi/scsiconf.c for more info on how this is done. This scheme works fine, but keep in mind that it of course only works for devices that are known to be weird. If you are the first to connect your bogus Mumbletech SCSI CDROM you might be the one that has to define which workaround is needed. After you got your Mumbletech working, please send the required workaround to the FreeBSD development team for inclusion in the next release of FreeBSD. Other Mumbletech owners will be grateful to you. Multiple LUN devices In some cases you come across devices that use multiple logical units (LUNs) on a single SCSI ID. In most cases FreeBSD only probes devices for LUN 0. An example are so called bridge - boards that connect 2 non-SCSI harddisks to a SCSI bus (e.g. an + boards that connect 2 non-SCSI hard disks to a SCSI bus (e.g. an Emulex MD21 found in old Sun systems). This means that any devices with LUNs != 0 are not normally found during device probe on system boot. To work around this problem you must add an appropriate entry in /sys/scsi/scsiconf.c and rebuild your kernel. Look for a struct that is initialized like below: { T_DIRECT, T_FIXED, "MAXTOR", "XT-4170S", "B5A", "mx1", SC_ONE_LU } For you Mumbletech BRIDGE2000 that has more than one LUN, acts as a SCSI disk and has firmware revision 123 you would add something like: { T_DIRECT, T_FIXED, "MUMBLETECH", "BRIDGE2000", "123", "sd", SC_MORE_LUS } The kernel on boot scans the inquiry data it receives against the table and acts accordingly. See the source for more info. Tagged command queuing Modern SCSI devices, particularly magnetic disks, support what is called tagged command queuing (TCQ). In a nutshell, TCQ allows the device to have multiple I/O requests outstanding at the same time. Because the device is intelligent, it can optimize its operations (like head positioning) based on its own request queue. On SCSI devices like RAID (Redundant Array of Independent Disks) arrays the TCQ function is indispensable to take advantage of the device's inherent parallelism. Each I/O request is uniquely identified by a tag (hence the name tagged command queuing) and this tag is used by FreeBSD to see which I/O in the device drivers queue is reported as complete by the device. It should be noted however that TCQ requires device driver support and that some devices implemented it not quite right in their firmware. This problem bit me once, and it leads to highly mysterious problems. In such cases, try to disable TCQ. Busmaster host adapters Most, but not all, SCSI host adapters are bus mastering controllers. This means that they can do I/O on their own without putting load onto the host CPU for data movement. This is of course an advantage for a multitasking operating system like FreeBSD. It must be noted however that there might be some rough edges. For instance an Adaptec 1542 controller can be set to use different transfer speeds on the host bus (ISA or AT in this case). The controller is settable to different rates because not all motherboards can handle the higher speeds. Problems like hang-ups, bad data etc might be the result of using a higher data transfer rate then your motherboard can stomach. The solution is of course obvious: switch to a lower data transfer rate and try if that works better. In the case of a Adaptec 1542, there is an option that can be put into the kernel config file to allow dynamic determination of the right, read: fastest feasible, transfer rate. This option is disabled by default: options "TUNE_1542" #dynamic tune of bus DMA speed Check the man pages for the host adapter that you use. Or better still, use the ultimate documentation (read: driver source). Tracking down problems The following list is an attempt to give a guideline for the most common SCSI problems and their solutions. It is by no means complete. Check for loose connectors and cables. Check and double check the location and number of your terminators. Check if your bus has at least one supplier of terminator power (especially with external terminators. Check if no double target IDs are used. Check if all devices to be used are powered up. Make a minimal bus config with as little devices as possible. If possible, configure your host adapter to use slow bus speeds. Disable tagged command queuing to make things as simple as possible (for a NCR host adapter based system see man ncrcontrol) If you can compile a kernel, make one with the SCSIDEBUG option, and try accessing the device with debugging turned on for that device. If your device does not even probe at startup, you may have to define the address of the device that is failing, and the desired debug level in /sys/scsi/scsidebug.h. If it probes but just does not work, you can use the &man.scsi.8; command to dynamically set a debug level to it in a running kernel (if SCSIDEBUG is defined). This will give you copious debugging output with which to confuse the gurus. See man 4 scsi for more exact information. Also look at man 8 scsi. Further reading If you intend to do some serious SCSI hacking, you might want to have the official standard at hand: Approved American National Standards can be purchased from ANSI at

13th Floor 11 West 42nd Street New York NY 10036 Sales Dept: (212) 642-4900

You can also buy many ANSI standards and most committee draft documents from Global Engineering Documents,

15 Inverness Way East Englewood CO, 80112-5704 Phone: (800) 854-7179 Outside USA and Canada: (303) 792-2181 Fax: (303) 792- 2192

Many X3T10 draft documents are available electronically on the SCSI BBS (719-574-0424) and on the ncrinfo.ncr.com anonymous FTP site. Latest X3T10 committee documents are: AT Attachment (ATA or IDE) [X3.221-1994] (Approved) ATA Extensions (ATA-2) [X3T10/948D Rev 2i] Enhanced Small Device Interface (ESDI) [X3.170-1990/X3.170a-1991] (Approved) Small Computer System Interface — 2 (SCSI-2) [X3.131-1994] (Approved) SCSI-2 Common Access Method Transport and SCSI Interface Module (CAM) [X3T10/792D Rev 11] Other publications that might provide you with additional information are: SCSI: Understanding the Small Computer System Interface, written by NCR Corporation. Available from: Prentice Hall, Englewood Cliffs, NJ, 07632 Phone: (201) 767-5937 ISBN 0-13-796855-8 Basics of SCSI, a SCSI tutorial written by Ancot Corporation Contact Ancot for availability information at: Phone: (415) 322-5322 Fax: (415) 322-0455 SCSI Interconnection Guide Book, an AMP publication (dated 4/93, Catalog 65237) that lists the various SCSI connectors and suggests cabling schemes. Available from AMP at (800) 522-6752 or (717) 564-0100 Fast Track to SCSI, A Product Guide written by Fujitsu. Available from: Prentice Hall, Englewood Cliffs, NJ, 07632 Phone: (201) 767-5937 ISBN 0-13-307000-X The SCSI Bench Reference, The SCSI Encyclopedia, and the SCSI Tutor, ENDL Publications, 14426 Black Walnut Court, Saratoga CA, 95070 Phone: (408) 867-6642 Zadian SCSI Navigator (quick ref. book) and Discover the Power of SCSI (First book along with a one-hour video and tutorial book), Zadian Software, Suite 214, 1210 S. Bascom Ave., San Jose, CA 92128, (408) 293-0800 On Usenet the newsgroups comp.periphs.scsi and comp.periphs are noteworthy places to look for more info. You can also find the SCSI-Faq there, which is posted periodically. Most major SCSI device and host adapter suppliers operate FTP sites and/or BBS systems. They may be valuable sources of information about the devices you own. * Disk/tape controllers * SCSI * IDE * Floppy Hard drives SCSI hard drives Contributed by &a.asami;. 17 February 1998. As mentioned in the SCSI section, virtually all SCSI hard drives sold today are SCSI-2 compliant and thus will work fine as long as you connect them to a supported SCSI host adapter. Most problems people encounter are either due to badly designed cabling (cable too long, star topology, etc.), insufficient termination, or defective parts. Please refer to the SCSI section first if your SCSI hard drive is not working. However, there are a couple of things you may want to take into account before you purchase SCSI hard drives for your system. Rotational speed Rotational speeds of SCSI drives sold today range from around 4,500RPM to 10,000RPM. Most of them are either 5,400RPM or 7,200RPM. Even though the 7,200RPM drives can generally transfer data faster, they run considerably hotter than their 5,400RPM counterparts. A large fraction of today's disk drive malfunctions are heat-related. If you do not have very good cooling in your PC case, you may want to stick with 5,400RPM or slower drives. Note that newer drives, with higher areal recording densities, can deliver much more bits per rotation than older ones. Today's top-of-line 5,400RPM drives can sustain a throughput comparable to 7,200RPM drives of one or two model generations ago. The number to find on the spec sheet for bandwidth is internal data (or transfer) rate. It is usually in megabits/sec so divide it by 8 and you will get the rough approximation of how much megabytes/sec you can get out of the drive. (If you are a speed maniac and want a 10,000RPM drive for your cute little PC, be my guest; however, those drives become extremely hot. Do not even think about it if you do not have a fan blowing air directly at the drive or a properly ventilated disk enclosure.) Obviously, the latest 10,000RPM drives and 7,200RPM drives can deliver more data than the latest 5,400RPM drives, so if absolute bandwidth is the necessity for your applications, you have little choice but to get the faster drives. Also, if you need low latency, faster drives are better; not only do they usually have lower average seek times, but also the rotational delay is one place where slow-spinning drives can never beat a faster one. (The average rotational latency is half the time it takes to rotate the drive once; thus, it is 3 milliseconds for 10,000RPM drives, 4.2ms for 7,200RPM drives and 5.6ms for 5,400RPM drives.) Latency is seek time plus rotational delay. Make sure you understand whether you need low latency or more accesses per second, though; in the latter case (e.g., news servers), it may not be optimal to purchase one big fast drive. You can achieve similar or even better results by using the ccd (concatenated disk) driver to create a striped disk array out of multiple slower drives for comparable overall cost. Make sure you have adequate air flow around the drive, especially if you are going to use a fast-spinning drive. You generally need at least 1/2" (1.25cm) of spacing above and below a drive. Understand how the air flows through your PC case. Most cases have the power supply suck the air out of the back. See where the air flows in, and put the drive where it will have the largest volume of cool air flowing around it. You may need to seal some unwanted holes or add a new fan for effective cooling. Another consideration is noise. Many 7,200 or faster drives generate a high-pitched whine which is quite unpleasant to most people. That, plus the extra fans often required for cooling, may make 7,200 or faster drives unsuitable for some office and home environments. Form factor Most SCSI drives sold today are of 3.5" form factor. They come in two different heights; 1.6" (half-height) or 1" (low-profile). The half-height drive is the same height as a CDROM drive. However, do not forget the spacing rule mentioned in the previous section. If you have three standard 3.5" drive bays, you will not be able to put three half-height drives in there (without frying them, that is). Interface The majority of SCSI hard drives sold today are Ultra or Ultra-wide SCSI. The maximum bandwidth of Ultra SCSI is 20MB/sec, and Ultra-wide SCSI is 40MB/sec. There is no difference in max cable length between Ultra and Ultra-wide; however, the more devices you have on the same bus, the sooner you will start having bus integrity problems. Unless you have a well-designed disk enclosure, it is not easy to make more than 5 or 6 Ultra SCSI drives work on a single bus. On the other hand, if you need to connect many drives, going for Fast-wide SCSI may not be a bad idea. That will have the same max bandwidth as Ultra (narrow) SCSI, while electronically it is much easier to get it right. My advice would be: if you want to connect many disks, get wide SCSI drives; they usually cost a little more but it may save you down the road. (Besides, if you can not afford the cost difference, you should not be building a disk array.) There are two variant of wide SCSI drives; 68-pin and 80-pin SCA (Single Connector Attach). The SCA drives do not have a separate 4-pin power connector, and also read the SCSI ID settings through the 80-pin connector. If you are really serious about building a large storage system, get SCA drives and a good SCA enclosure (dual power supply with at least one extra fan). They are more electronically sound than 68-pin counterparts because there is no stub of the SCSI bus inside the disk canister as in arrays built from 68-pin drives. They are easier to install too (you just need to screw the drive in the canister, instead of trying to squeeze in your fingers in a tight place to hook up all the little cables (like the SCSI ID and disk activity LED lines). * IDE hard drives Tape drives Contributed by &a.jmb;. 2 July 1996. General tape access commands &man.mt.1; provides generic access to the tape drives. Some of the more common commands are rewind, erase, and status. See the &man.mt.1; manual page for a detailed description. Controller Interfaces There are several different interfaces that support tape drives. The interfaces are SCSI, IDE, Floppy and Parallel Port. A wide variety of tape drives are available for these interfaces. Controllers are discussed in Disk/tape controllers. SCSI drives The &man.st.4; driver provides support for 8mm (Exabyte), 4mm (DAT: Digital Audio Tape), QIC (Quarter-Inch Cartridge), DLT (Digital Linear Tape), QIC Mini cartridge and 9-track (remember the big reels that you see spinning in Hollywood computer rooms) tape drives. See the &man.st.4; manual page for a detailed description. The drives listed below are currently being used by members of the FreeBSD community. They are not the only drives that will work with FreeBSD. They just happen to be the ones that we use. 4mm (DAT: Digital Audio Tape) Archive Python 28454 Archive Python 04687 HP C1533A HP C1534A HP 35450A HP 35470A HP 35480A SDT-5000 Wangtek 6200 8mm (Exabyte) EXB-8200 EXB-8500 EXB-8505 QIC (Quarter-Inch Cartridge) Archive Anaconda 2750 Archive Viper 60 Archive Viper 150 Archive Viper 2525 Tandberg TDC 3600 Tandberg TDC 3620 Tandberg TDC 3800 Tandberg TDC 4222 Wangtek 5525ES DLT (Digital Linear Tape) Digital TZ87 Mini-Cartridge Conner CTMS 3200 Exabyte 2501 Autoloaders/Changers Hewlett-Packard HP C1553A Autoloading DDS2 * IDE drives Floppy drives Conner 420R * Parallel port drives Detailed Information Archive Anaconda 2750 The boot message identifier for this drive is ARCHIVE ANCDA 2750 28077 -003 type 1 removable SCSI 2 This is a QIC tape drive. Native capacity is 1.35GB when using QIC-1350 tapes. This drive will read and write QIC-150 (DC6150), QIC-250 (DC6250), and QIC-525 (DC6525) tapes as well. Data transfer rate is 350kB/s using &man.dump.8;. Rates of 530kB/s have been reported when using Amanda Production of this drive has been discontinued. The SCSI bus connector on this tape drive is reversed from that on most other SCSI devices. Make sure that you have enough SCSI cable to twist the cable one-half turn before and after the Archive Anaconda tape drive, or turn your other SCSI devices upside-down. Two kernel code changes are required to use this drive. This drive will not work as delivered. If you have a SCSI-2 controller, short jumper 6. Otherwise, the drive behaves are a SCSI-1 device. When operating as a SCSI-1 device, this drive, locks the SCSI bus during some tape operations, including: fsf, rewind, and rewoffl. If you are using the NCR SCSI controllers, patch the file /usr/src/sys/pci/ncr.c (as shown below). Build and install a new kernel. *** 4831,4835 **** }; ! if (np->latetime>4) { /* ** Although we tried to wake it up, --- 4831,4836 ---- }; ! if (np->latetime>1200) { /* ** Although we tried to wake it up, Reported by: &a.jmb; Archive Python 28454 The boot message identifier for this drive is ARCHIVE Python 28454-XXX4ASB type 1 removable SCSI 2 density code 0x8c, 512-byte blocks This is a DDS-1 tape drive. Native capacity is 2.5GB on 90m tapes. Data transfer rate is XXX. This drive was repackaged by Sun Microsystems as model 595-3067. Reported by: Bob Bishop rb@gid.co.uk Throughput is in the 1.5 MByte/sec range, however this will drop if the disks and tape drive are on the same SCSI controller. Reported by: Robert E. Seastrom rs@seastrom.com Archive Python 04687 The boot message identifier for this drive is ARCHIVE Python 04687-XXX 6580 Removable Sequential Access SCSI-2 device This is a DAT-DDS-2 drive. Native capacity is 4GB when using 120m tapes. This drive supports hardware data compression. Switch 4 controls MRS (Media Recognition System). MRS tapes have stripes on the transparent leader. Switch 4 off enables MRS, on disables MRS. Parity is controlled by switch 5. Switch 5 on to enable parity control. Compression is enabled with Switch 6 off. It is possible to override compression with the SCSI MODE SELECT command (see &man.mt.1;). Data transfer rate is 800kB/s. Archive Viper 60 The boot message identifier for this drive is ARCHIVE VIPER 60 21116 -007 type 1 removable SCSI 1 This is a QIC tape drive. Native capacity is 60MB. Data transfer rate is XXX. Production of this drive has been discontinued. Reported by: Philippe Regnauld regnauld@hsc.fr Archive Viper 150 The boot message identifier for this drive is ARCHIVE VIPER 150 21531 -004 Archive Viper 150 is a known rogue type 1 removable SCSI 1. A multitude of firmware revisions exist for this drive. Your drive may report different numbers (e.g 21247 -005. This is a QIC tape drive. Native capacity is 150/250MB. Both 150MB (DC6150) and 250MB (DC6250) tapes have the recording format. The 250MB tapes are approximately 67% longer than the 150MB tapes. This drive can read 120MB tapes as well. It can not write 120MB tapes. Data transfer rate is 100kB/s This drive reads and writes DC6150 (150MB) and DC6250 (250MB) tapes. This drives quirks are known and pre-compiled into the scsi tape device driver (&man.st.4;). Under FreeBSD 2.2-CURRENT, use mt blocksize 512 to set the blocksize. (The particular drive had firmware revision 21247 -005. Other firmware revisions may behave differently) Previous versions of FreeBSD did not have this problem. Production of this drive has been discontinued. Reported by: Pedro A M Vazquez vazquez@IQM.Unicamp.BR &a.msmith; Archive Viper 2525 The boot message identifier for this drive is ARCHIVE VIPER 2525 25462 -011 type 1 removable SCSI 1 This is a QIC tape drive. Native capacity is 525MB. Data transfer rate is 180kB/s at 90 inches/sec. The drive reads QIC-525, QIC-150, QIC-120 and QIC-24 tapes. Writes QIC-525, QIC-150, and QIC-120. Firmware revisions prior to 25462 -011 are bug ridden and will not function properly. Production of this drive has been discontinued. Conner 420R The boot message identifier for this drive is Conner tape. This is a floppy controller, mini cartridge tape drive. Native capacity is XXXX Data transfer rate is XXX The drive uses QIC-80 tape cartridges. Reported by: Mark Hannon mark@seeware.DIALix.oz.au Conner CTMS 3200 The boot message identifier for this drive is CONNER CTMS 3200 7.00 type 1 removable SCSI 2. This is a mini cartridge tape drive. Native capacity is XXXX Data transfer rate is XXX The drive uses QIC-3080 tape cartridges. Reported by: Thomas S. Traylor tst@titan.cs.mci.com <ulink url="http://www.digital.com/info/Customer-Update/931206004.txt.html">DEC TZ87</ulink> The boot message identifier for this drive is DEC TZ87 (C) DEC 9206 type 1 removable SCSI 2 density code 0x19 This is a DLT tape drive. Native capacity is 10GB. This drive supports hardware data compression. Data transfer rate is 1.2MB/s. This drive is identical to the Quantum DLT2000. The drive firmware can be set to emulate several well-known drives, including an Exabyte 8mm drive. Reported by: &a.wilko; <ulink url="http://www.Exabyte.COM:80/Products/Minicartridge/2501/Rfeatures.html">Exabyte EXB-2501</ulink> The boot message identifier for this drive is EXABYTE EXB-2501 This is a mini-cartridge tape drive. Native capacity is 1GB when using MC3000XL mini cartridges. Data transfer rate is XXX This drive can read and write DC2300 (550MB), DC2750 (750MB), MC3000 (750MB), and MC3000XL (1GB) mini cartridges. WARNING: This drive does not meet the SCSI-2 specifications. The drive locks up completely in response to a SCSI MODE_SELECT command unless there is a formatted tape in the drive. Before using this drive, set the tape blocksize with &prompt.root; mt -f /dev/st0ctl.0 blocksize 1024 Before using a mini cartridge for the first time, the mini cartridge must be formated. FreeBSD 2.1.0-RELEASE and earlier: &prompt.root; /sbin/scsi -f /dev/rst0.ctl -s 600 -c "4 0 0 0 0 0" (Alternatively, fetch a copy of the scsiformat shell script from FreeBSD 2.1.5/2.2.) FreeBSD 2.1.5 and later: &prompt.root; /sbin/scsiformat -q -w /dev/rst0.ctl Right now, this drive cannot really be recommended for FreeBSD. Reported by: Bob Beaulieu ez@eztravel.com Exabyte EXB-8200 The boot message identifier for this drive is EXABYTE EXB-8200 252X type 1 removable SCSI 1 This is an 8mm tape drive. Native capacity is 2.3GB. Data transfer rate is 270kB/s. This drive is fairly slow in responding to the SCSI bus during boot. A custom kernel may be required (set SCSI_DELAY to 10 seconds). There are a large number of firmware configurations for this drive, some have been customized to a particular vendor's hardware. The firmware can be changed via EPROM replacement. Production of this drive has been discontinued. Reported by: &a.msmith; Exabyte EXB-8500 The boot message identifier for this drive is EXABYTE EXB-8500-85Qanx0 0415 type 1 removable SCSI 2 This is an 8mm tape drive. Native capacity is 5GB. Data transfer rate is 300kB/s. Reported by: Greg Lehey grog@lemis.de <ulink url="http://www.Exabyte.COM:80/Products/8mm/8505XL/Rfeatures.html">Exabyte EXB-8505</ulink> The boot message identifier for this drive is EXABYTE EXB-85058SQANXR1 05B0 type 1 removable SCSI 2 This is an 8mm tape drive which supports compression, and is upward compatible with the EXB-5200 and EXB-8500. Native capacity is 5GB. The drive supports hardware data compression. Data transfer rate is 300kB/s. Reported by: Glen Foster gfoster@gfoster.com Hewlett-Packard HP C1533A The boot message identifier for this drive is HP C1533A 9503 type 1 removable SCSI 2. This is a DDS-2 tape drive. DDS-2 means hardware data compression and narrower tracks for increased data capacity. Native capacity is 4GB when using 120m tapes. This drive supports hardware data compression. Data transfer rate is 510kB/s. This drive is used in Hewlett-Packard's SureStore 6000eU and 6000i tape drives and C1533A DDS-2 DAT drive. The drive has a block of 8 dip switches. The proper settings for FreeBSD are: 1 ON; 2 ON; 3 OFF; 4 ON; 5 ON; 6 ON; 7 ON; 8 ON. switch 1 switch 2 Result On On Compression enabled at power-on, with host control On Off Compression enabled at power-on, no host control Off On Compression disabled at power-on, with host control Off Off Compression disabled at power-on, no host control Switch 3 controls MRS (Media Recognition System). MRS tapes have stripes on the transparent leader. These identify the tape as DDS (Digital Data Storage) grade media. Tapes that do not have the stripes will be treated as write-protected. Switch 3 OFF enables MRS. Switch 3 ON disables MRS. See HP SureStore Tape Products and Hewlett-Packard Disk and Tape Technical Information for more information on configuring this drive. Warning: Quality control on these drives varies greatly. One FreeBSD core-team member has returned 2 of these drives. Neither lasted more than 5 months. Reported by: &a.se; Hewlett-Packard HP 1534A The boot message identifier for this drive is HP HP35470A T503 type 1 removable SCSI 2 Sequential-Access density code 0x13, variable blocks. This is a DDS-1 tape drive. DDS-1 is the original DAT tape format. Native capacity is 2GB when using 90m tapes. Data transfer rate is 183kB/s. The same mechanism is used in Hewlett-Packard's SureStore 2000i tape drive, C35470A DDS format DAT drive, C1534A DDS format DAT drive and HP C1536A DDS format DAT drive. The HP C1534A DDS format DAT drive has two indicator lights, one green and one amber. The green one indicates tape action: slow flash during load, steady when loaded, fast flash during read/write operations. The amber one indicates warnings: slow flash when cleaning is required or tape is nearing the end of its useful life, steady indicates an hard fault. (factory service required?) Reported by Gary Crutcher gcrutchr@nightflight.com Hewlett-Packard HP C1553A Autoloading DDS2 The boot message identifier for this drive is "". This is a DDS-2 tape drive with a tape changer. DDS-2 means hardware data compression and narrower tracks for increased data capacity. Native capacity is 24GB when using 120m tapes. This drive supports hardware data compression. Data transfer rate is 510kB/s (native). This drive is used in Hewlett-Packard's SureStore 12000e tape drive. The drive has two selectors on the rear panel. The selector closer to the fan is SCSI id. The other selector should be set to 7. There are four internal switches. These should be set: 1 ON; 2 ON; 3 ON; 4 OFF. At present the kernel drivers do not automatically change tapes at the end of a volume. This shell script can be used to change tapes: #!/bin/sh PATH="/sbin:/usr/sbin:/bin:/usr/bin"; export PATH usage() { echo "Usage: dds_changer [123456ne] raw-device-name echo "1..6 = Select cartridge" echo "next cartridge" echo "eject magazine" exit 2 } if [ $# -ne 2 ] ; then usage fi cdb3=0 cdb4=0 cdb5=0 case $1 in [123456]) cdb3=$1 cdb4=1 ;; n) ;; e) cdb5=0x80 ;; ?) usage ;; esac scsi -f $2 -s 100 -c "1b 0 0 $cdb3 $cdb4 $cdb5" Hewlett-Packard HP 35450A The boot message identifier for this drive is HP HP35450A -A C620 type 1 removable SCSI 2 Sequential-Access density code 0x13 This is a DDS-1 tape drive. DDS-1 is the original DAT tape format. Native capacity is 1.2GB. Data transfer rate is 160kB/s. Reported by: Mark Thompson mark.a.thompson@pobox.com Hewlett-Packard HP 35470A The boot message identifier for this drive is HP HP35470A 9 09 type 1 removable SCSI 2 This is a DDS-1 tape drive. DDS-1 is the original DAT tape format. Native capacity is 2GB when using 90m tapes. Data transfer rate is 183kB/s. The same mechanism is used in Hewlett-Packard's SureStore 2000i tape drive, C35470A DDS format DAT drive, C1534A DDS format DAT drive, and HP C1536A DDS format DAT drive. Warning: Quality control on these drives varies greatly. One FreeBSD core-team member has returned 5 of these drives. None lasted more than 9 months. Reported by: David Dawes dawes@rf900.physics.usyd.edu.au (9 09) Hewlett-Packard HP 35480A The boot message identifier for this drive is HP HP35480A 1009 type 1 removable SCSI 2 Sequential-Access density code 0x13. This is a DDS-DC tape drive. DDS-DC is DDS-1 with hardware data compression. DDS-1 is the original DAT tape format. Native capacity is 2GB when using 90m tapes. It cannot handle 120m tapes. This drive supports hardware data compression. Please refer to the section on HP C1533A for the proper switch settings. Data transfer rate is 183kB/s. This drive is used in Hewlett-Packard's SureStore 5000eU and 5000i tape drives and C35480A DDS format DAT drive.. This drive will occasionally hang during a tape eject operation (mt offline). Pressing the front panel button will eject the tape and bring the tape drive back to life. WARNING: HP 35480-03110 only. On at least two occasions this tape drive when used with FreeBSD 2.1.0, an IBM Server 320 and an 2940W SCSI controller resulted in all SCSI disk partitions being lost. The problem has not be analyzed or resolved at this time. <ulink url="http://www.sel.sony.com/SEL/ccpg/storage/tape/t5000.html">Sony SDT-5000</ulink> There are at least two significantly different models: one is a DDS-1 and the other DDS-2. The DDS-1 version is SDT-5000 3.02. The DDS-2 version is SONY SDT-5000 327M. The DDS-2 version has a 1MB cache. This cache is able to keep the tape streaming in almost any circumstances. The boot message identifier for this drive is SONY SDT-5000 3.02 type 1 removable SCSI 2 Sequential-Access density code 0x13 Native capacity is 4GB when using 120m tapes. This drive supports hardware data compression. Data transfer rate is depends upon the model or the drive. The rate is 630kB/s for the SONY SDT-5000 327M while compressing the data. For the SONY SDT-5000 3.02, the data transfer rate is 225kB/s. In order to get this drive to stream, set the blocksize to 512 bytes (mt blocksize 512) reported by Kenneth Merry ken@ulc199.residence.gatech.edu. SONY SDT-5000 327M information reported by Charles Henrich henrich@msu.edu. Reported by: &a.jmz; Tandberg TDC 3600 The boot message identifier for this drive is TANDBERG TDC 3600 =08: type 1 removable SCSI 2 This is a QIC tape drive. Native capacity is 150/250MB. This drive has quirks which are known and work around code is present in the scsi tape device driver (&man.st.4;). Upgrading the firmware to XXX version will fix the quirks and provide SCSI 2 capabilities. Data transfer rate is 80kB/s. IBM and Emerald units will not work. Replacing the firmware EPROM of these units will solve the problem. Reported by: &a.msmith; Tandberg TDC 3620 This is very similar to the Tandberg TDC 3600 drive. Reported by: &a.joerg; Tandberg TDC 3800 The boot message identifier for this drive is TANDBERG TDC 3800 =04Y Removable Sequential Access SCSI-2 device This is a QIC tape drive. Native capacity is 525MB. Reported by: &a.jhs; Tandberg TDC 4222 The boot message identifier for this drive is TANDBERG TDC 4222 =07 type 1 removable SCSI 2 This is a QIC tape drive. Native capacity is 2.5GB. The drive will read all cartridges from the 60 MB (DC600A) upwards, and write 150 MB (DC6150) upwards. Hardware compression is optionally supported for the 2.5 GB cartridges. This drives quirks are known and pre-compiled into the scsi tape device driver (&man.st.4;) beginning with FreeBSD 2.2-CURRENT. For previous versions of FreeBSD, use mt to read one block from the tape, rewind the tape, and then execute the backup program (mt fsr 1; mt rewind; dump ...) Data transfer rate is 600kB/s (vendor claim with compression), 350 KB/s can even be reached in start/stop mode. The rate decreases for smaller cartridges. Reported by: &a.joerg; Wangtek 5525ES The boot message identifier for this drive is WANGTEK 5525ES SCSI REV7 3R1 type 1 removable SCSI 1 density code 0x11, 1024-byte blocks This is a QIC tape drive. Native capacity is 525MB. Data transfer rate is 180kB/s. The drive reads 60, 120, 150, and 525MB tapes. The drive will not write 60MB (DC600 cartridge) tapes. In order to overwrite 120 and 150 tapes reliably, first erase (mt erase) the tape. 120 and 150 tapes used a wider track (fewer tracks per tape) than 525MB tapes. The extra width of the previous tracks is not overwritten, as a result the new data lies in a band surrounded on both sides by the previous data unless the tape have been erased. This drives quirks are known and pre-compiled into the scsi tape device driver (&man.st.4;). Other firmware revisions that are known to work are: M75D Reported by: Marc van Kempen marc@bowtie.nl REV73R1 Andrew Gordon Andrew.Gordon@net-tel.co.uk M75D Wangtek 6200 The boot message identifier for this drive is WANGTEK 6200-HS 4B18 type 1 removable SCSI 2 Sequential-Access density code 0x13 This is a DDS-1 tape drive. Native capacity is 2GB using 90m tapes. Data transfer rate is 150kB/s. Reported by: Tony Kimball alk@Think.COM * Problem drives CDROM drives Contributed by &a.obrien;. 23 November 1997. Generally speaking those in The FreeBSD Project prefer SCSI CDROM drives over IDE CDROM drives. However not all SCSI CDROM drives are equal. Some feel the quality of some SCSI CDROM drives have been deteriorating to that of IDE CDROM drives. Toshiba used to be the favored stand-by, but many on the SCSI mailing list have found displeasure with the 12x speed XM-5701TA as its volume (when playing audio CDROMs) is not controllable by the various audio player software. Another area where SCSI CDROM manufacturers are cutting corners is adherence to the SCSI specification. Many SCSI CDROMs will respond to multiple LUNs for its target address. Known violators include the 6x Teac CD-56S 1.0D.

Bootstrapping Vinum: A Foundation for Reliable Servers Robert A. Van Valzah 2001 Robert A. Van Valzah - $Date: 2001-10-29 23:20:40 $ GMT - $Id: article.sgml,v 1.2 2001-10-29 23:20:40 chern Exp $ + $Date: 2001-10-31 23:10:31 $ GMT + $Id: article.sgml,v 1.3 2001-10-31 23:10:31 chern Exp $ In the most abstract sense, these instructions show how to build a pair of disk drives where either one is adequate to keep your server running if the other fails. Life is better if they are both working, but your server will never die unless both disk drives die at once. If you choose ATAPI drives and use a fairly generic kernel, you can be confident that either of these drives can be plugged into most any main board to produce a working server in a pinch. The drives need not be identical. These techniques work equally well with SCSI drives as they do with ATAPI, but I will focus on ATAPI here because main boards with this interface are ubiquitous. After building the foundation of a reliable server as shown here, you can expand to as many disk drives as necessary to build the failure-resilient server of your dreams.

Introduction Any machine that is going to provide reliable service needs to have either redundant components on-line or a pool of off-line spares that can be promptly swapped in. Commodity PC hardware makes it affordable for even small organizations to have some spare parts available that could be pressed into service following the failure of production equipment. In many organizations, a failed power supply, NIC, memory, or main board could easily be swapped with a standby in a matter of minutes and be ready to return to production work. If a disk drive fails, however, it often has to be restored from a tape backup. This may take many hours. With disk drive capacities rising faster than tape drive capacities, the time needed to restore a failed disk drive seems to increase as technology progresses. &vinum.ap; is a volume manager for FreeBSD that provides a standard block I/O layer interface to the file system code just as any hardware device driver would. It works by managing partitions of type vinum and allows you to subdivide and group the space in such partitions into logical devices called volumes that can be used in the same way as disk partitions. Volumes can be configured for resilience, performance, or both. Experienced system administrators will immediately recognize the benefits of being able to configure each file system to match the way it is most often used. In some ways, Vinum is similar to &man.ccd.4;, but it is far more flexible and robust in the face of failures. It is only slightly more difficult to set up than &man.ccd.4;. &man.ccd.4; may meet your needs if you are only interested in concatenation.

Terminology Discussion of storage management can get very tricky simply because of the terminology involved. As we will see below, the terms disk, slice, partition, subdisk, and volume each refer to different things that present the same interface to a kernel function like swapping. The potential for confusion is compounded because the objects that these terms represent can be nested inside each other. I will refer to a physical disk drive as a spindle. A partition here means a BSD partition as maintained by disklabel. It does not refer to slices or BIOS partitions as maintained by fdisk.

Vinum Objects Vinum defines a hierarchy of four objects that it uses to manage storage (see ). Different combinations of these objects are used to achieve failure resilience, performance, and/or extra capacity. I will give a whirlwind tour of the objects here--see the Vinum web site for a more thorough description.

Vinum Objects and Architecture +-----+------+------+ | UFS | swap | Etc. | +---+-+------+----+ + | volume | | + V +-------------+ + | i plex | | + n +-------------+ + | u subdisk | | + m +-------------+ + | drive | | +-----------------+ + | Block I/O devices | +-------------------+ Vinum Objects and Architecture The top object, a vinum volume, implements a virtual disk that provides a standard block I/O layer interface to other parts of the kernel. The bottom object, a vinum drive, uses this same interface to request I/O from physical devices below it. In between these two (from top to bottom) we have objects called a vinum plex and a vinum subdisk. As you can probably guess from the name, a vinum subdisk is a contiguous subset of the space available on a vinum drive. It lets you subdivide a vinum drive in much the same way that a disk BSD partition lets you subdivide a BIOS slice. A plex allows subdisks to be grouped together making the space of all subdisks available as a single object. A plex can be organized with its constituent subdisks concatenated or striped. Both organizations are useful for spreading I/O requests across spindles since plexes reside on distinct spindles. A striped plex will switch spindles each time a multiple of the strip size is reached. A concatenated plex will switch spindles only when the end of a subdisk is reached. An important characteristic of a Vinum volume is that it can be made up of more than one plex. In this case, writes go to all plexes and a read may be satisfied by any plex. Configuring two or more plexes on distinct spindles yields a volume that is resilient to failure. Vinum maintains a configuration that defines instances of the above objects and the way they are related to each other. This configuration is automatically written to all spindles under Vinum management whenever it changes.

Vinum Volume/Plex Organization Although Vinum can manage any number of spindles, I will only cover scenarios with two spindles here for simplification. See to see how two spindles organized with Vinum compare to two spindles without Vinum. Characteristics of Two Spindles Organized with Vinum Organization Total Capacity Failure Resilient Peak Read Performance Peak Write Performance Concatenated Plexes Unchanged, but appears as a single drive No Unchanged Unchanged Striped Plexes (RAID-0) Unchanged, but appears as a single drive No 2x 2x Mirrored Volumes (RAID-1) 1/2, appearing as a single drive Yes 2x Unchanged

shows that striping yields the same capacity and lack of failure resilience as concatenation, but it has better peak read and write performance. Hence we will not be using concatenation in any of the examples here. Mirrored volumes provide the benefits of improved peak read performance and failure resilience--but this comes at a loss in capacity. Both concatenation and striping bring their benefits over a - single spindle at the cost of increased likelyhood of failure since + single spindle at the cost of increased likelihood of failure since more than one spindle is now involved. When three or more spindles are present, Vinum also supports rotated, block-interleaved parity (also called RAID-5) that provides better capacity than mirroring (but not quite as good as striping), better read performance than both mirroring and striping, and good failure resilience. There is, however, a substantial decrease in write performance with RAID-5. Most of the benefits become more pronounced with five or more spindles. The organizations described above may be combined to provide benefits that no single organization can match. For example, mirroring and striping can be combined to provide failure-resilience with very fast read performance.

Vinum History Vinum is a standard part of even a "minimum" FreeBSD distribution and it has been standard since 3.0-RELEASE. The official pronunciation of the name is VEE-noom. &vinum.ap; was inspired by the Veritas Volume Manager, but was not derived from it. The name is a play on that history and the Latin adage In Vino Veritas (Vino is the accusative form of Vinum). Literally translated, that is "Truth lies in wine" hinting that drunkards have a hard time lying. I have been using it in production on six different servers for over two years with no data loss. Like the rest of FreeBSD, Vinum provides "rock-stable performance." (On a personal note, I have seen Vinum panic when I misconfigured something, but I have never had any trouble in normal operation.) Greg Lehey wrote Vinum for FreeBSD, but he is seeking help in porting it to NetBSD and OpenBSD. Just like the rest of FreeBSD, Vinum is undergoing continuous development. Several subtle, but significant bugs have been fixed in recent releases. It is always best to use the most recent code base that meets your stability requirements.

Vinum Deployment Strategy Vinum, coupled with prudent partition management, lets you keep "warm-spare" spindles on-line so that failures are transparent to users. Failed spindles can be replaced during regular maintenance periods or whenever it is convenient. When all spindles are working, the server benefits from increased performance and capacity. Having redundant copies of your home directory does not help you if the spindle holding root, /usr, or swap fails on your server. Hence I focus here on building a simple foundation for a failure-resilient server covering the root, /usr, /home, and swap partitions. Vinum mirroring does not remove the need for making backups! Mirroring cannot help you recover from site disasters or the dreaded rm -r -f / command.

Why Bootstrap Vinum? It is possible to add Vinum to a server configuration after it is already in production use, but this is much harder than designing for it from the start. Ironically, Vinum is not supported by /stand/sysinstall and hence you cannot install /usr right onto a Vinum volume. Vinum currently does not support the root file system (this feature is in development). Hence it is a bit tricky to get started using Vinum, but these instructions take you though the process of planning for Vinum, installing FreeBSD without it, and then beginning to use it. I have come to call this whole process "bootstrapping Vinum." That is, the process of getting Vinum initially installed and operating to the point where you have met your resilience or performance goals. My purpose here is to document a Vinum bootstrapping method that I have found that works well for me.

Vinum Benefits The server foundation scenario I have chosen here allows me to show you examples of configuring for resilience on /usr and /home. Yet Vinum provides benefits other than resilience--namely performance, capacity, and manageability. It can significantly improve disk performance (especially under multi-user loads). Vinum can easily concatenate many smaller disks to produce the illusion of a single larger disk (but my server foundation scenario does not allow me to illustrate these benefits here). For servers with many spindles, Vinum provides substantial benefits in volume management, particularly when coupled with hot-pluggable hardware. Data can be moved from spindle to spindle while the system is running without loss of production time. Again, details of this will not be given here, but once you get your feet wet with Vinum, other documentation will help you do things like this. See "The Vinum Volume Manager" for a technical introduction to Vinum, &man.vinum.8; for a description of the vinum command, and &man.vinum.4; for a description of the vinum device driver and the way Vinum objects are named. Breaking up your disk space into smaller and smaller partitions has the benefit of allowing you to "tune" for the most common type of access and tends to keep disk hogs "within their pens." However it also causes some loss in total available disk space due to fragmentation.

Server Operation in Degraded Mode Some disk failures in this two-spindle scenario will result in Vinum automatically routing all disk I/O to the remaining good spindle. Others will require brief manual intervention on the console to configure the server for degraded mode operation and a quick reboot. Other than actual hardware repairs, most recovery work can be done while the server is running in multi-user degraded mode so there is as little production impact from failures as possible. I give the instructions in needed to configure the server for degraded mode operation in those cases where Vinum cannot do it automatically. I also give the instructions needed to return to normal operation once the failed hardware is repaired. You might call these instructions Vinum failure recovery techniques. I recommend practicing using these instructions by recovering from simulated failures. For each failure scenario, I also give tips below for simulating a failure even when your hardware is working well. Even a minimum Vinum system as described in below can be a good place to experiment with recovery techniques without impacting production equipment.

Hardware RAID vs. Vinum (Software RAID) Manual intervention is sometimes required to configure a server for degraded mode because Vinum is implemented in software that runs after the FreeBSD kernel is loaded. One disadvantage of such software RAID solutions is that there is nothing that can be done to hide spindle failures from the BIOS or the FreeBSD boot sequence. Hence the manual reconfiguration of the server for degraded operation mentioned above just informs the BIOS and boot sequence of failed spindles. Hardware RAID solutions generally have an advantage in that they require no such reconfiguration since spindle failures are hidden from the BIOS and boot sequence. Hardware RAID, however, may have some disadvantages that can be significant in some cases: The hardware RAID controller itself may become a single point of failure for the system. The data is usually kept in a proprietary format so that a disk drive cannot be simply plugged into another main board and booted. You often cannot mix and match drives with different sizes and interfaces. You are often limited to the number of drives supported by the hardware RAID controller (often only four or eight). In other words, &vinum.ap; may offer advantages in that there is no single point of failure, the drives can boot on most any main board, and you are free to mix and match as many drives using whatever interface you choose. Keep your kernel fairly generic (or at least keep /kernel.GENERIC around). This will improve the chances that you can come back up on "foreign" hardware more quickly. The pros and cons discussed above suggest that the root file system and swap partition are good candidates for hardware RAID if available. This is especially true for servers where it is difficult for administrators to get console access (recall that this is sometimes required to configure a server for degraded mode operation). A server with only software RAID is well suited to office and home environments where an administrator can be close at hand. A common myth is that hardware RAID is always faster than software RAID. Since it runs on the host CPU, Vinum often has more CPU power and memory available than a dedicated RAID controller would have. If performance is a prime concern, it is best to benchmark your application running on your CPU with your spindles using both hardware and software RAID systems before making a decision.

Hardware for Vinum These instructions may be timely since commodity PC hardware can now easily host several hundred megabytes of reasonably high-performance disk space at a low price. Many disk drive manufactures now sell 7,200 RPM disk drives with quite low seek times and high transfer rates through ATA-100 interfaces, all at very attractive prices. Four such drives, attached to a suitable main board and configured with Vinum and prudent partitioning, yields a failure-resilient, high performance disk server at a very reasonable cost. However, you can indeed get started with Vinum very simply. A minimum system can be as simple as an old CPU (even a 486 is fine) and a pair of drives that are 500 MB or more. They need not be the same size or even use the same interface (i.e., it is fine to mix ATAPI and SCSI). So get busy and give this a try today! You will have the foundation of a failure-resilient server running in an hour or so!

Bootstrapping Phases Greg Lehey suggested this bootstrapping method. It uses knowledge of how Vinum internally allocates disk space to avoid copying data. Instead, Vinum objects are configured so that they occupy the same disk space where /stand/sysinstall built file systems. The file systems are thus embedded within Vinum objects without copying. There are several distinct phases to the Vinum bootstrapping procedure. Each of these phases is presented in a separate section below. The section starts with a general overview of the phase and its goals. It then gives example steps for the two-spindle scenario presented here and advice on how to adapt them for your server. (If you are reading for a general understanding of Vinum bootstrapping, the example sections for each phase can safely be skipped.) The remainder of this section gives an overview of the entire bootstrapping process. Phase 1 involves planning and preparation. We will balance requirements for the server against available resources and make design tradeoffs. We will plan the transition from no Vinum to Vinum on just one spindle, to Vinum on two spindles. In phase 2, we will install a minimum FreeBSD system on a single spindle using partitions of type 4.2BSD (regular UFS file systems). Phase 3 will embed the non-root file systems from phase 2 in Vinum objects. Note that Vinum will be up and running at this point, but it cannot yet provide any resilience since it only has one spindle on which to store data. Finally in phase 4, we configure Vinum on a second spindle and make a backup copy of the root file system. This will give us resilience on all file systems.

Bootstrapping Phase 1: Planning and Preparation Our goal in this phase is to define the different partitions we will need and examine their requirements. We will also look at available disk drives and controllers and allocate partitions to them. Finally, we will determine the size of each partition and its use during the bootstrapping process. After this planning is complete, we can optionally prepare to use some tools that will make bootstrapping Vinum easier. Several key questions must be answered in this planning phase: What file system and partitions will be needed? How will they be used? How will we name each spindle? How will the partitions be ordered for each spindle? How will partitions be assigned to the spindles? How will partitions be configured? Resilience or performance? What technique will be used to achieve resilience? What spindles will be used? How will they be configured on the available controllers? How much space is required for each partition?

Phase 1 Example In this example, I will assume a scenario where we are building a minimal foundation for a failure-resilient server. Hence we will need at least root, /usr, /home, and swap partitions. The root, /usr, and /home file systems all need resilience since the server will not be much good without them. The swap partition needs performance first and generally does not need resilience since nothing it holds needs to be retained across a reboot.

Spindle Naming The kernel would refer to the master spindle on the primary and secondary ATA controllers as /dev/ad0 and /dev/ad2 respectively. This assumes that you have not removed the line options ATA_STATIC_ID from your kernel configuration. But Vinum also needs to have a name for each spindle that will stay the same name regardless of how it is attached to the CPU (i.e., if the drive moves, the Vinum name moves with the drive). Some recovery techniques documented below suggest moving a spindle from the secondary ATA controller to the primary ATA controller. (Indeed, the flexibility of making such moves is a key benefit of Vinum especially if you are managing a large number of spindles.) After such a drive/controller swap, the kernel will see what used to be /dev/ad2 as /dev/ad0 but Vinum will still call it by whatever name it had when it was attached to /dev/ad2 (i.e., when it was "created" or first made known to Vinum). Since connections can change, it is best to give each spindle a unique, abstract name that gives no hint of how it is attached. Avoid names that suggest a manufacturer, model number, physical location, or membership in a sequence (e.g. avoid names like upper, lower, etc., alpha, beta, etc., SCSI1, SCSI2, etc., or Seagate1, Seagate2 etc.). Such names are likely to lose their uniqueness or get out of sequence someday even if they seem like great names today. Once you have picked names for your spindles, label them with a permanent marker. If you have hot-swappable hardware, write the names on the sleds in which the spindles are mounted. This will significantly reduce the likelihood of error when you are moving spindles around later as part of failure recovery or routine system management procedures. In the instructions that follow, Vinum will name the root spindle YouCrazy and the rootback spindle UpWindow. I will only use /dev/ad0 when I want to refer to whichever of the two spindles is currently attached as /dev/ad0.

Partition Ordering Modern disk drives operate with fairly uniform areal density across the surface of the disk. That implies that more data is available under the heads without seeking on the outer cylinders than on the inner cylinders. We will allocate partitions most critical to system performance from these outer cylinders as /stand/sysinstall generally does. The root file system is traditionally the outermost, even though it generally is not as critical to system performance as others. (However root can have a larger impact on performance if it contains /tmp and /var as it does in this example.) The FreeBSD boot loaders assume that the root file system lives in the a partition. There is no requirement that the a partition start on the outermost cylinders, but this convention makes it easier to manage disk labels. Swap performance is critical so it comes next on our way toward the center. I/O operations here tend to be large and contiguous. Having as much data under the heads as possible avoids seeking while swapping. With all the smaller partitions out of the way, we finish up the disk with /home and /usr. Access patterns here tend not to be as intense as for other file systems (especially if there is an abundant supply of RAM and read cache hit rates are high). If the pair of spindles you have are large enough to allow for more than /home and /usr, it is fine to plan for additional file systems here.

Assigning Partitions to Spindles We will want to assign partitions to these spindles so that either can fail without loss of data on file systems configured for resilience. Reliability on /usr and /home is best achieved using Vinum mirroring. Resilience will have to come differently, however, for the root file system since Vinum is not a part of the FreeBSD boot sequence. Here we will have to settle for two identical partitions with a periodic copy from the primary to the backup secondary. The kernel already has support for interleaved swap across all available partitions so there is no need for help from Vinum here. /stand/sysinstall will automatically configure /etc/fstab for all swap partitions given. The &vinum.ap; bootstrapping method given below requires a pair of spindles that I will call the root spindle and the rootback spindle. The rootback spindle must be the same size or larger than the root spindle. These instructions first allocate all space on the root spindle and then allocate exactly that amount of space on a rootback spindle. (After &vinum.ap; is bootstrapped, there is nothing special about either of these spindles--they are interchangeable.) You can later use the remaining space on the rootback spindle for other file systems. If you have more than two spindles, the bootvinum Perl script and the procedure below will help you initialize them for use with &vinum.ap;. However you will have to figure out how to assign partitions to them on your own.

Assigning Space to Partitions For this example, I will use two spindles: one with 4,124,673 blocks (about 2 GB) on /dev/ad0 and one with 8,420,769 blocks (about 4 GB) on /dev/ad2. It is best to configure your two spindles on separate controllers so that both can operate in parallel and so that you will have failure resilience in case a controller dies. Note that mirrored volume write performance will be halved in cases where both spindles share a controller that requires they operate serially (as is often the case with ATA controllers). One spindle will be the master on the primary ATA controller and the other will be the master on the secondary ATA controller. Recall that we will be allocating space on the smaller spindle first and the larger spindle second.

Assigning Partitions on the Root Spindle We will allocate 200,000 blocks (about 93 MB) for a root file system on each spindle (/dev/ad0s1a and /dev/ad2s1a). We will initially allocate 200,265 blocks for a swap partition on each spindle, giving a total of about 186 MB of swap space (/dev/ad0s1b and /dev/ad2s1b). We will lose 265 blocks from each swap partition as part of the bootstrapping process. This is the size of the space used by Vinum to store configuration information. The space will be taken from swap and given to a vinum partition but will be unavailable for Vinum subdisks. I have done the partition allocation in nice round numbers of blocks just to emphasize where the 265 blocks go. There is nothing wrong with allocating space in MB if that is more convenient for you. This leaves 4,124,673 - 200,000 - 200,265 = 3,724,408 blocks (about 1,818 MB) on the root spindle for Vinum partitions (/dev/ad0s1e and /dev/ad2s1f). From this, allocate the 265 blocks for Vinum configuration information, 1,000,000 blocks (about 488 MB) for /home, and the remaining 2,724,408 blocks (about 1,330 MB) for /usr. See below to see this graphically. The left-hand side of below shows what spindle ad0 will look like at the end of phase 2. The right-hand side shows what it will look like at the end of phase 3.

Assigning Partitions on the Rootback Spindle The /rootback and swap partition sizes on the rootback spindle must match the root and swap partition sizes on the root spindle. That leaves 8,420,769 - 200,000 - 200,265 = 8,020,504 blocks for the Vinum partition. Mirrors of /home and /usr receive the same allocation as on the root spindle. That will leave an extra 2 GB or so that we can deal with later. See below to see this graphically. The left-hand side of below shows what spindle ad2 will look like at the beginning of phase 4. The right-hand side shows what it will look like at the end.

Preparation of Tools The bootvinum Perl script given below in will make the Vinum bootstrapping process much easier if you can run it on the machine being bootstrapped. It is over 200 lines and you would not want to type it in. At this point, I recommend that you copy it to a floppy or arrange some alternative method of making it readily available so that it can be available later when needed. For example: &prompt.root; fdformat -f 1440 /dev/fd0 &prompt.root; newfs_msdos -f 1440 /dev/fd0 &prompt.root; mount /dev/fd0 /mnt &prompt.root; cp /usr/share/examples/vinum/bootvinum /mnt XXX Someday, I would like this script to live in /usr/share/examples/vinum. Till then, please use this link to get a copy.

Bootstrapping Phase 2: Minimal OS Installation Our goal in this phase is to complete the smallest possible FreeBSD installation in such a way that we can later install Vinum. We will use only partitions of type 4.2BSD (i.e., regular UFS file systems) since that is the only type supported by /stand/sysinstall.

Phase 2 Example Start up the FreeBSD installation process by running /stand/sysinstall from installation media as you normally would. Fdisk partition all spindles as needed. Make sure to select BootMgr for all spindles. Partition the root spindle with appropriate block allocations as described above in . For this example on a 2 GB spindle, I will use 200,000 blocks for root, 200,265 blocks for swap, 1,000,000 blocks for /home, and the rest of the spindle (2,724,408 blocks) for /usr. (/stand/sysinstall should automatically assign these to /dev/ad0s1a, /dev/ad0s1b, /dev/ad0s1e, and /dev/ad0s1f by default.) If you prefer soft updates as I do and you are using 4.4-RELEASE or better, this is a good time to enable them. Partition the rootback spindle with the appropriate block allocations as described above in . For this example on a 4 GB spindle, I will use 200,000 blocks for /rootback, 200,265 blocks for swap, and the rest of the spindle (8,020,504 blocks) for /NOFUTURE. (/stand/sysinstall should automatically assign these to /dev/ad2s1e, /dev/ad2s1b, and /dev/ad2s1f by default.) We do not really want to have a /NOFUTURE UFS file system (we want a vinum partition instead), but that is the best choice we have for the space given the limitations of /stand/sysinstall. Mount point names beginning with NOFUTURE and rootback serve as sentinels to the bootstrapping script presented in below. Partition any other spindles with swap if desired and a single /NOFUTURExx file system. Select a minimum system install for now even if you want to end up with more distributions loaded later. Do not worry about system configuration options at this point--get Vinum set up and get the partitions in the right places first. Exit /stand/sysinstall and reboot. Do a quick test to verify that the minimum installation was successful. The left-hand side of above and the left-hand side of above show how the disks will look at this point.

Bootstrapping Phase 3: Root Spindle Setup Our goal in this phase is get Vinum set up and running on the root spindle. We will embed the existing /usr and /home file systems in a Vinum partition. Note that the Vinum volumes created will not yet be failure-resilient since we have only one underlying Vinum drive to hold them. The resulting system will automatically start Vinum as it boots to multi-user mode.

Phase 3 Example Login as root. We will need a directory in the root file system in which to keep a few files that will be used in the Vinum bootstrapping process. &prompt.root; mkdir /bootvinum &prompt.root; cd /bootvinum Several files need to be prepared for use in bootstrapping. I have written a Perl script that makes all the required files for you. Copy this script to /bootvinum by floppy disk, tape, network, or any convenient means and then run it. (If you cannot get this script copied onto the machine being bootstrapped, then see below for a manual alternative.) &prompt.root; cp /mnt/bootvinum . &prompt.root; ./bootvinum bootvinum produces no output when run successfully. If you get any errors, something may have gone wrong when you were creating partitions with /stand/sysinstall above. Running bootvinum will: Create /etc/fstab.vinum based on what it finds in your existing /etc/fstab Create new disk labels for each spindle mentioned in /etc/fstab and keep copies of the current disk labels Create files needed as input to vinum for building Vinum objects on each spindle Create many alternates to /etc/fstab.vinum that might come in handy should a spindle fail You may want to take a look at these files to learn more about the disk partitioning required for Vinum or to learn more about the commands needed to create Vinum objects. We now need to install new spindle partitioning for /dev/ad0. This requires that /dev/ad0s1b not be in use for swapping so we have to reboot in single-user mode. First, reboot the system. &prompt.root; reboot Next, enter single-user mode. Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel] in 8 seconds... Type '?' for a list of commands, 'help' for more detailed help. ok boot -s In single-user mode, install the new partitioning created above. &prompt.root; cd /bootvinum &prompt.root; disklabel -R ad0s1 disklabel.ad0s1 &prompt.root; disklabel -R ad2s1 disklabel.ad2s1 - If you have additional spindles, repeate the + If you have additional spindles, repeat the above commands as appropriate for them. We are about to start Vinum for the first time. It is going to want to create several device nodes under /dev/vinum so we will need to mount the root file system for read/write access. &prompt.root; fsck -p / &prompt.root; mount / Now it is time to create the Vinum objects that will embed the existing non-root file systems on the root spindle in a Vinum partition. This will load the Vinum kernel module and start Vinum as a side effect. &prompt.root; vinum create create.YouCrazy You should see a list of Vinum objects created that looks like the following: 1 drives: D YouCrazy State: up Device /dev/ad0s1h Avail: 0/1818 MB (0%) 2 volumes: V home State: up Plexes: 1 Size: 488 MB V usr State: up Plexes: 1 Size: 1330 MB 2 plexes: P home.p0 C State: up Subdisks: 1 Size: 488 MB P usr.p0 C State: up Subdisks: 1 Size: 1330 MB 2 subdisks: S home.p0.s0 State: up PO: 0 B Size: 488 MB S usr.p0.s0 State: up PO: 0 B Size: 1330 MB You should also see several kernel messages which state that the Vinum objects you have created are now up. Our non-root file systems should now be embedded in a Vinum partition and hence available through Vinum volumes. It is important to test that this embedding worked. &prompt.root; fsck -n /dev/vinum/home &prompt.root; fsck -n /dev/vinum/usr This should produce no errors. If it does produce errors do not fix them. Instead, go back and examine the root spindle partition tables before and after Vinum to see if you can spot the error. You can back out the partition table changes by using disklabel -R with the disklabel.*.b4vinum files. While we have the root file system mounted read/write, this is a good time to install /etc/fstab. &prompt.root; mv /etc/fstab /etc/fstab.b4vinum &prompt.root; cp /etc/fstab.vinum /etc/fstab We are now done with tasks requiring single-user mode, so it is safe to go multi-user from here on. &prompt.root; ^D Login as root. Edit /etc/rc.conf and add this line: start_vinum="YES"

Bootstrapping Phase 4: Rootback Spindle Setup Our goal in this phase is to get redundant copies of all data from the root spindle to the rootback spindle. We will first create the necessary Vinum objects on the rootback spindle. Then we will ask Vinum to copy the data from the root spindle to the rootback spindle. Finally, we use dump and restore to copy the root file system.

Phase 4 Example Now that Vinum is running on the root spindle, we can bring it up on the rootback spindle so that our Vinum volumes can become failure-resilient. &prompt.root; cd /bootvinum &prompt.root; vinum create create.UpWindow You should see a list of Vinum objects created that looks like the following: 2 drives: D YouCrazy State: up Device /dev/ad0s1h Avail: 0/1818 MB (0%) D UpWindow State: up Device /dev/ad2s1h Avail: 2096/3915 MB (53%) 2 volumes: V home State: up Plexes: 2 Size: 488 MB V usr State: up Plexes: 2 Size: 1330 MB 4 plexes: P home.p0 C State: up Subdisks: 1 Size: 488 MB P usr.p0 C State: up Subdisks: 1 Size: 1330 MB P home.p1 C State: faulty Subdisks: 1 Size: 488 MB P usr.p1 C State: faulty Subdisks: 1 Size: 1330 MB 4 subdisks: S home.p0.s0 State: up PO: 0 B Size: 488 MB S usr.p0.s0 State: up PO: 0 B Size: 1330 MB S home.p1.s0 State: stale PO: 0 B Size: 488 MB S usr.p1.s0 State: stale PO: 0 B Size: 1330 MB You should also see several kernel messages which state that some of the Vinum objects you have created are now up while others are faulty or stale. Now we ask Vinum to copy each of the subdisks on drive YouCrazy to drive UpWindow. This will change the state of the newly created Vinum subdisks from stale to up. It will also change the state of the newly created Vinum plexes from faulty to up. First, we do the new subdisk we added to /home. &prompt.root; vinum start -w home.p1.s0 reviving home.p1.s0 (time passes . . . ) home.p1.s0 is up by force home.p1 is up home.p1.s0 is up My 5,400 RPM EIDE spindles copied at about 3.5 MBytes/sec. Your mileage may vary. Next we do the new subdisk we added to /usr. &prompt.root; vinum -w start usr.p1.s0 reviving usr.p1.s0 (time passes . . . ) usr.p1.s0 is up by force usr.p1 is up usr.p1.s0 is up All Vinum objects should be in state up at this point. The output of vinum list should look like the following: 2 drives: D YouCrazy State: up Device /dev/ad0s1h Avail: 0/1818 MB (0%) D UpWindow State: up Device /dev/ad2s1h Avail: 2096/3915 MB (53%) 2 volumes: V home State: up Plexes: 2 Size: 488 MB V usr State: up Plexes: 2 Size: 1330 MB 4 plexes: P home.p0 C State: up Subdisks: 1 Size: 488 MB P usr.p0 C State: up Subdisks: 1 Size: 1330 MB P home.p1 C State: up Subdisks: 1 Size: 488 MB P usr.p1 C State: up Subdisks: 1 Size: 1330 MB 4 subdisks: S home.p0.s0 State: up PO: 0 B Size: 488 MB S usr.p0.s0 State: up PO: 0 B Size: 1330 MB S home.p1.s0 State: up PO: 0 B Size: 488 MB S usr.p1.s0 State: up PO: 0 B Size: 1330 MB Copy the root file system so that you will have a backup. &prompt.root; cd /rootback &prompt.root; dump 0f - / | restore rf - &prompt.root; rm restoresymtable &prompt.root; cd / You may see errors like this: ./tmp/rstdir1001216411: (inode 558) not found on tape cannot find directory inode 265 abort? [yn] n expected next file 492, got 491 They seem to cause no harm. I suspect they are a consequence of dumping the file system containing /tmp and/or the pipe connecting dump and restore. Make a directory on which we can mount a damaged root file system during the recovery process. &prompt.root; mkdir /rootbad Remove sentinel mount points that are now unused. &prompt.root; rmdir /NOFUTURE* Create empty &vinum.ap; drives on remaining spindles. &prompt.root; vinum create create.ThruBank &prompt.root; ... At this point, the reliable server foundation is complete. The right-hand side of above and the right-hand side of above show how the disks will look. You may want to do a quick reboot to multi-user and give it a quick test drive. This is also a good point to complete installation of other distributions beyond the minimal install. Add packages, ports, and users as required. Configure /etc/rc.conf as required. After you have completed your server configuration, remember to do one more copy of root to /rootback as shown above before placing the server into production. Make a schedule to refresh /rootback periodically. It may be a good idea to mount /rootback read-only for normal operation of the server. This does, however, complicate the periodic refresh a bit. Do not forget to watch /var/log/messages carefully for errors. Vinum may automatically avoid failed hardware in a way that users do not notice. You must watch for such failures and get them repaired before a second failure results in data loss. You may see Vinum noting damaged objects at server boot time.

Where to Go from Here? Now that you have established the foundation of a reliable server, there are several things you might want to try next.

Make a Vinum Volume with Remaining Space Following are the steps to create another Vinum volume with space remaining on the rootback spindle. This volume will not be resilient to spindle failure since it has only one plex on a single spindle. Create a file with the following contents: volume hope plex name hope.p0 org concat volume hope sd name hope.p0.s0 drive UpWindow plex hope.p0 len 0 Specifying a length of 0 for the hope.p0.s0 subdisk asks Vinum to use whatever space is left available on the underlying drive. Feed these commands into vinum . &prompt.root; vinum create filename Now we newfs the volume and mount it. &prompt.root; newfs -v /dev/vinum/hope &prompt.root; mkdir /hope &prompt.root; mount /dev/vinum/hope /hope Edit /etc/fstab if you want /hope mounted at boot time.

Try Out More Vinum Commands You might already be familiar with vinum to get a list of all Vinum objects. Try following it to see more detail. If you have more spindles and you want to bring them up as concatenated, mirrored, or striped volumes, then give vinum drivelist, vinum drivelist, or vinum drivelist a try. See &man.vinum.8; for sample configurations and important performance considerations before settling on a final organization for your additional spindles. The failure recovery instructions below will also give you some experience using more Vinum commands.

Failure Scenarios This section contains descriptions of various failure scenarios. For each scenario, there is a subsection on how to configure your server for degraded mode operation, how to recover from the failure, how to exit degraded mode, and how to simulate the failure. Make a hard copy of these instructions and leave them inside the CPU case, being careful not to interfere with ventilation.

Root file system on ad0 unusable, rest of drive ok We assume here that the boot blocks and disk label on /dev/ad0 are ok. If your BIOS can boot from a drive other than C:, you may be able to get around this limitation.

Configure Server for Degraded Mode Use BootMgr to load kernel from /dev/ad2s1a. Hit F5 in BootMgr to select Drive 1. Hit F1 to select FreeBSD. After the kernel is loaded, hit any key but enter to interrupt the boot sequence. Boot into single-user mode and allow explicit entry of a root file system. Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel] in 8 seconds... Type '?' for a list of commands, 'help' for more detailed help. ok boot -as Select /rootback as your root file system. Manual root file system specification: <fstype>:<device> Mount <device> using filesystem <fstype> e.g. ufs:/dev/da0s1a ? List valid disk boot devices <empty line> Abort manual input mountroot> ufs:/dev/ad2s1a Now that you are in single-user mode, change /etc/fstab to avoid the bad root file system. If you used the bootvinum Perl script from below, then these commands should configure your server for degraded mode. &prompt.root; fsck -p / &prompt.root; mount / &prompt.root; cd /etc &prompt.root; mv fstab fstab.bak &prompt.root; cp fstab_ad0s1_root_bad fstab &prompt.root; cd / &prompt.root; mount -o ro / &prompt.root; vinum start &prompt.root; fsck -p &prompt.root; ^D

Recovery Restore /dev/ad0s1a from backups or copy /rootback to it with these commands: &prompt.root; umount /rootbad &prompt.root; newfs /dev/ad0s1a &prompt.root; tunefs -n enable /dev/ad0s1a &prompt.root; mount /rootbad &prompt.root; cd /rootbad &prompt.root; dump 0f - / | restore rf - &prompt.root; rm restoresymtable

Exiting Degraded Mode Enter single-user mode. &prompt.root; shutdown now Put /etc/fstab back to normal and reboot. &prompt.root; cd /rootbad/etc &prompt.root; rm fstab &prompt.root; mv fstab.bak fstab &prompt.root; reboot Reboot and hit F1 to boot from /dev/ad0 when prompted by BootMgr.

Simulation This kind of failure can be simulated by shutting down to single-user mode and then booting as shown above in .

Drive ad2 Fails This section deals with the total failure of /dev/ad2.

Configure Server for Degraded Mode After the kernel is loaded, hit any key but Enter to interrupt the boot sequence. Boot into single-user mode. Hit [Enter] to boot immediately, or any other key for command prompt. Booting [kernel] in 8 seconds... Type '?' for a list of commands, 'help' for more detailed help. ok boot -s Change /etc/fstab to avoid the bad drive. If you used the bootvinum Perl script from below, then these commands should configure your server for degraded mode. &prompt.root; fsck -p / &prompt.root; mount / &prompt.root; cd /etc &prompt.root; mv fstab fstab.bak &prompt.root; cp fstab_only_have_ad0s1 fstab &prompt.root; cd / &prompt.root; mount -o ro / &prompt.root; vinum start &prompt.root; fsck -p &prompt.root; ^D If you do not have modified versions of /etc/fstab that are ready for use, then you can use ed to make one. Alternatively, you can fsck and mount /usr and then use your favorite editor.

Recovery We assume here that your server is up and running multi-user in degraded mode on just /dev/ad0 and that you have a new spindle now on /dev/ad2 ready to go. You will need a new spindle with enough room to hold root and swap partitions plus a Vinum partition large enough to hold /home and /usr. Create a BIOS partition (slice) on the new spindle. &prompt.root; /stand/sysinstall Select Custom. Select Partition. Select ad2. Create a FreeBSD (type 165) slice large enough to hold everything mentioned above. Write changes. Yes, you are absolutely sure. Select BootMgr. Quit Partitioning. Exit /stand/sysinstall. Create disk label partitioning based on current /dev/ad0 partitioning. &prompt.root; disklabel ad0 > /tmp/ad0 &prompt.root; disklabel -e ad2 This will drop you into your favorite editor. Copy the lines for the a and b partitions from /tmp/ad0 to the ad2 disklabel. Add the size of the a and b partitions to find the proper offset for the h partition. Subtract this offset from the size of the c partition to find the proper size for the h partition. Define an h partition with the size and offset calculated above. Set the fstype column to vinum. Save the file and quit your editor. Tell Vinum about the new drive. Ask Vinum to start an editor with a copy of the current configuration. &prompt.root; vinum create Uncomment the drive line referring to drive UpWindow and set device to /dev/ad2s1h. Save the file and quit your editor. Now that Vinum has two spindles again, revive the mirrors. &prompt.root; vinum start -w usr.p1.s0 &prompt.root; vinum start -w home.p1.s0 Now we need to restore /rootback to a current copy of the root file system. These commands will accomplish this. &prompt.root; newfs /dev/ad2s1a &prompt.root; tunefs -n enable /dev/ad2s1a &prompt.root; mount /dev/ad2s1a /mnt &prompt.root; cd /mnt &prompt.root; dump 0f - / | restore rf - &prompt.root; rm restoresymtable &prompt.root; cd / &prompt.root; umount /mnt

Exiting Degraded Mode Enter single-user mode. &prompt.root; shutdown now Return /etc/fstab to its normal state and reboot. &prompt.root; cd /etc &prompt.root; rm fstab &prompt.root; mv fstab.bak fstab &prompt.root; reboot

Simulation You can simulate this kind of failure by unplugging /dev/ad2, write-protecting it, or by this procedure: Shutdown to single-user mode. Unmount all non-root file systems. Clobber any existing Vinum configuration and partitioning on /dev/ad2. &prompt.root; vinum stop &prompt.root; dd if=/dev/zero of=/dev/ad2s1h count=512 &prompt.root; dd if=/dev/zero of=/dev/ad2 count=512

Drive ad0 Fails Some BIOSes can boot from drive 1 or drive 2 (often called C: or D:), while others can boot only from drive 1. If your BIOS can boot from either, the fastest road to recovery might be to boot directly from /dev/ad2 in single-user mode and install /etc/fsatb_only_have_ad2s1 as /etc/fstab. You would then have to adapt the /dev/ad2 failure recovery instructions from above. If your BIOS can only boot from drive one, then you will have to unplug drive YouCrazy from the controller for /dev/ad2 and plug it into the controller for /dev/ad0. Then continue with the instructions for /dev/ad2 failure recovery in above.

bootvinum Perl Script The bootvinum Perl script below reads /etc/fstab and current drive partitioning. It then writes several files in the current directory and several variants of /etc/fstab in /etc. These files significantly simplify the installation of Vinum and recovery from spindle failures. #!/usr/bin/perl -w use strict; use FileHandle; -my $config_tag1 = '$Id: article.sgml,v 1.2 2001-10-29 23:20:40 chern Exp $'; +my $config_tag1 = '$Id: article.sgml,v 1.3 2001-10-31 23:10:31 chern Exp $'; # Copyright (C) 2001 Robert A. Van Valzah # # Bootstrap Vinum # # Read /etc/fstab and current partitioning for all spindles mentioned there. # Generate files needed to mirror all file systems on root spindle. # A new partition table for each spindle # Input for the vinum create command to create Vinum objects on each spindle # A copy of fstab mounting Vinum volumes instead of BSD partitions # Copies of fstab altered for server's degraded modes of operation # See handbook for instructions on how to use the the files generated. # N.B. This bootstrapping method shrinks size of swap partition by the size # of Vinum's on-disk configuration (265 sectors). It embeds existing file # systems on the root spindle in Vinum objects without having to copy them. # Thanks to Greg Lehey for suggesting this bootstrapping method. # Expectations: # The root spindle must contain at least root, swap, and /usr partitions # The rootback spindle must have matching /rootback and swap partitions # Other spindles should only have a /NOFUTURE* file system and maybe swap # File systems named /NOFUTURE* will be replaced with Vinum drives # Change configuration variables below to suit your taste my $vip = 'h'; # VInum Partition my @drv = ('YouCrazy', 'UpWindow', 'ThruBank', # Vinum DRiVe names 'OutSnakes', 'MeWild', 'InMovie', 'HomeJames', 'DownPrices', 'WhileBlind'); # No configuration variables beyond this point my %vols; # One entry per Vinum volume to be created my @spndl; # One entry per SPiNDLe my $rsp; # Root SPindle (as in /dev/$rsp) my $rbsp; # RootBack SPindle (as in /dev/$rbsp) my $cfgsiz = 265; # Size of Vinum on-disk configuration info in sectors my $nxtpas = 2; # Next fsck pass number for non-root file systems # Parse fstab, generating the version we'll need for Vinum and noting # spindles in use. my $fsin = "/etc/fstab"; #my $fsin = "simu/fstab"; open(FSIN, "$fsin") || die("Couldn't open $fsin: $!\n"); my $fsout = "/etc/fstab.vinum"; open(FSOUT, ">$fsout") || die("Couldn't open $fsout for writing: $!\n"); while (<FSIN>) { my ($dev, $mnt, $fstyp, $opt, $dump, $pass) = split; next if $dev =~ /^#/; if ($mnt eq '/' || $mnt eq '/rootback' || $mnt =~ /^\/NOFUTURE/) { my $dn = substr($dev, 5, length($dev)-6); # Device Name without /dev/ push(@spndl, $dn) unless grep($_ eq $dn, @spndl); $rsp = $dn if $mnt eq '/'; next if $mnt =~ /^\/NOFUTURE/; } # Move /rootback from partition e to a if ($mnt =~ /^\/rootback/) { $dev =~ s/e$/a/; $pass = 1; $rbsp = substr($dev, 5, length($dev)-6); print FSOUT "$dev\t\t$mnt\t$fstyp\t$opt\t\t$dump\t$pass\n"; next; } # Move non-root file systems on smallest spindle into Vinum if (defined($rsp) && $dev =~ /^\/dev\/$rsp/ && $dev =~ /[d-h]$/) { $pass = $nxtpas++; print FSOUT "/dev/vinum$mnt\t\t$mnt\t\t$fstyp\t$opt\t\t$dump\t$pass\n"; $vols{$dev}->{mnt} = substr($mnt, 1); next; } print FSOUT $_; } close(FSOUT); die("Found more spindles than we have abstract names\n") if $#spndl > $#drv; die("Didn't find a root partition!\n") if !defined($rsp); die("Didn't find a /rootback partition!\n") if !defined($rbsp); # Table of server's Degraded Modes # One row per mode with hash keys # fn FileName # xpr eXPRession needed to convert fstab lines for this mode # cm1 CoMment 1 describing this mode # cm2 CoMment 2 describing this mode # FH FileHandle (dynamically initialized below) my @DM = ( { cm1 => "When we only have $rsp, comment out lines using $rbsp", fn => "/etc/fstab_only_have_$rsp", xpr => "s:^/dev/$rbsp:#\$&:", }, { cm1 => "When we only have $rbsp, comment out lines using $rsp and", cm2 => "rootback becomes root", fn => "/etc/fstab_only_have_$rbsp", xpr => "s:^/dev/$rsp:#\$&: || s:/rootback:/\t:", }, { cm1 => "When only $rsp root is bad, /rootback becomes root and", cm2 => "root becomes /rootbad", fn => "/etc/fstab_${rsp}_root_bad", xpr => "s:\t/\t:\t/rootbad: || s:/rootback:/\t:", }, ); # Initialize output FileHandles and write comments foreach my $dm (@DM) { my $fh = new FileHandle; $fh->open(">$dm->{fn}") || die("Can't write $dm->{fn}: $!\n"); print $fh "# $dm->{cm1}\n" if $dm->{cm1}; print $fh "# $dm->{cm2}\n" if $dm->{cm2}; $dm->{FH} = $fh; } # Parse the Vinum version of fstab written above and write versions needed # for server's degraded modes. open(FSOUT, "$fsout") || die("Couldn't open $fsout: $!\n"); while (<FSOUT>) { my $line = $_; foreach my $dm (@DM) { $_ = $line; eval $dm->{xpr}; print {$dm->{FH}} $_; } } # Parse partition table for each spindle and write versions needed for Vinum my $rootsiz; # ROOT partition SIZe my $swapsiz; # SWAP partition SIZe my $rspminoff; # Root SPindle MINimum OFFset of non-root, non-swap, non-c parts my $rspsiz; # Root SPindle SIZe my $rbspsiz; # RootBack SPindle SIZe foreach my $i (0..$#spndl) { my $dlin = "disklabel $spndl[$i] |"; # my $dlin = "simu/disklabel.$spndl[$i]"; open(DLIN, "$dlin") || die("Couldn't open $dlin: $!\n"); my $dlout = "disklabel.$spndl[$i]"; open(DLOUT, ">$dlout") || die("Couldn't open $dlout for writing: $!\n"); my $dlb4 = "$dlout.b4vinum"; open(DLB4, ">$dlb4") || die("Couldn't open $dlb4 for writing: $!\n"); my $minoff; # MINimum OFFset of non-root, non-swap, non-c partitions my $totsiz = 0; # TOTal SIZe of all non-root, non-swap, non-c partitions my $swapspndl = 0; # True if SWAP partition on this SPiNDLe while (<DLIN>) { print DLB4 $_; my ($part, $siz, $off, $fstyp, $fsiz, $bsiz, $bps) = split; if ($part && $part eq 'a:' && $spndl[$i] eq $rsp) { $rootsiz = $siz; } if ($part && $part eq 'e:' && $spndl[$i] eq $rbsp) { if ($rootsiz != $siz) { die("Rootback size ($siz) != root size ($rootsiz)\n"); } } if ($part && $part eq 'c:') { $rspsiz = $siz if $spndl[$i] eq $rsp; $rbspsiz = $siz if $spndl[$i] eq $rbsp; } # Make swap partition $cfgsiz sectors smaller if ($part && $part eq 'b:') { if ($spndl[$i] eq $rsp) { $swapsiz = $siz; } else { if ($swapsiz != $siz) { die("Swap partition sizes unequal across spindles\n"); } } printf DLOUT "%4s%9d%9d%10s\n", $part, $siz-$cfgsiz, $off, $fstyp; $swapspndl = 1; next; } # Move rootback spindle e partitions to a if ($part && $part eq 'e:' && $spndl[$i] eq $rbsp) { printf DLOUT "%4s%9d%9d%10s%9d%6d%6d\n", 'a:', $siz, $off, $fstyp, $fsiz, $bsiz, $bps; next; } # Delete non-root, non-swap, non-c partitions but note their minimum # offset and total size that're needed below. if ($part && $part =~ /^[d-h]:$/) { $minoff = $off unless $minoff; $minoff = $off if $off < $minoff; $totsiz += $siz; if ($spndl[$i] eq $rsp) { # If doing spindle containing root my $dev = "/dev/$spndl[$i]" . substr($part, 0, 1); $vols{$dev}->{siz} = $siz; $vols{$dev}->{off} = $off; $rspminoff = $minoff; } next; } print DLOUT $_; } if ($swapspndl) { # If there was a swap partition on this spindle # Make a Vinum partition the size of all non-root, non-swap, # non-c partitions + the size of Vinum's on-disk configuration. # Set its offset so that the start of the first subdisk it contains # coincides with the first file system we're embedding in Vinum. printf DLOUT "%4s%9d%9d%10s\n", "$vip:", $totsiz+$cfgsiz, $minoff-$cfgsiz, 'vinum'; } else { # No need to mess with size size and offset if there was no swap printf DLOUT "%4s%9d%9d%10s\n", "$vip:", $totsiz, $minoff, 'vinum'; } } die("Swap partition not found\n") unless $swapsiz; die("Swap partition not larger than $cfgsiz blocks\n") unless $swapsiz>$cfgsiz; die("Rootback spindle size not >= root spindle size\n") unless $rbspsiz>=$rspsiz; # Generate input to vinum create command needed for each spindle. foreach my $i (0..$#spndl) { my $cfn = "create.$drv[$i]"; # Create File Name open(CF, ">$cfn") || die("Can't open $cfn for writing: $!\n"); print CF "drive $drv[$i] device /dev/$spndl[$i]$vip\n"; next unless $spndl[$i] eq $rsp || $spndl[$i] eq $rbsp; foreach my $dev (keys(%vols)) { my $mnt = $vols{$dev}->{mnt}; my $siz = $vols{$dev}->{siz}; my $off = $vols{$dev}->{off}-$rspminoff+$cfgsiz; print CF "volume $mnt\n" if $spndl[$i] eq $rsp; print CF <<EOF; plex name $mnt.p$i org concat volume $mnt sd name $mnt.p$i.s0 drive $drv[$i] plex $mnt.p$i len ${siz}s driveoffset ${off}s EOF } } Manual Vinum Bootstrapping The bootvinum Perl script in makes life easier, but it may be necessary to manually perform some or all of the steps that it automates. This appendix describes how you would manually mimic the script. Make a copy of /etc/fstab to be customized. &prompt.root; cp /etc/fstab /etc/fstab.vinum Edit /etc/fstab.vinum. Change the device column of non-root partitions on the root spindle to /dev/vinum/mnt. Change the pass column of non-root partitions on the root spindle to 2, 3, etc. Delete any lines with mountpoint matching /NOFUTURE*. Change the device column of /rootback from e to a. Change the pass column of /rootback to 1. Prepare disklabels for editing: &prompt.root; cd /bootvinum &prompt.root; disklabel ad0s1 > disklabel.ad0s1 &prompt.root; cp disklabel.ad0s1 disklabel.ad0s1.b4vinum &prompt.root; disklabel ad2s1 > disklabel.ad2s1 &prompt.root; cp disklabel.ad2s1 disklabel.ad2s1.b4vinum Edit /etc/disklabel.ad?s1. On the root spindle: Decrease the size of the b partition by 265 blocks. Note the size and offset of the a and b partitions. Note the smallest offset for partitions d-h. Note the size and offset for all non-root, non-swap partitions (/home was probably on e and /usr was probably on f). Delete partitions d-h. Create a new h partition with offset 265 blocks less than the smallest offset for partitions d-h noted above. Set its size to the size of the c partition less the smallest offset for partitions d-h noted above + 265 blocks. Vinum can use any partition other than c. It is not strictly necessary to use h for all your Vinum partitions, but it is good practice to be consistent across all spindles. Set the fstype of this new partition to vinum. On the rootback spindle: Move the e partition to a. Verify that the size of the a and b partitions matches the root spindle. Note the smallest offset for partitions d-h. Delete partitions d-h. Create a new h partition with offset 265 blocks less than the smallest offset noted above for partitions d-h. Set its size to the size of the c partition less the smallest offset for partitions d-h noted above + 265 blocks. Set the fstype of this new partition to vinum. Create a file named create.YouCrazy that contains: drive YouCrazy device /dev/ad0s1h volume home plex name home.p0 org concat volume home sd name home.p0.s0 drive YouCrazy plex home.p0 len $hl driveoffset $ho volume usr plex name usr.p0 org concat volume usr sd name usr.p0.s0 drive YouCrazy plex usr.p0 len $ul driveoffset $uo Where: $hl is the length noted above for /home. $ho is the offset noted above for /home less the smallest offset noted above + 265 blocks. $ul is the length noted above for /usr. $uo is the offset noted above for /usr less the smallest offset noted above + 265 blocks. Create a file named create.UpWindow containing: drive UpWindow device /dev/ad2s1h plex name home.p1 org concat volume home sd name home.p1.s0 drive UpWindow plex home.p1 len $hl driveoffset $ho plex name usr.p1 org concat volume usr sd name usr.p1.s0 drive UpWindow plex usr.p1 len $ul driveoffset $uo Where $hl, $ho, $ul, and $uo are set as above. Acknowledgements I would like to thank Greg Lehey for writing &vinum.ap; and for providing very helpful comments on early drafts. Several others made helpful suggestions after reviewing later drafts including Dag-Erling Smørgrav, Michael Splendoria, Chern Lee, Stefan Aeschbacher, Fleming Froekjaer, Bernd Walter, Aleksey Baranov, and Doug Swarin.

Console Server

The Problem

Possible Solutions

Our Solution

Setting Up The Server

Patching the Stallion driver

Configuring a new kernel

Building World

Making The Devices

Compiling conserver

Using the port framework

From the source tarball

Configuring conserver

Setting conserver passwords

Starting conserver at system boot time

Keeping the log files trimmed

Cabling

RJ-45 colors

On Sun Systems And Break

Using a Serial Console on FreeBSD

Check the kernel configuration

Create the `/boot.conf` file

Edit `/etc/ttys`

Security Implications

On Conserver Versions

Links

Files

Man Pages

Change Log

Console Server

The Problem

Possible Solutions

Our Solution

Setting Up The Server

Patching the Stallion driver

Configuring a new kernel

Building World

Making The Devices

Compiling conserver

Using the port framework

From the source tarball

Configuring conserver

Setting conserver passwords

Starting conserver at system boot time

Keeping the log files trimmed

Cabling

RJ-45 colors

On Sun Systems And Break

Using a Serial Console on FreeBSD

Check the kernel configuration

Create the /boot.conf file

Edit /etc/ttys

Security Implications

On Conserver Versions

Links

Files

Man Pages

Change Log

Create the `/boot.conf` file

Edit `/etc/ttys`