diff --git a/documentation/content/en/articles/committers-guide/_index.adoc b/documentation/content/en/articles/committers-guide/_index.adoc index 47d7060859..744d3e31ed 100644 --- a/documentation/content/en/articles/committers-guide/_index.adoc +++ b/documentation/content/en/articles/committers-guide/_index.adoc @@ -1,3721 +1,3721 @@ --- title: Committer's Guide authors: - author: The FreeBSD Documentation Project copyright: 1999-2022 The FreeBSD Documentation Project description: Introductory information for FreeBSD committers trademarks: ["freebsd", "coverity", "git", "github", "gitlab", "ibm", "intel", "general"] weight: 25 tags: ["FreeBSD Committer's Guide", "Guide", "Community"] --- = Committer's Guide :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/committers-guide/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract This document provides information for the FreeBSD committer community. All new committers should read this document before they start, and existing committers are strongly encouraged to review it from time to time. Almost all FreeBSD developers have commit rights to one or more repositories. However, a few developers do not, and some of the information here applies to them as well. (For instance, some people only have rights to work with the Problem Report database.) Please see <> for more information. This document may also be of interest to members of the FreeBSD community who want to learn more about how the project works. ''' toc::[] [[admin]] == Administrative Details [.informaltable] [cols="1,1", frame="none"] |=== |_Login Methods_ |man:ssh[1], protocol 2 only |_Main Shell Host_ |`freefall.FreeBSD.org` |_Reference Machines_ |`ref*.FreeBSD.org`, `universe*.freeBSD.org` (see also link:https://www.FreeBSD.org/internal/machines/[FreeBSD Project Hosts]) |_SMTP Host_ |`smtp.FreeBSD.org:587` (see also <>). |`_src/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/src.git` |`_doc/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/doc.git` |`_ports/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/ports.git` |_Internal Mailing Lists_ |developers (technically called all-developers), doc-developers, doc-committers, ports-developers, ports-committers, src-developers, src-committers. (Each project repository has its own -developers and -committers mailing lists. Archives for these lists can be found in the files [.filename]#/local/mail/repository-name-developers-archive# and [.filename]#/local/mail/repository-name-committers-archive# on `freefall.FreeBSD.org`.) |_Core Team monthly reports_ |[.filename]#/home/core/public/reports# on the `FreeBSD.org` cluster. |_Ports Management Team monthly reports_ |[.filename]#/home/portmgr/public/monthly-reports# on the `FreeBSD.org` cluster. |_Noteworthy `src/` Git Branches:_ |`stable/n` (`n`-STABLE), `main` (-CURRENT) |=== man:ssh[1] is required to connect to the project hosts. For more information, see <>. Useful links: * link:https://www.FreeBSD.org/internal/[FreeBSD Project Internal Pages] * link:https://www.FreeBSD.org/internal/machines/[FreeBSD Project Hosts] * link:https://www.FreeBSD.org/administration/[FreeBSD Project Administrative Groups] [[pgpkeys]] == OpenPGP Keys for FreeBSD Cryptographic keys conforming to the OpenPGP (__Pretty Good Privacy__) standard are used by the FreeBSD project to authenticate committers. Messages carrying important information like public SSH keys can be signed with the OpenPGP key to prove that they are really from the committer. See https://nostarch.com/releases/pgp_release.pdf[PGP & GPG: Email for the Practical Paranoid by Michael Lucas] and http://en.wikipedia.org/wiki/Pretty_Good_Privacy[] for more information. [[pgpkeys-creating]] === Creating a Key Existing keys can be used, but should be checked with [.filename]#documentation/tools/checkkey.sh# first. In this case, make sure the key has a FreeBSD user ID. For those who do not yet have an OpenPGP key, or need a new key to meet FreeBSD security requirements, here we show how to generate one. [[pgpkeys-create-steps]] [.procedure] ==== . Install [.filename]#security/gnupg#. Enter these lines in [.filename]#~/.gnupg/gpg.conf# to set minimum acceptable defaults: + [.programlisting] .... fixed-list-mode keyid-format 0xlong personal-digest-preferences SHA512 SHA384 SHA256 SHA224 default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP Uncompressed verify-options show-uid-validity list-options show-uid-validity sig-notation issuer-fpr@notations.openpgp.fifthhorseman.net=%g cert-digest-algo SHA512 .... . Generate a key: + [source,shell] .... % gpg --full-gen-key gpg (GnuPG) 2.1.8; Copyright (C) 2015 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Warning: using insecure memory! Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 2048 <.> Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 3y <.> Key expires at Wed Nov 4 17:20:20 2015 MST Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Chucky Daemon <.> Email address: notreal@example.com Comment: You selected this USER-ID: "Chucky Daemon " Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o You need a Passphrase to protect your secret key. .... <.> 2048-bit keys with a three-year expiration provide adequate protection at present (2022-10). <.> A three year key lifespan is short enough to obsolete keys weakened by advancing computer power, but long enough to reduce key management problems. <.> Use your real name here, preferably matching that shown on government-issued ID to make it easier for others to verify your identity. Text that may help others identify you can be entered in the `Comment` section. + After the email address is entered, a passphrase is requested. Methods of creating a secure passphrase are contentious. Rather than suggest a single way, here are some links to sites that describe various methods: https://world.std.com/~reinhold/diceware.html[], https://www.iusmentis.com/security/passphrasefaq/[], https://xkcd.com/936/[], https://en.wikipedia.org/wiki/Passphrase[]. ==== Protect the private key and passphrase. If either the private key or passphrase may have been compromised or disclosed, immediately notify mailto:accounts@FreeBSD.org[accounts@FreeBSD.org] and revoke the key. Committing the new key is shown in <>. [[kerberos-ldap]] == Kerberos and LDAP web Password for FreeBSD Cluster The FreeBSD cluster requires a Kerberos password to access certain services. The Kerberos password also serves as the LDAP web password, since LDAP is proxying to Kerberos in the cluster. Some of the services which require this include: * https://bugs.freebsd.org/bugzilla[Bugzilla] * https://ci.freebsd.org[Jenkins] To create a new Kerberos account in the FreeBSD cluster, or to reset a Kerberos password for an existing account using a random password generator: [source,shell] .... % ssh kpasswd.freebsd.org .... [NOTE] ==== This must be done from a machine outside of the FreeBSD.org cluster. ==== A Kerberos password can also be set manually by logging into `freefall.FreeBSD.org` and running: [source,shell] .... % kpasswd .... [NOTE] ==== Unless the Kerberos-authenticated services of the FreeBSD.org cluster have been used previously, `Client unknown` will be shown. This error means that the `ssh kpasswd.freebsd.org` method shown above must be used first to initialize the Kerberos account. ==== [[committer.types]] == Commit Bit Types The FreeBSD repository has a number of components which, when combined, support the basic operating system source, documentation, third party application ports infrastructure, and various maintained utilities. When FreeBSD commit bits are allocated, the areas of the tree where the bit may be used are specified. Generally, the areas associated with a bit reflect who authorized the allocation of the commit bit. Additional areas of authority may be added at a later date: when this occurs, the committer should follow normal commit bit allocation procedures for that area of the tree, seeking approval from the appropriate entity and possibly getting a mentor for that area for some period of time. [.informaltable] [cols="1,1,1", frame="none"] |=== |__Committer Type__ |__Responsible__ |__Tree Components__ |src |core@ |src/ |doc |doceng@ |doc/, ports/, src/ documentation |ports |portmgr@ |ports/ |=== Commit bits allocated prior to the development of the notion of areas of authority may be appropriate for use in many parts of the tree. However, common sense dictates that a committer who has not previously worked in an area of the tree seek review prior to committing, seek approval from the appropriate responsible party, and/or work with a mentor. Since the rules regarding code maintenance differ by area of the tree, this is as much for the benefit of the committer working in an area of less familiarity as it is for others working on the tree. Committers are encouraged to seek review for their work as part of the normal development process, regardless of the area of the tree where the work is occurring. === Policy for Committer Activity in Other Trees * All committers may modify [.filename]#src/share/misc/committers-*.dot#, [.filename]#src/usr.bin/calendar/calendars/calendar.freebsd#, and [.filename]#ports/astro/xearth/files#. * doc committers may commit documentation changes to [.filename]#src# files, such as manual pages, READMEs, fortune databases, calendar files, and comment fixes without approval from a src committer, subject to the normal care and tending of commits. * Any committer may make changes to any other tree with an "Approved by" from a non-mentored committer with the appropriate bit. Mentored committers can provide a "Reviewed by" but not an "Approved by". * Committers can acquire an additional bit by the usual process of finding a mentor who will propose them to core, doceng, or portmgr, as appropriate. When approved, they will be added to 'access' and the normal mentoring period will ensue, which will involve a continuing of "Approved by" for some period. [[doc-blanket-approval]] ==== Documentation Implicit (Blanket) Approval Some types of fixes have "blanket approval" from the {doceng}, allowing any committer to fix those categories of problems on any part of the doc tree. These fixes do not need approval or review from a doc committer if the author doesn't have a doc commit bit. Blanket approval applies to these types of fixes: * Typos * Trivial fixes + Punctuation, URLs, dates, paths and file names with outdated or incorrect information, and other common mistakes that may confound the readers. Over the years, some implicit approvals were granted in the doc tree. This list shows the most common cases: * Changes in [.filename]#documentation/content/en/books/porters-handbook/versions/_index.adoc# + extref:{porters-handbook}versions/[__FreeBSD_version Values (Porter's Handbook)], mainly used for src committers. * Changes in [.filename]#doc/shared/contrib-additional.adoc# + extref:{contributors}[Additional FreeBSD Contributors, contrib-additional] maintenance. * All link:#commit-steps[Steps for New Committers], doc related * Security advisories; Errata Notices; Releases; + Used by {security-officer} and {re}. * Changes in [.filename]#website/content/en/donations/donors.adoc# + Used by {donations}. Before any commit, a build test is necessary; see the 'Overview' and 'The FreeBSD Documentation Build Process' sections of the extref:{fdp-primer}[FreeBSD Documentation Project Primer for New Contributors] for more details. [[git-primer]] == Git Primer [[git-basics]] === Git basics When one searches for "Git Primer" a number of good ones come up. Daniel Miessler's link:https://danielmiessler.com/study/git/[A git primer] and Willie Willus' link:https://gist.github.com/williewillus/068e9a8543de3a7ef80adb2938657b6b[Git - Quick Primer] are both good overviews. The Git book is also complete, but much longer https://git-scm.com/book/en/v2. There is also this website https://dangitgit.com/ for common traps and pitfalls of Git, in case you need guidance to fix things up. Finally, an introduction link:https://eagain.net/articles/git-for-computer-scientists/[targeted at computer scientists] has proven helpful to some at explaining the Git world view. This document will assume that you've read through it and will try not to belabor the basics (though it will cover them briefly). [[git-mini-primer]] === Git Mini Primer This primer is less ambitiously scoped than the old Subversion Primer, but should cover the basics. ==== Scope If you want to download FreeBSD, compile it from sources, and generally keep up to date that way, this primer is for you. It covers getting the sources, updating the sources, bisecting and touches briefly on how to cope with a few local changes. It covers the basics, and tries to give good pointers to more in-depth treatment for when the reader finds the basics insufficient. Other sections of this guide cover more advanced topics related to contributing to the project. The goal of this section is to highlight those bits of Git needed to track sources. They assume a basic understanding of Git. There are many primers for Git on the web, but the https://git-scm.com/book/en/v2[Git Book] provides one of the better treatments. [[git-mini-primer-getting-started]] ==== Getting Started For Developers This section describes the read-write access for committers to push the commits from developers or contributors. [[git-mini-daily-use]] ===== Daily use * Clone the repository: + [source,shell] .... % git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/${repo}.git .... + Then you should have the official mirrors as your remote: + [source,shell] .... % git remote -v freebsd https://git.freebsd.org/${repo}.git (fetch) freebsd https://git.freebsd.org/${repo}.git (push) .... * Configure the FreeBSD committer data: + The commit hook in repo.freebsd.org checks the "Commit" field matches the committer's information in FreeBSD.org. The easiest way to get the suggested config is by executing `/usr/local/bin/gen-gitconfig.sh` script on freefall: + [source,shell] .... % gen-gitconfig.sh [...] % git config user.name (your name in gecos) % git config user.email (your login)@FreeBSD.org .... * Set the push URL: + [source,shell] .... % git remote set-url --push freebsd git@gitrepo.freebsd.org:${repo}.git .... + Then you should have separated fetch and push URLs as the most efficient setup: + [source,shell] .... % git remote -v freebsd https://git.freebsd.org/${repo}.git (fetch) freebsd git@gitrepo.freebsd.org:${repo}.git (push) .... + Again, note that `gitrepo.freebsd.org` has been canonicalized to `repo.freebsd.org`. * Install commit message template hook: + [source,shell] .... % fetch https://cgit.freebsd.org/src/plain/tools/tools/git/hooks/prepare-commit-msg -o .git/hooks % chmod 755 .git/hooks/prepare-commit-msg .... [[admin-branch]] ===== "admin" branch The `access` and `mentors` files are stored in an orphan branch, `internal/admin`, in each repository. Following example is how to check out the `internal/admin` branch to a local branch named `admin`: [source,shell] .... % git config --add remote.freebsd.fetch '+refs/internal/*:refs/internal/*' % git fetch % git checkout -b admin internal/admin .... Alternatively, you can add a worktree for the `admin` branch: [source,shell] .... git worktree add -b admin ../${repo}-admin internal/admin .... For browsing `internal/admin` branch on web: `https://cgit.freebsd.org/${repo}/log/?h=internal/admin` For pushing, either specify the full refspec: [source,shell] .... git push freebsd HEAD:refs/internal/admin .... ==== Keeping Current With The FreeBSD src Tree [[keeping_current]] First step: cloning a tree. This downloads the entire tree. There are two ways to download. Most people will want to do a deep clone of the repository. However, there are times when you may wish to do a shallow clone. ===== Branch Names FreeBSD-CURRENT uses the `main` branch. `main` is the default branch. For FreeBSD-STABLE, branch names include `stable/12` and `stable/13`. For FreeBSD-RELEASE, release engineering branch names include `releng/12.4` and `releng/13.2`. https://www.freebsd.org/releng/[] shows: * `main` and `stable/⋯` branches open * `releng/⋯` branches, each of which is frozen when a release is tagged. Examples: * tag https://cgit.freebsd.org/src/tag/?h=release/13.1.0[release/13.1.0] on the https://cgit.freebsd.org/src/log/?h=releng/13.1[releng/13.1] branch * tag https://cgit.freebsd.org/src/tag/?h=release/13.2.0[release/13.2.0] on the https://cgit.freebsd.org/src/log/?h=releng/13.2[releng/13.2] branch. ===== Repositories Please see the <> for the latest information on where to get FreeBSD sources. $URL below can be obtained from that page. Note: The project doesn't use submodules as they are a poor fit for our workflows and development model. How we track changes in third-party applications is discussed elsewhere and generally of little concern to the casual user. ===== Deep Clone A deep clone pulls in the entire tree, as well as all the history and branches. It is the easiest to do. It also allows you to use Git's worktree feature to have all your active branches checked out into separate directories but with only one copy of the repository. [source,shell] .... % git clone -o freebsd $URL -b branch [] .... -- will create a deep clone. `branch` should be one of the branches listed in the previous section. If no `branch` is given: the default (`main`) will be used. If no `` is given: the name of the new directory will match the name of the repo ([.filename]#doc#, [.filename]#ports# or [.filename]#src#). You will want a deep clone if you are interested in the history, plan on making local changes, or plan on working on more than one branch. It is the easiest to keep up to date as well. If you are interested in the history, but are working with only one branch and are short on space, you can also use --single-branch to only download the one branch (though some merge commits will not reference the merged-from branch which may be important for some users who are interested in detailed versions of history). ===== Shallow Clone A shallow clone copies just the most current code, but none or little of the history. This can be useful when you need to build a specific revision of FreeBSD, or when you are just starting out and plan to track the tree more fully. You can also use it to limit history to only so many revisions. However, see below for a significant limitation of this approach. [source,shell] .... % git clone -o freebsd -b branch --depth 1 $URL [dir] .... This clones the repository, but only has the most recent version in the repository. The rest of the history is not downloaded. Should you change your mind later, you can do `git fetch --unshallow` to get the old history. [WARNING] ==== When you make a shallow clone, you will lose the commit count in your uname output. This can make it more difficult to determine if your system needs to be updated when a security advisory is issued. ==== ===== Building Once you've downloaded, building is done as described in the handbook, e.g.: [source,shell] .... % cd src % make buildworld % make buildkernel % make installkernel % make installworld .... so that won't be covered in depth here. If you want to build a custom kernel, extref:{handbook}[the kernel config section, kernelconfig] of the FreeBSD Handbook recommends creating a file MYKERNEL under sys/${ARCH}/conf with your changes against GENERIC. To have MYKERNEL disregarded by Git, it can be added to .git/info/exclude. ===== Updating To update both types of trees uses the same commands. This pulls in all the revisions since your last update. [source,shell] .... % git pull --ff-only .... will update the tree. In Git, a 'fast forward' merge is one that only needs to set a new branch pointer and doesn't need to re-create the commits. By always doing a fast forward merge/pull, you'll ensure that you have an exact copy of the FreeBSD tree. This will be important if you want to maintain local patches. See below for how to manage local changes. The simplest is to use `--autostash` on the `git pull` command, but more sophisticated options are available. ==== Selecting a Specific Version In Git, `git checkout` checks out both branches and specific versions. Git's versions are the long hashes rather than a sequential number. When you checkout a specific version, just specify the hash you want on the command line (the git log command can help you decide which hash you might want): [source,shell] .... % git checkout 08b8197a74 .... and you have that checked out. You will be greeted with a message similar to the following: [source,shell] .... Note: checking out '08b8197a742a96964d2924391bf9fdfeb788865d'. You are in a 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 08b8197a742a hook gpiokeys.4 to the build .... where the last line is generated from the hash you are checking out and the first line of the commit message from that revision. The hash can be abbreviated to the shortest unique length. Git itself is inconsistent about how many digits it displays. ==== Bisecting Sometimes, things go wrong. The last version worked, but the one you just updated to does not. A developer may ask you to bisect the problem to track down which commit caused the regression. Git makes bisecting changes easy with a powerful `git bisect` command. Here's a brief outline of how to use it. For more information, you can view https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination or https://git-scm.com/docs/git-bisect for more details. The man git-bisect page is good at describing what can go wrong, what to do when versions won't build, when you want to use terms other than 'good' and 'bad', etc, none of which will be covered here. `git bisect start --first-parent` will start the bisection process. Next, you need to tell a range to go through. `git bisect good XXXXXX` will tell it the working version and `git bisect bad XXXXX` will tell it the bad version. The bad version will almost always be HEAD (a special tag for what you have checked out). The good version will be the last one you checked out. The `--first-parent` argument is necessary so that subsequent `git bisect` commands do not try to check out a vendor branch which lacks the full FreeBSD source tree. [TIP] ==== If you want to know the last version you checked out, you should use `git reflog`: [source,shell] .... 5ef0bd68b515 (HEAD -> main, freebsd/main, freebsd/HEAD) HEAD@{0}: pull --ff-only: Fast-forward a8163e165c5b (upstream/main) HEAD@{1}: checkout: moving from b6fb97efb682994f59b21fe4efb3fcfc0e5b9eeb to main ... .... shows me moving the working tree to the `main` branch (a816...) and then updating from upstream (to 5ef0...). In this case, bad would be HEAD (or 5rf0bd68) and good would be a8163e165. As you can see from the output, HEAD@{1} also often works, but isn't foolproof if you have done other things to your Git tree after updating, but before you discover the need to bisect. ==== Set the 'good' version first, then set the bad (though the order doesn't matter). When you set the bad version, it will give you some statistics on the process: [source,shell] .... % git bisect start --first-parent % git bisect good a8163e165c5b % git bisect bad HEAD Bisecting: 1722 revisions left to test after this (roughly 11 steps) [c427b3158fd8225f6afc09e7e6f62326f9e4de7e] Fixup r361997 by balancing parens. Duh. .... You would then build/install that version. If it's good you'd type `git bisect good` otherwise `git bisect bad`. If the version doesn't compile, type `git bisect skip`. You will get a similar message to the above after each step. When you are done, report the bad version to the developer (or fix the bug yourself and send a patch). `git bisect reset` will end the process and return you back to where you started (usually tip of `main`). Again, the git-bisect manual (linked above) is a good resource for when things go wrong or for unusual cases. [[git-gpg-signing]] ==== Signing the commits, tags, and pushes, with GnuPG Git knows how to sign commits, tags, and pushes. When you sign a Git commit or a tag, you can prove that the code you submitted came from you and wasn't altered while you were transferring it. You also can prove that you submitted the code and not someone else. A more in-depth documentation on signing commits and tags can be found in the https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work[Git Tools - Signing Your Work] chapter of the Git's book. The rationale behind signing pushes can be found in the https://github.com/git/git/commit/a85b377d0419a9dfaca8af2320cc33b051cbed04[commit that introduced the feature]. The best way is to simply tell Git you always want to sign commits, tags, and pushes. You can do this by setting a few configuration variables: [source,shell] .... % git config --add user.signingKey LONG-KEY-ID % git config --add commit.gpgSign true % git config --add tag.gpgSign true % git config --add push.gpgSign if-asked .... // push.gpgSign should probably be set to `yes` once we enable it, or be set with --global, so that it is enabled for all repositories. [NOTE] ====== To avoid possible collisions, make sure you give a long key id to Git. You can get the long id with: `gpg --list-secret-keys --keyid-format LONG`. ====== [TIP] ====== To use specific subkeys, and not have GnuPG to resolve the subkey to a primary key, attach `!` to the key. For example, to encrypt for the subkey `DEADBEEF`, use `DEADBEEF!`. ====== ===== Verifying signatures Commit signatures can be verified by running either `git verify-commit `, or `git log --show-signature`. Tag signatures can be verified with `git verity-tag `, or `git tag -v `. //// Commented out for now until we decide what to do. Git pushes are a bit different, they live in a special ref in the repository. TODO: write how to verify them //// ==== Ports Considerations The ports tree operates the same way. The branch names are different and the repositories are in different locations. The cgit repository web interface for use with web browsers is at https://cgit.FreeBSD.org/ports/ . The production Git repository is at https://git.FreeBSD.org/ports.git and at ssh://anongit@git.FreeBSD.org/ports.git (or anongit@git.FreeBSD.org:ports.git). There is also a mirror on GitHub, see extref:{handbook}/mirrors[External mirrors, mirrors] for an overview. The _latest_ branch is `main`. The _quarterly_ branches are named `yyyyQn` for year 'yyyy' and quarter 'n'. [[port-commit-message-formats]] ===== Commit message formats A hook is available in the ports repository to help you write up your commit messages in https://cgit.freebsd.org/ports/tree/.hooks/prepare-commit-msg[.hooks/prepare-commit-message]. It can be enabled by running ``git config --add core.hooksPath .hooks``. The main point being that a commit message should be formatted in the following way: .... category/port: Summary. Description of why the changes where made. PR: 12345 .... [IMPORTANT] ==== The first line is the subject of the commit, it contains what port was changed, and a summary of the commit. It should contain 50 characters or less. A blank line should separate it from the rest of the commit message. The rest of the commit message should be wrapped at the 72 characters boundary. Another blank line should be added if there are any metadata fields, so that they are easily distinguishable from the commit message. ==== ==== Managing Local Changes This section addresses tracking local changes. If you have no local changes, you can stop reading now (it is the last section and OK to skip). One item that is important for all of them: all changes are local until pushed. Unlike Subversion, Git uses a distributed model. For users, for most things, there is very little difference. However, if you have local changes, you can use the same tool to manage them as you use to pull in changes from FreeBSD. All changes that you have not pushed are local and can easily be modified (git rebase, discussed below does this). ===== Keeping local changes The simplest way to keep local changes (especially trivial ones) is to use `git stash`. In its simplest form, you use `git stash` to record the changes (which pushes them onto the stash stack). Most people use this to save changes before updating the tree as described above. They then use `git stash apply` to re-apply them to the tree. The stash is a stack of changes that can be examined with `git stash list`. The git-stash man page (https://git-scm.com/docs/git-stash) has all the details. This method is suitable when you have tiny tweaks to the tree. When you have anything non trivial, you'll likely be better off keeping a local branch and rebasing. Stashing is also integrated with the `git pull` command: just add `--autostash` to the command line. ===== Keeping a local branch [[keeping_a_local_branch]] It is much easier to keep a local branch with Git than Subversion. In Subversion you need to merge the commit, and resolve the conflicts. This is manageable, but can lead to a convoluted history that's hard to upstream should that ever be necessary, or hard to replicate if you need to do so. Git also allows one to merge, along with the same problems. That's one way to manage the branch, but it's the least flexible. In addition to merging, Git supports the concept of 'rebasing' which avoids these issues. The `git rebase` command replays all the commits of a branch at a newer location on the parent branch. We will cover the most common scenarios that arise using it. ====== Create a branch Let's say you want to make a change to FreeBSD's ls command to never, ever do color. There are many reasons to do this, but this example will use that as a baseline. The FreeBSD ls command changes from time to time, and you'll need to cope with those changes. Fortunately, with Git rebase it usually is automatic. [source,shell] .... % cd src % git checkout main % git checkout -b no-color-ls % cd bin/ls % vi ls.c # hack the changes in % git diff # check the changes diff --git a/bin/ls/ls.c b/bin/ls/ls.c index 7378268867ef..cfc3f4342531 100644 --- a/bin/ls/ls.c +++ b/bin/ls/ls.c @@ -66,6 +66,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#undef COLORLS #ifdef COLORLS #include #include % # these look good, make the commit... % git commit ls.c .... The commit will pop you into an editor to describe what you've done. Once you enter that, you have your own **local** branch in the Git repo. Build and install it like you normally would, following the directions in the handbook. Git differs from other version control systems in that you have to tell it explicitly which files to commit. I have opted to do it on the commit command line, but you can also do it with `git add` which many of the more in depth tutorials cover. ====== Time to update When it is time to bring in a new version, it is almost the same as w/o the branches. You would update like you would above, but there is one extra command before you update, and one after. The following assumes you are starting with an unmodified tree. It is important to start rebasing operations with a clean tree (Git requires this). [source,shell] .... % git checkout main % git pull --ff-only % git rebase -i main no-color-ls .... This will bring up an editor that lists all the commits in it. For this example, do not change it at all. This is typically what you are doing while updating the baseline (though you also use the Git rebase command to curate the commits you have in the branch). Once you are done with the above, you have to move the commits to ls.c forward from the old version of FreeBSD to the newer one. Sometimes there are merge conflicts. That is OK. Do not panic. Instead, handle them the same as any other merge conflicts. To keep it simple, I will just describe a common issue that may arise. A pointer to a more complete treatment can be found at the end of this section. Let's say the includes changes upstream in a radical shift to terminfo as well as a name change for the option. When you updated, you might see something like this: [source,shell] .... Auto-merging bin/ls/ls.c CONFLICT (content): Merge conflict in bin/ls/ls.c error: could not apply 646e0f9cda11... no color ls Resolve all conflicts manually, mark them as resolved with "git add/rm ", then run "git rebase --continue". You can instead skip this commit: run "git rebase --skip". To abort and get back to the state before "git rebase", run "git rebase --abort". Could not apply 646e0f9cda11... no color ls .... which looks scary. If you bring up an editor, you will see it is a typical 3-way merge conflict resolution that you may be familiar with from other source code systems (the rest of ls.c has been omitted): [source,shell] <<<<<<< HEAD #ifdef COLORLS_NEW #include ======= #undef COLORLS #ifdef COLORLS #include >>>>>>> 646e0f9cda11... no color ls .... The new code is first, and your code is second. The right fix here is to just add a #undef COLORLS_NEW before #ifdef and then delete the old changes: [source,shell] .... #undef COLORLS_NEW #ifdef COLORLS_NEW #include .... save the file. The rebase was interrupted, so you have to complete it: [source,shell] .... % git add ls.c % git rebase --continue .... which tells Git that ls.c has been fixed and to continue the rebase operation. Since there was a conflict, you will get kicked into the editor to update the commit message if necessary. If the commit message is still accurate, just exit the editor. If you get stuck during the rebase, do not panic. git rebase --abort will take you back to a clean slate. It is important, though, to start with an unmodified tree. An aside: The above mentioned `git reflog` comes in handy here, as it will have a list of all the (intermediate) commits that you can view or inspect or cherry-pick. For more on this topic, https://www.freecodecamp.org/news/the-ultimate-guide-to-git-merge-and-git-rebase/ provides a rather extensive treatment. It is a good resource for issues that arise occasionally but are too obscure for this guide. ===== Switching to a Different FreeBSD Branch If you wish to shift from stable/12 to the current branch. If you have a deep clone, the following will suffice: [source,shell] .... % git checkout main % # build and install here... .... If you have a local branch, though, there are one or two caveats. First, rebase will rewrite history, so you will likely want to do something to save it. Second, jumping branches tends to cause more conflicts. If we pretend the example above was relative to stable/12, then to move to `main`, I'd suggest the following: [source,shell] .... % git checkout no-color-ls % git checkout -b no-color-ls-stable-12 # create another name for this branch % git rebase -i stable/12 no-color-ls --onto main .... What the above does is checkout no-color-ls. Then create a new name for it (no-color-ls-stable-12) in case you need to get back to it. Then you rebase onto the `main` branch. This will find all the commits to the current no-color-ls branch (back to where it meets up with the stable/12 branch) and then it will replay them onto the `main` branch creating a new no-color-ls branch there (which is why I had you create a place holder name). [[mfc-with-git]] === MFC (Merge From Current) Procedures ==== Summary MFC workflow can be summarized as `git cherry-pick -x` plus `git commit --amend` to adjust the commit message. For multiple commits, use `git rebase -i` to squash them together and edit the commit message. ==== Single commit MFC [source,shell] .... % git checkout stable/X % git cherry-pick -x $HASH --edit .... For MFC commits, for example a vendor import, you would need to specify one parent for cherry-pick purposes. Normally, that would be the "first parent" of the branch you are cherry-picking from, so: [source,shell] .... % git checkout stable/X % git cherry-pick -x $HASH -m 1 --edit .... If things go wrong, you'll either need to abort the cherry-pick with `git cherry-pick --abort` or fix it up and do a `git cherry-pick --continue`. Once the cherry-pick is finished, push with `git push`. If you get an error due to losing the commit race, use `git pull --rebase` and try to push again. ==== MFC to RELENG branch MFCs to branches that require approval require a bit more care. The process is the same for either a typical merge or an exceptional direct commit. * Merge or direct commit to the appropriate `stable/X` branch first before merging to the `releng/X.Y` branch. * Use the hash that's in the `stable/X` branch for the MFC to `releng/X.Y` branch. * Leave both "cherry picked from" lines in the commit message. * Be sure to add the `Approved by:` line when you are in the editor. [source,shell] .... % git checkout releng/13.0 % git cherry-pick -x $HASH --edit .... If you forget to to add the `Approved by:` line, you can do a `git commit --amend` to edit the commit message before you push the change. ==== Multiple commit MFC [source,shell] .... % git checkout -b tmp-branch stable/X % for h in $HASH_LIST; do git cherry-pick -x $h; done % git rebase -i stable/X # mark each of the commits after the first as 'squash' # Update the commit message to reflect all elements of commit, if necessary. # Be sure to retain the "cherry picked from" lines. % git push freebsd HEAD:stable/X .... If the push fails due to losing the commit race, rebase and try again: [source,shell] .... % git checkout stable/X % git pull % git checkout tmp-branch % git rebase stable/X % git push freebsd HEAD:stable/X .... Once the MFC is complete, you can delete the temporary branch: [source,shell] .... % git checkout stable/X % git branch -d tmp-branch .... ==== MFC a vendor import Vendor imports are the only thing in the tree that creates a merge commit in the `main` branch. Cherry picking merge commits into stable/XX presents an additional difficulty because there are two parents for a merge commit. Generally, you'll want the first parent's diff since that's the diff to `main` (though there may be some exceptions). [source,shell] .... % git cherry-pick -x -m 1 $HASH .... is typically what you want. This will tell cherry-pick to apply the correct diff. There are some, hopefully, rare cases where it's possible that the `main` branch was merged backwards by the conversion script. Should that be the case (and we've not found any yet), you'd change the above to `-m 2` to pickup the proper parent. Just do: [source,shell] .... % git cherry-pick --abort % git cherry-pick -x -m 2 $HASH .... to do that. The `--abort` will cleanup the failed first attempt. ==== Redoing a MFC If you do a MFC, and it goes horribly wrong and you want to start over, then the easiest way is to use `git reset --hard` like so: [source,shell] .... % git reset --hard freebsd/stable/12 .... though if you have some revs you want to keep, and others you don't, using `git rebase -i` is better. ==== Considerations when MFCing When committing source commits to stable and releng branches, we have the following goals: * Clearly mark direct commits distinct from commits that land a change from another branch. * Avoid introducing known breakage into stable and releng branches. * Allow developers to determine which changes have or have not been landed from one branch to another. With Subversion, we used the following practices to achieve these goals: * Using `MFC` and `MFS` tags to mark commits that merged changes from another branch. * Squashing fixup commits into the main commit when merging a change. * Recording mergeinfo so that `svn mergeinfo --show-revs` worked. With Git, we will need to use different strategies to achieve the same goals. This document aims to define best practices when merging source commits using Git that achieve these goals. In general, we aim to use Git's native support to achieve these goals rather than enforcing practices built on Subversion's model. One general note: due to technical differences with Git, we will not be using Git "merge commits" (created via `git merge`) in stable or releng branches. Instead, when this document refers to "merge commits", it means a commit originally made to `main` that is replicated or "landed" to a stable branch, or a commit from a stable branch that is replicated to a releng branch with some variation of `git cherry-pick`. ==== Finding Eligible Hashes to MFC Git provides some built-in support for this via the `git cherry` and `git log --cherry` commands. These commands compare the raw diffs of commits (but not other metadata such as log messages) to determine if two commits are identical. This works well when each commit from `main` is landed as a single commit to a stable branch, but it falls over if multiple commits from `main` are squashed together as a single commit to a stable branch. The project makes extensive use of `git cherry-pick -x` with all lines preserved to work around these difficulties and is working on automated tooling to take advantage of this. ==== Commit message standards ===== Marking MFCs The project has adopted the following practice for marking MFCs: * Use the `-x` flag with `git cherry-pick`. This adds a line to the commit message that includes the hash of the original commit when merging. Since it is added by Git directly, committers do not have to manually edit the commit log when merging. When merging multiple commits, keep all the "cherry picked from" lines. ===== Trim Metadata? One area that was not clearly documented with Subversion (or even CVS) is how to format metadata in log messages for MFC commits. Should it include the metadata from the original commit unchanged, or should it be altered to reflect information about the MFC commit itself? Historical practice has varied, though some of the variance is by field. For example, MFCs that are relevant to a PR generally include the PR field in the MFC so that MFC commits are included in the bug tracker's audit trail. Other fields are less clear. For example, Phabricator shows the diff of the last commit tagged to a review, so including Phabricator URLs replaces the main commit with the landed commits. The list of reviewers is also not clear. If a reviewer has approved a change to `main`, does that mean they have approved the MFC commit? Is that true if it's identical code only, or with merely trivial rework? It's clearly not true for more extensive reworks. Even for identical code what if the commit doesn't conflict but introduces an ABI change? A reviewer may have ok'd a commit for `main` due to the ABI breakage but may not approve of merging the same commit as-is. One will have to use one's best judgment until clear guidelines can be agreed upon. For MFCs regulated by re@, new metadata fields are added, such as the Approved by tag for approved commits. This new metadata will have to be added via `git commit --amend` or similar after the original commit has been reviewed and approved. We may also want to reserve some metadata fields in MFC commits such as Phabricator URLs for use by re@ in the future. Preserving existing metadata provides a very simple workflow. Developers use `git cherry-pick -x` without having to edit the log message. If instead we choose to adjust metadata in MFCs, developers will have to edit log messages explicitly via the use of `git cherry-pick --edit` or `git commit --amend`. However, as compared to svn, at least the existing commit message can be pre-populated and metadata fields can be added or removed without having to re-enter the entire commit message. The bottom line is that developers will likely need to curate their commit message for MFCs that are non-trivial. [[vendor-import-git]] === Vendor Imports with Git This section describes the vendor import procedure with Git in detail. ==== Branch naming convention All vendor branches and tags start with `vendor/`. These branches and tags are visible by default. [NOTE] ==== This chapter follows the convention that the `freebsd` origin is the origin name for the official FreeBSD Git repository. If you use a different convention, replace `freebsd` with the name you use instead in the examples below. ==== We will explore an example for updating NetBSD's mtree that is in our tree. The vendor branch for this is `vendor/NetBSD/mtree`. ==== Updating an old vendor import The vendor trees usually have only the subset of the third-party software that is appropriate to FreeBSD. These trees are usually tiny in comparison to the FreeBSD tree. Git worktrees are thus quite small and fast and the preferred method to use. Make sure that whatever directory you choose below (the `../mtree`) does not currently exist. [source,shell] .... % git worktree add ../mtree vendor/NetBSD/mtree .... ==== Update the Sources in the Vendor Branch Prepare a full, clean tree of the vendor sources. Import everything but merge only what is needed. This example assumes the NetBSD source is checked out from their GitHub mirror in `~/git/NetBSD`. Note that "upstream" might have added or removed files, so we want to make sure deletions are propagated as well. package:net/rsync[] is commonly installed, so I'll use that. [source,shell] .... % cd ../mtree % rsync -va --del --exclude=".git" ~/git/NetBSD/usr.sbin/mtree/ . % git add -A % git status ... % git diff --staged ... % git commit -m "Vendor import of NetBSD's mtree at 2020-12-11" [vendor/NetBSD/mtree 8e7aa25fcf1] Vendor import of NetBSD's mtree at 2020-12-11 7 files changed, 114 insertions(+), 82 deletions(-) % git tag -a vendor/NetBSD/mtree/20201211 .... Note: I run the `git diff` and `git status` commands to make sure nothing weird was present. Also I used `-m` to illustrate, but you should compose a proper message in an editor (using a commit message template). It is also important to create an annotated tag using `git tag -a`, otherwise the push will be rejected. Only annotated tags are allowed to be pushed. The annotated tag gives you a chance to enter a commit message. Enter the version you are importing, along with any salient new features or fixes in that version. ==== Updating the FreeBSD Copy At this point you can push the import to `vendor` into our repo. [source,shell] .... % git push --follow-tags freebsd vendor/NetBSD/mtree .... `--follow-tags` tells `git push` to also push tags associated with the locally committed revision. ==== Updating the FreeBSD source tree Now you need to update the mtree in FreeBSD. The sources live in `contrib/mtree` since it is upstream software. [source,shell] .... % cd ../src % git subtree merge -P contrib/mtree vendor/NetBSD/mtree .... This would generate a subtree merge commit of `contrib/mtree` against the local `vendor/NetBSD/mtree` branch. If there were conflicts, you would need to fix them before committing. Include details about the changes being merged in the merge commit message. ==== Rebasing your change against latest FreeBSD source tree Because the current policy recommends against using merges, if the upstream FreeBSD `main` moved forward before you get a chance to push, you would have to redo the merge. Regular `git rebase` or `git pull --rebase` doesn't know how to rebase a merge commit **as a merge commit**, so instead of that you would have to recreate the commit. The following steps should be taken to easily recreate the merge commit as if `git rebase --merge-commits` worked properly: * cd to the top of the repo * Create a side branch `XXX` with the **contents** of the merged tree. * Update this side branch `XXX` to be merged and up-to-date with FreeBSD's `main` branch. ** In the worst case scenario, you would still have to resolve merge conflicts, if there was any, but this should be really rare. ** Resolve conflicts, and collapse multiple commits down to 1 if need be (without conflicts, there's no collapse needed) * checkout `main` * create a branch `YYY` (allows for easier unwinding if things go wrong) * Re-do the subtree merge * Instead of resolving any conflicts from the subtree merge, checkout the contents of XXX on top of it. ** The trailing `.` is important, as is being at the top level of the repo. ** Rather than switching branches to XXX, it splats the contents of XXX on top of the repo * Commit the results with the prior commit message (the example assumes there's only one merge on the XXX branch). * Make sure the branches are the same. * Do whatever review you need, including having others check it out if you think that's needed. * Push the commit, if you 'lost the race' again, just redo these steps again (see below for a recipe) * Delete the branches once the commit is upstream. They are throw-a-way. The commands one would use, following the above example of mtree, would be like so (the `#` starts a comment to help link commands to descriptions above): [source,shell] .... % cd ../src # CD to top of tree % git checkout -b XXX # create new throw-away XXX branch for merge % git fetch freebsd # Get changes from upstream from upstream % git merge freebsd/main # Merge the changes and resolve conflicts % git checkout -b YYY freebsd/main # Create new throw-away YYY branch for redo % git subtree merge -P contrib/mtree vendor/NetBSD/mtree # Redo subtree merge % git checkout XXX . # XXX branch has the conflict resolution % git commit -c XXX~1 # -c reuses the commit message from commit before rebase % git diff XXX YYY # Should be empty % git show YYY # Should only have changes you want, and be a merge commit from vendor branch .... Note: if things go wrong with the commit, you can reset the `YYY` branch by reissuing the checkout command that created it with -B to start over: [source,shell] .... % git checkout -B YYY freebsd/main # Create new throw-away YYY branch if starting over is just going to be easier .... ==== Pushing the changes Once you think you have a set of changes that are good, you can push it to a fork off GitHub or GitLab for others to review. One nice thing about Git is that it allows you to publish rough drafts of your work for others to review. While Phabricator is good for content review, publishing the updated vendor branch and merge commits lets others check the details as they will eventually appear in the repository. After review, when you are sure it is a good change, you can push it to the FreeBSD repo: [source,shell] .... % git push freebsd YYY:main # put the commit on upstream's 'main' branch % git branch -D XXX # Throw away the throw-a-way branches. % git branch -D YYY .... Note: I used `XXX` and `YYY` to make it obvious they are terrible names and should not leave your machine. If you use such names for other work, then you'll need to pick different names, or risk losing the other work. There is nothing magic about these names. Upstream will not allow you to push them, but never the less, please pay attention to the exact commands above. Some commands use syntax that differs only slightly from typical uses and that different behavior is critical to this recipe working. ==== How to redo things if need be If you've tried to do the push in the previous section and it fails, then you should do the following to 'redo' things. This sequence keeps the commit with the commit message always at XXX~1 to make committing easier. [source,shell] .... % git checkout -B XXX YYY # recreate that throw-away-branch XXX and switch to it % git merge freebsd/main # Merge the changes and resolve conflicts % git checkout -B YYY freebsd/main # Recreate new throw-away YYY branch for redo % git subtree merge -P contrib/mtree vendor/NetBSD/mtree # Redo subtree merge % git checkout XXX . # XXX branch has the conflict resolution % git commit -c XXX~1 # -c reuses the commit message from commit before rebase .... Then go check it out as above and push as above when ready. === Creating a new vendor branch There are a number of ways to create a new vendor branch. The recommended way is to create a new repository and then merge that with FreeBSD. If one is importing `glorbnitz` into the FreeBSD tree, release 3.1415. For the sake of simplicity, we will not trim this release. It is a simple user command that puts the nitz device into different magical glorb states and is small enough trimming will not save much. ==== Create the repo [source,shell] .... % cd /some/where % mkdir glorbnitz % cd glorbnitz % git init % git checkout -b vendor/glorbnitz .... At this point, you have a new repo, where all new commits will go on the `vendor/glorbnitz` branch. Git experts can also do this right in their FreeBSD clone, using `git checkout --orphan vendor/glorbnitz` if they are more comfortable with that. ==== Copy the sources in Since this is a new import, you can just cp the sources in, or use tar or even rsync as shown above. And we will add everything, assuming no dot files. [source,shell] .... % cp -r ~/glorbnitz/* . % git add * .... At this point, you should have a pristine copy of glorbnitz ready to commit. [source,shell] .... % git commit -m "Import GlorbNitz frobnosticator revision 3.1415" .... As above, I used `-m` for simplicity, but you should likely create a commit message that explains what a Glorb is and why you'd use a Nitz to get it. Not everybody will know so, for your actual commit, you should follow the <> section instead of emulating the brief style used here. ==== Now import it into our repository Now you need to import the branch into our repository. [source,shell] .... % cd /path/to/freebsd/repo/src % git remote add glorbnitz /some/where/glorbnitz % git fetch glorbnitz vendor/glorbnitz .... Note the vendor/glorbnitz branch is in the repo. At this point the `/some/where/glorbnitz` can be deleted, if you like. It was only a means to an end. // perhaps the real treasure was the friends it made along the way... ==== Tag and push Steps from here on out are much the same as they are in the case of updating a vendor branch, though without the updating the vendor branch step. [source,shell] .... % git worktree add ../glorbnitz vendor/glorbnitz % cd ../glorbnitz % git tag --annotate vendor/glorbnitz/3.1415 # Make sure the commit is good with "git show" % git push --follow-tags freebsd vendor/glorbnitz .... By 'good' we mean: . All the right files are present . None of the wrong files are present . The vendor branch points at something sensible . The tag looks good, and is annotated . The commit message for the tag has a quick summary of what's new since the last tag ==== Time to finally merge it into the base tree [source,shell] .... % cd ../src % git subtree add -P contrib/glorbnitz vendor/glorbnitz # Make sure the commit is good with "git show" % git commit --amend # one last sanity check on commit message % git push freebsd .... Here 'good' means: . All the right files, and none of the wrong ones, were merged into contrib/glorbnitz. . No other changes are in the tree. . The commit messages look <>. It should contain a summary of what's changed since the last merge to the FreeBSD `main` branch and any caveats. . UPDATING should be updated if there is anything of note, such as user visible changes, important upgrade concerns, etc. [NOTE] ==== This hasn't connected `glorbnitz` to the build yet. How so do that is specific to the software being imported and is beyond the scope of this tutorial. ==== ===== Keeping current So, time passes. It's time now to update the tree for the latest changes upstream. When you checkout `main` make sure that you have no diffs. It's a lot easier to commit those to a branch (or use `git stash`) before doing the following. If you are used to `git pull`, we strongly recommend using the `--ff-only` option, and further setting it as the default option. Alternatively, `git pull --rebase` is useful if you have changes staged in the `main` branch. [source,shell] .... % git config --global pull.ff only .... You may need to omit the --global if you want this setting to apply to only this repository. [source,shell] .... % cd freebsd-src % git checkout main % git pull (--ff-only|--rebase) .... There is a common trap, that the combination command `git pull` will try to perform a merge, which would sometimes creates a merge commit that didn't exist before. This can be harder to recover from. The longer form is also recommended. [source,shell] .... % cd freebsd-src % git checkout main % git fetch freebsd % git merge --ff-only freebsd/main .... These commands reset your tree to the `main` branch, and then update it from where you pulled the tree from originally. It's important to switch to `main` before doing this so it moves forward. Now, it's time to move the changes forward: [source,shell] .... % git rebase -i main working .... This will bring up an interactive screen to change the defaults. For now, just exit the editor. Everything should just apply. If not, then you'll need to resolve the diffs. https://docs.github.com/en/free-pro-team@latest/github/using-git/resolving-merge-conflicts-after-a-git-rebase[This github document] can help you navigate this process. [[git-push-upstream]] ===== Time to push changes upstream First, ensure that the push URL is properly configured for the upstream repository. [source,shell] .... % git remote set-url --push freebsd ssh://git@gitrepo.freebsd.org/src.git .... Then, verify that user name and email are configured right. We require that they exactly match the passwd entry in FreeBSD cluster. Use [source,shell] .... freefall% gen-gitconfig.sh .... on freefall.freebsd.org to get a recipe that you can use directly, assuming /usr/local/bin is in the PATH. The below command merges the `working` branch into the upstream `main` branch. It's important that you curate your changes to be just like you want them in the FreeBSD source repo before doing this. This syntax pushes the `working` branch to `main`, moving the `main` branch forward. You will only be able to do this if this results in a linear change to `main` (e.g. no merges). [source,shell] .... % git push freebsd working:main .... If your push is rejected due to losing a commit race, rebase your branch before trying again: [source,shell] .... % git checkout working % git fetch freebsd % git rebase freebsd/main % git push freebsd working:main .... [[git-push-upstream-alt]] ===== Time to push changes upstream (alternative) Some people find it easier to merge their changes to their local `main` before pushing to the remote repository. Also, `git arc stage` moves changes from a branch to the local `main` when you need to do a subset of a branch. The instructions are similar to the prior section: [source,shell] .... % git checkout main % git merge --ff-only `working` % git push freebsd .... If you lose the race, then try again with [source,shell] .... % git pull --rebase % git push freebsd .... These commands will fetch the most recent `freebsd/main` and then rebase the local `main` changes on top of that, which is what you want when you lose the commit race. Note: merging vendor branch commits will not work with this technique. ===== Finding the Subversion Revision You'll need to make sure that you've fetched the notes (see the <> for details). Once you have these, notes will show up in the git log command like so: [source,shell] .... % git log .... If you have a specific version in mind, you can use this construct: [source,shell] .... % git log --grep revision=XXXX .... to find the specific revision. The hex number after 'commit' is the hash you can use to refer to this commit. [[git-faq]] === Git FAQ This section provides a number of targeted answers to questions that are likely to come up often for users and developers. [NOTE] ==== We use the common convention of having the origin for the FreeBSD repository being 'freebsd' rather than the default 'origin' to allow people to use that for their own development and to minimize "whoops" pushes to the wrong repository. ==== ==== Users ===== How do I track -current and -stable with only one copy of the repository? **Q:** Although disk space is not a huge issue, it's more efficient to use only one copy of the repository. With SVN mirroring, I could checkout multiple trees from the same repository. How do I do this with Git? **A:** You can use Git worktrees. There's a number of ways to do this, but the simplest way is to use a clone to track -current, and a worktree to track stable releases. While using a 'bare repository' has been put forward as a way to cope, it's more complicated and will not be documented here. First, you need to clone the FreeBSD repository, shown here cloning into `freebsd-current` to reduce confusion. $URL is whatever mirror works best for you: [source,shell] .... % git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' $URL freebsd-current .... then once that's cloned, you can simply create a worktree from it: [source,shell] .... % cd freebsd-current % git worktree add ../freebsd-stable-12 stable/12 .... this will checkout `stable/12` into a directory named `freebsd-stable-12` that's a peer to the `freebsd-current` directory. Once created, it's updated very similarly to how you might expect: [source,shell] .... % cd freebsd-current % git checkout main % git pull --ff-only # changes from upstream now local and current tree updated % cd ../freebsd-stable-12 % git merge --ff-only freebsd/stable/12 # now your stable/12 is up to date too .... I recommend using `--ff-only` because it's safer and you avoid accidentally getting into a 'merge nightmare' where you have an extra change in your tree, forcing a complicated merge rather than a simple one. Here's https://adventurist.me/posts/00296[a good writeup] that goes into more detail. ==== Developers ===== Ooops! I committed to `main`, instead of another branch. **Q:** From time to time, I goof up and mistakenly commit to the `main` branch. What do I do? **A:** First, don't panic. Second, don't push. In fact, you can fix almost anything if you haven't pushed. All the answers in this section assume no push has happened. The following answer assumes you committed to `main` and want to create a branch called `issue`: [source,shell] .... % git branch issue # Create the 'issue' branch % git reset --hard freebsd/main # Reset 'main' back to the official tip % git checkout issue # Back to where you were .... ===== Ooops! I committed something to the wrong branch! **Q:** I was working on feature on the `wilma` branch, but accidentally committed a change relevant to the `fred` branch in 'wilma'. What do I do? **A:** The answer is similar to the previous one, but with cherry picking. This assumes there's only one commit on wilma, but will generalize to more complicated situations. It also assumes that it's the last commit on wilma (hence using wilma in the `git cherry-pick` command), but that too can be generalized. [source,shell] .... # We're on branch wilma % git checkout fred # move to fred branch % git cherry-pick wilma # copy the misplaced commit % git checkout wilma # go back to wilma branch % git reset --hard HEAD^ # move what wilma refers to back 1 commit .... Git experts would first rewind the wilma branch by 1 commit, switch over to fred and then use `git reflog` to see what that 1 deleted commit was and cherry-pick it over. **Q:** But what if I want to commit a few changes to `main`, but keep the rest in `wilma` for some reason? **A:** The same technique above also works if you are wanting to 'land' parts of the branch you are working on into `main` before the rest of the branch is ready (say you noticed an unrelated typo, or fixed an incidental bug). You can cherry pick those changes into `main`, then push to the parent repository. Once you've done that, cleanup couldn't be simpler: just `git rebase -i`. Git will notice you've done this and skip the common changes automatically (even if you had to change the commit message or tweak the commit slightly). There's no need to switch back to wilma to adjust it: just rebase! **Q:** I want to split off some changes from branch `wilma` into branch `fred` **A:** The more general answer would be the same as the previous. You'd checkout/create the `fred` branch, cherry pick the changes you want from `wilma` one at a time, then rebase `wilma` to remove those changes you cherry picked. `git rebase -i main wilma` will toss you into an editor, and remove the `pick` lines that correspond to the commits you copied to `fred`. If all goes well, and there are no conflicts, you're done. If not, you'll need to resolve the conflicts as you go. The other way to do this would be to checkout `wilma` and then create the branch `fred` to point to the same point in the tree. You can then `git rebase -i` both these branches, selecting the changes you want in `fred` or `wilma` by retaining the pick likes, and deleting the rest from the editor. Some people would create a tag/branch called `pre-split` before starting in case something goes wrong in the split. You can undo it with the following sequence: [source,shell] .... % git checkout pre-split # Go back % git branch -D fred # delete the fred branch % git checkout -B wilma # reset the wilma branch % git branch -d pre-split # Pretend it didn't happen .... The last step is optional. If you are going to try again to split, you'd omit it. **Q:** But I did things as I read along and didn't see your advice at the end to create a branch, and now `fred` and `wilma` are all screwed up. How do I find what `wilma` was before I started. I don't know how many times I moved things around. **A:** All is not lost. You can figure out it, so long as it hasn't been too long, or too many commits (hundreds). So I created a wilma branch and committed a couple of things to it, then decided I wanted to split it into fred and wilma. Nothing weird happened when I did that, but let's say it did. The way to look at what you've done is with the `git reflog`: [source,shell] .... % git reflog 6ff9c25 (HEAD -> wilma) HEAD@{0}: rebase -i (finish): returning to refs/heads/wilma 6ff9c25 (HEAD -> wilma) HEAD@{1}: rebase -i (start): checkout main 869cbd3 HEAD@{2}: rebase -i (start): checkout wilma a6a5094 (fred) HEAD@{3}: rebase -i (finish): returning to refs/heads/fred a6a5094 (fred) HEAD@{4}: rebase -i (pick): Encourage contributions 1ccd109 (freebsd/main, main) HEAD@{5}: rebase -i (start): checkout main 869cbd3 HEAD@{6}: rebase -i (start): checkout fred 869cbd3 HEAD@{7}: checkout: moving from wilma to fred 869cbd3 HEAD@{8}: commit: Encourage contributions ... % .... Here we see the changes I've made. You can use it to figure out where things went wrong. I'll just point out a few things here. The first one is that HEAD@{X} is a 'commitish' thing, so you can use that as an argument to a command. Although if that command commits anything to the repository, the X numbers change. You can also use the hash (first column). Next, 'Encourage contributions' was the last commit I made to `wilma` before I decided to split things up. You can also see the same hash is there when I created the `fred` branch to do that. I started by rebasing `fred` and you see the 'start', each step, and the 'finish' for that process. While we don't need it here, you can figure out exactly what happened. Fortunately, to fix this, you can follow the prior answer's steps, but with the hash `869cbd3` instead of `pre-split`. While that seems a bit verbose, it's easy to remember since you're doing one thing at a time. You can also stack: [source,shell] .... % git checkout -B wilma 869cbd3 % git branch -D fred .... and you are ready to try again. The `checkout -B` with the hash combines checking out and creating a branch for it. The `-B` instead of `-b` forces the movement of a pre-existing branch. Either way works, which is what's great (and awful) about Git. One reason I tend to use `git checkout -B xxxx hash` instead of checking out the hash, and then creating / moving the branch is purely to avoid the slightly distressing message about detached heads: [source,shell] .... % git checkout 869cbd3 M faq.md Note: checking out '869cbd3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 869cbd3 Encourage contributions % git checkout -B wilma .... this produces the same effect, but I have to read a lot more and severed heads aren't an image I like to contemplate. ===== Ooops! I did a `git pull` and it created a merge commit, what do I do? **Q:** I was on autopilot and did a `git pull` for my development tree and that created a merge commit on `main`. How do I recover? **A:** This can happen when you invoke the pull with your development branch checked out. Right after the pull, you will have the new merge commit checked out. Git supports a `HEAD^#` syntax to examine the parents of a merge commit: [source,shell] .... git log --oneline HEAD^1 # Look at the first parent's commits git log --oneline HEAD^2 # Look at the second parent's commits .... From those logs, you can easily identify which commit is your development work. Then you simply reset your branch to the corresponding `HEAD^#`: [source,shell] .... git reset --hard HEAD^2 .... **Q:** But I also need to fix my `main` branch. How do I do that? **A:** Git keeps track of the remote repository branches in a `freebsd/` namespace. To fix your `main` branch, just make it point to the remote's `main`: [source,shell] .... git branch -f main freebsd/main .... There's nothing magical about branches in Git: they are just labels on a graph that are automatically moved forward by making commits. So the above works because you're just moving a label. There's no metadata about the branch that needs to be preserved due to this. ===== Mixing and matching branches **Q:** So I have two branches `worker` and `async` that I'd like to combine into one branch called `feature` while maintaining the commits in both. **A:** This is a job for cherry pick. [source,shell] .... % git checkout worker % git checkout -b feature # create a new branch % git cherry-pick main..async # bring in the changes .... You now have a new branch called `feature`. This branch combines commits from both branches. You can further curate it with `git rebase`. **Q:** I have a branch called `driver` and I'd like to break it up into `kernel` and `userland` so I can evolve them separately and commit each branch as it becomes ready. **A:** This takes a little bit of prep work, but `git rebase` will do the heavy lifting here. [source,shell] .... % git checkout driver # Checkout the driver % git checkout -b kernel # Create kernel branch % git checkout -b userland # Create userland branch .... Now you have two identical branches. So, it's time to separate out the commits. We'll assume first that all the commits in `driver` go into either the `kernel` or the `userland` branch, but not both. [source,shell] .... % git rebase -i main kernel .... and just include the changes you want (with a 'p' or 'pick' line) and just delete the commits you don't (this sounds scary, but if worse comes to worse, you can throw this all away and start over with the `driver` branch since you've not yet moved it). [source,shell] .... % git rebase -i main userland .... and do the same thing you did with the `kernel` branch. **Q:** Oh great! I followed the above and forgot a commit in the `kernel` branch. How do I recover? **A:** You can use the `driver` branch to find the hash of the commit is missing and cherry pick it. [source,shell] .... % git checkout kernel % git log driver % git cherry-pick $HASH .... **Q:** OK. I have the same situation as the above, but my commits are all mixed up. I need parts of one commit to go to one branch and the rest to go to the other. In fact, I have several. Your rebase method to select sounds tricky. **A:** In this situation, you'd be better off to curate the original branch to separate out the commits, and then use the above method to split the branch. So let's assume that there's just one commit with a clean tree. You can either use `git rebase` with an `edit` line, or you can use this with the commit on the tip. The steps are the same either way. The first thing we need to do is to back up one commit while leaving the changes uncommitted in the tree: [source,shell] .... % git reset HEAD^ .... Note: Do not, repeat do not, add `--hard` here since that also removes the changes from your tree. Now, if you are lucky, the change needing to be split up falls entirely along file lines. In that case you can just do the usual `git add` for the files in each group than do a `git commit`. Note: when you do this, you'll lose the commit message when you do the reset, so if you need it for some reason, you should save a copy (though `git log $HASH` can recover it). If you are not lucky, you'll need to split apart files. There's another tool to do that which you can apply one file at a time. [source,shell] .... git add -i foo/bar.c .... will step through the diffs, prompting you, one at time, whether to include or exclude the hunk. Once you're done, `git commit` and you'll have the remainder in your tree. You can run it multiple times as well, and even over multiple files (though I find it easier to do one file at a time and use the `git rebase -i` to fold the related commits together). ==== Cloning and Mirroring **Q:** I'd like to mirror the entire Git repository, how do I do that? **A:** If all you want to do is mirror, then [source,shell] .... % git clone --mirror $URL .... will do the trick. However, there are two disadvantages to this if you want to use it for anything other than a mirror you'll reclone. First, this is a 'bare repository' which has the repository database, but no checked out worktree. This is great for mirroring, but terrible for day to day work. There's a number of ways around this with `git worktree`: [source,shell] .... % git clone --mirror https://git.freebsd.org/ports.git ports.git % cd ports.git % git worktree add ../ports main % git worktree add ../quarterly branches/2020Q4 % cd ../ports .... But if you aren't using your mirror for further local clones, then it's a poor match. The second disadvantage is that Git normally rewrites the refs (branch name, tags, etc) from upstream so that your local refs can evolve independently of upstream. This means that you'll lose changes if you are committing to this repository on anything other than private project branches. **Q:** So what can I do instead? **A:** Well, you can stuff all of the upstream repository's refs into a private namespace in your local repository. Git clones everything via a 'refspec' and the default refspec is: [source,shell] .... fetch = +refs/heads/*:refs/remotes/freebsd/* .... which says just fetch the branch refs. However, the FreeBSD repository has a number of other things in it. To see those, you can add explicit refspecs for each ref namespace, or you can fetch everything. To setup your repository to do that: [source,shell] .... git config --add remote.freebsd.fetch '+refs/*:refs/freebsd/*' .... which will put everything in the upstream repository into your local repository's `refs/freebsd/` namespace. Please note, that this also grabs all the unconverted vendor branches and the number of refs associated with them is quite large. You'll need to refer to these 'refs' with their full name because they aren't in and of Git's regular namespaces. [source,shell] .... git log refs/freebsd/vendor/zlib/1.2.10 .... would look at the log for the vendor branch for zlib starting at 1.2.10. === Collaborating with others One of the keys to good software development on a project as large as FreeBSD is the ability to collaborate with others before you push your changes to the tree. -The FreeBSD project's Git repositories do not, yet, allow user-created branches to be pushed to the repository, and therefore if you wish to share your changes with others you must use another mechanism, such as a hosted GitLab or GitHub, in order to share changes in a user-generated branch. +The FreeBSD project's Git repositories do not, yet, allow user-created branches to be pushed to the repository, and therefore if you wish to share your changes with others you must use another mechanism, such as a hosted GitLab or GitHub, to share changes in a user-generated branch. The following instructions show how to set up a user-generated branch, based on the FreeBSD `main` branch, and push it to GitHub. Before you begin, make sure that your local Git repo is up to date and has the correct origins set <> [source,shell] ```` % git remote -v freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) ```` The first step is to create a fork of https://github.com/freebsd/freebsd-src[FreeBSD] on GitHub following these https://docs.github.com/en/github/getting-started-with-github/fork-a-repo[guidelines]. The destination of the fork should be your own, personal, GitHub account (gvnn3 in my case). Now add a remote on your local system that points to your fork: [source,shell] .... % git remote add github git@github.com:gvnn3/freebsd-src.git % git remote -v github git@github.com:gvnn3/freebsd-src.git (fetch) github git@github.com:gvnn3/freebsd-src.git (push) freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) .... With this in place you can create a branch <> [source,shell] .... % git checkout -b gnn-pr2001-fix .... Make whatever modifications you wish in your branch. Build, test, and once you're ready to collaborate with others it's time to push your changes into your hosted branch. Before you can push you'll have to set the appropriate upstream, as Git will tell you the first time you try to push to your +github+ remote: [source,shell] .... % git push github fatal: The current branch gnn-pr2001-fix has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream github gnn-pr2001-fix .... Setting the push as +git+ advises allows it to succeed: [source,shell] .... % git push --set-upstream github gnn-feature Enumerating objects: 20486, done. Counting objects: 100% (20486/20486), done. Delta compression using up to 8 threads Compressing objects: 100% (12202/12202), done. Writing objects: 100% (20180/20180), 56.25 MiB | 13.15 MiB/s, done. Total 20180 (delta 11316), reused 12972 (delta 7770), pack-reused 0 remote: Resolving deltas: 100% (11316/11316), completed with 247 local objects. remote: remote: Create a pull request for 'gnn-feature' on GitHub by visiting: remote: https://github.com/gvnn3/freebsd-src/pull/new/gnn-feature remote: To github.com:gvnn3/freebsd-src.git * [new branch] gnn-feature -> gnn-feature Branch 'gnn-feature' set up to track remote branch 'gnn-feature' from 'github'. .... Subsequent changes to the same branch will push correctly by default: [source,shell] .... % git push Enumerating objects: 4, done. Counting objects: 100% (4/4), done. Delta compression using up to 8 threads Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 314 bytes | 1024 bytes/s, done. Total 3 (delta 1), reused 1 (delta 0), pack-reused 0 remote: Resolving deltas: 100% (1/1), completed with 1 local object. To github.com:gvnn3/freebsd-src.git 9e5243d7b659..cf6aeb8d7dda gnn-feature -> gnn-feature .... At this point your work is now in your branch on +GitHub+ and you can share the link with other collaborators. [[github-pull-land]] === Landing a github pull request This section documents how to land a GitHub pull request that's submitted against the FreeBSD Git mirrors at GitHub. While this is not an official way to submit patches at this time, sometimes good fixes come in this way and it is easiest just to bring them into a committer's tree and have them pushed into the FreeBSD's tree from there. Similar steps can be used to pull branches from other repositories and land those. When committing pull requests from others, one should take extra care to examine all the changes to ensure they are exactly as represented. Before beginning, make sure that the local Git repo is up to date and has the correct origins set <> In addition, make sure to have the following origins: [source,shell] .... % git remote -v freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) github https://github.com/freebsd/freebsd-src (fetch) github https://github.com/freebsd/freebsd-src (fetch) .... Often pull requests are simple: requests that contain only a single commit. In this case, a streamlined approach may be used, though the approach in the prior section will also work. Here, a branch is created, the change is cherry picked, the commit message adjusted, and sanity-checked before being pushed. The branch `staging` is used in this example but it can be any name. This technique works for any number of commits in the pull request, especially when the changes apply cleanly to the FreeBSD tree. However, when there's multiple commits, especially when minor adjustments are needed, `git rebase -i` works better than `git cherry-pick`. Briefly, these commands create a branch; cherry-picks the changes from the pull request; tests it; adjusts the commit messages; and fast forward merges it back to `main`. The PR number is `$PR` below. When adjusting the message, add `Pull Request: https://github.com/freebsd-src/pull/$PR`. All pull requests committed to the FreeBSD repository should be reviewed by at least one person. This need not be the person committing it, but in that case the person committing it should trust the other reviewers competence to review the commit. Committers that do a code review of pull requests before pushing them into the repo should add a `Reviewed by:` line to the commit, because in this case it is not implicit. Add anybody that reviews and approves the commit on github to `Reviewed by:` as well. As always, care should be taken to ensure the change does what it is supposed to, and that no malicious code is present. [NOTE] ====== In addition, please check to make sure that the pull request author name is not anonymous. Github's web editing interface generates names like: [source,shell] .... Author: github-user <38923459+github-user@users.noreply.github.com> .... A polite request to the author for a better name and/or email should be made. Extra care should be taken to ensure no style issue or malicious code is introduced. ====== [source,shell] .... % git fetch github pull/$PR/head:staging % git rebase -i main staging # to move the staging branch forward, adjust commit message here % git checkout main % git pull --ff-only # to get the latest if time has passed % git checkout main % git merge --ff-only staging % git push freebsd --push-option=confirm-author .... [.procedure] ==== For complicated pull requests that have multiple commits with conflicts, follow the following outline. . checkout the pull request `git checkout github/pull/XXX` . create a branch to rebase `git checkout -b staging` . rebase the `staging` branch to the latest `main` with `git rebase -i main staging` . resolve conflicts and do whatever testing is needed . fast forward the `staging` branch into `main` as above . final sanity check of changes to make sure all is well . push to FreeBSD's Git repository. This will also work when bringing branches developed elsewhere into the local tree for committing. ==== Once finished with the pull request, close it using GitHub's web interface. It is worth noting that if your `github` origin uses `https://`, the only step you'll need a GitHub account for is closing the pull request. [[vcs-history]] == Version Control History The project has moved to <>. The FreeBSD source repository switched from CVS to Subversion on May 31st, 2008. The first real SVN commit is __r179447__. The source repository switched from Subversion to Git on December 23rd, 2020. The last real svn commit is __r368820__. The first real git commit hash is __5ef5f51d2bef80b0ede9b10ad5b0e9440b60518c__. The FreeBSD `doc/www` repository switched from CVS to Subversion on May 19th, 2012. The first real SVN commit is __r38821__. The documentation repository switched from Subversion to Git on December 8th, 2020. The last SVN commit is __r54737__. The first real git commit hash is __3be01a475855e7511ad755b2defd2e0da5d58bbe__. The FreeBSD `ports` repository switched from CVS to Subversion on July 14th, 2012. The first real SVN commit is __r300894__. The ports repository switched from Subversion to Git on April 6, 2021. The last SVN commit is __r569609__ The first real git commit hash is __ed8d3eda309dd863fb66e04bccaa513eee255cbf__. [[conventions]] == Setup, Conventions, and Traditions There are a number of things to do as a new developer. The first set of steps is specific to committers only. These steps must be done by a mentor for those who are not committers. [[conventions-committers]] === For New Committers Those who have been given commit rights to the FreeBSD repositories must follow these steps. * Get mentor approval before committing each of these changes! * All [.filename]#src# commits go to FreeBSD-CURRENT first before being merged to FreeBSD-STABLE. The FreeBSD-STABLE branch must maintain ABI and API compatibility with earlier versions of that branch. Do not merge changes that break this compatibility. [[commit-steps]] [.procedure] ==== *Steps for New Committers* . Add an Author Entity + [.filename]#doc/shared/authors.adoc# - Add an author entity. Later steps depend on this entity, and missing this step will cause the [.filename]#doc/# build to fail. This is a relatively easy task, but remains a good first test of version control skills. . Update the List of Developers and Contributors + [.filename]#doc/shared/contrib-committers.adoc# - Add an entry, which will then appear in the "Developers" section of the extref:{contributors}[Contributors List, staff-committers]. Entries are sorted by last name. + [.filename]#doc/shared/contrib-additional.adoc# - _Remove_ the entry. Entries are sorted by first name. . Add a News Item + [.filename]#doc/website/data/en/news/news.toml# - Add an entry. Look for the other entries that announce new committers and follow the format. Use the date from the commit bit approval email. . Add a PGP Key + `{des}` has written a shell script ([.filename]#doc/documentation/tools/addkey.sh#) to make this easier. See the https://cgit.freebsd.org/doc/plain/documentation/static/pgpkeys/README[README] file for more information. + Use [.filename]#doc/documentation/tools/checkkey.sh# to verify that keys meet minimal best-practices standards. + After adding and checking a key, add both updated files to source control and then commit them. Entries in this file are sorted by last name. + [NOTE] ====== It is very important to have a current PGP/GnuPG key in the repository. The key may be required for positive identification of a committer. For example, the `{admins}` might need it for account recovery. A complete keyring of `FreeBSD.org` users is available for download from link:https://docs.FreeBSD.org/pgpkeys/pgpkeys.txt[https://docs.FreeBSD.org/pgpkeys/pgpkeys.txt]. ====== . Update Mentor and Mentee Information + [.filename]#src/share/misc/committers-.dot# - Add an entry to the current committers section, where _repository_ is `doc`, `ports`, or `src`, depending on the commit privileges granted. + Add an entry for each additional mentor/mentee relationship in the bottom section. . Generate a Kerberos Password + See <> to generate or set a Kerberos account for use with other FreeBSD services like the link:https://bugs.freebsd.org/bugzilla/[bug-tracking database] (you get a bug-tracking account as part of that step). . Optional: Enable Wiki Account + link:https://wiki.freebsd.org[FreeBSD Wiki] Account - A wiki account allows sharing projects and ideas. Those who do not yet have an account can follow instructions on the link:https://wiki.freebsd.org/Wiki/About[Wiki/About page] to obtain one. Contact mailto:wiki-admin@FreeBSD.org[wiki-admin@FreeBSD.org] if you need help with your Wiki account. . Optional: Update Wiki Information + Wiki Information - After gaining access to the wiki, some people add entries to the https://wiki.freebsd.org/HowWeGotHere[How We Got Here], https://wiki.freebsd.org/IRC/Nicknames[IRC Nicks], https://wiki.freebsd.org/Community/Dogs[Dogs of FreeBSD], and or https://wiki.freebsd.org/Community/Cats[Cats of FreeBSD] pages. . Optional: Update Ports with Personal Information + [.filename]#ports/astro/xearth/files/freebsd.committers.markers# and [.filename]#src/usr.bin/calendar/calendars/calendar.freebsd# - Some people add entries for themselves to these files to show where they are located or the date of their birthday. . Optional: Prevent Duplicate Mailings + Subscribers to {dev-commits-doc-all}, {dev-commits-ports-all} or {dev-commits-src-all} might wish to unsubscribe to avoid receiving duplicate copies of commit messages and followups. ==== [[conventions-everyone]] === For Everyone [[conventions-everyone-steps]] [.procedure] ==== . Introduce yourself to the other developers, otherwise no one will have any idea who you are or what you are working on. The introduction need not be a comprehensive biography, just write a paragraph or two about who you are, what you plan to be working on as a developer in FreeBSD, and who will be your mentor. Email this to the {developers-name} and you will be on your way! . Log into `freefall.FreeBSD.org` and create a [.filename]#/var/forward/user# (where _user_ is your username) file containing the e-mail address where you want mail addressed to _yourusername_@FreeBSD.org to be forwarded. This includes all of the commit messages as well as any other mail addressed to the {committers-name} and the {developers-name}. Really large mailboxes which have taken up permanent residence on `freefall` may get truncated without warning if space needs to be freed, so forward it or save it elsewhere. + [NOTE] ====== If your e-mail system uses SPF with strict rules, you should exclude `mx2.FreeBSD.org` from SPF checks. ====== + Due to the severe load dealing with SPAM places on the central mail servers that do the mailing list processing, the front-end server does do some basic checks and will drop some messages based on these checks. At the moment proper DNS information for the connecting host is the only check in place but that may change. Some people blame these checks for bouncing valid email. To have these checks turned off for your email, create a file named [.filename]#~/.spam_lover# on `freefall.FreeBSD.org`. + [NOTE] ====== Those who are developers but not committers will not be subscribed to the committers or developers mailing lists. The subscriptions are derived from the access rights. ====== ==== [[smtp-setup]] ==== SMTP Access Setup For those willing to send e-mail messages through the FreeBSD.org infrastructure, follow the instructions below: [.procedure] ==== . Point your mail client at `smtp.FreeBSD.org:587`. . Enable STARTTLS. . Ensure your `From:` address is set to `_yourusername_@FreeBSD.org`. . For authentication, you can use your FreeBSD Kerberos username and password (see <>). The `_yourusername_/mail` principal is preferred, as it is only valid for authenticating to mail resources. + [NOTE] ====== Do not include `@FreeBSD.org` when entering in your username. ====== + .Additional Notes [NOTE] ====== * Will only accept mail from `_yourusername_@FreeBSD.org`. If you are authenticated as one user, you are not permitted to send mail from another. * A header will be appended with the SASL username: (`Authenticated sender: _username_`). * Host has various rate limits in place to cut down on brute force attempts. ====== ==== [[smtp-setup-local-mta]] ===== Using a Local MTA to Forward Emails to the FreeBSD.org SMTP Service It is also possible to use a local MTA to forward locally sent emails to the FreeBSD.org SMTP servers. [[smtp-setup-local-postfix]] .Using Postfix [example] ==== To tell a local Postfix instance that anything from `_yourusername_@FreeBSD.org` should be forwarded to the FreeBSD.org servers, add this to your [.filename]#main.cf#: [.programlisting] .... sender_dependent_relayhost_maps = hash:/usr/local/etc/postfix/relayhost_maps smtp_sasl_auth_enable = yes smtp_sasl_security_options = noanonymous smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd smtp_use_tls = yes .... Create [.filename]#/usr/local/etc/postfix/relayhost_maps# with the following content: [.programlisting] .... yourusername@FreeBSD.org [smtp.freebsd.org]:587 .... Create [.filename]#/usr/local/etc/postfix/sasl_passwd# with the following content: [.programlisting] .... [smtp.freebsd.org]:587 yourusername:yourpassword .... If the email server is used by other people, you may want to prevent them from sending e-mails from your address. To achieve this, add this to your [.filename]#main.cf#: [.programlisting] .... smtpd_sender_login_maps = hash:/usr/local/etc/postfix/sender_login_maps smtpd_sender_restrictions = reject_known_sender_login_mismatch .... Create [.filename]#/usr/local/etc/postfix/sender_login_maps# with the following content: [.programlisting] .... yourusername@FreeBSD.org yourlocalusername .... Where _yourlocalusername_ is the SASL username used to connect to the local instance of Postfix. ==== [[smtp-setup-local-opensmtpd]] .Using OpenSMTPD [example] ==== To tell a local OpenSMTPD instance that anything from `_yourusername_@FreeBSD.org` should be forwarded to the FreeBSD.org servers, add this to your [.filename]#smtpd.conf#: [.programlisting] .... action "freebsd" relay host smtp+tls://freebsd@smtp.freebsd.org:587 auth match from any auth yourlocalusername mail-from "_yourusername_@freebsd.org" for any action "freebsd" .... Where _yourlocalusername_ is the SASL username used to connect to the local instance of OpenSMTPD. Create [.filename]#/usr/local/etc/mail/secrets# with the following content: [.programlisting] .... freebsd yourusername:yourpassword .... ==== [[smtp-setup-local-exim]] .Using Exim [example] ==== To direct a local Exim instance to forward all mail from `_example_@FreeBSD.org` to FreeBSD.org servers, add this to Exim [.filename]#configuration#: [.programlisting] .... Routers section: (at the top of the list): freebsd_send: driver = manualroute domains = !+local_domains transport = freebsd_smtp route_data = ${lookup {${lc:$sender_address}} lsearch {/usr/local/etc/exim/freebsd_send}} Transport Section: freebsd_smtp: driver = smtp tls_certificate= tls_privatekey= tls_require_ciphers = EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+AESGCM:EECDH:EDH+AESGCM:EDH+aRSA:HIGH:!MEDIUM:!LOW:!aNULL:!eNULL:!LOW:!RC4:!MD5:!EXP:!PSK:!SRP:!DSS dkim_domain = dkim_selector = dkim_private_key= dnssec_request_domains = * hosts_require_auth = smtp.freebsd.org Authenticators: fixed_plain: driver = plaintext public_name = PLAIN client_send = ^example/mail^examplePassword .... Create [.filename]#/usr/local/etc/exim/freebsd_send# with the following content: [.programlisting] .... example@freebsd.org:smtp.freebsd.org::587 .... ==== [[mentors]] === Mentors All new developers have a mentor assigned to them for the first few months. A mentor is responsible for teaching the mentee the rules and conventions of the project and guiding their first steps in the developer community. The mentor is also personally responsible for the mentee's actions during this initial period. For committers: do not commit anything without first getting mentor approval. Document that approval with an `Approved by:` line in the commit message. When the mentor decides that a mentee has learned the ropes and is ready to commit on their own, the mentor announces it with a commit to [.filename]#mentors#. This file is in the [.filename]#admin# orphan branch of each repository. Detailed information on how to access these branches can be found in <>. [[pre-commit-review]] == Pre-Commit Review Code review is one way to increase the quality of software. The following guidelines apply to commits to the `main` (-CURRENT) branch of the `src` repository. Other branches and the `ports` and `docs` trees have their own review policies, but these guidelines generally apply to commits requiring review: * All non-trivial changes should be reviewed before they are committed to the repository. * Reviews may be conducted by email, in Bugzilla, in Phabricator, or by another mechanism. Where possible, reviews should be public. * The developer responsible for a code change is also responsible for making all necessary review-related changes. * Code review can be an iterative process, which continues until the patch is ready to be committed. Specifically, once a patch is sent out for review, it should receive an explicit "looks good" before it is committed. So long as it is explicit, this can take whatever form makes sense for the review method. * Timeouts are not a substitute for review. Sometimes code reviews will take longer than you would hope for, especially for larger features. Accepted ways to speed up review times for your patches are: * Review other people's patches. If you help out, everybody will be more willing to do the same for you; goodwill is our currency. * Ping the patch. If it is urgent, provide reasons why it is important to you to get this patch landed and ping it every couple of days. If it is not urgent, the common courtesy ping rate is one week. Remember that you are asking for valuable time from other professional developers. * Ask for help on mailing lists, IRC, etc. Others may be able to either help you directly, or suggest a reviewer. * Split your patch into multiple smaller patches that build on each other. The smaller your patch, the higher the probability that somebody will take a quick look at it. + When making large changes, it is helpful to keep this in mind from the beginning of the effort as breaking large changes into smaller ones is often difficult after the fact. Developers should participate in code reviews as both reviewers and reviewees. If someone is kind enough to review your code, you should return the favor for someone else. Note that while anyone is welcome to review and give feedback on a patch, only an appropriate subject-matter expert can approve a change. This will usually be a committer who works with the code in question on a regular basis. In some cases, no subject-matter expert may be available. In those cases, a review by an experienced developer is sufficient when coupled with appropriate testing. [[commit-log-message]] == Commit Log Messages This section contains some suggestions and traditions for how commit logs are formatted. === Why are commit messages important? When you commit a change in Git, Subversion, or another version control system (VCS), you're prompted to write some text describing the commit -- a commit message. How important is this commit message? Should you spend some significant effort writing it? Does it really matter if you write simply fixed a bug? Most projects have more than one developer and last for some length of time. Commit messages are a very important method of communicating with other developers, in the present and for the future. FreeBSD has hundreds of active developers and hundreds of thousands of commits spanning decades of history. Over that time the developer community has learned how valuable good commit messages are; sometimes these are hard-learned lessons. Commit messages serve at least three purposes: * Communicating with other developers + FreeBSD commits generate email to various mailing lists. These include the commit message along with a copy of the patch itself. Commit messages are also viewed through commands like git log. These serve to make other developers aware of changes that are ongoing; that other developer may want to test the change, may have an interest in the topic and will want to review in more detail, or may have their own projects underway that would benefit from interaction. * Making Changes Discoverable + In a large project with a long history it may be difficult to find changes of interest when investigating an issue or change in behaviour. Verbose, detailed commit messages allow searches for changes that might be relevant. For example, `git log --since 1year --grep 'USB timeout'`. * Providing historical documentation + Commit messages serve to document changes for future developers, perhaps years or decades later. This future developer may even be you, the original author. A change that seems obvious today may be decidedly not so much later on. The `git blame` command annotates each line of a source file with the change (hash and subject line) that brought it in. Having established the importance, here are elements of a good FreeBSD commit message: === Start with a subject line Commit messages should start with a single-line subject that briefly summarizes the change. The subject should, by itself, allow the reader to quickly determine if the change is of interest or not. === Keep subject lines short The subject line should be as short as possible while still retaining the required information. This is to make browsing Git log more efficient, and so that git log --oneline can display the short hash and subject on a single 80-column line. A good rule of thumb is to stay below 63 characters, and aim for about 50 or fewer if possible. === Prefix the subject line with a component, if applicable If the change relates to a specific component the subject line may be prefixed with that component name and a colon (:). ✓ `foo: Add -k option to keep temporary data` Include the prefix in the 63-character limit suggested above, so that `git log --oneline` avoids wrapping. === Capitalize the first letter of the subject Capitalize the first letter of the subject itself. The prefix, if any, is not capitalized unless necessary (e.g., `USB:` is capitalized). === Do not end the subject line with punctuation Do not end with a period or other punctuation. In this regard the subject line is like a newspaper headline. === Separate the subject and body with a blank line Separate the body from the subject with a blank line. Some trivial commits do not require a body, and will have only a subject. ✓ `ls: Fix typo in usage text` === Limit messages to 72 columns `git log` and `git format-patch` indent the commit message by four spaces. Wrapping at 72 columns provides a matching margin on the right edge. Limiting messages to 72 characters also keeps the commit message in formatted patches below RFC 2822's suggested email line length limit of 78 characters. This limit works well with a variety of tools that may render commit messages; line wrapping might be inconsistent with longer line length. === Use the present tense, imperative mood This facilitates short subject lines and provides consistency, including with automatically generated commit messages (e.g., as generated by git revert). This is important when reading a list of commit subjects. Think of the subject as finishing the sentence "when applied, this change will ...". ✓ `foo: Implement the -k (keep) option` + ✗ `foo: Implemented the -k option` + ✗ `This change implements the -k option in foo` + ✗ `-k option added` === Focus on what and why, not how Explain what the change accomplishes and why it is being done, rather than how. Do not assume that the reader is familiar with the issue. Explain the background and motivation for the change. Include benchmark data if you have it. If there are limitations or incomplete aspects of the change, describe them in the commit message. === Consider whether parts of the commit message could be code comments instead Sometimes while writing a commit message you may find yourself writing a sentence or two explaining some tricky or confusing aspect of the change. When this happens consider whether it would be valuable to have that explanation as a comment in the code itself. === Write commit messages for your future self While writing the commit message for a change you have all of the context in mind - what prompted the change, alternate approaches that were considered and rejected, limitations of the change, and so on. Imagine yourself revisiting the change a year or two in the future, and write the commit message in a way that would provide that necessary context. === Commit messages should stand alone You may include references to mailing list postings, benchmark result web sites, or code review links. However, the commit message should contain all of the relevant information in case these references are no longer available in the future. Similarly, a commit may refer to a previous commit, for example in the case of a bug fix or revert. In addition to the commit identifier (revision or hash), include the subject line from the referenced commit (or another suitable brief reference). With each VCS migration (from CVS to Subversion to Git) revision identifiers from previous systems may become difficult to follow. === Include appropriate metadata in a footer As well as including an informative message with each commit, some additional information may be needed. This information consists of one or more lines containing the key word or phrase, a colon, tabs for formatting, and then the additional information. The key words or phrases are: [.informaltable] [cols="20%,80%", frame="none"] |=== |`PR:` |The problem report (if any) which is affected (typically, by being closed) by this commit. Multiple PRs may be specified on one line, separated by commas or spaces. |`Reported by:` |The name and e-mail address of the person that reported the issue; for developers, just the username on the FreeBSD cluster. Typically used when there is no PR, for example if the issue was reported on a mailing list. |`Submitted by:` |This has been deprecated with git; submitted patches should have the author set by using `git commit --author` with a full name and valid email. |`Reviewed by:` a| The name and e-mail address of the person or people that reviewed the change; for developers, just the username on the FreeBSD cluster. If a patch was submitted to a mailing list for review, and the review was favorable, then just include the list name. If the reviewer is not a member of the project, provide the name, email, and if ports an external role like maintainer: Reviewed by a developer: [source,shell] .... Reviewed by: username .... Reviewed by a ports maintainer that is not a developer: [source,shell] .... Reviewed by: Full Name (maintainer) .... |`Tested by:` |The name and e-mail address of the person or people that tested the change; for developers, just the username on the FreeBSD cluster. |`Approved by:` a| The name and e-mail address of the person or people that approved the change; for developers, just the username on the FreeBSD cluster. There are several cases where approval is customary: * while a new committer is under mentorship * commits to an area of the tree covered by the LOCKS file (src) * during a release cycle * committing to a repo where you do not hold a commit bit (e.g. src committer committing to docs) While under mentorship, get mentor approval before the commit. Enter the mentor's username in this field, and note that they are a mentor: [source,shell] .... Approved by: username-of-mentor (mentor) .... If a team approved these commits then include the team name followed by the username of the approver in parentheses. For example: [source,shell] .... Approved by: re (username) .... |`Obtained from:` |The name of the project (if any) from which the code was obtained. Do not use this line for the name of an individual person. |`Fixes:` |The Git short hash and the title line of a commit that is fixed by this change as returned by `git log -n 1 --oneline GIT-COMMIT-HASH`. |`MFC after:` |To receive an e-mail reminder to MFC at a later date, specify the number of days, weeks, or months after which an MFC is planned. |`MFC to:` |If the commit should be merged to a subset of stable branches, specify the branch names. |`MFH:` |If the commit is to be merged into a ports quarterly branch name, specify the quarterly branch. For example `2021Q2`. |`Relnotes:` |If the change is a candidate for inclusion in the release notes for the next release from the branch, set to `yes`. |`Security:` |If the change is related to a security vulnerability or security exposure, include one or more references or a description of the issue. If possible, include a VuXML URL or a CVE ID. |`Event:` |The description for the event where this commit was made. If this is a recurring event, add the year or even the month to it. For example, this could be `FooBSDcon 2019`. The idea behind this line is to put recognition to conferences, gatherings, and other types of meetups and to show that these are useful to have. Please do not use the `Sponsored by:` line for this as that is meant for organizations sponsoring certain features or developers working on them. |`Sponsored by:` |Sponsoring organizations for this change, if any. Separate multiple organizations with commas. If only a portion of the work was sponsored, or different amounts of sponsorship were provided to different authors, please give appropriate credit in parentheses after each sponsor name. For example, `Example.com (alice, code refactoring), Wormulon (bob), Momcorp (cindy)` shows that Alice was sponsored by Example.com to do code refactoring, while Wormulon sponsored Bob's work and Momcorp sponsored Cindy's work. Other authors were either not sponsored or chose not to list sponsorship. |`Pull Request:` |This change was submitted as a pull request or merge request against one of FreeBSD's public read-only Git repositories. It should include the entire URL to the pull request, as these often act as code reviews for the code. For example: `https://github.com/freebsd/freebsd-src/pull/745` |`Signed-off-by:` |ID certifies compliance with https://developercertificate.org/ |`Differential Revision:` |The full URL of the Phabricator review. This line __must be the last line__. For example: `https://reviews.freebsd.org/D1708`. |=== .Commit Log for a Commit Based on a PR [example] ==== The commit is based on a patch from a PR submitted by John Smith. The commit message "PR" field is filled. [.programlisting] .... ... PR: 12345 .... The committer sets the author of the patch with `git commit --author "John Smith "`. ==== .Commit Log for a Commit Needing Review [example] ==== The virtual memory system is being changed. After posting patches to the appropriate mailing list (in this case, `freebsd-arch`) and the changes have been approved. [.programlisting] .... ... Reviewed by: -arch .... ==== .Commit Log for a Commit Needing Approval [example] ==== Commit a port, after working with the listed MAINTAINER, who said to go ahead and commit. [.programlisting] .... ... Approved by: abc (maintainer) .... Where _abc_ is the account name of the person who approved. ==== .Commit Log for a Commit Bringing in Code from OpenBSD [example] ==== Committing some code based on work done in the OpenBSD project. [.programlisting] .... ... Obtained from: OpenBSD .... ==== .Commit Log for a Change to FreeBSD-CURRENT with a Planned Commit to FreeBSD-STABLE to Follow at a Later Date. [example] ==== Committing some code which will be merged from FreeBSD-CURRENT into the FreeBSD-STABLE branch after two weeks. [.programlisting] .... ... MFC after: 2 weeks .... Where _2_ is the number of days, weeks, or months after which an MFC is planned. The _weeks_ option may be `day`, `days`, `week`, `weeks`, `month`, `months`. ==== It is often necessary to combine these. Consider the situation where a user has submitted a PR containing code from the NetBSD project. Looking at the PR, the developer sees it is not an area of the tree they normally work in, so they have the change reviewed by the `arch` mailing list. Since the change is complex, the developer opts to MFC after one month to allow adequate testing. The extra information to include in the commit would look something like .Example Combined Commit Log [example] ==== [.programlisting] .... PR: 54321 Reviewed by: -arch Obtained from: NetBSD MFC after: 1 month Relnotes: yes .... ==== [[pref-license]] == Preferred License for New Files The FreeBSD Project's full license policy can be found at link:https://www.FreeBSD.org/internal/software-license/[https://www.FreeBSD.org/internal/software-license]. The rest of this section is intended to help you get started. As a rule, when in doubt, ask. It is much easier to give advice than to fix the source tree. The FreeBSD Project suggests and uses this text as the preferred license scheme: [.programlisting] .... /*- * SPDX-License-Identifier: BSD-2-Clause * * Copyright (c) [year] [your name] * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * [id for your version control system, if any] */ .... The FreeBSD project strongly discourages the so-called "advertising clause" in new code. Due to the large number of contributors to the FreeBSD project, complying with this clause for many commercial vendors has become difficult. If you have code in the tree with the advertising clause, please consider removing it. In fact, please consider using the above license for your code. The FreeBSD project discourages completely new licenses and variations on the standard licenses. New licenses require the approval of {core-email} to reside in the `src` repository. The more different licenses that are used in the tree, the more problems that this causes to those wishing to utilize this code, typically from unintended consequences from a poorly worded license. Project policy dictates that code under some non-BSD licenses must be placed only in specific sections of the repository, and in some cases, compilation must be conditional or even disabled by default. For example, the GENERIC kernel must be compiled under only licenses identical to or substantially similar to the BSD license. GPL, APSL, CDDL, etc, licensed software must not be compiled into GENERIC. Developers are reminded that in open source, getting "open" right is just as important as getting "source" right, as improper handling of intellectual property has serious consequences. Any questions or concerns should immediately be brought to the attention of the core team. [[tracking.license.grants]] == Keeping Track of Licenses Granted to the FreeBSD Project Various software or data exist in the repositories where the FreeBSD project has been granted a special license to be able to use them. A case in point are the Terminus fonts for use with man:vt[4]. Here the author Dimitar Zhekov has allowed us to use the "Terminus BSD Console" font under a 2-clause BSD license rather than the regular Open Font License he normally uses. It is clearly sensible to keep a record of any such license grants. To that end, the {core-email} has decided to keep an archive of them. Whenever the FreeBSD project is granted a special license we require the {core-email} to be notified. Any developers involved in arranging such a license grant, please send details to the {core-email} including: * Contact details for people or organizations granting the special license. * What files, directories etc. in the repositories are covered by the license grant including the revision numbers where any specially licensed material was committed. * The date the license comes into effect from. Unless otherwise agreed, this will be the date the license was issued by the authors of the software in question. * The license text. * A note of any restrictions, limitations or exceptions that apply specifically to FreeBSD's usage of the licensed material. * Any other relevant information. Once the {core-email} is satisfied that all the necessary details have been gathered and are correct, the secretary will send a PGP-signed acknowledgment of receipt including the license details. This receipt will be persistently archived and serve as our permanent record of the license grant. The license archive should contain only details of license grants; this is not the place for any discussions around licensing or other subjects. Access to data within the license archive will be available on request to the {core-email}. [[spdx.tags]] == SPDX Tags in the tree The project uses https://spdx.dev[SPDX] tags in our source base. At present, these tags are indented to help automated tools reconstruct license requirements mechanically. All _SPDX-License-Identifier_ tags in the tree should be considered to be informative. All files in the FreeBSD source tree with these tags also have a copy of the license which governs use of that file. In the event of a discrepancy, the verbatim license is controlling. The project tries to follow the https://spdx.github.io/spdx-spec/[SPDX Specification, Version 2.2]. How to mark source files and valid algebraic expressions are found in https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/[Appendix IV] and https://spdx.github.io/spdx-spec/appendix-V-using-SPDX-short-identifiers-in-source-files/[Appendix V]. The project draws identifiers from SPDX's list of valid https://spdx.org/licenses/[short license identifiers]. The project uses only the _SPDX-License-Identifier_ tag. As of March 2021, approximately 25,000 out of 90,000 files in the tree have been marked. [[developer.relations]] == Developer Relations When working directly on your own code or on code which is already well established as your responsibility, then there is probably little need to check with other committers before jumping in with a commit. When working on a bug in an area of the system which is clearly orphaned (and there are a few such areas, to our shame), the same applies. When modifying parts of the system which are maintained, formally or informally, consider asking for a review just as a developer would have before becoming a committer. For ports, contact the listed `MAINTAINER` in the [.filename]#Makefile#. To determine if an area of the tree is maintained, check the MAINTAINERS file at the root of the tree. If nobody is listed, scan the revision history to see who has committed changes in the past. To list the names and email addresses of all commit authors for a given file in the last 2 years and the number of commits each has authored, ordered by descending number of commits, use: [source,shell] ---- % git -C /path/to/repo shortlog -sne --since="2 years" -- relative/path/to/file ---- If queries go unanswered or the committer otherwise indicates a lack of interest in the area affected, go ahead and commit it. [IMPORTANT] ==== Avoid sending private emails to maintainers. Other people might be interested in the conversation, not just the final output. ==== If there is any doubt about a commit for any reason at all, have it reviewed before committing. Better to have it flamed then and there rather than when it is part of the repository. If a commit does results in controversy erupting, it may be advisable to consider backing the change out again until the matter is settled. Remember, with a version control system we can always change it back. Do not impugn the intentions of others. If they see a different solution to a problem, or even a different problem, it is probably not because they are stupid, because they have questionable parentage, or because they are trying to destroy hard work, personal image, or FreeBSD, but basically because they have a different outlook on the world. Different is good. Disagree honestly. Argue your position from its merits, be honest about any shortcomings it may have, and be open to seeing their solution, or even their vision of the problem, with an open mind. Accept correction. We are all fallible. When you have made a mistake, apologize and get on with life. Do not beat up yourself, and certainly do not beat up others for your mistake. Do not waste time on embarrassment or recrimination, just fix the problem and move on. Ask for help. Seek out (and give) peer reviews. One of the ways open source software is supposed to excel is in the number of eyeballs applied to it; this does not apply if nobody will review code. [[if-in-doubt]] == If in Doubt... When unsure about something, whether it be a technical issue or a project convention be sure to ask. If you stay silent you will never make progress. If it relates to a technical issue ask on the public mailing lists. Avoid the temptation to email the individual person that knows the answer. This way everyone will be able to learn from the question and the answer. For project specific or administrative questions ask, in order: * Your mentor or former mentor. * An experienced committer on IRC, email, etc. * Any team with a "hat", as they can give you a definitive answer. * If still not sure, ask on {developers-name}. Once your question is answered, if no one pointed you to documentation that spelled out the answer to your question, document it, as others will have the same question. [[bugzilla]] == Bugzilla The FreeBSD Project utilizes Bugzilla for tracking bugs and change requests. Be sure that if you commit a fix or suggestion found in the PR database to close it. It is also considered nice if you take time to close any PRs associated with your commits, if appropriate. Committers with non-``FreeBSD.org`` Bugzilla accounts can have the old account merged with the `FreeBSD.org` account by following these steps: [.procedure] ==== . Log in using your old account. . Open new bug. Choose `Services` as the Product, and `Bug Tracker` as the Component. In bug description list accounts you wish to be merged. . Log in using `FreeBSD.org` account and post comment to newly opened bug to confirm ownership. See <> for more details on how to generate or set a password for your `FreeBSD.org` account. . If there are more than two accounts to merge, post comments from each of them. ==== You can find out more about Bugzilla at: * extref:{pr-guidelines}[FreeBSD Problem Report Handling Guidelines] * link:https://www.FreeBSD.org/support/[https://www.FreeBSD.org/support] [[phabricator]] == Phabricator The FreeBSD Project utilizes https://reviews.freebsd.org[Phabricator] for code review requests. See the https://wiki.freebsd.org/Phabricator[Phabricator wiki page] for details. Committers with non-``FreeBSD.org`` Phabricator accounts can have the old account renamed to the ``FreeBSD.org`` account by following these steps: [.procedure] ==== . Change your Phabricator account email to your `FreeBSD.org` email. . Open new bug on our bug tracker using your `FreeBSD.org` account, see <> for more information. Choose `Services` as the Product, and `Code Review` as the Component. In bug description request that your Phabricator account be renamed, and provide a link to your Phabricator user. For example, `https://reviews.freebsd.org/p/bob_example.com/` ==== [IMPORTANT] ==== Phabricator accounts cannot be merged, please do not open a new account. ==== [[people]] == Who's Who Besides the repository meisters, there are other FreeBSD project members and teams whom you will probably get to know in your role as a committer. Briefly, and by no means all-inclusively, these are: `{doceng}`:: doceng is the group responsible for the documentation build infrastructure, approving new documentation committers, and ensuring that the FreeBSD website and documentation on the FTP site is up to date with respect to the Subversion tree. It is not a conflict resolution body. The vast majority of documentation related discussion takes place on the {freebsd-doc}. More details regarding the doceng team can be found in its https://www.FreeBSD.org/internal/doceng/[charter]. Committers interested in contributing to the documentation should familiarize themselves with the extref:{fdp-primer}[Documentation Project Primer]. `{re-members}`:: These are the members of the `{re}`. This team is responsible for setting release deadlines and controlling the release process. During code freezes, the release engineers have final authority on all changes to the system for whichever branch is pending release status. If there is something you want merged from FreeBSD-CURRENT to FreeBSD-STABLE (whatever values those may have at any given time), these are the people to talk to about it. `{so}`:: `{so-name}` is the link:https://www.FreeBSD.org/security/[FreeBSD Security Officer] and oversees the `{security-officer}`. {committers-name}:: {dev-src-all}, {dev-ports-all} and {dev-doc-all} are the mailing lists that the version control system uses to send commit messages to. _Never_ send email directly to these lists. Only send replies to this list when they are short and are directly related to a commit. {developers-name}:: All committers are subscribed to -developers. This list was created to be a forum for the committers "community" issues. Examples are Core voting, announcements, etc. + The {developers-name} is for the exclusive use of FreeBSD committers. To develop FreeBSD, committers must have the ability to openly discuss matters that will be resolved before they are publicly announced. Frank discussions of work in progress are not suitable for open publication and may harm FreeBSD. + All FreeBSD committers are expected not to not publish or forward messages from the {developers-name} outside the list membership without permission of all of the authors. Violators will be removed from the {developers-name}, resulting in a suspension of commit privileges. Repeated or flagrant violations may result in permanent revocation of commit privileges. + This list is _not_ intended as a place for code reviews or for any technical discussion. In fact using it as such hurts the FreeBSD Project as it gives a sense of a closed list where general decisions affecting all of the FreeBSD using community are made without being "open". Last, but not least __never, never ever, email the {developers-name} and CC:/BCC: another FreeBSD list__. Never, ever email another FreeBSD email list and CC:/BCC: the {developers-name}. Doing so can greatly diminish the benefits of this list. [[ssh.guide]] == SSH Quick-Start Guide [.procedure] ==== . If you do not wish to type your password in every time you use man:ssh[1], and you use keys to authenticate, man:ssh-agent[1] is there for your convenience. If you want to use man:ssh-agent[1], make sure that you run it before running other applications. X users, for example, usually do this from their [.filename]#.xsession# or [.filename]#.xinitrc#. See man:ssh-agent[1] for details. . Generate a key pair using man:ssh-keygen[1]. The key pair will wind up in your [.filename]#$HOME/.ssh/# directory. + [IMPORTANT] ====== Only ECDSA, Ed25519 or RSA keys are supported. ====== . Send your public key ([.filename]#$HOME/.ssh/id_ecdsa.pub#, [.filename]#$HOME/.ssh/id_ed25519.pub#, or [.filename]#$HOME/.ssh/id_rsa.pub#) to the person setting you up as a committer so it can be put into [.filename]#yourlogin# in [.filename]#/etc/ssh-keys/# on `freefall`. ==== Now man:ssh-add[1] can be used for authentication once per session. It prompts for the private key's pass phrase, and then stores it in the authentication agent (man:ssh-agent[1]). Use `ssh-add -d` to remove keys stored in the agent. Test with a simple remote command: `ssh freefall.FreeBSD.org ls /usr`. For more information, see package:security/openssh-portable[], man:ssh[1], man:ssh-add[1], man:ssh-agent[1], man:ssh-keygen[1], and man:scp[1]. For information on adding, changing, or removing man:ssh[1] keys, see https://wiki.freebsd.org/clusteradm/ssh-keys[this article]. [[coverity]] == Coverity(R) Availability for FreeBSD Committers All FreeBSD developers can obtain access to Coverity analysis results of all FreeBSD Project software. All who are interested in obtaining access to the analysis results of the automated Coverity runs, can sign up at http://scan.coverity.com/[Coverity Scan]. The FreeBSD wiki includes a mini-guide for developers who are interested in working with the Coverity(R) analysis reports: https://wiki.freebsd.org/CoverityPrevent[https://wiki.freebsd.org/CoverityPrevent]. Please note that this mini-guide is only readable by FreeBSD developers, so if you cannot access this page, you will have to ask someone to add you to the appropriate Wiki access list. Finally, all FreeBSD developers who are going to use Coverity(R) are always encouraged to ask for more details and usage information, by posting any questions to the mailing list of the FreeBSD developers. [[rules]] == The FreeBSD Committers' Big List of Rules Everyone involved with the FreeBSD project is expected to abide by the _Code of Conduct_ available from link:https://www.FreeBSD.org/internal/code-of-conduct/[https://www.FreeBSD.org/internal/code-of-conduct]. As committers, you form the public face of the project, and how you behave has a vital impact on the public perception of it. This guide expands on the parts of the _Code of Conduct_ specific to committers. . Respect other committers. . Respect other contributors. . Discuss any significant change _before_ committing. . Respect existing maintainers (if listed in the `MAINTAINER` field in [.filename]#Makefile# or in [.filename]#MAINTAINER# in the top-level directory). . Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. . Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the FreeBSD-STABLE branch as outlined for the maintainer in rule #5. . Do not fight in public with other committers; it looks bad. . Respect all code freezes and read the `committers` and `developers` mailing lists in a timely manner so you know when a code freeze is in effect. . When in doubt on any procedure, ask first! . Test your changes before committing them. . Do not commit to contributed software without _explicit_ approval from the respective maintainers. As noted, breaking some of these rules can be grounds for suspension or, upon repeated offense, permanent removal of commit privileges. Individual members of core have the power to temporarily suspend commit privileges until core as a whole has the chance to review the issue. In case of an "emergency" (a committer doing damage to the repository), a temporary suspension may also be done by the repository meisters. Only a 2/3 majority of core has the authority to suspend commit privileges for longer than a week or to remove them permanently. This rule does not exist to set core up as a bunch of cruel dictators who can dispose of committers as casually as empty soda cans, but to give the project a kind of safety fuse. If someone is out of control, it is important to be able to deal with this immediately rather than be paralyzed by debate. In all cases, a committer whose privileges are suspended or revoked is entitled to a "hearing" by core, the total duration of the suspension being determined at that time. A committer whose privileges are suspended may also request a review of the decision after 30 days and every 30 days thereafter (unless the total suspension period is less than 30 days). A committer whose privileges have been revoked entirely may request a review after a period of 6 months has elapsed. This review policy is _strictly informal_ and, in all cases, core reserves the right to either act on or disregard requests for review if they feel their original decision to be the right one. In all other aspects of project operation, core is a subset of committers and is bound by the __same rules__. Just because someone is in core this does not mean that they have special dispensation to step outside any of the lines painted here; core's "special powers" only kick in when it acts as a group, not on an individual basis. As individuals, the core team members are all committers first and core second. === Details [[respect]] . Respect other committers. + This means that you need to treat other committers as the peer-group developers that they are. Despite our occasional attempts to prove the contrary, one does not get to be a committer by being stupid and nothing rankles more than being treated that way by one of your peers. Whether we always feel respect for one another or not (and everyone has off days), we still have to _treat_ other committers with respect at all times, on public forums and in private email. + Being able to work together long term is this project's greatest asset, one far more important than any set of changes to the code, and turning arguments about code into issues that affect our long-term ability to work harmoniously together is just not worth the trade-off by any conceivable stretch of the imagination. + To comply with this rule, do not send email when you are angry or otherwise behave in a manner which is likely to strike others as needlessly confrontational. First calm down, then think about how to communicate in the most effective fashion for convincing the other persons that your side of the argument is correct, do not just blow off some steam so you can feel better in the short term at the cost of a long-term flame war. Not only is this very bad "energy economics", but repeated displays of public aggression which impair our ability to work well together will be dealt with severely by the project leadership and may result in suspension or termination of your commit privileges. The project leadership will take into account both public and private communications brought before it. It will not seek the disclosure of private communications, but it will take it into account if it is volunteered by the committers involved in the complaint. + All of this is never an option which the project's leadership enjoys in the slightest, but unity comes first. No amount of code or good advice is worth trading that away. . Respect other contributors. + You were not always a committer. At one time you were a contributor. Remember that at all times. Remember what it was like trying to get help and attention. Do not forget that your work as a contributor was very important to you. Remember what it was like. Do not discourage, belittle, or demean contributors. Treat them with respect. They are our committers in waiting. They are every bit as important to the project as committers. Their contributions are as valid and as important as your own. After all, you made many contributions before you became a committer. Always remember that. + Consider the points raised under <> and apply them also to contributors. . Discuss any significant change _before_ committing. + The repository is not where changes are initially submitted for correctness or argued over, that happens first in the mailing lists or by use of the Phabricator service. The commit will only happen once something resembling consensus has been reached. This does not mean that permission is required before correcting every obvious syntax error or manual page misspelling, just that it is good to develop a feel for when a proposed change is not quite such a no-brainer and requires some feedback first. People really do not mind sweeping changes if the result is something clearly better than what they had before, they just do not like being _surprised_ by those changes. The very best way of making sure that things are on the right track is to have code reviewed by one or more other committers. + When in doubt, ask for review! . Respect existing maintainers if listed. + Many parts of FreeBSD are not "owned" in the sense that any specific individual will jump up and yell if you commit a change to "their" area, but it still pays to check first. One convention we use is to put a maintainer line in the [.filename]#Makefile# for any package or subtree which is being actively maintained by one or more people; see extref:{developers-handbook}[Source Tree Guidelines and Policies, policies] for documentation on this. Where sections of code have several maintainers, commits to affected areas by one maintainer need to be reviewed by at least one other maintainer. In cases where the "maintainer-ship" of something is not clear, look at the repository logs for the files in question and see if someone has been working recently or predominantly in that area. . Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. + This may be hard to swallow in times of conflict (when each side is convinced that they are in the right, of course) but a version control system makes it unnecessary to have an ongoing dispute raging when it is far easier to simply reverse the disputed change, get everyone calmed down again and then try to figure out what is the best way to proceed. If the change turns out to be the best thing after all, it can be easily brought back. If it turns out not to be, then the users did not have to live with the bogus change in the tree while everyone was busily debating its merits. People _very_ rarely call for back-outs in the repository since discussion generally exposes bad or controversial changes before the commit even happens, but on such rare occasions the back-out should be done without argument so that we can get immediately on to the topic of figuring out whether it was bogus or not. . Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the FreeBSD-STABLE branch as outlined in rule #5. + This is another "do not argue about it" issue since it is the release engineer who is ultimately responsible (and gets beaten up) if a change turns out to be bad. Please respect this and give the release engineer your full cooperation when it comes to the FreeBSD-STABLE branch. The management of FreeBSD-STABLE may frequently seem to be overly conservative to the casual observer, but also bear in mind the fact that conservatism is supposed to be the hallmark of FreeBSD-STABLE and different rules apply there than in FreeBSD-CURRENT. There is also really no point in having FreeBSD-CURRENT be a testing ground if changes are merged over to FreeBSD-STABLE immediately. Changes need a chance to be tested by the FreeBSD-CURRENT developers, so allow some time to elapse before merging unless the FreeBSD-STABLE fix is critical, time sensitive or so obvious as to make further testing unnecessary (spelling fixes to manual pages, obvious bug/typo fixes, etc.) In other words, apply common sense. + Changes to the security branches (for example, `releng/9.3`) must be approved by a member of the `{security-officer}`, or in some cases, by a member of the `{re}`. . Do not fight in public with other committers; it looks bad. + This project has a public image to uphold and that image is very important to all of us, especially if we are to continue to attract new members. There will be occasions when, despite everyone's very best attempts at self-control, tempers are lost and angry words are exchanged. The best thing that can be done in such cases is to minimize the effects of this until everyone has cooled back down. Do not air angry words in public and do not forward private correspondence or other private communications to public mailing lists, mail aliases, instant messaging channels or social media sites. What people say one-to-one is often much less sugar-coated than what they would say in public, and such communications therefore have no place there - they only serve to inflame an already bad situation. If the person sending a flame-o-gram at least had the grace to send it privately, then have the grace to keep it private yourself. If you feel you are being unfairly treated by another developer, and it is causing you anguish, bring the matter up with core rather than taking it public. Core will do its best to play peace makers and get things back to sanity. In cases where the dispute involves a change to the codebase and the participants do not appear to be reaching an amicable agreement, core may appoint a mutually-agreeable third party to resolve the dispute. All parties involved must then agree to be bound by the decision reached by this third party. . Respect all code freezes and read the `committers` and `developers` mailing list on a timely basis so you know when a code freeze is in effect. + Committing unapproved changes during a code freeze is a really big mistake and committers are expected to keep up-to-date on what is going on before jumping in after a long absence and committing 10 megabytes worth of accumulated stuff. People who abuse this on a regular basis will have their commit privileges suspended until they get back from the FreeBSD Happy Reeducation Camp we run in Greenland. . When in doubt on any procedure, ask first! + Many mistakes are made because someone is in a hurry and just assumes they know the right way of doing something. If you have not done it before, chances are good that you do not actually know the way we do things and really need to ask first or you are going to completely embarrass yourself in public. There is no shame in asking "how in the heck do I do this?" We already know you are an intelligent person; otherwise, you would not be a committer. . Test your changes before committing them. + If your changes are to the kernel, make sure you can still compile both GENERIC and LINT. If your changes are anywhere else, make sure you can still make world. If your changes are to a branch, make sure your testing occurs with a machine which is running that code. If you have a change which also may break another architecture, be sure and test on all supported architectures. Please ensure your change works for <>. Please refer to the https://www.FreeBSD.org/internal/[FreeBSD Internal Page] for a list of available resources. As other architectures are added to the FreeBSD supported platforms list, the appropriate shared testing resources will be made available. . Do not commit to contributed software without _explicit_ approval from the respective maintainers. + Contributed software is anything under the [.filename]#src/contrib#, [.filename]#src/crypto#, or [.filename]#src/sys/contrib# trees. + The trees mentioned above are for contributed software usually imported onto a vendor branch. Committing something there may cause unnecessary headaches when importing newer versions of the software. As a general consider sending patches upstream to the vendor. Patches may be committed to FreeBSD first with permission of the maintainer. + Reasons for modifying upstream software range from wanting strict control over a tightly coupled dependency to lack of portability in the canonical repository's distribution of their code. Regardless of the reason, effort to minimize the maintenance burden of fork is helpful to fellow maintainers. Avoid committing trivial or cosmetic changes to files since it makes every merge thereafter more difficult: such patches need to be manually re-verified every import. + If a particular piece of software lacks a maintainer, you are encouraged to take up ownership. If you are unsure of the current maintainership email {freebsd-arch} and ask. === Policy on Multiple Architectures FreeBSD has added several new architecture ports during recent release cycles and is truly no longer an i386(TM) centric operating system. In an effort to make it easier to keep FreeBSD portable across the platforms we support, core has developed this mandate: [.blockquote] Our 32-bit reference platform is i386, and our 64-bit reference platform is amd64. Major design work (including major API and ABI changes) must prove itself on at least one 32-bit and at least one 64-bit platform, preferably the primary reference platforms, before it may be committed to the source tree. Developers should also be aware of our Tier Policy for the long term support of hardware architectures. The rules here are intended to provide guidance during the development process, and are distinct from the requirements for features and architectures listed in that section. The Tier rules for feature support on architectures at release-time are more strict than the rules for changes during the development process. [[compilers]] === Policy on Multiple Compilers FreeBSD builds with both Clang and GCC. The project does this in a careful and controlled way to maximize benefits from this extra work, while keeping the extra work to a minimum. Supporting both Clang and GCC improves the flexibility our users have. These compilers have different strengths and weaknesses, and supporting both allows users to pick the best one for their needs. Clang and GCC support similar dialects of C and C++, necessitating a relatively small amount of conditional code. The project gains increased code coverage and improves the code quality by using features from both compilers. The project is able to build in more user environments and leverage more CI environments by supporting this range, increasing convenience for users and giving them more tools to test with. By carefully constraining the range of versions supported to modern versions of these compilers, the project avoids unduely increasing the testing matrix. Older and obscure compilers, as well as older dialects of the langauges, have extremely limited support that allow user programs to build with them, but without constraining the base system to being built with them. The exact balance continues to evolve to ensure the benefits of extra work remain greater than the burdens it imposes. The project used to support really old Intel compilers or old GCC versions, but we traded supporting those obsolete compilers for a carefully selected range of modern compilers. This section documents where we use different compilers, and the expectations around that. The FreeBSD project provides an in-tree Clang compiler. Due to being in the tree, this compiler is the most supported compiler. All changes must compile with it, prior to commit. Complete testing, as appropriate for the change, should be done with this compiler. At any moment in time, the FreeBSD project also supports one or more out-of-tree compilers. At present, this is GCC 12.x. Ideally, committers should test compile with this compiler, especially for large or risky changes. This compiler is available as the `${TARGET_ARCH}-gcc${VERSION}` package, such as package:devel/freebsd-gcc12@aarch64[aarch64-gcc12] or package:devel/freebsd-gcc12@riscv64[riscv64-gcc12]. The project runs automated CI jobs to build everything with these compilers. Committers are expected to fix the jobs they break with their changes. Committers may test build with, for example `CROSS_TOOLCHAIN=aarch64-gcc12` or `CROSS_TOOLCHAIN=llvm15` where necessary. The FreeBSD project also has some CI pipelines on github. For pull requests on github and some branches pushed to the github forks, a number of cross compilation jobs run. These test FreeBSD building using a version of Clang that sometimes lags the in-tree compiler by a major version for a time. The FreeBSD project is also upgrading compilers. Both Clang and GCC are fast moving targets. Some work to change things in the tree, for example removing the old-style K&R function declarations and definitions, will land in the tree prior to the compiler landing. Committers should try to be mindful about this and be receptive to looking into problems with their code or changes with these new compilers. Also, just after a new compiler version hits the tree, people may need to compile things with the old version if there was an undetected regression suspected. In addition to the compiler, LLVM's LLD and GNU's binutils are used indirectly by the compiler. Committers should be mindful of variations in assembler syntax and features of the linkers and ensure both variants work. These components will be tested as part of FreeBSD's CI jobs for Clang or GCC. The FreeBSD project provides headers and libraries that allow other compilers to be used to build software not in the base system. These headers have support for making the environment as strict as the standard, supporting prior dialects of ANSI-C back to C89, and other edge cases our large ports collection has uncovered. This support constrains retirement of older standards in places like header files, but does not constrain updating the base system to newer dialects. Nor does it require the base system to compile with these older standards as a whole. Breaking this support will cause packages in the ports collection to fail, so should be avoided where possible, and promptly fixed when it is easy to do so. The FreeBSD build system currently accommodates these different environments. As new warnings are added to compilers, the project tries to fix them. However, sometimes these warnings require extensive rework, so are suppressed in some way by using make variables that evaluate to the proper thing depending on the compiler version. Developers should be mindful of this, and ensure any compiler specific flags are properly conditionalized. ==== Current Compiler Versions The in-tree compiler is currently Clang 15.x. Currently, GCC 12 and Clang 12, 13, 14 and 15 are tested in the github and project's CI jenkins jobs. Work is underway to get the tree ready for Clang 16. The oldest project supported branch has Clang 12, so the bootstrap portions of the build must work for Clang major versions 12 to 15. === Other Suggestions When committing documentation changes, use a spell checker before committing. For all XML docs, verify that the formatting directives are correct by running `make lint` and package:textproc/igor[]. For manual pages, run package:sysutils/manck[] and package:textproc/igor[] over the manual page to verify all of the cross references and file references are correct and that the man page has all of the appropriate `MLINKS` installed. Do not mix style fixes with new functionality. A style fix is any change which does not modify the functionality of the code. Mixing the changes obfuscates the functionality change when asking for differences between revisions, which can hide any new bugs. Do not include whitespace changes with content changes in commits to [.filename]#doc/#. The extra clutter in the diffs makes the translators' job much more difficult. Instead, make any style or whitespace changes in separate commits that are clearly labeled as such in the commit message. === Deprecating Features When it is necessary to remove functionality from software in the base system, follow these guidelines whenever possible: . Mention is made in the manual page and possibly the release notes that the option, utility, or interface is deprecated. Use of the deprecated feature generates a warning. . The option, utility, or interface is preserved until the next major (point zero) release. . The option, utility, or interface is removed and no longer documented. It is now obsolete. It is also generally a good idea to note its removal in the release notes. === Privacy and Confidentiality . Most FreeBSD business is done in public. + FreeBSD is an _open_ project. Which means that not only can anyone use the source code, but that most of the development process is open to public scrutiny. . Certain sensitive matters must remain private or held under embargo. + There unfortunately cannot be complete transparency. As a FreeBSD developer you will have a certain degree of privileged access to information. Consequently you are expected to respect certain requirements for confidentiality. Sometimes the need for confidentiality comes from external collaborators or has a specific time limit. Mostly though, it is a matter of not releasing private communications. . The Security Officer has sole control over the release of security advisories. + Where there are security problems that affect many different operating systems, FreeBSD frequently depends on early access to be able to prepare advisories for coordinated release. Unless FreeBSD developers can be trusted to maintain security, such early access will not be made available. The Security Officer is responsible for controlling pre-release access to information about vulnerabilities, and for timing the release of all advisories. He may request help under condition of confidentiality from any developer with relevant knowledge to prepare security fixes. . Communications with Core are kept confidential for as long as necessary. + Communications to core will initially be treated as confidential. Eventually however, most of Core's business will be summarized into the monthly or quarterly core reports. Care will be taken to avoid publicising any sensitive details. Records of some particularly sensitive subjects may not be reported on at all and will be retained only in Core's private archives. . Non-disclosure Agreements may be required for access to certain commercially sensitive data. + Access to certain commercially sensitive data may only be available under a Non-Disclosure Agreement. The FreeBSD Foundation legal staff must be consulted before any binding agreements are entered into. . Private communications must not be made public without permission. + Beyond the specific requirements above there is a general expectation not to publish private communications between developers without the consent of all parties involved. Ask permission before forwarding a message onto a public mailing list, or posting it to a forum or website that can be accessed by other than the original correspondents. . Communications on project-only or restricted access channels must be kept private. + Similarly to personal communications, certain internal communications channels, including FreeBSD Committer only mailing lists and restricted access IRC channels are considered private communications. Permission is required to publish material from these sources. . Core may approve publication. + Where it is impractical to obtain permission due to the number of correspondents or where permission to publish is unreasonably withheld, Core may approve release of such private matters that merit more general publication. [[archs]] == Support for Multiple Architectures FreeBSD is a highly portable operating system intended to function on many different types of hardware architectures. Maintaining clean separation of Machine Dependent (MD) and Machine Independent (MI) code, as well as minimizing MD code, is an important part of our strategy to remain agile with regards to current hardware trends. Each new hardware architecture supported by FreeBSD adds substantially to the cost of code maintenance, toolchain support, and release engineering. It also dramatically increases the cost of effective testing of kernel changes. As such, there is strong motivation to differentiate between classes of support for various architectures while remaining strong in a few key architectures that are seen as the FreeBSD "target audience". === Statement of General Intent The FreeBSD Project targets "production quality commercial off-the-shelf (COTS) workstation, server, and high-end embedded systems". By retaining a focus on a narrow set of architectures of interest in these environments, the FreeBSD Project is able to maintain high levels of quality, stability, and performance, as well as minimize the load on various support teams on the project, such as the ports team, documentation team, security officer, and release engineering teams. Diversity in hardware support broadens the options for FreeBSD consumers by offering new features and usage opportunities, but these benefits must always be carefully considered in terms of the real-world maintenance cost associated with additional platform support. The FreeBSD Project differentiates platform targets into four tiers. Each tier includes a list of guarantees consumers may rely on as well as obligations by the Project and developers to fulfill those guarantees. These lists define the minimum guarantees for each tier. The Project and developers may provide additional levels of support beyond the minimum guarantees for a given tier, but such additional support is not guaranteed. Each platform target is assigned to a specific tier for each stable branch. As a result, a platform target might be assigned to different tiers on concurrent stable branches. === Platform Targets Support for a hardware platform consists of two components: kernel support and userland Application Binary Interfaces (ABIs). Kernel platform support includes things needed to run a FreeBSD kernel on a hardware platform such as machine-dependent virtual memory management and device drivers. A userland ABI specifies an interface for user processes to interact with a FreeBSD kernel and base system libraries. A userland ABI includes system call interfaces, the layout and semantics of public data structures, and the layout and semantics of arguments passed to subroutines. Some components of an ABI may be defined by specifications such as the layout of C++ exception objects or calling conventions for C functions. A FreeBSD kernel also uses an ABI (sometimes referred to as the Kernel Binary Interface (KBI)) which includes the semantics and layouts of public data structures and the layout and semantics of arguments to public functions within the kernel itself. A FreeBSD kernel may support multiple userland ABIs. For example, FreeBSD's amd64 kernel supports FreeBSD amd64 and i386 userland ABIs as well as Linux x86_64 and i386 userland ABIs. A FreeBSD kernel should support a "native" ABI as the default ABI. The native "ABI" generally shares certain properties with the kernel ABI such as the C calling convention, sizes of basic types, etc. Tiers are defined for both kernels and userland ABIs. In the common case, a platform's kernel and FreeBSD ABIs are assigned to the same tier. === Tier 1: Fully-Supported Architectures Tier 1 platforms are the most mature FreeBSD platforms. They are supported by the security officer, release engineering, and Ports Management Team. Tier 1 architectures are expected to be Production Quality with respect to all aspects of the FreeBSD operating system, including installation and development environments. The FreeBSD Project provides the following guarantees to consumers of Tier 1 platforms: * Official FreeBSD release images will be provided by the release engineering team. * Binary updates and source patches for Security Advisories and Errata Notices will be provided for supported releases. * Source patches for Security Advisories will be provided for supported branches. * Binary updates and source patches for cross-platform Security Advisories will typically be provided at the time of the announcement. * Changes to userland ABIs will generally include compatibility shims to ensure correct operation of binaries compiled against any stable branch where the platform is Tier 1. These shims might not be enabled in the default install. If compatibility shims are not provided for an ABI change, the lack of shims will be clearly documented in the release notes. * Changes to certain portions of the kernel ABI will include compatibility shims to ensure correct operation of kernel modules compiled against the oldest supported release on the branch. Note that not all parts of the kernel ABI are protected. * Official binary packages for third party software will be provided by the ports team. For embedded architectures, these packages may be cross-built from a different architecture. * Most relevant ports should either build or have the appropriate filters to prevent inappropriate ones from building. * New features which are not inherently platform-specific will be fully functional on all Tier 1 architectures. * Features and compatibility shims used by binaries compiled against older stable branches may be removed in newer major versions. Such removals will be clearly documented in the release notes. * Tier 1 platforms should be fully documented. Basic operations will be documented in the FreeBSD Handbook. * Tier 1 platforms will be included in the source tree. * Tier 1 platforms should be self-hosting either via the in-tree toolchain or an external toolchain. If an external toolchain is required, official binary packages for an external toolchain will be provided. To maintain maturity of Tier 1 platforms, the FreeBSD Project will maintain the following resources to support development: * Build and test automation support either in the FreeBSD.org cluster or some other location easily available for all developers. Embedded platforms may substitute an emulator available in the FreeBSD.org cluster for actual hardware. * Inclusion in the `make universe` and `make tinderbox` targets. * Dedicated hardware in one of the FreeBSD clusters for package building (either natively or via qemu-user). Collectively, developers are required to provide the following to maintain the Tier 1 status of a platform: * Changes to the source tree should not knowingly break the build of a Tier 1 platform. * Tier 1 architectures must have a mature, healthy ecosystem of users and active developers. * Developers should be able to build packages on commonly available, non-embedded Tier 1 systems. This can mean either native builds if non-embedded systems are commonly available for the platform in question, or it can mean cross-builds hosted on some other Tier 1 architecture. * Changes cannot break the userland ABI. If an ABI change is required, ABI compatibility for existing binaries should be provided via use of symbol versioning or shared library version bumps. * Changes merged to stable branches cannot break the protected portions of the kernel ABI. If a kernel ABI change is required, the change should be modified to preserve functionality of existing kernel modules. === Tier 2: Developmental and Niche Architectures Tier 2 platforms are functional, but less mature FreeBSD platforms. They are not supported by the security officer, release engineering, and Ports Management Team. Tier 2 platforms may be Tier 1 platform candidates that are still under active development. Architectures reaching end of life may also be moved from Tier 1 status to Tier 2 status as the availability of resources to continue to maintain the system in a Production Quality state diminishes. Well-supported niche architectures may also be Tier 2. The FreeBSD Project provides the following guarantees to consumers of Tier 2 platforms: * The ports infrastructure should include basic support for Tier 2 architectures sufficient to support building ports and packages. This includes support for basic packages such as ports-mgmt/pkg, but there is no guarantee that arbitrary ports will be buildable or functional. * New features which are not inherently platform-specific should be feasible on all Tier 2 architectures if not implemented. * Tier 2 platforms will be included in the source tree. * Tier 2 platforms should be self-hosting either via the in-tree toolchain or an external toolchain. If an external toolchain is required, official binary packages for an external toolchain will be provided. * Tier 2 platforms should provide functional kernels and userlands even if an official release distribution is not provided. To maintain maturity of Tier 2 platforms, the FreeBSD Project will maintain the following resources to support development: * Inclusion in the `make universe` and `make tinderbox` targets. Collectively, developers are required to provide the following to maintain the Tier 2 status of a platform: * Changes to the source tree should not knowingly break the build of a Tier 2 platform. * Tier 2 architectures must have an active ecosystem of users and developers. * While changes are permitted to break the userland ABI, the ABI should not be broken gratuitously. Significant userland ABI changes should be restricted to major versions. * New features that are not yet implemented on Tier 2 architectures should provide a means of disabling them on those architectures. === Tier 3: Experimental Architectures Tier 3 platforms have at least partial FreeBSD support. They are _not_ supported by the security officer, release engineering, and Ports Management Team. Tier 3 platforms are architectures in the early stages of development, for non-mainstream hardware platforms, or which are considered legacy systems unlikely to see broad future use. Initial support for Tier 3 platforms may exist in a separate repository rather than the main source repository. The FreeBSD Project provides no guarantees to consumers of Tier 3 platforms and is not committed to maintaining resources to support development. Tier 3 platforms may not always be buildable, nor are any kernel or userland ABIs considered stable. === Unsupported Architectures Other platforms are not supported in any form by the project. The project previously described these as Tier 4 systems. After a platform transitions to unsupported, all support for the platform is removed from the source, ports and documentation trees. Note that ports support should remain as long as the platform is supported in a branch supported by ports. === Policy on Changing the Tier of an Architecture Systems may only be moved from one tier to another by approval of the FreeBSD Core Team, which shall make that decision in collaboration with the Security Officer, Release Engineering, and ports management teams. For a platform to be promoted to a higher tier, any missing support guarantees must be satisfied before the promotion is completed. [[ports]] == Ports Specific FAQ [[ports-qa-adding]] === Adding a New Port [[ports-qa-add-new]] ==== How do I add a new port? Adding a port to the tree is relatively simple. Once the port is ready to be added, as explained later <>, you need to add the port's directory entry in the category's [.filename]#Makefile#. In this [.filename]#Makefile#, ports are listed in alphabetical order and added to the `SUBDIR` variable, like this: [.programlisting] .... SUBDIR += newport .... Once the port and its category's Makefile are ready, the new port can be committed: [source,shell] .... % git add category/Makefile category/newport % git commit % git push .... [TIP] ==== Don't forget to <>; a specific hook has been developed to verify the category's [.filename]#Makefile#. ==== [[ports-qa-add-new-extra]] ==== Any other things I need to know when I add a new port? Check the port, preferably to make sure it compiles and packages correctly. The extref:{porters-handbook}testing[Porters Handbook's Testing Chapter] contains more detailed instructions. See the extref:{porters-handbook}testing[Portclippy / Portfmt, testing-portclippy] and the extref:{porters-handbook}testing[Poudriere, testing-poudriere] sections. You do not necessarily have to eliminate all warnings but make sure you have fixed the simple ones. If the port came from a submitter who has not contributed to the Project before, add that person's name to the extref:{contributors}[Additional Contributors, contrib-additional] section of the FreeBSD Contributors List. Close the PR if the port came in as a PR. To close a PR, change the state to `Issue Resolved` and the resolution as `Fixed`. [NOTE] ==== If for some reason using extref:{porters-handbook}testing[Poudriere, testing-poudriere] to test the new port is not possible, the bare minimum of testing includes this sequence: [source,shell] .... # make install # make package # make deinstall # pkg add package you built above # make deinstall # make reinstall # make package .... Note that poudriere is the reference for package building, it the port does not build in poudriere, it will be removed. ==== [[ports-qa-removing]] === Removing an Existing Port [[ports-qa-remove-one]] ==== How do I remove an existing port? First, please read the section about repository copies. Before you remove the port, you have to verify there are no other ports depending on it. * Make sure there is no dependency on the port in the ports collection: ** The port's PKGNAME appears in exactly one line in a recent INDEX file. ** No other ports contains any reference to the port's directory or PKGNAME in their Makefiles + [TIP] ==== When using Git, consider using man:git-grep[1], it is much faster than `grep -r`. ==== + * Then, remove the port: + [.procedure] ==== * Remove the port's files and directory with `git rm`. * Remove the `SUBDIR` listing of the port in the parent directory [.filename]#Makefile#. * Add an entry to [.filename]#ports/MOVED#. * Remove the port from [.filename]#ports/LEGAL# if it is there. ==== Alternatively, you can use the rmport script, from [.filename]#ports/Tools/scripts#. This script was written by {vd}. When sending questions about this script to the {freebsd-ports}, please also CC {crees}, the current maintainer. [[ports-qa-move-port]] === How do I move a port to a new location? [.procedure] ==== . Perform a thorough check of the ports collection for any dependencies on the old port location/name, and update them. Running `grep` on [.filename]#INDEX# is not enough because some ports have dependencies enabled by compile-time options. A full man:git-grep[1] of the ports collection is recommended. . Remove the `SUBDIR` entry from the old category Makefile and add a `SUBDIR` entry to the new category Makefile. . Add an entry to [.filename]#ports/MOVED#. . Search for entries in xml files inside [.filename]#ports/security/vuxml# and adjust them accordingly. In particular, check for previous packages with the new name which version could include the new port. . Move the port with `git mv`. . Commit the changes. ==== [[ports-qa-copy-port]] === How do I copy a port to a new location? [.procedure] ==== . Copy port with `cp -R old-cat/old-port new-cat/new-port`. . Add the new port to the [.filename]#new-cat/Makefile#. . Change stuff in [.filename]#new-cat/new-port#. . Commit the changes. ==== [[ports-qa-freeze]] === Ports Freeze [[ports-qa-freeze-what]] ==== What is a “ports freeze”? A “ports freeze” was a restricted state the ports tree was put in before a release. It was used to ensure a higher quality for the packages shipped with a release. It usually lasted a couple of weeks. During that time, build problems were fixed, and the release packages were built. This practice is no longer used, as the packages for the releases are built from the current stable, quarterly branch. For more information on how to merge commits to the quarterly branch, see <>. [[ports-qa-quarterly]] === Quarterly Branches [[ports-qa-misc-request-mfh]] ==== What is the procedure to request authorization for merging a commit to the quarterly branch? As of November 30, 2020, there is no need to seek explicit approval to commit to the quarterly branch. [[ports-qa-misc-commit-mfh]] ==== What is the procedure for merging commits to the quarterly branch? Merging commits to the quarterly branch (a process we call MFH for a historical reason) is very similar to MFC'ing a commit in the src repository, so basically: [source,shell] .... % git checkout 2021Q2 % git cherry-pick -x $HASH (verify everything is OK, for example by doing a build test) % git push .... where `$HASH` is the hash of the commit you want to copy over to the quarterly branch. The `-x` parameter ensures the hash `$HASH` of the `main` branch is included in the new commit message of the quarterly branch. [[ports-qa-new-category]] === Creating a New Category [[ports-qa-new-category-how]] ==== What is the procedure for creating a new category? Please see extref:{porters-handbook}[Proposing a New Category, proposing-categories] in the Porter's Handbook. Once that procedure has been followed and the PR has been assigned to the {portmgr}, it is their decision whether or not to approve it. If they do, it is their responsibility to: [.procedure] ==== . Perform any needed moves. (This only applies to physical categories.) . Update the `VALID_CATEGORIES` definition in [.filename]#ports/Mk/bsd.port.mk#. . Assign the PR back to you. ==== [[ports-qa-new-category-physical]] ==== What do I need to do to implement a new physical category? [.procedure] ==== . Upgrade each moved port's [.filename]#Makefile#. Do not connect the new category to the build yet. + To do this, you will need to: + [.procedure] ====== . Change the port's `CATEGORIES` (this was the point of the exercise, remember?) The new category is listed first. This will help to ensure that the PKGORIGIN is correct. . Run a `make describe`. Since the top-level `make index` that you will be running in a few steps is an iteration of `make describe` over the entire ports hierarchy, catching any errors here will save you having to re-run that step later on. . If you want to be really thorough, now might be a good time to run man:portlint[1]. ====== + . Check that the ``PKGORIGIN``s are correct. The ports system uses each port's `CATEGORIES` entry to create its `PKGORIGIN`, which is used to connect installed packages to the port directory they were built from. If this entry is wrong, common port tools like man:pkg-version[8] and man:portupgrade[1] fail. + To do this, use the [.filename]#chkorigin.sh# tool: `env PORTSDIR=/path/to/ports sh -e /path/to/ports/Tools/scripts/chkorigin.sh`. This will check every port in the ports tree, even those not connected to the build, so you can run it directly after the move operation. Hint: do not forget to look at the ``PKGORIGIN``s of any slave ports of the ports you just moved! . On your own local system, test the proposed changes: first, comment out the SUBDIR entries in the old ports' categories' [.filename]##Makefile##s; then enable building the new category in [.filename]#ports/Makefile#. Run make checksubdirs in the affected category directories to check the SUBDIR entries. Next, in the [.filename]#ports/# directory, run make index. This can take over 40 minutes on even modern systems; however, it is a necessary step to prevent problems for other people. . Once this is done, you can commit the updated [.filename]#ports/Makefile# to connect the new category to the build and also commit the [.filename]#Makefile# changes for the old category or categories. . Add appropriate entries to [.filename]#ports/MOVED#. . Update the documentation by modifying: ** the extref:{porters-handbook}[list of categories, PORTING-CATEGORIES] in the Porter's Handbook + . Only once all the above have been done, and no one is any longer reporting problems with the new ports, should the old ports be deleted from their previous locations in the repository. ==== ==== What do I need to do to implement a new virtual category? This is much simpler than a physical category. Only a few modifications are needed: * the extref:{porters-handbook}[list of categories, PORTING-CATEGORIES] in the Porter's Handbook [[ports-qa-misc-questions]] === Miscellaneous Questions [[ports-qa-misc-blanket-approval]] ==== Are there changes that can be committed without asking the maintainer for approval? Blanket approval for most ports applies to these types of fixes: * Most infrastructure changes to a port (that is, modernizing, but not changing the functionality). For example, the blanket covers converting to new `USES` macros, enabling verbose builds, and switching to new ports system syntaxes. * Trivial and _tested_ build and runtime fixes. * Documentations or metadata changes to ports, like [.filename]#pkg-descr# or `COMMENT`. [IMPORTANT] ==== Exceptions to this are anything maintained by the {portmgr}, or the {security-officer}. No unauthorized commits may ever be made to ports maintained by those groups. ==== [[ports-qa-misc-correctly-building]] ==== How do I know if my port is building correctly or not? The packages are built multiple times each week. If a port fails, the maintainer will receive an email from `pkg-fallout@FreeBSD.org`. Reports for all the package builds (official, experimental, and non-regression) are aggregated at link:pkg-status.FreeBSD.org[pkg-status.FreeBSD.org]. [[ports-qa-misc-INDEX]] ==== I added a new port. Do I need to add it to the [.filename]#INDEX#? No. The file can either be generated by running `make index`, or a pre-generated version can be downloaded with `make fetchindex`. [[ports-qa-misc-no-touch]] ==== Are there any other files I am not allowed to touch? Any file directly under [.filename]#ports/#, or any file under a subdirectory that starts with an uppercase letter ([.filename]#Mk/#, [.filename]#Tools/#, etc.). In particular, the {portmgr} is very protective of [.filename]#ports/Mk/bsd.port*.mk# so do not commit changes to those files unless you want to face their wrath. [[ports-qa-misc-updated-distfile]] ==== What is the proper procedure for updating the checksum for a port distfile when the file changes without a version change? When the checksum for a distribution file is updated due to the author updating the file without changing the port revision, the commit message includes a summary of the relevant diffs between the original and new distfile to ensure that the distfile has not been corrupted or maliciously altered. If the current version of the port has been in the ports tree for a while, a copy of the old distfile will usually be available on the ftp servers; otherwise the author or maintainer should be contacted to find out why the distfile has changed. [[ports-exp-run]] ==== How can an experimental test build of the ports tree (exp-run) be requested? An exp-run must be completed before patches with a significant ports impact are committed. The patch can be against the ports tree or the base system. Full package builds will be done with the patches provided by the submitter, and the submitter is required to fix detected problems _(fallout)_ before commit. [.procedure] ==== . Go to the link:https://bugs.freebsd.org/submit[Bugzilla new PR page]. . Select the product your patch is about. . Fill in the bug report as normal. Remember to attach the patch. . If at the top it says “Show Advanced Fields” click on it. It will now say “Hide Advanced Fields”. Many new fields will be available. If it already says “Hide Advanced Fields”, no need to do anything. . In the “Flags” section, set the “exp-run” one to `?`. As for all other fields, hovering the mouse over any field shows more details. . Submit. Wait for the build to run. . {portmgr} will reply with a possible fallout. . Depending on the fallout: ** If there is no fallout, the procedure stops here, and the change can be committed, pending any other approval required. ... If there is fallout, it _must_ be fixed, either by fixing the ports directly in the ports tree, or adding to the submitted patch. ... When this is done, go back to step 6 saying the fallout was fixed and wait for the exp-run to be run again. Repeat as long as there are broken ports. ==== [[non-committers]] == Issues Specific to Developers Who Are Not Committers A few people who have access to the FreeBSD machines do not have commit bits. Almost all of this document will apply to these developers as well (except things specific to commits and the mailing list memberships that go with them). In particular, we recommend that you read: * <> * <> + [NOTE] ==== Get your mentor to add you to the "Additional Contributors" ([.filename]#doc/shared/contrib-additional.adoc#), if you are not already listed there. ==== * <> * <> * <> [[google-analytics]] == Information About Google Analytics As of December 12, 2012, Google Analytics was enabled on the FreeBSD Project website to collect anonymized usage statistics regarding usage of the site. [NOTE] ==== As of March 3, 2022, Google Analytics was removed from the FreeBSD Project. ==== [[misc]] == Miscellaneous Questions === How do I access people.FreeBSD.org to put up personal or project information? `people.FreeBSD.org` is the same as `freefall.FreeBSD.org`. Just create a [.filename]#public_html# directory. Anything you place in that directory will automatically be visible under https://people.FreeBSD.org/[https://people.FreeBSD.org/]. === Where are the mailing list archives stored? The mailing lists are archived under [.filename]#/local/mail# on `freefall.FreeBSD.org`. === I would like to mentor a new committer. What process do I need to follow? See the https://www.freebsd.org/internal/new-account/[New Account Creation Procedure] document on the internal pages. [[benefits]] == Benefits and Perks for FreeBSD Committers [[benefits-recognition]] === Recognition Recognition as a competent software engineer is the longest lasting value. In addition, getting a chance to work with some of the best people that every engineer would dream of meeting is a great perk! [[benefits-freebsdmall]] === FreeBSD Mall FreeBSD committers can get a free 4-CD or DVD set at conferences from http://www.freebsdmall.com[FreeBSD Mall, Inc.]. [[benefits-gandi]] === `Gandi.net` https://gandi.net[Gandi] provides website hosting, cloud computing, domain registration, and X.509 certificate services. Gandi offers an E-rate discount to all FreeBSD developers. -In order to streamline the process of getting the discount first set up a Gandi account, fill in the billing information and select the currency. +To streamline the process of getting the discount first set up a Gandi account, fill in the billing information and select the currency. Then send an mail to mailto:non-profit@gandi.net[non-profit@gandi.net] using your `@freebsd.org` mail address, and indicate your Gandi handle. [[benefits-rsync]] === `rsync.net` https://rsync.net[rsync.net] provides cloud storage for offsite backup that is optimized for UNIX users. Their service runs entirely on FreeBSD and ZFS. rsync.net offers a free-forever 500 GB account to FreeBSD developers. Simply sign up at https://www.rsync.net/freebsd.html[https://www.rsync.net/freebsd.html] using your `@freebsd.org` address to receive this free account. [[benefits-jetbrains]] === `JetBrains` https://www.jetbrains.com[JetBrains] is a software development company which makes tools for software developers and project managers. The company offers many integrated development environments (IDEs) for different programming languages. JetBrains offers 100 free yearly licenses for all https://www.jetbrains.com/products[JetBrains IDE products]. Simply sign up at https://account.jetbrains.com/a/322tl3z7[https://account.jetbrains.com/a/322tl3z7] using your `@freebsd.org` address and the account will have a license attached to it automatically. Once the account is active, use it in any of the products to activate them and you're done. [IMPORTANT] ==== Please, only use these licences personally and do not share them with anyone outside of the FreeBSD project, as that would be a violation of the terms of the donation. ==== diff --git a/documentation/content/en/articles/contributing/_index.adoc b/documentation/content/en/articles/contributing/_index.adoc index f4d6a9c657..ceefd904e5 100644 --- a/documentation/content/en/articles/contributing/_index.adoc +++ b/documentation/content/en/articles/contributing/_index.adoc @@ -1,598 +1,598 @@ --- title: Contributing to FreeBSD authors: - author: Jordan Hubbard - author: Sam Lawrance - author: Mark Linimon description: How to contribute to the FreeBSD Project trademarks: ["freebsd", "ieee", "general"] weight: 15 tags: ["Contributing", "FreeBSD", "Non-Programmer Tasks", "Programmer Tasks"] --- = Contributing to FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/contributing/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract This article describes the different ways in which an individual or organization may contribute to the FreeBSD Project. ''' toc::[] So you want to contribute to FreeBSD? That is great! FreeBSD _relies_ on the contributions of its user base to survive. Your contributions are not only appreciated, they are vital to FreeBSD's continued growth. A large and growing number of international contributors, of greatly varying ages and areas of technical expertise, develop FreeBSD. There is always more work to be done than there are people available to do it, and more help is always appreciated. As a volunteer, what you do is limited only by what you want to do. However, we do ask that you are aware of what other members of the FreeBSD community will expect of you. You may want to take this into account before deciding to volunteer. The FreeBSD project is responsible for an entire operating system environment, rather than just a kernel or a few scattered utilities. As such, our [.filename]#TODO# lists span a very wide range of tasks: from documentation, beta testing and presentation, to the system installer and highly specialized types of kernel development. People of any skill level, in almost any area, can almost certainly help the project. Commercial entities engaged in FreeBSD-related enterprises are also encouraged to contact us. Do you need a special extension to make your product work? You will find us receptive to your requests, given that they are not too outlandish. Are you working on a value-added product? Please let us know! We may be able to work cooperatively on some aspect of it. The free software world is challenging many existing assumptions about how software is developed, sold, and maintained, and we urge you to at least give it a second look. [[contrib-what]] == What Is Needed The following list of tasks and sub-projects represents something of an amalgam of various [.filename]#TODO# lists and user requests. [[non-programmer-tasks]] === Ongoing Non-Programmer Tasks Many people who are involved in FreeBSD are not programmers. The Project includes documentation writers, Web designers, and support people. All that these people need to contribute is an investment of time and a willingness to learn. . Read through the FAQ and Handbook periodically. If anything is poorly explained, ambiguous, out of date or incorrect, let us know. Even better, send us a fix (AsciiDoc is not difficult to learn, but there is no objection to plain text submissions). . Help translate FreeBSD documentation into your native language. If documentation already exists for your language, you can help translate additional documents or verify that the translations are up-to-date and correct. First take a look at the extref:{fdp-primer}[Translations FAQ, translations] in the FreeBSD Documentation Project Primer. You are not committing yourself to translating every single FreeBSD document by doing this - as a volunteer, you can do as much or as little translation as you desire. Once someone begins translating, others almost always join the effort. If you only have the time or energy to translate one part of the documentation, please translate the installation instructions. . Read the {freebsd-questions} occasionally (or even regularly). It can be very satisfying to share your expertise and help people solve their problems; sometimes you may even learn something new yourself! These forums can also be a source of ideas for things to improve upon. [[ongoing-programmer-tasks]] === Ongoing Programmer Tasks Most of the tasks listed here may require a considerable investment of time, an in-depth knowledge of the FreeBSD kernel, or both. However, there are also many useful tasks which are suitable for "weekend hackers". . If you run FreeBSD-CURRENT and have a good Internet connection, there is a machine `current.FreeBSD.org` which builds a full release once a day-every now and again, try to install the latest release from it and report any failures in the process. . Read the {freebsd-bugs}. There may be a problem you can comment constructively on or with patches you can test. Or you could even try to fix one of the problems yourself. . If you know of any bug fixes which have been successfully applied to -CURRENT but have not been merged into -STABLE after a decent interval (normally a couple of weeks), send the committer a polite reminder. . Move contributed software to [.filename]#src/contrib# in the source tree. . Make sure code in [.filename]#src/contrib# is up to date. . Build the source tree (or just part of it) with extra warnings enabled and clean up the warnings. A list of build warnings can also be found from our https://ci.freebsd.org[CI] by selecting a build and checking "LLVM/Clang Warnings". . Fix warnings for ports which do deprecated things like using `gets()` or including [.filename]#malloc.h#. . If you have contributed any ports and you had to make FreeBSD-specific changes, send your patches back to the original authors (this will make your life easier when they bring out the next version). . Get copies of formal standards like POSIX(R). Compare FreeBSD's behavior to that required by the standard. If the behavior differs, particularly in subtle or obscure corners of the specification, send in a PR about it. If you are able, figure out how to fix it and include a patch in the PR. If you think the standard is wrong, ask the standards body to consider the question. . Suggest further tasks for this list! === Work through the PR Database The https://bugs.FreeBSD.org/search/[FreeBSD PR list] shows all the current active problem reports and requests for enhancement that have been submitted by FreeBSD users. The PR database includes both programmer and non-programmer tasks. Look through the open PRs, and see if anything there takes your interest. Some of these might be very simple tasks that just need an extra pair of eyes to look over them and confirm that the fix in the PR is a good one. Others might be much more complex, or might not even have a fix included at all. Start with the PRs that have not been assigned to anyone else. If a PR is assigned to someone else, but it looks like something you can handle, email the person it is assigned to and ask if you can work on it-they might already have a patch ready to be tested, or further ideas that you can discuss with them. === Ongoing Ports Tasks The Ports Collection is a perpetual work in progress. We want to provide our users with an easy to use, up to date, high quality repository of third party software. We need people to donate some of their time and effort to help us achieve this goal. Anyone can get involved, and there are lots of different ways to do so. Contributing to ports is an excellent way to help "give back" something to the project. Whether you are looking for an ongoing role, or a fun challenge for a rainy day, we would love to have your help! There are a number of easy ways you can contribute to keeping the ports tree up to date and in good working order: * Find some cool or useful software and extref:{porters-handbook}[create a port] for it. * There are a large number of ports that have no maintainer. Become a maintainer and <>. * If you have created or adopted a port, be aware of <>. * When you are looking for a quick challenge you could <>. === Pick one of the items from the Ideas page The https://wiki.freebsd.org/IdeasPage[FreeBSD list of projects and ideas for volunteers] is also available for people willing to contribute to the FreeBSD project. The list is being regularly updated and contains items for both programmers and non-programmers with information about each project. [[contrib-how]] == How to Contribute Contributions to the system generally fall into one or more of the following 5 categories: [[contrib-general]] === Bug Reports and General Commentary An idea or suggestion of _general_ technical interest should be mailed to the {freebsd-hackers}. Likewise, people with an interest in such things (and a tolerance for a _high_ volume of mail!) may subscribe to the {freebsd-hackers}. See extref:{handbook}[The FreeBSD Handbook, eresources-mail] for more information about this and other mailing lists. If you are submitting a simple patch to the src repo, please consider submitting it to the project's GitHub mirror as https://github.com/freebsd/freebsd-src/pulls[a pull request]. Suitable submissions should: * It is ready or nearly ready to be committed. A committer should be able to land this patch with less than 10 minutes of additional work. * It passes all the GitHub CI jobs. * You can respond to feedback quickly. * It touches fewer than about 10 files and the changes are less than about 200 lines. Changes larger than this may be OK, or you may be asked to submit multiple pull requests of a more manageable size. * Each logical change is a separate commit within the pull request. Commit messages for each change should follow extref:{committers-guide}#commit-log-message[commit log guide]. * All commits have your name and valid email address as you'd like to see them in the FreeBSD repository as the author. Fake github.com addresses cannot be used. * The scope of the pull request should not change during review. If the review suggests changes that expand the scope, please create an independent pull request. * Fixup commits should be squashed with the commit they are fixing. Each commit in your branch should be suitable for FreeBSD's repository. * Commits should include one or more `Signed-off-by:` lines with full name and email address certifying https://developercertificate.org/[Developer Certificate of Origin]. When updating pull request, please rebase with a forced push rather than a merge commit. More complex changes may be submitted as pull requests, but they may be closed if they are too large, too unwieldy, become inactive, need further discussion in the community, or need extensive revision. Please avoid creating large, wide-ranging cleanup patches: they are too large and lack the focus needed for a good review. Misdirected patches may be redirected to a more appropriate forum for the patch to be resolved. Pull requests submitted to the ports repository may or may not see action, based on the whims of developers. For now, you will have a better experience if you follow the ports submission process <>. The docs team also accepts pull requests via GitHub, but has not established any policy for them yet. If you find a bug or are submitting a specific change, please report it using the https://bugs.FreeBSD.org/submit/[bug submission form]. Try to fill-in each field of the bug report. Unless they exceed 65KB, include any patches directly in the report. If the patch is suitable to be applied to the source tree put `[PATCH]` in the synopsis of the report. When including patches, _do not_ use cut-and-paste because cut-and-paste turns tabs into spaces and makes them unusable. When patches are a lot larger than 20KB, consider compressing them (eg. with man:gzip[1] or man:bzip2[1]) prior to uploading them. After filing a report, you should receive confirmation along with a tracking number. Keep this tracking number so that you can update us with details about the problem. See also extref:{problem-reports}[this article] on how to write good problem reports. === Changes to the Documentation Changes to the documentation are overseen by the {freebsd-doc}. Please look at the extref:{fdp-primer}[FreeBSD Documentation Project Primer] for complete instructions. Send submissions and changes (even small ones are welcome!) using the same method as any other bug report. === Changes to Existing Source Code An addition or change to the existing source code is a somewhat trickier affair and depends a lot on how far out of date you are with the current state of FreeBSD development. There is a special on-going release of FreeBSD known as "FreeBSD-CURRENT" which is made available in a variety of ways for the convenience of developers working actively on the system. See extref:{handbook}[The FreeBSD Handbook, current-stable] for more information about getting and using FreeBSD-CURRENT. Working from older sources unfortunately means that your changes may sometimes be too obsolete or too divergent for easy re-integration into FreeBSD. Chances of this can be minimized somewhat by subscribing to the {freebsd-announce} and the {freebsd-current} lists, where discussions on the current state of the system take place. Assuming that you can manage to secure fairly up-to-date sources to base your changes on, the next step is to produce a set of diffs to send to the FreeBSD maintainers. This is done with the man:diff[1] command. The preferred man:diff[1] format for submitting patches is the unified output format generated by `diff -u`. [source,shell] .... % diff -u oldfile newfile .... or [source,shell] .... % diff -u -r -N olddir newdir .... would generate a set of unified diffs for the given source file or directory hierarchy. See man:diff[1] for more information. Once you have a set of diffs (which you may test with the man:patch[1] command), you should submit them for inclusion with FreeBSD as a bug report. _Do not_ just send the diffs to the {freebsd-hackers} or they will get lost! We greatly appreciate your submission (this is a volunteer project!); because we are busy, we may not be able to address it immediately, but it will remain in the PR database until we do. Indicate your submission by including `[PATCH]` in the synopsis of the report. If you feel it appropriate (e.g. you have added, deleted, or renamed files), bundle your changes into a `tar` file. Archives created with man:shar[1] are also welcome. If your change is of a potentially sensitive nature, such as if you are unsure of copyright issues governing its further distribution then you should send it to {core-email} directly rather than submitting as a bug report. The {core-email} reaches a much smaller group of people who do much of the day-to-day work on FreeBSD. Note that this group is also _very busy_ and so you should only send mail to them where it is truly necessary. Please refer to man:intro[9] and man:style[9] for some information on coding style. We would appreciate it if you were at least aware of this information before submitting code. === New Code or Major Value-Added Packages In the case of a significant contribution of a large body work, or the addition of an important new feature to FreeBSD, it becomes almost always necessary to either send changes as tar files or upload them to a web or FTP site for other people to access. If you do not have access to a web or FTP site, ask on an appropriate FreeBSD mailing list for someone to host the changes for you. When working with large amounts of code, the touchy subject of copyrights also invariably comes up. FreeBSD prefers free software licenses such as BSD or ISC. Copyleft licenses such as GPLv2 are sometimes permitted. The complete listing can be found on the link:https://www.FreeBSD.org/internal/software-license/[core team licensing policy] page. === Money or Hardware We are always very happy to accept donations to further the cause of the FreeBSD Project and, in a volunteer effort like ours, a little can go a long way! Donations of hardware are also very important to expanding our list of supported peripherals since we generally lack the funds to buy such items ourselves. [[donations]] ==== Donating Funds The https://www.freebsdfoundation.org[FreeBSD Foundation] is a non-profit, tax-exempt foundation established to further the goals of the FreeBSD Project. As a 501(c)3 entity, the Foundation is generally exempt from US federal income tax as well as Colorado State income tax. Donations to a tax-exempt entity are often deductible from taxable federal income. Donations may be sent in check form to: [.address] **** The FreeBSD Foundation + 3980 Broadway Street + STE #103-107 + Boulder CO 80304 + USA **** The FreeBSD Foundation is also able to accept https://www.freebsdfoundation.org/donate/[online donations] through various payment options. More information about the FreeBSD Foundation can be found in https://people.FreeBSD.org/~jdp/foundation/announcement.html[The FreeBSD Foundation -- an Introduction]. To contact the Foundation by email, write to mailto:info@FreeBSDFoundation.org[info@FreeBSDFoundation.org]. ==== Donating Hardware The FreeBSD Project happily accepts donations of hardware that it can find good use for. If you are interested in donating hardware, please contact the link:https://www.FreeBSD.org/donations/[Donations Liaison Office]. [[ports-contributing]] == Contributing to ports [[adopt-port]] === Adopting an unmaintained port ==== Choosing an unmaintained port Taking over maintainership of ports that are unmaintained is a great way to get involved. Unmaintained ports are only updated and fixed when somebody volunteers to work on them. There are a large number of unmaintained ports. It is a good idea to start with adopting a port that you use regularly. Unmaintained ports have their `MAINTAINER` set to `ports@FreeBSD.org`. Many unmaintained ports can have pending updates, this can be seen at the https://portscout.freebsd.org/ports@freebsd.org.html[FreeBSD Ports distfile scanner]. On https://portsfallout.com/fallout?port=&maintainer=ports%40FreeBSD.org[PortsFallout] can be seen a list of unmaintained ports with errors. Some ports affect a large number of others due to dependencies and slave port relationships. Generally, we want people to have some experience before they maintain such ports. You can find out whether or not a port has dependencies or slave ports by looking at a master index of ports called [.filename]#INDEX#. (The name of the file varies by release of FreeBSD; for instance, [.filename]#INDEX-8#.) Some ports have conditional dependencies that are not included in a default [.filename]#INDEX# build. We expect you to be able to recognize such ports by looking through other ports' [.filename]#Makefile#'s. ==== How to adopt the port First make sure you understand your <>. Also read the extref:{porters-handbook}[Porter's Handbook]. _Please do not commit yourself to more than you feel you can comfortably handle._ You may request maintainership of any unmaintained port as soon as you wish. Simply set `MAINTAINER` to your own email address and send a PR (Problem Report) with the change. If the port has build errors or needs updating, you may wish to include any other changes in the same PR. This will help because many committers are less willing to assign maintainership to someone who does not have a known track record with FreeBSD. Submitting PRs that fix build errors or update ports are the best ways to establish one. File your PR with category `ports` and class `change-request`. A committer will examine your PR, commit the changes, and finally close the PR. Sometimes this process can take a little while (committers are volunteers, too :). [[maintain-port]] === The challenge for port maintainers This section will give you an idea of why ports need to be maintained and outline the responsibilities of a port maintainer. [[why-maintenance]] ==== Why ports require maintenance Creating a port is a once-off task. Ensuring that a port is up to date and continues to build and run requires an ongoing maintenance effort. Maintainers are the people who dedicate some of their time to meeting these goals. The foremost reason ports need maintenance is to bring the latest and greatest in third party software to the FreeBSD community. An additional challenge is to keep individual ports working within the Ports Collection framework as it evolves. As a maintainer, you will need to manage the following challenges: -* *New software versions and updates.* New versions and updates of existing ported software become available all the time, and these need to be incorporated into the Ports Collection in order to provide up-to-date software. +* *New software versions and updates.* New versions and updates of existing ported software become available all the time, and these need to be incorporated into the Ports Collection to provide up-to-date software. * *Changes to dependencies.* If significant changes are made to the dependencies of your port, it may need to be updated so that it will continue to work correctly. * *Changes affecting dependent ports.* If other ports depend on a port that you maintain, changes to your port may require coordination with other maintainers. * *Interaction with other users, maintainers and developers.* Part of being a maintainer is taking on a support role. You are not expected to provide general support (but we welcome it if you choose to do so). What you should provide is a point of coordination for FreeBSD-specific issues regarding your ports. * *Bug hunting.* A port may be affected by bugs which are specific to FreeBSD. You will need to investigate, find, and fix these bugs when they are reported. Thoroughly testing a port to identify problems before they make their way into the Ports Collection is even better. * *Changes to ports infrastructure and policy.* Occasionally the systems that are used to build ports and packages are updated or a new recommendation affecting the infrastructure is made. You should be aware of these changes in case your ports are affected and require updating. * *Changes to the base system.* FreeBSD is under constant development. Changes to software, libraries, the kernel or even policy changes can cause flow-on change requirements to ports. ==== Maintainer responsibilities ===== Keep your ports up to date This section outlines the process to follow to keep your ports up to date. This is an overview. More information about upgrading a port is available in the extref:{porters-handbook}[Porter's Handbook]. [.procedure] ==== . Watch for updates + Monitor the upstream vendor for new versions, updates and security fixes for the software. Announcement mailing lists or news web pages are useful for doing this. Sometimes users will contact you and ask when your port will be updated. If you are busy with other things or for any reason just cannot update it at the moment, ask if they will help you by submitting an update. + You may also receive automated email from the `FreeBSD Ports Version Check` informing you that a newer version of your port's distfile is available. More information about that system (including how to stop future emails) will be provided in the message. . Incorporate changes + When they become available, incorporate the changes into the port. You need to be able to generate a patch between the original port and your updated port. . Review and test + Thoroughly review and test your changes: ** Build, install and test your port on as many platforms and architectures as you can. It is common for a port to work on one branch or platform and fail on another. ** Make sure your port's dependencies are complete. The recommended way of doing this is by installing your own ports tinderbox. See <> for more information. ** Check that the packing list is up to date. This involves adding in any new files and directories and removing unused entries. ** Verify your port using man:portlint[1] as a guide. See <> for important information about using portlint. ** Consider whether changes to your port might cause any other ports to break. If this is the case, coordinate the changes with the maintainers of those ports. This is especially important if your update changes the shared library version; in this case, at the very least, the dependent ports will need to get a `PORTREVISION` bump so that they will automatically be upgraded by automated tools such as portmaster or man:portupgrade[1]. . Submit changes + Send your update by submitting a PR with an explanation of the changes and a patch containing the differences between the original port and the updated one. Please refer to extref:{problem-reports}[Writing FreeBSD Problem Reports] for information on how to write a really good PR. + [NOTE] ====== Please do not submit a man:shar[1] archive of the entire port; instead, use man:git-format-patch[1] or man:diff[1] `-ruN`. In this way, committers can much more easily see exactly what changes are being made. The Porter's Handbook section on extref:{porters-handbook}[Upgrading, port-upgrading] has more information. ====== . Wait + At some stage a committer will deal with your PR. It may take minutes, or it may take one or two weeks - so please be patient. If it takes any longer, please seek for help on mailing lists ({freebsd-ports}), IRC: #bsdports on EFNet or #freebsd-ports on Libera for example. . Give feedback + If a committer finds a problem with your changes, they will most likely refer it back to you. A prompt response will help get your PR committed faster, and is better for maintaining a thread of conversation when trying to resolve any problems. . And Finally + Your changes will be committed and your port will have been updated. The PR will then be closed by the committer. That's it! ==== ===== Ensure your ports continue to build correctly This section is about discovering and fixing problems that stop your ports from building correctly. FreeBSD only guarantees that the Ports Collection works on the `-STABLE` branches. In theory, you should be able to get by with running the latest release of each stable branch (since the ABIs are not supposed to change) but if you can run the branch, that is even better. Since the majority of FreeBSD installations run on PC-compatible machines (what is termed the `i386` architecture), we expect you to keep the port working on that architecture. We prefer that ports also work on the `amd64` architecture running native. It is completely fair to ask for help if you do not have one of these machines. [NOTE] ==== The usual failure modes for non-`x86` machines are that the original programmers assumed that, for instance, pointers are `int`-s, or that a relatively lax older gcc compiler was being used. More and more, application authors are reworking their code to remove these assumptions - but if the author is not actively maintaining their code, you may need to do this yourself. ==== These are the tasks you need to perform to ensure your port is able to be built: [.procedure] ==== . Watch for build failures + Check your mail for mail from `pkg-fallout@FreeBSD.org` and the http://portscout.FreeBSD.org[distfiles scanner] to see if any of the port which are failing to build are out of date. . Collect information + Once you are aware of a problem, collect information to help you fix it. Build errors reported by `pkg-fallout` are accompanied by logs which will show you where the build failed. If the failure was reported to you by a user, ask them to send you information which may help in diagnosing the problem, such as: ** Build logs ** The commands and options used to build the port (including options set in [.filename]#/etc/make.conf#) ** A list of packages installed on their system as shown by man:pkg-info[8] ** The version of FreeBSD they are running as shown by man:uname[1] `-a` ** When their ports collection was last updated ** When their ports tree and [.filename]#INDEX# was last updated . Investigate and find a solution + Unfortunately there is no straightforward process to follow to do this. Remember, though: if you are stuck, ask for help! The {freebsd-ports} is a good place to start, and the upstream developers are often very helpful. . Submit changes + Just as with updating a port, you should now incorporate changes, review and test, submit your changes in a PR, and provide feedback if required. . Send patches to upstream authors + In some cases, you will have to make patches to the port to make it run on FreeBSD. Some (but not all) upstream authors will accept such patches back into their code for the next release. If so, this may even help their users on other BSD-based systems as well and perhaps save duplicated effort. Please consider sending any applicable patches to the authors as a courtesy. ==== ===== Investigate bug reports and PRs related to your port This section is about discovering and fixing bugs. FreeBSD-specific bugs are generally caused by assumptions about the build and runtime environments that do not apply to FreeBSD. You are less likely to encounter a problem of this type, but it can be more subtle and difficult to diagnose. These are the tasks you need to perform to ensure your port continues to work as intended: [.procedure] ==== . Respond to bug reports + Bugs may be reported to you through email via the https://bugs.FreeBSD.org/search/[Problem Report database]. Bugs may also be reported directly to you by users. + You should respond to PRs and other reports within 14 days, but please try not to take that long. Try to respond as soon as possible, even if it is just to say you need some more time before you can work on the PR. + If you have not responded after 14 days, any committer may commit from a PR that you have not responded to via a `maintainer-timeout`. . Collect information + If the person reporting the bug has not also provided a fix, you need to collect the information that will allow you to generate one. + If the bug is reproducible, you can collect most of the required information yourself. If not, ask the person who reported the bug to collect the information for you, such as: ** A detailed description of their actions, expected program behavior and actual behavior ** Copies of input data used to trigger the bug ** Information about their build and execution environment - for example, a list of installed packages and the output of man:env[1] ** Core dumps ** Stack traces . Eliminate incorrect reports + Some bug reports may be incorrect. For example, the user may have simply misused the program; or their installed packages may be out of date and require updating. Sometimes a reported bug is not specific to FreeBSD. In this case report the bug to the upstream developers. If the bug is within your capabilities to fix, you can also patch the port so that the fix is applied before the next upstream release. . Find a solution + As with build errors, you will need to sort out a fix to the problem. Again, remember to ask if you are stuck! . Submit or approve changes + Just as with updating a port, you should now incorporate changes, review and test, and submit your changes in a PR (or send a follow-up if a PR already exists for the problem). If another user has submitted changes in the PR, you can also send a follow-up saying whether or not you approve the changes. ==== ===== Providing support Part of being a maintainer is providing support - not for the software in general - but for the port and any FreeBSD-specific quirks and problems. Users may contact you with questions, suggestions, problems and patches. Most of the time their correspondence will be specific to FreeBSD. Occasionally you may have to invoke your skills in diplomacy, and kindly point users seeking general support to the appropriate resources. Less frequently you will encounter a person asking why the `RPMS` are not up to date or how can they get the software to run under Foo Linux. Take the opportunity to tell them that your port is up to date (if it is, of course!), and suggest that they try FreeBSD. Sometimes users and developers will decide that you are a busy person whose time is valuable and do some of the work for you. For example, they might: * submit a PR or send you patches to update your port, * investigate and perhaps provide a fix to a PR, or * otherwise submit changes to your port. In these cases your main obligation is to respond in a timely manner. Again, the timeout for non-responsive maintainers is 14 days. After this period changes may be committed unapproved. They have taken the trouble to do this for you; so please try to at least respond promptly. Then review, approve, modify or discuss their changes with them as soon as possible. If you can make them feel that their contribution is appreciated (and it should be) you will have a better chance persuading them to do more things for you in the future :-). [[fix-broken]] === Finding and fixing a broken port There are some really good places to find a port that needs some attention. You can use the https://bugs.freebsd.org/search[web interface] to the Problem Report database to search through and view unresolved PRs. The majority of ports PRs are updates, but with a little searching and skimming over synopses you should be able to find something interesting to work on (the `sw-bug` class is a good place to start). https://portsfallout.com/[PortsFallout] shows port issues gathered from the FreeBSD package building. It is OK to send changes for a maintained port as well, but remember to ask the maintainer in case they are already working on the problem. Once you have found a bug or problem, collect information, investigate and fix! If there is an existing PR, follow up to that. Otherwise create a new PR. Your changes will be reviewed and, if everything checks out, committed. [[mortal-coil]] === When to call it quits As your interests and commitments change, you may find that you no longer have time to continue some (or all) of your ports contributions. That is fine! Please let us know if you are no longer using a port or have otherwise lost time or interest in being a maintainer. In this way we can go ahead and allow other people to try to work on existing problems with the port without waiting for your response. Remember, FreeBSD is a volunteer project, so if maintaining a port is no fun any more, it is probably time to let someone else do it! In any case, the Ports Management Team (`portmgr`) reserves the right to reset your maintainership if you have not actively maintained your port in some time. (Currently, this is set to 3 months.) By this, we mean that there are unresolved problems or pending updates that have not been worked on during that time. [[resources]] === Resources for ports maintainers and contributors The extref:{porters-handbook}[Porter's Handbook] is your hitchhiker's guide to the ports system. Keep it handy! extref:{problem-reports}[Writing FreeBSD Problem Reports] describes how to best formulate and submit a PR. In 2005 more than eleven thousand ports PRs were submitted! Following this article will greatly assist us in reducing the time needed to handle your PRs. The https://bugs.freebsd.org/bugzilla/query.cgi[Problem Report database]. The http://portscout.FreeBSD.org[FreeBSD Ports distfile scanner (portscout)] can show you ports for which the distfiles are not fetchable. You can check on your own ports or use it to find ports that need their `MASTER_SITES` updated. package:ports-mgmt/poudriere[] is the most thorough way to test a port through the entire cycle of installation, packaging, and deinstallation. Documentation is located at the https://github.com/freebsd/poudriere[poudriere github repository] man:portlint[1] is an application which can be used to verify that your port conforms to many important stylistic and functional guidelines. portlint is a simple heuristic application, so you should use it __only as a guide__. If portlint suggests changes which seem unreasonable, consult the extref:{porters-handbook}[Porter's Handbook] or ask for advice. The {freebsd-ports} is for general ports-related discussion. It is a good place to ask for help. You can link:https://lists.freebsd.org/[subscribe, or read and search the list archives]. Reading the archives of the {freebsd-ports-bugs} and the {svn-ports-head} may also be of interest. https://portsfallout.com/[PortsFallout] is a place to help in searching for the https://lists.freebsd.org/archives/freebsd-pkg-fallout/[FreeBSD package-fallout archive]. [[ideas-contributing]] == Getting Started in Other Areas Looking for something interesting to get started that is not mentioned elsewhere in this article? The FreeBSD Project has several Wiki pages containing areas within which new contributors can get ideas on how to get started. The https://wiki.freebsd.org/JuniorJobs[Junior Jobs] page has a list of projects that might be of interest to people just getting started in FreeBSD, and want to work on interesting things to get their feet wet. The https://wiki.freebsd.org/IdeasPage[Ideas Page] contains various "nice to have" or "interesting" things to work on in the Project. diff --git a/documentation/content/en/articles/contributors/_index.adoc b/documentation/content/en/articles/contributors/_index.adoc index 55230b53dc..41516b19fc 100644 --- a/documentation/content/en/articles/contributors/_index.adoc +++ b/documentation/content/en/articles/contributors/_index.adoc @@ -1,212 +1,212 @@ --- title: Contributors to FreeBSD description: A list of organizations and individuals who have contributed to FreeBSD trademarks: ["freebsd", "sun", "general"] tags: ["Contributors", "FreeBSD", "individuals", "organizations"] --- = Contributors to FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/contributors/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} :include-path: content/{{% lang %}}/articles/contributors/ :include-contrib-386bsd: shared/contrib-386bsd.adoc :include-contrib-additional: shared/contrib-additional.adoc :include-contrib-committers: shared/contrib-committers.adoc :include-contrib-corealumni: shared/contrib-corealumni.adoc :include-contrib-develalumni: shared/contrib-develalumni.adoc :include-contrib-portmgralumni: shared/contrib-portmgralumni.adoc endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] :include-path: :include-contrib-386bsd: ../../../../shared/contrib-386bsd.adoc :include-contrib-additional: ../../../../shared/contrib-additional.adoc :include-contrib-committers: ../../../../shared/contrib-committers.adoc :include-contrib-corealumni: ../../../../shared/contrib-corealumni.adoc :include-contrib-develalumni: ../../../../shared/contrib-develalumni.adoc :include-contrib-portmgralumni: ../../../../shared/contrib-portmgralumni.adoc endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] :include-path: :include-contrib-386bsd: ../../../../../shared/contrib-386bsd.adoc :include-contrib-additional: ../../../../../shared/contrib-additional.adoc :include-contrib-committers: ../../../../../shared/contrib-committers.adoc :include-contrib-corealumni: ../../../../../shared/contrib-corealumni.adoc :include-contrib-develalumni: ../../../../../shared/contrib-develalumni.adoc :include-contrib-portmgralumni: ../../../../../shared/contrib-portmgralumni.adoc endif::[] [.abstract-title] Abstract This article lists individuals and organizations who have made a contribution to FreeBSD. To see the current list of FreeBSD Committers you can take a look at the following <>. ''' toc::[] [[staff-committers]] == The FreeBSD Developers This list, which includes all members of the Core Team, names everyone who has commit privileges for one or more of the three source trees (doc, ports and src). To see the current Core Team members you can take a look at the link:https://www.freebsd.org/administration/#t-core[administration page]. (in alphabetical order by last name): include::{include-contrib-committers}[] [[contrib-corealumni]] == Core Team Alumni The following people were members of the FreeBSD core team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::{include-contrib-corealumni}[] [[contrib-develalumni]] == Development Team Alumni The following people were members of the FreeBSD development team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::{include-contrib-develalumni}[] [[contrib-portmgralumni]] == Ports Management Team Alumni The following people were members of the FreeBSD portmgr team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::{include-contrib-portmgralumni}[] [[contrib-develinmemoriam]] == Development Team: In Memoriam During the many years that the FreeBSD Project has been in existence, sadly, some of our developers have passed away. Here are some remembrances. _In rough reverse chronological order of their passing:_ include::{include-path}contrib-develinmemoriam.adoc[] [[contrib-derived]] == Derived Software Contributors This software was originally derived from William F. Jolitz's 386BSD release 0.1, though almost none of the original 386BSD specific code remains. This software has been essentially re-implemented from the 4.4BSD-Lite release provided by the Computer Science Research Group (CSRG) at the University of California, Berkeley and associated academic contributors. There are also portions of NetBSD and OpenBSD that have been integrated into FreeBSD as well, and we would therefore like to thank all the contributors to NetBSD and OpenBSD for their work. [[contrib-additional]] == Additional FreeBSD Contributors (in alphabetical order by first name): include::{include-contrib-additional}[] [[contrib-386bsd]] == 386BSD Patch Kit Patch Contributors (in alphabetical order by first name): include::{include-contrib-386bsd}[] [[donors]] == Donors Gallery The FreeBSD Foundation thanks https://freebsdfoundation.org/our-donors/donors/[financial and in-kind donors]. The https://www.freebsd.org/donations/[FreeBSD Donations Liaison] area includes a https://www.freebsd.org/donations/donors/[list of donated hardware]. The FreeBSD Project thanks all donors! [NOTE] ==== As of 2010, the section below was several years out-of-date. ==== === Contributors to the central server project The following individuals and businesses made it possible for the FreeBSD Project to build a new central server machine, which has replaced `freefall.FreeBSD.org` at one point, by donating the following items: * {mbarkah} and his employer, http://www.hemi.com/[Hemisphere Online], donated a _Pentium Pro (P6) 200MHz CPU_. * http://www.asacomputers.com/[ASA Computers] donated a _Tyan 1662 motherboard_. * Joe McGuckin of http://www.via.net/[ViaNet Communications] donated a _Kingston ethernet controller_. * Jack O'Neill donated an _NCR 53C875 SCSI controller card_. * Ulf Zimmermann of http://www.Alameda.net/[Alameda Networks] donated _128MB of memory_, a _4 Gb disk drive and the case_. === Direct funding The following individuals and businesses have generously contributed direct funding to the project: * Annelise Anderson . * {dillon}. * http://www.bluemountain.com/[Blue Mountain Arts]. * http://www.epilogue.com/[Epilogue Technology Corporation]. * {sef}. * http://www.gta.com/[Global Technology Associates, Inc]. * Don Scott Wilde. * Gianmarco Giovannelli . * Josef C. Grosch . * Robert T. Morris. * {chuckr}. * Kenneth P. Stox of http://www.imagescape.com/[Imaginary Landscape, LLC.]. * Dmitry S. Kohmanyuk . * http://www.cdrom.co.jp/[Laser5] of Japan (a portion of the profits from sales of their various FreeBSD CDROMs). * http://www.mmjp.or.jp/fuki/[Fuki Shuppan Publishing Co.] donated a portion of their profits from _Hajimete no FreeBSD_ (FreeBSD, Getting started) to the FreeBSD and XFree86 projects. * http://www.ascii.co.jp/[ASCII Corp.] donated a portion of their profits from several FreeBSD-related books to the FreeBSD project. * http://www.yokogawa.co.jp/[Yokogawa Electric Corp] has generously donated significant funding to the FreeBSD project. * http://www.buffnet.net/[BuffNET]. * http://www.pacificsolutions.com/[Pacific Solutions]. * http://www.siemens.de/[Siemens AG] via Andre Albsmeier . * Chris Silva . === Hardware contributors The following individuals and businesses have generously contributed hardware for testing and device driver development/support: * BSDi for providing the Pentium P5-90 and 486/DX2-66 EISA/VL systems that are being used for our development work, to say nothing of the network access and other donations of hardware resources. * http://www.compaq.com[Compaq] has donated a variety of Alpha systems to the FreeBSD Project. Among the many generous donations are 4 AlphaStation DS10s, an AlphaServer DS20, AlphaServer 2100s, an AlphaServer 4100, 8 500Mhz Personal Workstations, 4 433Mhz Personal Workstations, and more! These machines are used for release engineering, package building, SMP development, and general development on the Alpha architecture. * TRW Financial Systems, Inc. provided 130 PCs, three 68 GB file servers, twelve Ethernets, two routers and an ATM switch for debugging the diskless code * Dermot McDonnell donated the Toshiba XM3401B CDROM drive currently used in _freefall_. * Chuck Robey contributed his floppy tape streamer for experimental work. -* Larry Altneu and {wilko}, provided Wangtek and Archive QIC-02 tape drives in order to improve the [.filename]#wt# driver. +* Larry Altneu and {wilko}, provided Wangtek and Archive QIC-02 tape drives to improve the [.filename]#wt# driver. * Ernst Winter (http://berklix.org/ewinter/[Deceased]) contributed a 2.88 MB floppy drive to the project. This will hopefully increase the pressure for rewriting the floppy disk driver. * http://www.tekram.com/[Tekram Technologies] sent one each of their DC-390, DC-390U and DC-390F FAST and ULTRA SCSI host adapter cards for regression testing of the NCR and AMD drivers with their cards. They are also to be applauded for making driver sources for free operating systems available from their FTP server link:ftp://ftp.tekram.com/scsi/FreeBSD/[ftp://ftp.tekram.com/scsi/FreeBSD/]. * Larry M. Augustin contributed not only a Symbios Sym8751S SCSI card, but also a set of data books, including one about the forthcoming Sym53c895 chip with Ultra-2 and LVD support, and the latest programming manual with information on how to safely use the advanced features of the latest Symbios SCSI chips. Thanks a lot! * {kuku} donated an FX120 12 speed Mitsumi CDROM drive for IDE CDROM driver development. -* Mike Tancsa donated four various ATM PCI cards in order to help increase support of these cards as well as help support the development effort of the netatm ATM stack. +* Mike Tancsa donated four various ATM PCI cards to help increase support of these cards as well as help support the development effort of the netatm ATM stack. === Special contributors * http://www.osd.bsdi.com/[BSDi] (formerly Walnut Creek CDROM) has donated almost more than we can say (see the 'About the FreeBSD Project' section of the extref:{handbook}[FreeBSD Handbook] for more details). In particular, we would like to thank them for the original hardware used for `freefall.FreeBSD.org`, our primary development machine, and for `thud.FreeBSD.org`, a testing and build box. We are also indebted to them for funding various contributors over the years and providing us with unrestricted use of their T1 connection to the Internet. * The http://www.interface-business.de/[interface business GmbH, Dresden] has been patiently supporting {joerg} who has often preferred FreeBSD work over paid work, and used to fall back to their (quite expensive) EUnet Internet connection whenever his private connection became too slow or flaky to work with it. * http://www.bsdi.com/[Berkeley Software Design, Inc.] has contributed their DOS emulator code to the remaining BSD world, which is used in the `doscmd` command. diff --git a/documentation/content/en/articles/cups/_index.adoc b/documentation/content/en/articles/cups/_index.adoc index 58930951c9..d16f8da5de 100644 --- a/documentation/content/en/articles/cups/_index.adoc +++ b/documentation/content/en/articles/cups/_index.adoc @@ -1,278 +1,278 @@ --- title: CUPS on FreeBSD authors: - author: Chess Griffin email: chess@chessgriffin.com description: How to install and use CUPS on FreeBSD trademarks: ["freebsd", "general"] tags: ["CUPS", "FreeBSD", "Guide", "configuration", "introduction"] --- = CUPS on FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/cups/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract An article about configuring CUPS on FreeBSD. ''' toc::[] [[printing-cups]] == An Introduction to the Common Unix Printing System (CUPS) CUPS, the Common UNIX Printing System, provides a portable printing layer for UNIX(R)-based operating systems. It has been developed by Easy Software Products to promote a standard printing solution for all UNIX(R) vendors and users. CUPS uses the Internet Printing Protocol (IPP) as the basis for managing print jobs and queues. The Line Printer Daemon (LPD), Server Message Block (SMB), and AppSocket (aka JetDirect) protocols are also supported with reduced functionality. CUPS adds network printer browsing and PostScript Printer Description (PPD) based printing options to support real-world printing under UNIX(R). As a result, CUPS is ideally-suited for sharing and accessing printers in mixed environments of FreeBSD, Linux(R), Mac OS(R) X, or Windows(R). The main site for CUPS is http://www.cups.org/[http://www.cups.org/]. [[printing-cups-install]] == Installing the CUPS Print Server To install CUPS using a precompiled binary, issue the following command from a root terminal: [source,shell] .... # pkg install cups .... Other optional, but recommended, packages are package:print/gutenprint[] and package:print/hplip[], both of which add drivers and utilities for a variety of printers. Once installed, the CUPS configuration files can be found in the directory [.filename]#/usr/local/etc/cups#. [[printing-cups-configuring-server]] == Configuring the CUPS Print Server -After installation, a few files must be edited in order to configure the CUPS server. +After installation, a few files must be edited to configure the CUPS server. First, create or modify, as the case may be, the file [.filename]#/etc/devfs.rules# and add the following information to set the proper permissions on all potential printer devices and to associate printers with the `cups` user group: [.programlisting] .... [system=10] add path 'unlpt*' mode 0660 group cups add path 'ulpt*' mode 0660 group cups add path 'lpt*' mode 0660 group cups add path 'usb/X.Y.Z' mode 0660 group cups .... [NOTE] ==== Note that _X_, _Y_, and _Z_ should be replaced with the target USB device listed in the [.filename]#/dev/usb# directory that corresponds to the printer. To find the correct device, examine the output of man:dmesg[8], where [.filename]#ugenX.Y# lists the printer device, which is a symbolic link to a USB device in [.filename]#/dev/usb#. ==== Next, add two lines to [.filename]#/etc/rc.conf# as follows: [.programlisting] .... cupsd_enable="YES" devfs_system_ruleset="system" .... These two entries will start the CUPS print server on boot and invoke the local devfs rule created above, respectively. -In order to enable CUPS printing under certain Microsoft(R) Windows(R) clients, the line below should be uncommented in [.filename]#/usr/local/etc/cups/mime.types# and [.filename]#/usr/local/etc/cups/mime.convs#: +To enable CUPS printing under certain Microsoft(R) Windows(R) clients, the line below should be uncommented in [.filename]#/usr/local/etc/cups/mime.types# and [.filename]#/usr/local/etc/cups/mime.convs#: [.programlisting] .... application/octet-stream .... Once these changes have been made, the man:devfs[8] and CUPS systems must both be restarted, either by rebooting the computer or issuing the following two commands in a root terminal: [source,shell] .... # /etc/rc.d/devfs restart # /usr/local/etc/rc.d/cupsd restart .... [[printing-cups-configuring-printers]] == Configuring Printers on the CUPS Print Server After the CUPS system has been installed and configured, the administrator can begin configuring the local printers attached to the CUPS print server. This part of the process is very similar, if not identical, to configuring CUPS printers on other UNIX(R)-based operating systems, such as a Linux(R) distribution. The primary means for managing and administering the CUPS server is through the web-based interface, which can be found by launching a web browser and entering http://localhost:631[http://localhost:631] in the browser's URL bar. If the CUPS server is on another machine on the network, substitute the server's local IP address for `localhost`. The CUPS web interface is fairly self-explanatory, as there are sections for managing printers and print jobs, authorizing users, and more. Additionally, on the right-hand side of the Administration screen are several check-boxes allowing easy access to commonly-changed settings, such as whether to share published printers connected to the system, whether to allow remote administration of the CUPS server, and whether to allow users additional access and privileges to the printers and print jobs. Adding a printer is generally as easy as clicking "Add Printer" at the Administration screen of the CUPS web interface, or clicking one of the "New Printers Found" buttons also at the Administration screen. When presented with the "Device" drop-down box, simply select the desired locally-attached printer, and then continue through the process. If one has added the package:print/gutenprint-cups[] or package:print/hplip[] ports or packages as referenced above, then additional print drivers will be available in the subsequent screens that might provide more stability or features. [[printing-cups-clients]] == Configuring CUPS Clients Once the CUPS server has been configured and printers have been added and published to the network, the next step is to configure the clients, or the machines that are going to access the CUPS server. If one has a single desktop machine that is acting as both server and client, then much of this information may not be needed. [[printing-cups-clients-unix]] === UNIX(R) Clients CUPS will also need to be installed on your UNIX(R) clients. Once CUPS is installed on the clients, then CUPS printers that are shared across the network are often automatically discovered by the printer managers for various desktop environments such as GNOME or KDE. Alternatively, one can access the local CUPS interface on the client machine at http://localhost:631[http://localhost:631] and click on "Add Printer" in the Administration section. When presented with the "Device" drop-down box, simply select the networked CUPS printer, if it was automatically discovered, or select `ipp` or `http` and enter the IPP or HTTP URI of the networked CUPS printer, usually in one of the two following syntaxes: [.programlisting] .... ipp://server-name-or-ip/printers/printername .... [.programlisting] .... http://server-name-or-ip:631/printers/printername .... If the CUPS clients have difficulty finding other CUPS printers shared across the network, sometimes it is helpful to add or create a file [.filename]#/usr/local/etc/cups/client.conf# with a single entry as follows: [.programlisting] .... ServerName server-ip .... In this case, _server-ip_ would be replaced by the local IP address of the CUPS server on the network. [[printing-cups-clients-windows]] === Windows(R) Clients Versions of Windows(R) prior to XP did not have the capability to natively network with IPP-based printers. However, Windows(R) XP and later versions do have this capability. Therefore, to add a CUPS printer in these versions of Windows(R) is quite easy. Generally, the Windows(R) administrator will run the Windows(R) `Add Printer` wizard, select `Network Printer` and then enter the URI in the following syntax: [.programlisting] .... http://server-name-or-ip:631/printers/printername .... If one has an older version of Windows(R) without native IPP printing support, then the general means of connecting to a CUPS printer is to use package:net/samba413[] and CUPS together, which is a topic outside the scope of this chapter. [[printing-cups-troubleshooting]] == CUPS Troubleshooting Difficulties with CUPS often lies in permissions. First, double check the man:devfs[8] permissions as outlined above. Next, check the actual permissions of the devices created in the file system. It is also helpful to make sure your user is a member of the `cups` group. If the permissions check boxes in the Administration section of the CUPS web interface do not seem to be working, another fix might be to manually backup the main CUPS configuration file located at [.filename]#/usr/local/etc/cups/cupsd.conf# and edit the various configuration options and try different combinations of configuration options. One sample [.filename]#/usr/local/etc/cups/cupsd.conf# to test is listed below. Please note that this sample [.filename]#cupsd.conf# sacrifices security for easier configuration; once the administrator successfully connects to the CUPS server and configures the clients, it is advisable to revisit this configuration file and begin locking down access. [.programlisting] .... # Log general information in error_log - change "info" to "debug" for # troubleshooting... LogLevel info # Administrator user group... SystemGroup wheel # Listen for connections on Port 631. Port 631 #Listen localhost:631 Listen /var/run/cups.sock # Show shared printers on the local network. Browsing On BrowseOrder allow,deny #BrowseAllow @LOCAL BrowseAllow 192.168.1.* # change to local LAN settings BrowseAddress 192.168.1.* # change to local LAN settings # Default authentication type, when authentication is required... DefaultAuthType Basic DefaultEncryption Never # comment this line to allow encryption # Allow access to the server from any machine on the LAN Order allow,deny #Allow localhost Allow 192.168.1.* # change to local LAN settings # Allow access to the admin pages from any machine on the LAN #Encryption Required Order allow,deny #Allow localhost Allow 192.168.1.* # change to local LAN settings # Allow access to configuration files from any machine on the LAN AuthType Basic Require user @SYSTEM Order allow,deny #Allow localhost Allow 192.168.1.* # change to local LAN settings # Set the default printer/job policies... # Job-related operations must be done by the owner or an administrator... Require user @OWNER @SYSTEM Order deny,allow # All administration operations require an administrator to authenticate... AuthType Basic Require user @SYSTEM Order deny,allow # Only the owner or an administrator can cancel or authenticate a job... Require user @OWNER @SYSTEM Order deny,allow Order deny,allow .... diff --git a/documentation/content/en/articles/explaining-bsd/_index.adoc b/documentation/content/en/articles/explaining-bsd/_index.adoc index b188f0a2b0..ca7b1c02e5 100644 --- a/documentation/content/en/articles/explaining-bsd/_index.adoc +++ b/documentation/content/en/articles/explaining-bsd/_index.adoc @@ -1,233 +1,233 @@ --- title: Explaining BSD authors: - author: Greg Lehey email: grog@FreeBSD.org description: Brief explanation about BSD trademarks: ["freebsd", "amd", "apple", "intel", "linux", "opengroup", "sun", "unix", "general"] tags: ["Explaining BSD", "BSD", "FreeBSD", "operating system"] --- = Explaining BSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/explaining-bsd/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract In the open source world, the word "Linux" is almost synonymous with "Operating System", but it is not the only open source UNIX(R) operating system. So what is the secret? Why is BSD not better known? This white paper addresses these and other questions. Throughout this paper, differences between BSD and Linux will be noted __like this__. ''' toc::[] [[what-is-bsd]] == What is BSD? BSD stands for "Berkeley Software Distribution". It is the name of distributions of source code from the University of California, Berkeley, which were originally extensions to AT&T's Research UNIX(R) operating system. Several open source operating system projects are based on a release of this source code known as 4.4BSD-Lite. In addition, they comprise a number of packages from other Open Source projects, including notably the GNU project. The overall operating system comprises: * The BSD kernel, which handles process scheduling, memory management, symmetric multi-processing (SMP), device drivers, etc. * The C library, the base API for the system. + __The BSD C library is based on code from Berkeley, not the GNU project.__ * Utilities such as shells, file utilities, compilers and linkers. + __Some of the utilities are derived from the GNU project, others are not.__ * The X Window system, which handles graphical display. + The X Window system used in most versions of BSD is maintained by the http://www.X.org/[X.Org project]. FreeBSD allows the user to choose from a variety of desktop environments, such as Gnome, KDE, or Xfce; and lightweight window managers like Openbox, Fluxbox, or Awesome. * Many other programs and utilities. [[what-a-real-unix]] == What, a real UNIX(R)? The BSD operating systems are not clones, but open source derivatives of AT&T's Research UNIX(R) operating system, which is also the ancestor of the modern UNIX(R) System V. This may surprise you. How could that happen when AT&T has never released its code as open source? It is true that AT&T UNIX(R) is not open source, and in a copyright sense BSD is very definitely _not_ UNIX(R), but on the other hand, AT&T has imported sources from other projects, noticeably the Computer Sciences Research Group (CSRG) of the University of California in Berkeley, CA. Starting in 1976, the CSRG started releasing tapes of their software, calling them _Berkeley Software Distribution_ or __BSD__. Initial BSD releases consisted mainly of user programs, but that changed dramatically when the CSRG landed a contract with the Defense Advanced Research Projects Agency (DARPA) to upgrade the communications protocols on their network, ARPANET. The new protocols were known as the __Internet Protocols__, later _TCP/IP_ after the most important protocols. The first widely distributed implementation was part of 4.2BSD, in 1982. In the course of the 1980s, a number of new workstation companies sprang up. Many preferred to license UNIX(R) rather than developing operating systems for themselves. In particular, Sun Microsystems licensed UNIX(R) and implemented a version of 4.2BSD, which they called SunOS(TM). When AT&T themselves were allowed to sell UNIX(R) commercially, they started with a somewhat bare-bones implementation called System III, to be quickly followed by System V. The System V code base did not include networking, so all implementations included additional software from the BSD, including the TCP/IP software, but also utilities such as the _csh_ shell and the _vi_ editor. Collectively, these enhancements were known as the __Berkeley Extensions__. The BSD tapes contained AT&T source code and thus required a UNIX(R) source license. By 1990, the CSRG's funding was running out, and it faced closure. Some members of the group decided to release the BSD code, which was Open Source, without the AT&T proprietary code. This finally happened with the __Networking Tape 2__, usually known as __Net/2__. Net/2 was not a complete operating system: about 20% of the kernel code was missing. One of the CSRG members, William F. Jolitz, wrote the remaining code and released it in early 1992 as __386BSD__. At the same time, another group of ex-CSRG members formed a commercial company called http://www.bsdi.com/[Berkeley Software Design Inc.] and released a beta version of an operating system called http://www.bsdi.com/[BSD/386], which was based on the same sources. The name of the operating system was later changed to BSD/OS. 386BSD never became a stable operating system. Instead, two other projects split off from it in 1993: http://www.NetBSD.org/[NetBSD] and link:https://www.FreeBSD.org/[FreeBSD]. The two projects originally diverged due to differences in patience waiting for improvements to 386BSD: the NetBSD people started early in the year, and the first version of FreeBSD was not ready until the end of the year. In the meantime, the code base had diverged sufficiently to make it difficult to merge. In addition, the projects had different aims, as we will see below. In 1996, http://www.OpenBSD.org/[OpenBSD] split off from NetBSD, and in 2003, http://www.dragonflybsd.org/[DragonFlyBSD] split off from FreeBSD. [[why-is-bsd-not-better-known]] == Why is BSD not better known? For a number of reasons, BSD is relatively unknown: . The BSD developers are often more interested in polishing their code than marketing it. . Much of Linux's popularity is due to factors external to the Linux projects, such as the press, and to companies formed to provide Linux services. Until recently, the open source BSDs had no such proponents. . In 1992, AT&T sued http://www.bsdi.com/[BSDI], the vendor of BSD/386, alleging that the product contained AT&T-copyrighted code. The case was settled out of court in 1994, but the spectre of the litigation continues to haunt people. In March 2000 an article published on the web claimed that the court case had been "recently settled". + One detail that the lawsuit did clarify is the naming: in the 1980s, BSD was known as "BSD UNIX(R)". With the elimination of the last vestige of AT&T code from BSD, it also lost the right to the name UNIX(R). Thus you will see references in book titles to "the 4.3BSD UNIX(R) operating system" and "the 4.4BSD operating system". [[comparing-bsd-and-linux]] == Comparing BSD and Linux So what is really the difference between, say, Debian Linux and FreeBSD? For the average user, the difference is surprisingly small: Both are UNIX(R) like operating systems. Both are developed by non-commercial projects (this does not apply to many other Linux distributions, of course). In the following section, we will look at BSD and compare it to Linux. The description applies most closely to FreeBSD, which accounts for an estimated 80% of the BSD installations, but the differences from NetBSD, OpenBSD and DragonFlyBSD are small. === Who owns BSD? No one person or corporation owns BSD. It is created and distributed by a community of highly technical and committed contributors all over the world. Some of the components of BSD are Open Source projects in their own right and managed by different project maintainers. === How is BSD developed and updated? The BSD kernels are developed and updated following the Open Source development model. Each project maintains a publicly accessible _source tree_ which contains all source files for the project, including documentation and other incidental files. Users can obtain a complete copy of any version. A large number of developers worldwide contribute to improvements to BSD. They are divided into three kinds: -* _Contributors_ write code or documentation. They are not permitted to commit (add code) directly to the source tree. In order for their code to be included in the system, it must be reviewed and checked in by a registered developer, known as a __committer__. -* _Committers_ are developers with write access to the source tree. In order to become a committer, an individual must show ability in the area in which they are active. +* _Contributors_ write code or documentation. They are not permitted to commit (add code) directly to the source tree. For their code to be included in the system, it must be reviewed and checked in by a registered developer, known as a __committer__. +* _Committers_ are developers with write access to the source tree. To become a committer, an individual must show ability in the area in which they are active. + It is at the individual committer's discretion whether they should obtain authority before committing changes to the source tree. In general, an experienced committer may make changes which are obviously correct without obtaining consensus. For example, a documentation project committer may correct typographical or grammatical errors without review. On the other hand, developers making far-reaching or complicated changes are expected to submit their changes for review before committing them In extreme cases, a core team member with a function such as Principal Architect may order that changes be removed from the tree, a process known as _backing out_. All committers receive mail describing each individual commit, so it is not possible to commit secretly. -* The _Core team_. FreeBSD and NetBSD each have a core team which manages the project. The core teams developed in the course of the projects, and their role is not always well-defined. It is not necessary to be a developer in order to be a core team member, though it is normal. The rules for the core team vary from one project to the other, but in general they have more say in the direction of the project than non-core team members have. +* The _Core team_. FreeBSD and NetBSD each have a core team which manages the project. The core teams developed in the course of the projects, and their role is not always well-defined. It is not necessary to be a developer to be a core team member, though it is normal. The rules for the core team vary from one project to the other, but in general they have more say in the direction of the project than non-core team members have. This arrangement differs from Linux in a number of ways: . No one person controls the content of the system. In practice, this difference is overrated, since the Principal Architect can require that code be backed out, and even in the Linux project several people are permitted to make changes. . On the other hand, there _is_ a central repository, a single place where you can find the entire operating system sources, including all older versions. . BSD projects maintain the entire "Operating System", not only the kernel. This distinction is only marginally useful: neither BSD nor Linux is useful without applications. The applications used under BSD are frequently the same as the applications used under Linux. . As a result of the formalized maintenance of a single SVN source tree, BSD development is clear, and it is possible to access any version of the system by release number or by date. SVN also allows incremental updates to the system: for example, the FreeBSD repository is updated about 100 times a day. Most of these changes are small. === BSD releases FreeBSD, NetBSD and OpenBSD provide the system in three different "releases". As with Linux, releases are assigned a number such as 1.4.1 or 3.5. In addition, the version number has a suffix indicating its purpose: . The development version of the system is called _CURRENT_. FreeBSD assigns a number to CURRENT, for example FreeBSD 5.0-CURRENT. NetBSD uses a slightly different naming scheme and appends a single-letter suffix which indicates changes in the internal interfaces, for example NetBSD 1.4.3G. OpenBSD does not assign a number ("OpenBSD-current"). All new development on the system goes into this branch. . At regular intervals, between two and four times a year, the projects bring out a _RELEASE_ version of the system, which is available on CD-ROM and for free download from FTP sites, for example OpenBSD 2.6-RELEASE or NetBSD 1.4-RELEASE. The RELEASE version is intended for end users and is the normal version of the system. NetBSD also provides _patch releases_ with a third digit, for example NetBSD 1.4.2. . As bugs are found in a RELEASE version, they are fixed, and the fixes are added to the SVN tree. In FreeBSD, the resultant version is called the _STABLE_ version, while in NetBSD and OpenBSD it continues to be called the RELEASE version. Smaller new features can also be added to this branch after a period of test in the CURRENT branch. Security and other important bug fixes are also applied to all supported RELEASE versions. _By contrast, Linux maintains two separate code trees: the stable version and the development version. Stable versions have an even minor version number, such as 2.0, 2.2 or 2.4. Development versions have an odd minor version number, such as 2.1, 2.3 or 2.5. In each case, the number is followed by a further number designating the exact release. In addition, each vendor adds their own userland programs and utilities, so the name of the distribution is also important. Each distribution vendor also assigns version numbers to the distribution, so a complete description might be something like "TurboLinux 6.0 with kernel 2.2.14"_ === What versions of BSD are available? In contrast to the numerous Linux distributions, there are only four major open source BSDs. Each BSD project maintains its own source tree and its own kernel. In practice, though, there appear to be fewer divergences between the userland code of the projects than there is in Linux. It is difficult to categorize the goals of each project: the differences are very subjective. Basically, * FreeBSD aims for high performance and ease of use by end users, and is a favourite of web content providers. It runs on a link:https://www.FreeBSD.org/platforms/[number of platforms] and has significantly more users than the other projects. * NetBSD aims for maximum portability: "of course it runs NetBSD". It runs on machines from palmtops to large servers, and has even been used on NASA space missions. It is a particularly good choice for running on old non-Intel(R) hardware. * OpenBSD aims for security and code purity: it uses a combination of the open source concept and rigorous code reviews to create a system which is demonstrably correct, making it the choice of security-conscious organizations such as banks, stock exchanges and US Government departments. Like NetBSD, it runs on a number of platforms. * DragonFlyBSD aims for high performance and scalability under everything from a single-node UP system to a massively clustered system. DragonFlyBSD has several long-range technical goals, but focus lies on providing a SMP-capable infrastructure that is easy to understand, maintain and develop for. There are also two additional BSD UNIX(R) operating systems which are not open source, BSD/OS and Apple's Mac OS(R) X: * BSD/OS was the oldest of the 4.4BSD derivatives. It was not open source, though source code licenses were available at relatively low cost. It resembled FreeBSD in many ways. Two years after the acquisition of BSDi by Wind River Systems, BSD/OS failed to survive as an independent product. Support and source code may still be available from Wind River, but all new development is focused on the VxWorks embedded operating system. * http://www.apple.com/macosx/server/[Mac OS(R) X] is the latest version of the operating system for Apple(R)'s Mac(R) line. The BSD core of this operating system, http://developer.apple.com/darwin/[Darwin], is available as a fully functional open source operating system for x86 and PPC computers. The Aqua/Quartz graphics system and many other proprietary aspects of Mac OS(R) X remain closed-source, however. Several Darwin developers are also FreeBSD committers, and vice-versa. === How does the BSD license differ from the GNU Public license? Linux is available under the http://www.fsf.org/copyleft/gpl.html[GNU General Public License] (GPL), which is designed to eliminate closed source software. In particular, any derivative work of a product released under the GPL must also be supplied with source code if requested. By contrast, the http://www.opensource.org/licenses/bsd-license.html[BSD license] is less restrictive: binary-only distributions are allowed. This is particularly attractive for embedded applications. === What else should I know? Since fewer applications are available for BSD than Linux, the BSD developers created a Linux compatibility package, which allows Linux programs to run under BSD. -The package includes both kernel modifications, in order to correctly perform Linux system calls, and Linux compatibility files such as the C library. +The package includes both kernel modifications, to correctly perform Linux system calls, and Linux compatibility files such as the C library. There is no noticeable difference in execution speed between a Linux application running on a Linux machine and a Linux application running on a BSD machine of the same speed. The "all from one supplier" nature of BSD means that upgrades are much easier to handle than is frequently the case with Linux. BSD handles library version upgrades by providing compatibility modules for earlier library versions, so it is possible to run binaries which are several years old with no problems. === Which should I use, BSD or Linux? What does this all mean in practice? Who should use BSD, who should use Linux? This is a very difficult question to answer. Here are some guidelines: * "If it ain't broke, don't fix it": If you already use an open source operating system, and you are happy with it, there is probably no good reason to change. * BSD systems, in particular FreeBSD, can have notably higher performance than Linux. But this is not across the board. In many cases, there is little or no difference in performance. In some cases, Linux may perform better than FreeBSD. * In general, BSD systems have a better reputation for reliability, mainly as a result of the more mature code base. * BSD projects have a better reputation for the quality and completeness of their documentation. The various documentation projects aim to provide actively updated documentation, in many languages, and covering all aspects of the system. * The BSD license may be more attractive than the GPL. * BSD can execute most Linux binaries, while Linux can not execute BSD binaries. Many BSD implementations can also execute binaries from other UNIX(R) like systems. As a result, BSD may present an easier migration route from other systems than Linux would. === Who provides support, service, and training for BSD? BSDi / http://www.freebsdmall.com[FreeBSD Mall, Inc.] have been providing support contracts for FreeBSD for nearly a decade. In addition, each of the projects has a list of consultants for hire: link:https://www.FreeBSD.org/commercial/consult_bycat/[FreeBSD], http://www.netbsd.org/gallery/consultants.html[NetBSD], and http://www.openbsd.org/support.html[OpenBSD]. diff --git a/documentation/content/en/articles/filtering-bridges/_index.adoc b/documentation/content/en/articles/filtering-bridges/_index.adoc index 1d9f76ba83..ae744fb88d 100644 --- a/documentation/content/en/articles/filtering-bridges/_index.adoc +++ b/documentation/content/en/articles/filtering-bridges/_index.adoc @@ -1,294 +1,294 @@ --- title: Filtering Bridges authors: - author: Alex Dupre email: ale@FreeBSD.org description: Configuring firewalls and filtering on FreeBSD hosts acting as bridges rather than routers trademarks: ["freebsd", "3com", "intel", "general"] tags: ["network", "filtering", "bridges", "FreeBSD"] --- = Filtering Bridges :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/filtering-bridges/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract Often it is useful to divide one physical network (like an Ethernet) into two separate segments without having to create subnets, and use a router to link them together. The device that connects the two networks in this way is called a bridge. -A FreeBSD system with two network interfaces is enough in order to act as a bridge. +A FreeBSD system with two network interfaces is enough to act as a bridge. A bridge works by scanning the addresses of MAC level (Ethernet addresses) of the devices connected to each of its network interfaces and then forwarding the traffic between the two networks only if the source and the destination are on different segments. Under many points of view a bridge is similar to an Ethernet switch with only two ports. ''' toc::[] [[filtering-bridges-why]] == Why use a filtering bridge? More and more frequently, thanks to the lowering costs of broad band Internet connections (xDSL) and also because of the reduction of available IPv4 addresses, many companies are connected to the Internet 24 hours on 24 and with few (sometimes not even a power of 2) IP addresses. In these situations it is often desirable to have a firewall that filters incoming and outgoing traffic from and towards Internet, but a packet filtering solution based on router may not be applicable, either due to subnetting issues, the router is owned by the connectivity supplier (ISP), or because it does not support such functionalities. In these scenarios the use of a filtering bridge is highly advised. A bridge-based firewall can be configured and inserted between the xDSL router and your Ethernet hub/switch without any IP numbering issues. [[filtering-bridges-how]] == How to Install Adding bridge functionalities to a FreeBSD system is not difficult. Since 4.5 release it is possible to load such functionalities as modules instead of having to rebuild the kernel, simplifying the procedure a great deal. In the following subsections I will explain both installation ways. [IMPORTANT] ==== _Do not_ follow both instructions: a procedure _excludes_ the other one. Select the best choice according to your needs and abilities. ==== Before going on, be sure to have at least two Ethernet cards that support the promiscuous mode for both reception and transmission, since they must be able to send Ethernet packets with any address, not just their own. Moreover, to have a good throughput, the cards should be PCI bus mastering cards. The best choices are still the Intel EtherExpress(TM) Pro, followed by the 3Com(R) 3c9xx series. -To simplify the firewall configuration it may be useful to have two cards of different manufacturers (using different drivers) in order to distinguish clearly which interface is connected to the router and which to the inner network. +To simplify the firewall configuration it may be useful to have two cards of different manufacturers (using different drivers) to distinguish clearly which interface is connected to the router and which to the inner network. [[filtering-bridges-kernel]] === Kernel Configuration So you have decided to use the older but well tested installation method. To begin, you have to add the following rows to your kernel configuration file: [.programlisting] .... options BRIDGE options IPFIREWALL options IPFIREWALL_VERBOSE .... The first line is to compile the bridge support, the second one is the firewall and the third one is the logging functions of the firewall. Now it is necessary to build and install the new kernel. You may find detailed instructions in the extref:{handbook}[Building and Installing a Custom Kernel, kernelconfig-building] section of the FreeBSD Handbook. [[filtering-bridges-modules]] === Modules Loading If you have chosen to use the new and simpler installation method, the only thing to do now is add the following row to [.filename]#/boot/loader.conf#: [.programlisting] .... bridge_load="YES" .... In this way, during the system startup, the [.filename]#bridge.ko# module will be loaded together with the kernel. It is not required to add a similar row for the [.filename]#ipfw.ko# module, since it will be loaded automatically after the execution of the steps in the following section. [[filtering-bridges-finalprep]] == Final Preparation -Before rebooting in order to load the new kernel or the required modules (according to the previously chosen installation method), you have to make some changes to the [.filename]#/etc/rc.conf# configuration file. +Before rebooting to load the new kernel or the required modules (according to the previously chosen installation method), you have to make some changes to the [.filename]#/etc/rc.conf# configuration file. The default rule of the firewall is to reject all IP packets. -Initially we will set up an `open` firewall, in order to verify its operation without any issue related to packet filtering (in case you are going to execute this procedure remotely, such configuration will avoid you to remain isolated from the network). +Initially we will set up an `open` firewall, to verify its operation without any issue related to packet filtering (in case you are going to execute this procedure remotely, such configuration will avoid you to remain isolated from the network). Put these lines in [.filename]#/etc/rc.conf#: [.programlisting] .... firewall_enable="YES" firewall_type="open" firewall_quiet="YES" firewall_logging="YES" .... The first row will enable the firewall (and will load the module [.filename]#ipfw.ko# if it is not compiled in the kernel), the second one to set up it in `open` mode (as explained in [.filename]#/etc/rc.firewall#), the third one to not show rules loading and the fourth one to enable logging support. About the configuration of the network interfaces, the most used way is to assign an IP to only one of the network cards, but the bridge will work equally even if both interfaces or none has a configured IP. In the last case (IP-less) the bridge machine will be still more hidden, as inaccessible from the network: to configure it, you have to login from console or through a third network interface separated from the bridge. Sometimes, during the system startup, some programs require network access, say for domain resolution: in this case it is necessary to assign an IP to the external interface (the one connected to Internet, where DNS server resides), since the bridge will be activated at the end of the startup procedure. It means that the [.filename]#fxp0# interface (in our case) must be mentioned in the ifconfig section of the [.filename]#/etc/rc.conf# file, while the [.filename]#xl0# is not. Assigning an IP to both the network cards does not make much sense, unless, during the start procedure, applications should access to services on both Ethernet segments. There is another important thing to know. When running IP over Ethernet, there are actually two Ethernet protocols in use: one is IP, the other is ARP. ARP does the conversion of the IP address of a host into its Ethernet address (MAC layer). -In order to allow the communication between two hosts separated by the bridge, it is necessary that the bridge will forward ARP packets. +To allow the communication between two hosts separated by the bridge, it is necessary that the bridge will forward ARP packets. Such protocol is not included in the IP layer, since it exists only with IP over Ethernet. The FreeBSD firewall filters exclusively on the IP layer and therefore all non-IP packets (ARP included) will be forwarded without being filtered, even if the firewall is configured to not permit anything. Now it is time to reboot the system and use it as before: there will be some new messages about the bridge and the firewall, but the bridge will not be activated and the firewall, being in `open` mode, will not avoid any operations. If there are any problems, you should sort them out now before proceeding. [[filtering-bridges-enabling]] == Enabling the Bridge At this point, to enable the bridge, you have to execute the following commands (having the shrewdness to replace the names of the two network interfaces [.filename]#fxp0# and [.filename]#xl0# with your own ones): [source,shell] .... # sysctl net.link.ether.bridge.config=fxp0:0,xl0:0 # sysctl net.link.ether.bridge.ipfw=1 # sysctl net.link.ether.bridge.enable=1 .... The first row specifies which interfaces should be activated by the bridge, the second one will enable the firewall on the bridge and finally the third one will enable the bridge. At this point you should be able to insert the machine between two sets of hosts without compromising any communication abilities between them. -If so, the next step is to add the `net.link.ether.bridge._[blah]_=_[blah]_` portions of these rows to the [.filename]#/etc/sysctl.conf# file, in order to have them execute at startup. +If so, the next step is to add the `net.link.ether.bridge._[blah]_=_[blah]_` portions of these rows to the [.filename]#/etc/sysctl.conf# file, to have them execute at startup. [[filtering-bridges-ipfirewall]] == Configuring The Firewall -Now it is time to create your own file with custom firewall rules, in order to secure the inside network. +Now it is time to create your own file with custom firewall rules, to secure the inside network. There will be some complication in doing this because not all of the firewall functionalities are available on bridged packets. Furthermore, there is a difference between the packets that are in the process of being forwarded and packets that are being received by the local machine. In general, incoming packets are run through the firewall only once, not twice as is normally the case; in fact they are filtered only upon receipt, so rules that use `out` or `xmit` will never match. Personally, I use `in via` which is an older syntax, but one that has a sense when you read it. Another limitation is that you are restricted to use only `pass` or `drop` commands for packets filtered by a bridge. Sophisticated things like `divert`, `forward` or `reject` are not available. Such options can still be used, but only on traffic to or from the bridge machine itself (if it has an IP address). New in FreeBSD 4.0, is the concept of stateful filtering. This is a big improvement for UDP traffic, which typically is a request going out, followed shortly thereafter by a response with the exact same set of IP addresses and port numbers (but with source and destination reversed, of course). For firewalls that have no statekeeping, there is almost no way to deal with this sort of traffic as a single session. But with a firewall that can "remember" an outgoing UDP packet and, for the next few minutes, allow a response, handling UDP services is trivial. The following example shows how to do it. It is possible to do the same thing with TCP packets. This allows you to avoid some denial of service attacks and other nasty tricks, but it also typically makes your state table grow quickly in size. Let's look at an example setup. Note first that at the top of [.filename]#/etc/rc.firewall# there are already standard rules for the loopback interface [.filename]#lo0#, so we should not have to care for them anymore. Custom rules should be put in a separate file (say [.filename]#/etc/rc.firewall.local#) and loaded at system startup, by modifying the row of [.filename]#/etc/rc.conf# where we defined the `open` firewall: [.programlisting] .... firewall_type="/etc/rc.firewall.local" .... [IMPORTANT] ==== You have to specify the _full_ path, otherwise it will not be loaded with the risk to remain isolated from the network. ==== For our example imagine to have the [.filename]#fxp0# interface connected towards the outside (Internet) and the [.filename]#xl0# towards the inside (LAN). The bridge machine has the IP `1.2.3.4` (it is not possible that your ISP can give you an address quite like this, but for our example it is good). [.programlisting] .... # Things that we have kept state on before get to go through in a hurry add check-state # Throw away RFC 1918 networks add drop all from 10.0.0.0/8 to any in via fxp0 add drop all from 172.16.0.0/12 to any in via fxp0 add drop all from 192.168.0.0/16 to any in via fxp0 # Allow the bridge machine to say anything it wants # (if the machine is IP-less do not include these rows) add pass tcp from 1.2.3.4 to any setup keep-state add pass udp from 1.2.3.4 to any keep-state add pass ip from 1.2.3.4 to any # Allow the inside hosts to say anything they want add pass tcp from any to any in via xl0 setup keep-state add pass udp from any to any in via xl0 keep-state add pass ip from any to any in via xl0 # TCP section # Allow SSH add pass tcp from any to any 22 in via fxp0 setup keep-state # Allow SMTP only towards the mail server add pass tcp from any to relay 25 in via fxp0 setup keep-state # Allow zone transfers only by the secondary name server [dns2.nic.it] add pass tcp from 193.205.245.8 to ns 53 in via fxp0 setup keep-state # Pass ident probes. It is better than waiting for them to timeout add pass tcp from any to any 113 in via fxp0 setup keep-state # Pass the "quarantine" range add pass tcp from any to any 49152-65535 in via fxp0 setup keep-state # UDP section # Allow DNS only towards the name server add pass udp from any to ns 53 in via fxp0 keep-state # Pass the "quarantine" range add pass udp from any to any 49152-65535 in via fxp0 keep-state # ICMP section # Pass 'ping' add pass icmp from any to any icmptypes 8 keep-state # Pass error messages generated by 'traceroute' add pass icmp from any to any icmptypes 3 add pass icmp from any to any icmptypes 11 # Everything else is suspect add drop log all from any to any .... Those of you who have set up firewalls before may notice some things missing. In particular, there are no anti-spoofing rules, in fact we did _not_ add: [.programlisting] .... add deny all from 1.2.3.4/8 to any in via fxp0 .... That is, drop packets that are coming in from the outside claiming to be from our network. This is something that you would commonly do to be sure that someone does not try to evade the packet filter, by generating nefarious packets that look like they are from the inside. The problem with that is that there is _at least_ one host on the outside interface that you do not want to ignore: the router. But usually, the ISP anti-spoofs at their router, so we do not need to bother that much. The last rule seems to be an exact duplicate of the default rule, that is, do not let anything pass that is not specifically allowed. But there is a difference: all suspected traffic will be logged. There are two rules for passing SMTP and DNS traffic towards the mail server and the name server, if you have them. Obviously the whole rule set should be flavored to personal taste, this is only a specific example (rule format is described accurately in the man:ipfw[8] man page). -Note that in order for "relay" and "ns" to work, name service lookups must work _before_ the bridge is enabled. +Note that for "relay" and "ns" to work, name service lookups must work _before_ the bridge is enabled. This is an example of making sure that you set the IP on the correct network card. Alternatively it is possible to specify the IP address instead of the host name (required if the machine is IP-less). People that are used to setting up firewalls are probably also used to either having a `reset` or a `forward` rule for ident packets (TCP port 113). Unfortunately, this is not an applicable option with the bridge, so the best thing is to simply pass them to their destination. As long as that destination machine is not running an ident daemon, this is relatively harmless. The alternative is dropping connections on port 113, which creates some problems with services like IRC (the ident probe must timeout). The only other thing that is a little weird that you may have noticed is that there is a rule to let the bridge machine speak, and another for internal hosts. Remember that this is because the two sets of traffic will take different paths through the kernel and into the packet filter. The inside net will go through the bridge, while the local machine will use the normal IP stack to speak. Thus the two rules to handle the different cases. The `in via fxp0` rules work for both paths. In general, if you use `in via` rules throughout the filter, you will need to make an exception for locally generated packets, because they did not come in via any of our interfaces. [[filtering-bridges-contributors]] == Contributors Many parts of this article have been taken, updated and adapted from an old text about bridging, edited by Nick Sayer. A pair of inspirations are due to an introduction on bridging by Steve Peterson. A big thanks to Luigi Rizzo for the implementation of the bridge code in FreeBSD and for the time he has dedicated to me answering all of my related questions. A thanks goes out also to Tom Rhodes who looked over my job of translation from Italian (the original language of this article) into English. diff --git a/documentation/content/en/articles/freebsd-releng/_index.adoc b/documentation/content/en/articles/freebsd-releng/_index.adoc index 02452258df..69a43a8d46 100644 --- a/documentation/content/en/articles/freebsd-releng/_index.adoc +++ b/documentation/content/en/articles/freebsd-releng/_index.adoc @@ -1,891 +1,891 @@ --- title: FreeBSD Release Engineering authors: - author: Glen Barber email: gjb@FreeBSD.org organizations: - organization: The FreeBSD Foundation webpage: https://www.freebsdfoundation.org/ - organization: Rubicon Communications, LLC (Netgate) webpage: https://www.netgate.com/ description: Describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization trademarks: ["freebsd", "intel", "general"] tags: ["releases", "engineering", "process", "FreeBSD"] --- = FreeBSD Release Engineering :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :teamBugmeister: FreeBSD Bugmeister Team :teamDoceng: FreeBSD Documentation Engineering Team :teamPortmgr: FreeBSD Ports Management Team :teamPostmaster: FreeBSD Postmaster Team :teamRe: FreeBSD Release Engineering Team :teamSecteam: FreeBSD Security Team :branchHead: head/ :branchStable: stable/ :branchStablex: stable/12/ :branchReleng: releng/ :branchRelengx: releng/12.0/ :branchReleasex: release/12.0.0/ :branchRevision: 12.0 :images-path: articles/freebsd-releng/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract This article describes the release engineering process of the FreeBSD Project. [NOTE] ==== This document has not yet been updated to describe the current release procedures of the FreeBSD Release Engineering team following the transition from Subversion to Git. ==== ''' toc::[] [[introduction]] == Introduction to the FreeBSD Release Engineering Process Development of FreeBSD has a very specific workflow. In general, all changes to the FreeBSD base system are committed to the {branchHead} branch, which reflects the top of the source tree. After a reasonable testing period, changes can then be merged to the {branchStable} branches. The default minimum timeframe before merging to {branchStable} branches is three (3) days. Although a general rule to wait a minimum of three days before merging from {branchHead}, there are a few special circumstances where an immediate merge may be necessary, such as a critical security fix, or a bug fix that directly inhibits the release build process. After several months, and the number of changes in the {branchStable} branch have grown significantly, it is time to release the next version of FreeBSD. These releases have been historically referred to as "point" releases. In between releases from the {branchStable} branches, approximately every two (2) years, a release will be cut directly from {branchHead}. These releases have been historically referred to as "dot-zero" releases. This article will highlight the workflow and responsibilities of the {teamRe} for both "dot-zero" and "point"' releases. The following sections of this article describe: <>:: General information and preparation before starting the release cycle. <>:: Website Changes During the Release Cycle <>:: Terminology and general information, such as the "code slush" and "code freeze", used throughout this document. <>:: The Release Engineering process for a "dot-zero" release. <>:: The Release Engineering process for a "point" release. <>:: Information related to the specific procedures to build installation medium. <>:: Procedures to publish installation medium. <>:: Wrapping up the release cycle. [[releng-prep]] == General Information and Preparation Approximately two months before the start of the release cycle, the {teamRe} decides on a schedule for the release. The schedule includes the various milestone points of the release cycle, such as freeze dates, branch dates, and build dates. For example: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | Milestone | Anticipated Date |{branchHead} slush: |May 27, 2016 |{branchHead} freeze: |June 10, 2016 |{branchHead} KBI freeze: |June 24, 2016 |`doc/` tree slush [1]: |June 24, 2016 |Ports quarterly branch [2]: |July 1, 2016 |{branchStablex} branch: |July 8, 2016 |`doc/` tree tag [3]: |July 8, 2016 |BETA1 build starts: |July 8, 2016 |{branchHead} thaw: |July 9, 2016 |BETA2 build starts: |July 15, 2016 |BETA3 build starts [*]: |July 22, 2016 |{branchRelengx} branch: |July 29, 2016 |RC1 build starts: |July 29, 2016 |{branchStablex} thaw: |July 30, 2016 |RC2 build starts: |August 5, 2016 |Final Ports package builds [4]: |August 6, 2016 |Ports release tag: |August 12, 2016 |RC3 build starts [*]: |August 12, 2016 |RELEASE build starts: |August 19, 2016 |RELEASE announcement: |September 2, 2016 |=== [NOTE] ==== Items marked with "[*]" are "as needed". ==== . The `doc/` tree slush is coordinated by the {teamDoceng}. . The Ports quarterly branch used is determined by when the final `RC` build is planned. A new quarterly branch is created on the first day of the quarter, so this metric should be used when taking the release cycle milestones into account. The quarterly branch is created by the {teamPortmgr}. . The `doc/` tree is tagged by the {teamDoceng}. . The final Ports package build is done by the {teamPortmgr} after the final (or what is expected to be final) `RC` build. [NOTE] ==== If the release is being created from an existing {branchStable} branch, the KBI freeze date can be excluded, since the KBI is already considered frozen on established {branchStable} branches. ==== When writing the release cycle schedule, a number of things need to be taken into consideration, in particular milestones where the target date depends on predefined milestones upon which there is a dependency. For example, the Ports Collection release tag originates from the active quarterly branch at the time of the last `RC`. This in part defines which quarterly branch is used, when the release tag can happen, and what revision of the ports tree is used for the final `RELEASE` build. After general agreement on the schedule, the {teamRe} emails the schedule to the FreeBSD Developers. It is somewhat typical that many developers will inform the {teamRe} about various works-in-progress. In some cases, an extension for the in-progress work will be requested, and in other cases, a request for "blanket approval" to a particular subset of the tree will be made. When such requests are made, it is important to make sure timelines (even if estimated) are discussed. For blanket approvals, the length of time for the blanket approval should be made clear. For example, a FreeBSD developer may request blanket approvals from the start of the code slush until the start of the `RC` builds. [NOTE] ==== -In order to keep track of blanket approvals, the {teamRe} uses an internal repository to keep a running log of such requests, which defines the area upon which a blanket approval was granted, the author(s), when the blanket approval expires, and the reason the approval was granted. +To keep track of blanket approvals, the {teamRe} uses an internal repository to keep a running log of such requests, which defines the area upon which a blanket approval was granted, the author(s), when the blanket approval expires, and the reason the approval was granted. One example of this is granting blanket approval to [.filename]#release/doc/# to all {teamRe} members until the final `RC` to update the release notes and other release-related documentation. ==== [NOTE] ==== The {teamRe} also uses this repository to track pending approval requests that are received just prior to starting various builds during the release cycle, which the Release Engineer specifies the cutoff period with an email to the FreeBSD developers. ==== Depending on the underlying set of code in question, and the overall impact the set of code has on FreeBSD as a whole, such requests may be approved or denied by the {teamRe}. The same applies to work-in-progress extensions. For example, in-progress work for a new device driver that is otherwise isolated from the rest of the tree may be granted an extension. A new scheduler, however, may not be feasible, especially if such dramatic changes do not exist in another branch. The schedule is also added to the Project website, in the `doc/` repository, in [.filename]#~/website/content/en/releases/{branchRevision}R/schedule.adoc#. This file is continuously updated as the release cycle progresses. [NOTE] ==== In most cases, the [.filename]#schedule.adoc# can be copied from a prior release and updated accordingly. ==== In addition to adding [.filename]#schedule.adoc# to the website, [.filename]#~/shared/releases.adoc# is also updated to add the link to the schedule to various subpages, as well as enabling the link to the schedule on the Project website index page. The schedule is also linked from [.filename]#~/website/content/en/releng/_index.adoc#. Approximately one month prior to the scheduled "code slush", the {teamRe} sends a reminder email to the FreeBSD Developers. [[releng-terms]] == Release Engineering Terminology This section describes some of the terminology used throughout the rest of this document. [[releng-terms-code-slush]] === The Code Slush Although the code slush is not a hard freeze on the tree, the {teamRe} requests that bugs in the existing code base take priority over new features. The code slush does not enforce commit approvals to the branch. [[releng-terms-code-freeze]] === The Code Freeze The code freeze marks the point in time where all commits to the branch require explicit approval from the {teamRe}. The FreeBSD Subversion repository contains several hooks to perform sanity checks before any commit is actually committed to the tree. One of these hooks will evaluate if committing to a particular branch requires specific approval. To enforce commit approvals by the {teamRe}, the Release Engineer updates [.filename]#base/svnadmin/conf/approvers#, and commits the change back to the repository. Once this is done, any change to the branch must include an "Approved by:" line in the commit message. The "Approved by:" line must match the second column in [.filename]#base/svnadmin/conf/approvers#, otherwise the commit will be rejected by the repository hooks. [NOTE] ==== During the code freeze, FreeBSD committers are urged to follow the link:https://wiki.freebsd.org/Releng/ChangeRequestGuidelines[Change Request Guidelines]. ==== [[releng-terms-kbi-freeze]] === The KBI/KPI Freeze KBI/KPI stability implies that the caller of a function across two different releases of software that implement the function results in the same end state. The caller, whether it is a process, thread, or function, expects the function to operate in a certain way, otherwise the KBI/KPI stability on the branch is broken. [[releng-website]] == Website Changes During the Release Cycle This section describes the changes to the website that should occur as the release cycle progresses. [NOTE] ==== The files specified throughout this section are relative to the `head/` branch of the `doc` repository in Subversion. ==== [[releng-website-prerelease]] === Website Changes Before the Release Cycle Begins When the release cycle schedule is available, these files need to be updated to enable various different functionalities on the FreeBSD Project website: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#~/shared/releases.adoc# |Change `beta-upcoming` from `IGNORE` to `INCLUDE` |[.filename]#~/shared/releases.adoc# |Change `beta-testing` from `IGNORE` to `INCLUDE` |=== [[releng-website-beta-rc]] === Website Changes During `BETA` or `RC` When transitioning from `PRERELEASE` to `BETA`, these files need to be updated to enable the "Help Test" block on the download page. All files are relative to [.filename]#head/# in the `doc` repository: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#share/releases.adoc# |Update `betarel-vers` to `BETA__1__` |[.filename]#~/website/data/en/news/news.toml# |Add an entry announcing the `BETA` |[.filename]#~/website/static/security/advisory-template.txt# |Add the new `BETA`, `RC`, or final `RELEASE` to the template |[.filename]#~/website/static/security/errata-template.txt# |Add the new `BETA`, `RC`, or final `RELEASE` to the template |=== Once the {branchRelengx} branch is created, the various release-related documents need to be generated and manually added to the `doc/` repository. Within [.filename]#release/doc#, invoke to generate [.filename]#errata.html#, [.filename]#hardware.html#, [.filename]#readme.html#, and [.filename]#relnotes.html# pages, which are then added to [.filename]#doc/head/en_US.ISO8859-1/htdocs/releases/X.YR/#, where _X.Y_ represents the major and minor version number of the release. The `fbsd:nokeywords` property must be set to `on` on the newly-added files before the pre-commit hooks will allow them to be added to the repository. [NOTE] ==== The relevant release-related documents exist in the [.filename]#doc# repository for FreeBSD 12.x and later. ==== [[releng-ports-beta-rc]] === Ports Changes During `BETA`, `RC`, and the Final `RELEASE` For each build during the release cycle, the `MANIFEST` files containing the `SHA256` of the various distribution sets, such as `base.txz`, `kernel.txz`, and so on, are added to the package:misc/freebsd-release-manifests[] port. This allows utilities other than , such as package:ports-mgmt/poudriere[], to safely use these distribution sets by providing a mechanism through which the checksums can be verified. [[releng-head]] == Release from {branchHead} This section describes the general procedures of the FreeBSD release cycle from the {branchHead} branch. [[releng-head-builds-alpha]] === FreeBSD "`ALPHA`" Builds Starting with the FreeBSD 10.0-RELEASE cycle, the notion of "`ALPHA`" builds was introduced. Unlike the `BETA` and `RC` builds, `ALPHA` builds are not included in the FreeBSD Release schedule. The idea behind `ALPHA` builds is to provide regular FreeBSD-provided builds before the creation of the {branchStable} branch. FreeBSD `ALPHA` snapshots should be built approximately once a week. For the first `ALPHA` build, the `BRANCH` value in [.filename]#sys/conf/newvers.sh# needs to be changed from `CURRENT` to `ALPHA1`. For subsequent `ALPHA` builds, increment each `ALPHA__N__` value by one. See <> for information on building the `ALPHA` images. [[releng-head-branching]] === Creating the {branchStablex} Branch When creating the {branchStable} branch, several changes are required in both the new {branchStable} branch and the {branchHead} branch. The files listed are relative to the repository root. To create the new {branchStablex} branch in Subversion: [source,shell,subs="attributes"] .... % svn cp ^/head {branchStablex} .... Once the {branchStablex} branch has been committed, make the following edits: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#stable/12/UPDATING# |Update the FreeBSD version, and remove the notice about `WITNESS` |[.filename]#stable/12/contrib/jemalloc/include/jemalloc/jemalloc_FreeBSD.h# a| [source,shell,subs="attributes"] .... #ifndef MALLOC_PRODUCTION #define MALLOC_PRODUCTION #endif .... |[.filename]#stable/12/lib/clang/llvm.build.mk# |Uncomment `-DNDEBUG` |[.filename]#stable/12/sys/\*/conf/GENERIC*# |Remove debugging support |[.filename]#stable/12/sys/*/conf/MINIMAL# |Remove debugging support |[.filename]#stable/12/release/release.conf.sample# |Update `SRCBRANCH` |[.filename]#stable/12/sys/*/conf/GENERIC-NODEBUG# |Remove these kernel configurations |[.filename]#stable/12/sys/arm/conf/std.arm*# |Remove debugging options |[.filename]#stable/12/sys/conf/newvers.sh# |Update the `BRANCH` value to reflect `BETA1` |[.filename]#stable/12/share/mk/src.opts.mk# |Move `REPRODUCIBLE_BUILD` from `\__DEFAULT_NO_OPTIONS` to `__DEFAULT_YES_OPTIONS` |[.filename]#stable/12/share/mk/src.opts.mk# |Move `LLVM_ASSERTIONS` from `\__DEFAULT_YES_OPTIONS` to `__DEFAULT_NO_OPTIONS` (FreeBSD 13.x and later only) |[.filename]#stable/12/libexec/rc/rc.conf# |Set `dumpdev` from `AUTO` to `NO` (it is configurable via for those that want it enabled by default) |[.filename]#stable/12/release/Makefile# |Remove the `debug.witness.trace` entries |=== Then in the {branchHead} branch, which will now become a new major version: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#head/UPDATING# |Update the FreeBSD version |[.filename]#head/sys/conf/newvers.sh# |Update the `BRANCH` value to reflect `CURRENT`, and increment `REVISION` |[.filename]#head/Makefile.inc1# |Update `TARGET_TRIPLE` and `MACHINE_TRIPLE` |[.filename]#head/sys/sys/param.h# |Update `__FreeBSD_version` |[.filename]#head/gnu/usr.bin/cc/cc_tools/freebsd-native.h# |Update `FBSD_MAJOR` and `FBSD_CC_VER` |[.filename]#head/contrib/gcc/config.gcc# |Append the `freebsdversion.h` section |[.filename]#head/lib/clang/llvm.build.mk# |Update the value of `OS_VERSION` |[.filename]#head/lib/clang/freebsd_cc_version.h# |Update `FREEBSD_CC_VERSION` |[.filename]#head/lib/clang/include/lld/Common/Version.inc# |Update `LLD_REVISION_STRING` |[.filename]#head/Makefile.libcompat# |Update `LIB32CPUFLAGS` |=== [[releng-stable]] == Release from {branchStable} This section describes the general procedures of the FreeBSD release cycle from an extablished {branchStable} branch. [[releng-stable-slush]] === FreeBSD `stable` Branch Code Slush In preparation for the code freeze on a `stable` branch, several files need to be updated to reflect the release cycle is officially in progress. These files are all relative to the top-most level of the stable branch: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#sys/conf/newvers.sh# |Update the `BRANCH` value to reflect `PRERELEASE` |[.filename]#Makefile.inc1# |Update `TARGET_TRIPLE` |[.filename]#lib/clang/llvm.build.mk# |Update `OS_VERSION` |[.filename]#Makefile.libcompat# |Update `LIB32CPUFLAGS` |=== [[releng-stable-builds-beta]] === FreeBSD `BETA` Builds Following the code slush, the next phase of the release cycle is the code freeze. This is the point at which all commits to the stable branch require explicit approval from the {teamRe}. This is enforced by pre-commit hooks in the Subversion repository by editing [.filename]#base/svnadmin/conf/approvers# to include a regular expression matching the {branchStablex} branch for the release: [.programlisting,subs="attributes"] .... ^/{branchStablex} re ^/{branchRelengx} re .... [NOTE] ==== There are two general exceptions to requiring commit approval during the release cycle. -The first is any change that needs to be committed by the Release Engineer in order to proceed with the day-to-day workflow of the release cycle, the other is security fixes that may occur during the release cycle. +The first is any change that needs to be committed by the Release Engineer to proceed with the day-to-day workflow of the release cycle, the other is security fixes that may occur during the release cycle. ==== Once the code freeze is in effect, the next build from the branch is labeled `BETA1`. This is done by updating the `BRANCH` value in [.filename]#sys/conf/newvers.sh# from `PRERELEASE` to `BETA1`. Once this is done, the first set of `BETA` builds are started. Subsequent `BETA` builds do not require updates to any files other than [.filename]#sys/conf/newvers.sh#, incrementing the `BETA` build number. [[releng-stable-branching]] === Creating the {branchRelengx} Branch When the first `RC` (Release Candidate) build is ready to begin, the {branchReleng} branch is created. -This is a multi-step process that must be done in a specific order, in order to avoid anomalies such as overlaps with `__FreeBSD_version` values, for example. +This is a multi-step process that must be done in a specific order, to avoid anomalies such as overlaps with `__FreeBSD_version` values, for example. The paths listed below are relative to the repository root. The order of commits and what to change are: [source,shell,subs="attributes"] .... % svn cp ^/{branchStablex} {branchRelengx} .... [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#releng/12.0/sys/conf/newvers.sh# |Change `BETA__X__` to `RC1` |[.filename]#releng/12.0/sys/sys/param.h# |Update `__FreeBSD_version` |[.filename]#releng/12.0/sys/conf/kern.opts.mk |Move `REPRODUCIBLE_BUILD` from `__DEFAULT_NO_OPTIONS` to `__DEFAULT_YES_OPTIONS` |[.filename]#releng/12.0/etc/pkg/FreeBSD.conf# |Replace `latest` with `quarterly` as the default package repository location |[.filename]#releng/12.0/release/pkg_repos/release-dvd.conf# |Replace `latest` with `quarterly` as the default package repository location |[.filename]#stable/12/sys/conf/newvers.sh# |Update `BETA__X__` with `PRERELEASE` |[.filename]#stable/12/sys/sys/param.h# |Update `__FreeBSD_version` |[.filename]#svnadmin/conf/approvers# |Add a new approvers line for the releng branch as was done for the stable branch |=== [source,shell,subs="attributes"] .... % svn propdel -R svn:mergeinfo {branchRelengx} % svn commit {branchRelengx} % svn commit {branchStablex} .... Now that two new `__FreeBSD_version` values exist, also update [.filename]#~/documentation/content/en/books/porters-handbook/versions/chapter.adoc# in the Documentation Project repository. After the first `RC` build has completed and tested, the {branchStable} branch can be "thawed" by removing (or commenting) the ^/{branchStablex} entry in [.filename]#svnadmin/conf/approvers#. Following the availability of the first `RC`, {teamBugmeister} should be emailed to add the new FreeBSD `-RELEASE` to the `versions` available in the drop-down menu shown in the bug tracker. [[releng-building]] == Building FreeBSD Installation Media This section describes the general procedures producing FreeBSD development snapshots and releases. [[releng-build-scripts]] === Release Build Scripts This section describes the build scripts used by {teamRe} to produce development snapshots and releases. [[releng-build-scripts-single]] ==== The [.filename]#release.sh# Script Prior to FreeBSD 9.0-RELEASE, [.filename]#src/release/Makefile# was updated to support , and the [.filename]#src/release/generate-release.sh# script was introduced as a wrapper to automate invoking the targets. Prior to FreeBSD 9.2-RELEASE, [.filename]#src/release/release.sh# was introduced, which heavily based on [.filename]#src/release/generate-release.sh# included support to specify configuration files to override various options and environment variables. Support for configuration files provided support for cross building each architecture for a release by specifying a separate configuration file for each invocation. As a brief example of using [.filename]#src/release/release.sh# to build a single release in [.filename]#/scratch#: [source,shell,subs="attributes"] .... # /bin/sh /usr/src/release/release.sh .... As a brief example of using [.filename]#src/release/release.sh# to build a single, cross-built release using a different target directory, create a custom [.filename]#release.conf# containing: [.programlisting,subs="attributes"] .... # release.sh configuration for powerpc/powerpc64 CHROOTDIR="/scratch-powerpc64" TARGET="powerpc" TARGET_ARCH="powerpc64" KERNEL="GENERIC64" .... Then invoke [.filename]#src/release/release.sh# as: [source,shell,subs="attributes"] .... # /bin/sh /usr/src/release/release.sh -c $HOME/release.conf .... See and [.filename]#src/release/release.conf.sample# for more details and example usage. [[releng-build-scripts-multiple]] ==== The [.filename]#thermite.sh# Wrapper Script -In order to make cross building the full set of architectures supported on a given branch faster, easier, and reduce human error factors, a wrapper script around [.filename]#src/release/release.sh# was written to iterate through the various combinations of architectures and invoke [.filename]#src/release/release.sh# using a configuration file specific to that architecture. +To make cross building the full set of architectures supported on a given branch faster, easier, and reduce human error factors, a wrapper script around [.filename]#src/release/release.sh# was written to iterate through the various combinations of architectures and invoke [.filename]#src/release/release.sh# using a configuration file specific to that architecture. The wrapper script is called [.filename]#thermite.sh#, which is available in the FreeBSD Subversion repository at `svn://svn.freebsd.org/base/user/gjb/thermite/`, in addition to configuration files used to build {branchHead} and {branchStablex} development snapshots. Using [.filename]#thermite.sh# is covered in <> and <>. Each architecture and individual kernel have their own configuration file used by [.filename]#release.sh#. Each branch has its own [.filename]#defaults-X.conf# configuration which contains entries common throughout each architecture, where overrides or special variables are set and/or overridden in the per-build files. The per-build configuration file naming scheme is in the form of [.filename]#${revision}-${TARGET_ARCH}-${KERNCONF}-${type}.conf#, where the uppercase variables are equivalent to what uses in the build system, and lowercase variables are set within the configuration files, mapping to the major version of the respective branch. Each branch also has its own [.filename]#builds-X.conf# configuration, which is used by [.filename]#thermite.sh#. The [.filename]#thermite.sh# script iterates through each ${revision}, ${TARGET_ARCH}, ${KERNCONF}, and ${type} value, creating a master list of what to build. However, a given combination from the list will only be built if the respective configuration file exists, which is where the naming scheme above is relevant. There are two paths of file sourcing: * [.filename]#builds-12.conf# - [.filename]#main.conf# + This controls [.filename]#thermite.sh# behavior * [.filename]#12-amd64-GENERIC-snap.conf# - [.filename]#defaults-12.conf# - [.filename]#main.conf# + This controls [.filename]#release/release.sh# behavior within the build [NOTE] ==== The [.filename]#builds-12.conf#, [.filename]#defaults-12.conf#, and [.filename]#main.conf# configuration files exist to reduce repetition between the various per-build files. ==== [[releng-build-snapshot]] === Building FreeBSD Development Snapshots The official release build machines have a specific filesystem layout, which using ZFS, [.filename]#thermite.sh# takes heavy advantage of with clones and snapshots, ensuring a pristine build environment. The build scripts reside in [.filename]#/releng/scripts-snapshot/scripts# or [.filename]#/releng/scripts-release/scripts# respectively, to avoid collisions between an `RC` build from a releng branch versus a `STABLE` snapshot from the respective stable branch. A separate dataset exists for the final build images, [.filename]#/snap/ftp#. This directory contains both snapshots and releases directories. They are only used if the `EVERYTHINGISFINE` variable is defined in [.filename]#main.conf#. [NOTE] ==== The `EVERYTHINGISFINE` variable name was chosen to avoid colliding with a variable that might be possibly set in the user environment, accidentally enabling the behavior that depends on it being defined. ==== As [.filename]#thermite.sh# iterates through the master list of combinations and locates the per-build configuration file, a ZFS dataset is created under [.filename]#/releng#, such as [.filename]#/releng/12-amd64-GENERIC-snap#. The `src/`, `ports/`, and `doc/` trees are checked out to separate ZFS datasets, such as [.filename]#/releng/12-src-snap#, which are then cloned and mounted into the respective build datasets. This is done to avoid checking out a given tree more than once. Assuming these filesystem paths, [.filename]#thermite.sh# would be invoked as: [source,shell,subs="attributes"] .... # cd /releng/scripts-snapshot/scripts # ./setrev.sh -b {branchStablex} # ./zfs-cleanup.sh -c ./builds-12.conf # ./thermite.sh -c ./builds-12.conf .... Once the builds have completed, additional helper scripts are available to generate development snapshot emails which are sent to the `freebsd-snapshots@freebsd.org` mailing list: [source,shell,subs="attributes"] .... # cd /releng/scripts-snapshot/scripts # ./get-checksums.sh -c ./builds-12.conf | ./generate-email.pl > snapshot-12-mail .... [NOTE] ==== The generated output should be double-checked for correctness, and the email itself should be PGP signed, in-line. ==== [NOTE] ==== These helper scripts only apply to development snapshot builds. Announcements during the release cycle (excluding the final release announcement) are created from an email template. A sample of the email template currently used can be found link:here[here]. ==== [[releng-build-release]] === Building FreeBSD Releases Similar to building FreeBSD development snapshots, [.filename]#thermite.sh# would be invoked the same way. The difference between development snapshots and release builds, `BETA` and `RC` included, is that the configuration files must be named with `release` instead of `snap` as the type, as mentioned above. In addition, the `BUILDTYPE` and `types` must be changed from `snap` to `release` in [.filename]#defaults-12.conf# and [.filename]#builds-12.conf#, respectively. When building `BETA`, `RC`, and the final `RELEASE`, also statically set `BUILDSVNREV` to the revision on the branch reflecting the name change, `BUILDDATE` to the date the builds are started in `YYYYMMDD` format. If the `doc/` and `ports/` trees have been tagged, also set `PORTBRANCH` and `DOCBRANCH` to the relevant tag path in the Subversion repository, replacing `HEAD` with the last changed revision. Also set `releasesrc` in [.filename]#builds-12.conf# to the relevant branch, such as {branchStablex} or {branchRelengx}. During the release cycle, a copy of [.filename]#CHECKSUM.SHA512# and [.filename]#CHECKSUM.SHA256# for each architecture are stored in the {teamRe} internal repository in addition to being included in the various announcement emails. Each [.filename]#MANIFEST# containing the hashes of [.filename]#base.txz#, [.filename]#kernel.txz#, etc. are added to package:misc/freebsd-release-manifests[] in the Ports Collection, as well. In preparation for the release build, several files need to be updated: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#sys/conf/newvers.sh# |Update the `BRANCH` value to `RELEASE` |[.filename]#UPDATING# |Add the anticipated announcement date |[.filename]#lib/csu/common/crtbrand.S# |Replace `__FreeBSD_version` with the value in [.filename]#sys/sys/param.h# |=== After building the final `RELEASE`, the {branchRelengx} branch is tagged as {branchReleasex} using the revision from which the `RELEASE` was built. Similar to creating the {branchStablex} and {branchRelengx} branches, this is done with `svn cp`. From the repository root: [source,shell,subs="attributes"] .... % svn cp ^/{branchRelengx}@r306420 {branchReleasex} % svn commit {branchReleasex} .... [[releng-mirrors]] == Publishing FreeBSD Installation Media to Project Mirrors This section describes the procedure to publish FreeBSD development snapshots and releases to the Project mirrors. [[releng-mirrors-staging]] === Staging FreeBSD Installation Media Images Staging FreeBSD snapshots and releases is a two part process: * Creating the directory structure to match the hierarchy on `ftp-master` + If `EVERYTHINGISFINE` is defined in the build configuration files, [.filename]#main.conf# in the case of the build scripts referenced above, this happens automatically in the after the build is complete, creating the directory structure in [.filename]#${DESTDIR}/R/ftp-stage# with a path structure matching what is expected on `ftp-master`. This is equivalent to running the following in the directly: + [source,shell,subs="attributes"] .... # make -C /usr/src/release -f Makefile.mirrors EVERYTHINGISFINE=1 ftp-stage .... + After each architecture is built, [.filename]#thermite.sh# will rsync the [.filename]#${DESTDIR}/R/ftp-stage# from the build to [.filename]#/snap/ftp/snapshots# or [.filename]#/snap/ftp/releases# on the build host, respectively. * Copying the files to a staging directory on `ftp-master` before moving the files into [.filename]#pub/# to begin propagation to the Project mirrors + Once all builds have finished, [.filename]#/snap/ftp/snapshots#, or [.filename]#/snap/ftp/releases# for a release, is polled by `ftp-master` using rsync to [.filename]#/archive/tmp/snapshots# or [.filename]#/archive/tmp/releases#, respectively. + [NOTE] ==== On `ftp-master` in the FreeBSD Project infrastructure, this step requires `root` level access, as this step must be executed as the `archive` user. ==== [[releng-mirrors-publishing]] === Publishing FreeBSD Installation Media Once the images are staged in [.filename]#/archive/tmp/#, they are ready to be made public by putting them in [.filename]#/archive/pub/FreeBSD#. -In order to reduce propagation time, is used to create hard links from [.filename]#/archive/tmp# to [.filename]#/archive/pub/FreeBSD#. +To reduce propagation time, is used to create hard links from [.filename]#/archive/tmp# to [.filename]#/archive/pub/FreeBSD#. [NOTE] ==== -In order for this to be effective, both [.filename]#/archive/tmp# and [.filename]#/archive/pub# must reside on the same logical filesystem. +For this to be effective, both [.filename]#/archive/tmp# and [.filename]#/archive/pub# must reside on the same logical filesystem. ==== -There is a caveat, however, where rsync must be used after in order to correct the symbolic links in [.filename]#pub/FreeBSD/snapshots/ISO-IMAGES# which will replace with a hard link, increasing the propagation time. +There is a caveat, however, where rsync must be used after to correct the symbolic links in [.filename]#pub/FreeBSD/snapshots/ISO-IMAGES# which will replace with a hard link, increasing the propagation time. [NOTE] ==== As with the staging steps, this requires `root` level access, as this step must be executed as the `archive` user. ==== As the `archive` user: [source,shell,subs="attributes"] .... % cd /archive/tmp/snapshots % pax -r -w -l . /archive/pub/FreeBSD/snapshots % /usr/local/bin/rsync -avH /archive/tmp/snapshots/* /archive/pub/FreeBSD/snapshots/ .... Replace _snapshots_ with _releases_ as appropriate. [[releng-wrapup]] == Wrapping up the Release Cycle This section describes general post-release tasks. [[releng-wrapup-en]] === Post-Release Errata Notices As the release cycle approaches conclusion, it is common to have several EN (Errata Notice) candidates to address issues that were discovered late in the cycle. Following the release, the {teamRe} and the {teamSecteam} revisit changes that were not approved prior to the final release, and depending on the scope of the change in question, may issue an EN. [NOTE] ==== The actual process of issuing ENs is handled by the {teamSecteam}. ==== To request an Errata Notice after a release cycle has completed, a developer should fill out the https://www.freebsd.org/security/errata-template.txt[Errata Notice template], in particular the `Background`, `Problem Description`, `Impact`, and if applicable, `Workaround` sections. The completed Errata Notice template should be emailed together with either a patch against the {branchReleng} branch or a list of revisions from the {branchStable} branch. For Errata Notice requests immediately following the release, the request should be emailed to both the {teamRe} and the {teamSecteam}. Once the {branchReleng} branch has been handed over to the {teamSecteam} as described in <>, Errata Notice requests should be sent to the {teamSecteam}. [[releng-wrapup-handoff]] === Handoff to the {teamSecteam} Roughly two weeks following the release, the Release Engineer updates [.filename]#svnadmin/conf/approvers# changing the approver column from `re` to `(so|security-officer)` for the {branchRelengx} branch. [[releng-eol]] == Release End-of-Life This section describes the website-related files to update when a release reaches EoL (End-of-Life). [[releng-eol-website]] === Website Updates for End-of-Life When a release reaches End-of-Life, references to that release should be removed and/or updated on the website: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File | What to Change |[.filename]#~/website/themes/beastie/layouts/index.html# |Remove `u-relXXX-announce` and `u-relXXX-announce` references. |[.filename]#~/website/content/en/releases/_index.adoc# |Move the `u-relXXX-*` variables from the supported release list to the Legacy Releases list. |[.filename]#~/website/content/en/releng/_index.adoc# |Update the appropriate releng branch to refelect the branch is no longer supported. |[.filename]#~/website/content/en/security/_index.adoc# |Remove the branch from the supported branch list. |[.filename]#~/website/content/en/where.adoc# |Remove the URLs for the release. |[.filename]#~/website/themes/beastie/layouts/partials/sidenav.html# |Remove `u-relXXX-announce` and `u-relXXX-announce` references. |[.filename]#~/website/static/security/advisory-template.txt# |Remove references to the release and releng branch. |[.filename]#~/website/static/security/errata-template.txt# |Remove references to the release and releng branch. |=== diff --git a/documentation/content/en/articles/ipsec-must/_index.adoc b/documentation/content/en/articles/ipsec-must/_index.adoc index 361f70406f..d4de24e1a9 100644 --- a/documentation/content/en/articles/ipsec-must/_index.adoc +++ b/documentation/content/en/articles/ipsec-must/_index.adoc @@ -1,294 +1,294 @@ --- title: Independent Verification of IPsec Functionality in FreeBSD authors: - author: David Honig email: honig@sprynet.com description: Independent Verification of IPsec Functionality in FreeBSD trademarks: ["freebsd", "opengroup", "general"] tags: ["IPsec", "verification", "FreeBSD"] --- = Independent Verification of IPsec Functionality in FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/ipsec-must/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract You installed IPsec and it seems to be working. How do you know? I describe a method for experimentally verifying that IPsec is working. ''' toc::[] [[problem]] == The Problem First, lets assume you have <>. How do you know it is <>? Sure, your connection will not work if it is misconfigured, and it will work when you finally get it right. man:netstat[1] will list it. But can you independently confirm it? [[solution]] == The Solution First, some crypto-relevant info theory: . Encrypted data is uniformly distributed, i.e., has maximal entropy per symbol; . Raw, uncompressed data is typically redundant, i.e., has sub-maximal entropy. Suppose you could measure the entropy of the data to- and from- your network interface. Then you could see the difference between unencrypted data and encrypted data. This would be true even if some of the data in "encrypted mode" was not encrypted---as the outermost IP header must be if the packet is to be routable. [[MUST]] === MUST Ueli Maurer's "Universal Statistical Test for Random Bit Generators"(https://web.archive.org/web/20011115002319/http://www.geocities.com/SiliconValley/Code/4704/universal.pdf[MUST]) quickly measures the entropy of a sample. It uses a compression-like algorithm. <> for a variant which measures successive (~quarter megabyte) chunks of a file. [[tcpdump]] === Tcpdump We also need a way to capture the raw network data. A program called man:tcpdump[1] lets you do this, if you have enabled the _Berkeley Packet Filter_ interface in your <>. The command: [source,shell] .... tcpdump -c 4000 -s 10000 -w dumpfile.bin .... will capture 4000 raw packets to _dumpfile.bin_. Up to 10,000 bytes per packet will be captured in this example. [[experiment]] == The Experiment Here is the experiment: [.procedure] ==== . Open a window to an IPsec host and another window to an insecure host. . Now start <>. . In the "secure" window, run the UNIX(R) command man:yes[1], which will stream the `y` character. After a while, stop this. Switch to the insecure window, and repeat. After a while, stop. . Now run <> on the captured packets. You should see something like the following. The important thing to note is that the secure connection has 93% (6.7) of the expected value (7.18), and the "normal" connection has 29% (2.1) of the expected value. + [source,shell] .... % tcpdump -c 4000 -s 10000 -w ipsecdemo.bin % uliscan ipsecdemo.bin Uliscan 21 Dec 98 L=8 256 258560 Measuring file ipsecdemo.bin Init done Expected value for L=8 is 7.1836656 6.9396 -------------------------------------------------------- 6.6177 ----------------------------------------------------- 6.4100 --------------------------------------------------- 2.1101 ----------------- 2.0838 ----------------- 2.0983 ----------------- .... ==== [[caveat]] == Caveat This experiment shows that IPsec _does_ seem to be distributing the payload data __uniformly__, as encryption should. However, the experiment described here _cannot_ detect many possible flaws in a system (none of which do I have any evidence for). These include poor key generation or exchange, data or keys being visible to others, use of weak algorithms, kernel subversion, etc. Study the source; know the code. [[IPsec]] == IPsec---Definition Internet Protocol security extensions to IPv4; required for IPv6. A protocol for negotiating encryption and authentication at the IP (host-to-host) level. SSL secures only one application socket; SSH secures only a login; PGP secures only a specified file or message. IPsec encrypts everything between two hosts. [[ipsec-install]] == Installing IPsec Most of the modern versions of FreeBSD have IPsec support in their base source. So you will need to include the `IPSEC` option in your kernel config and, after kernel rebuild and reinstall, configure IPsec connections using man:setkey[8] command. A comprehensive guide on running IPsec on FreeBSD is provided in extref:{handbook}[FreeBSD Handbook, ipsec]. [[kernel]] == src/sys/i386/conf/KERNELNAME -This needs to be present in the kernel config file in order to capture network data with man:tcpdump[1]. +This needs to be present in the kernel config file to capture network data with man:tcpdump[1]. Be sure to run man:config[8] after adding this, and rebuild and reinstall. [.programlisting] .... device bpf .... [[code]] == Maurer's Universal Statistical Test (for block size=8 bits) You can find the same code at https://web.archive.org/web/20031204230654/http://www.geocities.com:80/SiliconValley/Code/4704/uliscanc.txt[this link]. [.programlisting] .... /* ULISCAN.c ---blocksize of 8 1 Oct 98 1 Dec 98 21 Dec 98 uliscan.c derived from ueli8.c This version has // comments removed for Sun cc This implements Ueli M Maurer's "Universal Statistical Test for Random Bit Generators" using L=8 Accepts a filename on the command line; writes its results, with other info, to stdout. Handles input file exhaustion gracefully. Ref: J. Cryptology v 5 no 2, 1992 pp 89-105 also on the web somewhere, which is where I found it. -David Honig honig@sprynet.com Usage: ULISCAN filename outputs to stdout */ #define L 8 #define V (1< #include int main(argc, argv) int argc; char **argv; { FILE *fptr; int i,j; int b, c; int table[V]; double sum = 0.0; int iproduct = 1; int run; extern double log(/* double x */); printf("Uliscan 21 Dec 98 \nL=%d %d %d \n", L, V, MAXSAMP); if (argc < 2) { printf("Usage: Uliscan filename\n"); exit(-1); } else { printf("Measuring file %s\n", argv[1]); } fptr = fopen(argv[1],"rb"); if (fptr == NULL) { printf("Can't find %s\n", argv[1]); exit(-1); } for (i = 0; i < V; i++) { table[i] = 0; } for (i = 0; i < Q; i++) { b = fgetc(fptr); table[b] = i; } printf("Init done\n"); printf("Expected value for L=8 is 7.1836656\n"); run = 1; while (run) { sum = 0.0; iproduct = 1; if (run) for (i = Q; run && i < Q + K; i++) { j = i; b = fgetc(fptr); if (b < 0) run = 0; if (run) { if (table[b] > j) j += K; sum += log((double)(j-table[b])); table[b] = i; } } if (!run) printf("Premature end of file; read %d blocks.\n", i - Q); sum = (sum/((double)(i - Q))) / log(2.0); printf("%4.4f ", sum); for (i = 0; i < (int)(sum*8.0 + 0.50); i++) printf("-"); printf("\n"); /* refill initial table */ if (0) { for (i = 0; i < Q; i++) { b = fgetc(fptr); if (b < 0) { run = 0; } else { table[b] = i; } } } } } .... diff --git a/documentation/content/en/articles/license-guide/_index.adoc b/documentation/content/en/articles/license-guide/_index.adoc index e6d91491dc..44fbb70d00 100644 --- a/documentation/content/en/articles/license-guide/_index.adoc +++ b/documentation/content/en/articles/license-guide/_index.adoc @@ -1,331 +1,331 @@ --- title: FreeBSD Licensing Policy authors: - author: Warner Losh email: imp@FreeBSD.org trademarks: ["freebsd", "general"] --- = FreeBSD License Policies :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: ''' toc::[] [[intro]] [[pref-license]] == Preferred License for New Files The rest of this section is intended to help you get started. As a rule, when in doubt, ask. It is much easier to receive advice than to fix the source tree. The FreeBSD Project makes use of both explicit licenses (where the verbatim text of the license is reproduced in each file) and detached licenses (where a tag in the file specifies the license, as described in this document). The FreeBSD Project uses this text as the preferred license: [.programlisting] .... /*- * Copyright (c) [year] [your name] * * SPDX-License-Identifier: BSD-2-Clause */ .... The FreeBSD project does not allow using the "advertising clause" in new code. Due to the large number of contributors to the FreeBSD project, complying with this clause for many commercial vendors has become difficult. If you have code in the tree with the advertising clause, please consider switching to a license without it. New contributions to FreeBSD should use the BSD-2-Clause license. The FreeBSD project discourages completely new licenses and variations on the standard licenses. New licenses require the approval of {core-email} to reside in the main repository. In the past, non-standard licenses have generated more problems than standard ones. Poor drafting of non-standard licenses often causes more unintended consequences, so they are unlikely to be approved by {core-email}. The FreeBSD project is standardizing on the BSD-2-Clause license, as published by SPDX. In addition, project policy requires that code licensed under some non-BSD licenses must be placed in specific sections of the repository. For some licenses, compilation must be conditional or disabled by default. For example, code in the static part of the GENERIC kernel must be licensed under the BSD or substantially similar licenses. GPL, APSL, CDDL, etc, licensed software must not be compiled into the static GENERIC kernel. Code with these licenses may be used in pre-compiled modules, however. Developers are reminded that, in open source, getting "open" correct is just as important as getting "source" correct. Improper handling of intellectual property has serious consequences. Any questions or concerns should immediately be brought to the attention of {core-email}. [[license-policy]] == Software License Policy The following sections outline the project's Software License Policies in detail. -For the most part we expect developers to read, understand and utilize the sections above this one in order to apply appropriate licenses to their contributions. +For the most part we expect developers to read, understand and utilize the sections above this one to apply appropriate licenses to their contributions. The rest of this document details the philosophical background to the policies as well as the policies in great detail. As always, if the text below is confusing or you need help with applying these policies, please reach out to {core-email}. === Guiding Principles The FreeBSD Project aims to produce a complete, BSD-licensed operating system allowing consumers of the system to produce derivative products without constraint or further license obligations. We invite and greatly appreciate the contribution of both changes and additions under the two-clause BSD license, and encourage the adoption of this license by other open source projects. Use of the BSD license is key to encouraging the adoption of advanced operating system technology, and on many notable occasions has been pivotal to widespread use of new technology. We accept however that compelling reasons exist to allow differently-licensed software to be included in the FreeBSD source tree. We require software licensed under some non-BSD licenses to be carefully isolated in the source tree so that it cannot contaminate BSD-only components. Such cautious management encourages licensing clarity and facilitates the production of BSD-only derivative products. Unless a special exception is made, no existing BSD-licensed components may be replaced with more restrictively licensed software. We encourage FreeBSD and third party developers to seek the relicensing, dual-licensing, or reimplementing of critical components under the BSD license instead. Such would ease their more integral adoption into the FreeBSD operating system. === Policy * The import of new software licensed under any licenses other than the BSD license and BSD-Like Licenses (as defined below) requires the prior approval of the FreeBSD Core Team. Requests for import must include: ** A list of features or bug fixes that the new version or patches contain, along with evidence that our users need those features. PRs or references to mailing list discussions are ideal forms of evidence. ** This process should be used for all software imports, not just those that require Core Team review. The mere existence of a new version does not justify an import of software to source or ports. ** A list of FreeBSD branches that may be affected. Expansions of scope require a new request to and approval from the FreeBSD Core Team. * The Apache License 2.0 is acceptable for use in some cases. The Core Team must approve the import of new Apache License licensed components or the change of license of existing components to the Apache License. ** This license is approved for the following components: *** LLVM toolchain and (with LLVM Exceptions) runtime components. * The BSD+Patent License is acceptable for use in some cases. The Core Team must approve the import of new BSD+Patent License licensed components or the change of license of existing components to the BSD+Patent License. ** This license is approved for the following components: *** EDK2 derived code related to UEFI functionality * The Common Development and Distribution License (CDDL) is acceptable for use in some cases. The Core Team must approve the import of new CDDL licensed components or the change of license of existing components to the CDDL. ** This license is approved for the following components: *** DTrace *** ZFS filesystem, including kernel support and userland utilities * Historically, the phrase 'All Rights Reserved.' was included in all copyright notices. -All the BSD releases had it, in order to comply with the https://en.wikipedia.org/wiki/Buenos_Aires_Convention[Buenos Aires Convention of 1910] in the Americas. +All the BSD releases had it, to comply with the https://en.wikipedia.org/wiki/Buenos_Aires_Convention[Buenos Aires Convention of 1910] in the Americas. With the ratification of the https://en.wikipedia.org/wiki/Berne_Convention[Berne Convention] in 2000 by Nicaragua, the Buenos Aires Convention -- and the phrase -- became obsolete. As such, the FreeBSD project recommends that new code omit the phrase and encourages existing copyright holders to remove it. In 2018, the project updated its templates to remove it. * Initially, many items in the FreeBSD tree were marked with BSD-2-Clause-FreeBSD. However, SPDX has obsoleted the license as a variant; and the SPDX text of the obsolete tag differs enough from the standard FreeBSD license that it shouldn't be used. A review of its current use is ongoing. ==== Acceptable licenses The following licenses are considered to be acceptable BSD-Like Licenses for the purpose of this Policy. Deviations or the use of any other license must be approved by the FreeBSD Core Team: * The 2 clause version of the BSD license [.programlisting] .... /*- * Copyright (c) [year] [your name] * * SPDX-License-Identifier: BSD-2-Clause */ .... * The 3 clause version of the BSD license [.programlisting] .... /*- * Copyright (c) [year] [your name] * * SPDX-License-Identifier: BSD-3-Clause */ .... * The ISC License [.programlisting] .... /*- * Copyright (c) [year] [copyright holder] * * SPDX-License-Identifier: ISC */ .... * The MIT License [.programlisting] .... /*- * Copyright (c) [year] [copyright holders] * * SPDX-License-Identifier: MIT */ .... == Software Collection License The FreeBSD Project licenses its compilation of software as described in [.filename]#COPYRIGHT# under the BSD-2-Clause license. This license does not supersede the license of individual files, which is described below. Files that do not have an explicit license are licensed under the BSD-2-Clause license. == License File Location To comply with the https://reuse.software/[REUSE Software] standard as much as possible, all license files will be stored in the [.filename]#LICENSES/# directory of the repository. There are three subdirectories under this top level directory. The [.filename]#LICENSES/text/# subdirectory contains, in detached form, the text of all the licenses that are allowed in the FreeBSD software collection. These files are stored using the SPDX-License-Identifier name followed by .txt. The [.filename]#LICENSES/exceptions/# subdirectory has the text of all exceptions that are allowed in detached form in the FreeBSD software collection. These files are stored using the exception identifier name followed by .txt. The [.filename]#LICENSES/other/# contains, in detached form, the license files references in SPDX-License-Identifier expressions, but aren't otherwise allowed as detached licenses. All such files must appear at least once in the FreeBSD software collection, and should be removed when the last file that references them is removed. Licenses that have no adequate SPDX matching license must be in [.filename]#LICENSES/other/# and have a filename that starts with LicenseRef- followed by a unique idstring. No such files have currently been identified, but if they are, a full list will appear here. The FreeBSD Project currently does not make use of the `DEP5` files described in the `REUSE Software` standard. The FreeBSD Project has not marked all the files in the tree yet in accordance with this standard, as described later in this document. The FreeBSD Project has not yet included these files in its repositories since this policy is still evolving. [[individual-files]] == Individual Files License Each individual file in the FreeBSD software collection has its own copyright and license. How they are marked varies and is described in this section. A copyright notice identifies who claims the legal copyright to a file. These are provided on a best effort basis by the project. Because copyrights may be legally transferred, the current copyright holder may differ from what is listed in the file. A license is a legal document between the contributor and the users of the software granting permission to use the copyrighted portions of the software, subject to certain terms and conditions set forth in the license. Licenses can be expressed in one of two ways in the FreeBSD software collection. Licenses can be explicit in a file. When a license grant is explicit in the file, that file may be used, copied, and modified in accordance with that license. Licenses can also be expressed indirectly, where the text of the license is elsewhere. The project uses the Software Package Data Exchange (SPDX) license identifiers for this purpose, as described in the following subsections. SPDX license identifiers are managed by the SPDX Workgroup at the Linux Foundation, and have been agreed on by partners throughout the industry, tool vendors, and legal teams. For further information see https://spdx.org/ and the following sections for how the FreeBSD Project uses them. Entities that contribute fixes and enhancements to the software collection without an explicit license agree to license those changes under the terms that apply to the modified file(s). Project policy, in line with industry practice, only includes a copyright notice from significant contributors to the files in the collection. There are four types of files in the FreeBSD software collection: . Files that have only an explicit copyright notice and license. . Files that have both an explicit copyright notice and license, and a SPDX-License-Identifier tag. . Files that have only a copyright notice and an SPDX-License-Identifier tag, but no explicit license. . Files that lack any copyright or license at all. === Only Copyright and License Many files in the FreeBSD software collection have both a copyright notice and an explicit license contained in the file. In these cases, the license contained in the file governs. === Copyright and License with SPDX-License-Identifier expression Some files in the FreeBSD software collection contain a copyright statement, an SPDX-License-Identifier tag and an explicit license. The explicit license takes precedence over the SPDX-License-Identifier tag. The SPDX-License-Identifier tag is the project's best effort attempt to characterize the license, but is only informative for automated tools. See <> for how to interpret the expression. === Only Copyright and SPDX-License-Identifier expression. Some files in the tree contain detached licenses. These files contain only a copyright notice and an SPDX-License-Identifier expression, but no explicit license. See <> for how to interpret the expression. Note: the expressions allowed for detached licenses by the project are a subset of the expressions used informationally or that are defined by the standard. The license for files containing only the SPDX-License-Identifier should be construed to be . Start the license with the copyright notice from the file. Include all the copyright holders. . For each sub-expression, copy the license text from [.filename]#LICENSE/text/`id`.txt#. When exceptions are present, append them from [.filename]#src/share/license/exceptions/`id`.txt#. SPDX-License-Identifier expressions should be construed as described in the SPDX standard. Where `id` is the SPDX short license identifier from the `Identifier` column of https://spdx.org/licenses/[SPDX Identifiers] or https://spdx.org/licenses/exceptions-index.html[license exception]. If there is no file in [.filename]#LICENSE/#, then that license or exception cannot be specified as a detached license under this section. When reading the license text that is detached from a file, a number of considerations must be taken to make the detached license make sense. . Any reference to a copyright notice shall refer to the copyright notice constructed from the licensed file, not from any copyright notice in the license text file itself. Many SPDX files have sample copyright notices that are understood to be examples only. . When names of entities are referred to in the license text, they shall be construed to apply to the list of all copyright holders listed in the copyright notices of the licensed file. For example, the BSD-4-clause license contains the phrase "This product includes software developed by the organization". The phrase 'the organization' should be replaced by the copyright holders. . When the SPDX offers variations of the license, it is understood the license in the [.filename]#LICENSE/# file represents the exact version of the license selected. The SPDX standard exists to match families of licenses and these variations help match similar licenses that the SPDX organization believes to be legally identical. For licenses that have slight variations in text, the SPDX has guidelines to match them. These guidelines are not relevant here. Contributors wishing to license under a variant of a SPDX license not contained verbatim in [.filename]#LICENSE/# cannot use the detached option and must specify the license explicitly. === Files without Copyright or any License Marking Some files cannot have suitable comments added to them. In such cases, a license may be found in [.filename]#file.ext.license#. For example, a file named [.filename]#foo.jpg# may have a license in [.filename]#foo.jpg.license#, following the REUSE Software conventions. Files created by the project that lack a copyright notice are understood to fall under the blanket copyright and licensing in [.filename]#COPYRIGHT#. Either the file is a mere recitation of facts, not protectable by Copyright Law, or the content is so trivial as to not warrant the overhead of an explicit license. Files that lack marking and have more than a trivial amount of copyrightable material, or whose author believes them to be improperly marked, should be brought to the attention of the FreeBSD core team. It is the strong policy of the FreeBSD Project to comply with all appropriate licenses. In the future, all such files will be marked explicitly, or follow the REUSE Software [.filename]#.license# convention. [[expressions]] === SPDX-License-Identifier Expressions An 'SPDX License expression' is used in two contexts in the FreeBSD software collection. First, its full form is used for files that have explicit license statements contained within the file as well as a summarizing SPDX-License-Identifier expression. In this context, the full power of these expressions may be used. Second, in a restricted form described above, it is used to denote the actual license for a given file. In the second context, only a subset of this expression is allowed by the project. An `SPDX License sub-expression` is either an SPDX short form license identifier from the https://spdx.org/licenses/[SPDX License List], or the combination of two SPDX short form license identifiers separated by "WITH" when a https://spdx.org/licenses/exceptions-index.html[license exception] applies. When multiple licenses apply, an expression consists of keywords "AND", "OR" separating sub-expressions and surrounded by "(", ")" . The https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/[full specification of expressions] spells out all the details and takes precedence when it conflicts with the simplified treatment of this section. Some license identifiers, like [L]GPL, have the option to use only that version, or any later version. SPDX defines the suffix `-or-later` to mean that version of the license or a later version. It defines `-only` to mean only that specific version of the file. There is an old convention to have no suffix (which means what the new '-only' suffix means, but which people confuse for `-or-later`). In addition, affixing a `+` suffix was meant to mean `-or-later`. New files in FreeBSD should not use these two conventions. Old files that use this convention should be converted as appropriate. [.programlisting] .... // SPDX-License-Identifier: GPL-2.0-only // SPDX-License-Identifier: LGPL-2.1-or-later .... `WITH` should be used when a license modifier is needed. In the FreeBSD project, a number of files from LLVM have an exception to the Apache 2.0 license: [.programlisting] .... // SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception .... https://spdx.org/licenses/exceptions-index.html[Exception tags] are managed by SPDX. License exceptions can only be applied to certain licenses, as specified in the exception. `OR` should be used if the file has a choice of license and one license is selected. For example, some dtsi files are available under dual licenses: [.programlisting] .... // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause .... `AND` should be used if the file has multiple licenses whose terms all apply to use the file. For example, if code has been incorporated by several projects, each with their own license: [.programlisting] .... // SPDX-License-Identifier: BSD-2-Clause AND MIT .... diff --git a/documentation/content/en/articles/linux-emulation/_index.adoc b/documentation/content/en/articles/linux-emulation/_index.adoc index b695d7bc81..7fefba22e0 100644 --- a/documentation/content/en/articles/linux-emulation/_index.adoc +++ b/documentation/content/en/articles/linux-emulation/_index.adoc @@ -1,1436 +1,1436 @@ --- title: Linux® emulation in FreeBSD authors: - author: Roman Divacky email: rdivacky@FreeBSD.org description: A technical description about the internals of the Linux emulation layer in FreeBSD trademarks: ["freebsd", "ibm", "adobe", "netbsd", "realnetworks", "oracle", "linux", "sun", "general"] tags: ["Emulation", "Linuxulator", "kernel", "FreeBSD"] --- = Linux(R) emulation in FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/linux-emulation/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract This masters thesis deals with updating the Linux(R) emulation layer (the so called _Linuxulator_). The task was to update the layer to match the functionality of Linux(R) 2.6. As a reference implementation, the Linux(R) 2.6.16 kernel was chosen. The concept is loosely based on the NetBSD implementation. Most of the work was done in the summer of 2006 as a part of the Google Summer of Code students program. The focus was on bringing the _NPTL_ (new POSIX(R) thread library) support into the emulation layer, including _TLS_ (thread local storage), _futexes_ (fast user space mutexes), _PID mangling_, and some other minor things. Many small problems were identified and fixed in the process. My work was integrated into the main FreeBSD source repository and will be shipped in the upcoming 7.0R release. We, the emulation development team, are working on making the Linux(R) 2.6 emulation the default emulation layer in FreeBSD. ''' toc::[] [[intro]] == Introduction In the last few years the open source UNIX(R) based operating systems started to be widely deployed on server and client machines. Among these operating systems I would like to point out two: FreeBSD, for its BSD heritage, time proven code base and many interesting features and Linux(R) for its wide user base, enthusiastic open developer community and support from large companies. FreeBSD tends to be used on server class machines serving heavy duty networking tasks with less usage on desktop class machines for ordinary users. While Linux(R) has the same usage on servers, but it is used much more by home based users. This leads to a situation where there are many binary only programs available for Linux(R) that lack support for FreeBSD. Naturally, a need for the ability to run Linux(R) binaries on a FreeBSD system arises and this is what this thesis deals with: the emulation of the Linux(R) kernel in the FreeBSD operating system. During the Summer of 2006 Google Inc. sponsored a project which focused on extending the Linux(R) emulation layer (the so called Linuxulator) in FreeBSD to include Linux(R) 2.6 facilities. This thesis is written as a part of this project. [[inside]] == A look inside... In this section we are going to describe every operating system in question. How they deal with syscalls, trapframes etc., all the low-level stuff. We also describe the way they understand common UNIX(R) primitives like what a PID is, what a thread is, etc. In the third subsection we talk about how UNIX(R) on UNIX(R) emulation could be done in general. [[what-is-unix]] === What is UNIX(R) UNIX(R) is an operating system with a long history that has influenced almost every other operating system currently in use. Starting in the 1960s, its development continues to this day (although in different projects). UNIX(R) development soon forked into two main ways: the BSDs and System III/V families. They mutually influenced themselves by growing a common UNIX(R) standard. Among the contributions originated in BSD we can name virtual memory, TCP/IP networking, FFS, and many others. The System V branch contributed to SysV interprocess communication primitives, copy-on-write, etc. UNIX(R) itself does not exist any more but its ideas have been used by many other operating systems world wide thus forming the so called UNIX(R)-like operating systems. These days the most influential ones are Linux(R), Solaris, and possibly (to some extent) FreeBSD. There are in-company UNIX(R) derivatives (AIX, HP-UX etc.), but these have been more and more migrated to the aforementioned systems. Let us summarize typical UNIX(R) characteristics. [[tech-details]] === Technical details Every running program constitutes a process that represents a state of the computation. Running process is divided between kernel-space and user-space. Some operations can be done only from kernel space (dealing with hardware etc.), but the process should spend most of its lifetime in the user space. The kernel is where the management of the processes, hardware, and low-level details take place. The kernel provides a standard unified UNIX(R) API to the user space. The most important ones are covered below. [[kern-proc-comm]] ==== Communication between kernel and user space process Common UNIX(R) API defines a syscall as a way to issue commands from a user space process to the kernel. The most common implementation is either by using an interrupt or specialized instruction (think of `SYSENTER`/`SYSCALL` instructions for ia32). Syscalls are defined by a number. For example in FreeBSD, the syscall number 85 is the man:swapon[2] syscall and the syscall number 132 is man:mkfifo[2]. Some syscalls need parameters, which are passed from the user-space to the kernel-space in various ways (implementation dependant). Syscalls are synchronous. Another possible way to communicate is by using a _trap_. Traps occur asynchronously after some event occurs (division by zero, page fault etc.). A trap can be transparent for a process (page fault) or can result in a reaction like sending a _signal_ (division by zero). [[proc-proc-comm]] ==== Communication between processes There are other APIs (System V IPC, shared memory etc.) but the single most important API is signal. Signals are sent by processes or by the kernel and received by processes. Some signals can be ignored or handled by a user supplied routine, some result in a predefined action that cannot be altered or ignored. [[proc-mgmt]] ==== Process management Kernel instances are processed first in the system (so called init). Every running process can create its identical copy using the man:fork[2] syscall. Some slightly modified versions of this syscall were introduced but the basic semantic is the same. Every running process can morph into some other process using the man:exec[3] syscall. Some modifications of this syscall were introduced but all serve the same basic purpose. Processes end their lives by calling the man:exit[2] syscall. Every process is identified by a unique number called PID. Every process has a defined parent (identified by its PID). [[thread-mgmt]] ==== Thread management Traditional UNIX(R) does not define any API nor implementation for threading, while POSIX(R) defines its threading API but the implementation is undefined. Traditionally there were two ways of implementing threads. Handling them as separate processes (1:1 threading) or envelope the whole thread group in one process and managing the threading in userspace (1:N threading). Comparing main features of each approach: 1:1 threading * - heavyweight threads * - the scheduling cannot be altered by the user (slightly mitigated by the POSIX(R) API) * + no syscall wrapping necessary * + can utilize multiple CPUs 1:N threading * + lightweight threads * + scheduling can be easily altered by the user * - syscalls must be wrapped * - cannot utilize more than one CPU [[what-is-freebsd]] === What is FreeBSD? The FreeBSD project is one of the oldest open source operating systems currently available for daily use. It is a direct descendant of the genuine UNIX(R) so it could be claimed that it is a true UNIX(R) although licensing issues do not permit that. The start of the project dates back to the early 1990's when a crew of fellow BSD users patched the 386BSD operating system. Based on this patchkit a new operating system arose named FreeBSD for its liberal license. Another group created the NetBSD operating system with different goals in mind. We will focus on FreeBSD. FreeBSD is a modern UNIX(R)-based operating system with all the features of UNIX(R). Preemptive multitasking, multiuser facilities, TCP/IP networking, memory protection, symmetric multiprocessing support, virtual memory with merged VM and buffer cache, they are all there. One of the interesting and extremely useful features is the ability to emulate other UNIX(R)-like operating systems. As of December 2006 and 7-CURRENT development, the following emulation functionalities are supported: * FreeBSD/i386 emulation on FreeBSD/amd64 * FreeBSD/i386 emulation on FreeBSD/ia64 * Linux(R)-emulation of Linux(R) operating system on FreeBSD * NDIS-emulation of Windows networking drivers interface * NetBSD-emulation of NetBSD operating system * PECoff-support for PECoff FreeBSD executables * SVR4-emulation of System V revision 4 UNIX(R) Actively developed emulations are the Linux(R) layer and various FreeBSD-on-FreeBSD layers. Others are not supposed to work properly nor be usable these days. [[freebsd-tech-details]] ==== Technical details FreeBSD is traditional flavor of UNIX(R) in the sense of dividing the run of processes into two halves: kernel space and user space run. There are two types of process entry to the kernel: a syscall and a trap. There is only one way to return. In the subsequent sections we will describe the three gates to/from the kernel. The whole description applies to the i386 architecture as the Linuxulator only exists there but the concept is similar on other architectures. The information was taken from [1] and the source code. [[freebsd-sys-entries]] ===== System entries FreeBSD has an abstraction called an execution class loader, which is a wedge into the man:execve[2] syscall. This employs a structure `sysentvec`, which describes an executable ABI. It contains things like errno translation table, signal translation table, various functions to serve syscall needs (stack fixup, coredumping, etc.). Every ABI the FreeBSD kernel wants to support must define this structure, as it is used later in the syscall processing code and at some other places. System entries are handled by trap handlers, where we can access both the kernel-space and the user-space at once. [[freebsd-syscalls]] ===== Syscalls Syscalls on FreeBSD are issued by executing interrupt `0x80` with register `%eax` set to a desired syscall number with arguments passed on the stack. When a process issues an interrupt `0x80`, the `int0x80` syscall trap handler is issued (defined in [.filename]#sys/i386/i386/exception.s#), which prepares arguments (i.e. copies them on to the stack) for a call to a C function man:syscall[2] (defined in [.filename]#sys/i386/i386/trap.c#), which processes the passed in trapframe. The processing consists of preparing the syscall (depending on the `sysvec` entry), determining if the syscall is 32-bit or 64-bit one (changes size of the parameters), then the parameters are copied, including the syscall. Next, the actual syscall function is executed with processing of the return code (special cases for `ERESTART` and `EJUSTRETURN` errors). Finally an `userret()` is scheduled, switching the process back to the users-pace. The parameters to the actual syscall handler are passed in the form of `struct thread *td`, `struct syscall args *` arguments where the second parameter is a pointer to the copied in structure of parameters. [[freebsd-traps]] ===== Traps Handling of traps in FreeBSD is similar to the handling of syscalls. Whenever a trap occurs, an assembler handler is called. It is chosen between alltraps, alltraps with regs pushed or calltrap depending on the type of the trap. This handler prepares arguments for a call to a C function `trap()` (defined in [.filename]#sys/i386/i386/trap.c#), which then processes the occurred trap. After the processing it might send a signal to the process and/or exit to userland using `userret()`. [[freebsd-exits]] ===== Exits Exits from kernel to userspace happen using the assembler routine `doreti` regardless of whether the kernel was entered via a trap or via a syscall. This restores the program status from the stack and returns to the userspace. [[freebsd-unix-primitives]] ===== UNIX(R) primitives FreeBSD operating system adheres to the traditional UNIX(R) scheme, where every process has a unique identification number, the so called _PID_ (Process ID). PID numbers are allocated either linearly or randomly ranging from `0` to `PID_MAX`. The allocation of PID numbers is done using linear searching of PID space. Every thread in a process receives the same PID number as result of the man:getpid[2] call. There are currently two ways to implement threading in FreeBSD. The first way is M:N threading followed by the 1:1 threading model. The default library used is M:N threading (`libpthread`) and you can switch at runtime to 1:1 threading (`libthr`). The plan is to switch to 1:1 library by default soon. Although those two libraries use the same kernel primitives, they are accessed through different API(es). The M:N library uses the `kse_*` family of syscalls while the 1:1 library uses the `thr_*` family of syscalls. Due to this, there is no general concept of thread ID shared between kernel and userspace. Of course, both threading libraries implement the pthread thread ID API. Every kernel thread (as described by `struct thread`) has td tid identifier but this is not directly accessible from userland and solely serves the kernel's needs. It is also used for 1:1 threading library as pthread's thread ID but handling of this is internal to the library and cannot be relied on. As stated previously there are two implementations of threading in FreeBSD. The M:N library divides the work between kernel space and userspace. Thread is an entity that gets scheduled in the kernel but it can represent various number of userspace threads. M userspace threads get mapped to N kernel threads thus saving resources while keeping the ability to exploit multiprocessor parallelism. Further information about the implementation can be obtained from the man page or [1]. The 1:1 library directly maps a userland thread to a kernel thread thus greatly simplifying the scheme. None of these designs implement a fairness mechanism (such a mechanism was implemented but it was removed recently because it caused serious slowdown and made the code more difficult to deal with). [[what-is-linux]] === What is Linux(R) Linux(R) is a UNIX(R)-like kernel originally developed by Linus Torvalds, and now being contributed to by a massive crowd of programmers all around the world. From its mere beginnings to today, with wide support from companies such as IBM or Google, Linux(R) is being associated with its fast development pace, full hardware support and benevolent dictator model of organization. Linux(R) development started in 1991 as a hobbyist project at University of Helsinki in Finland. Since then it has obtained all the features of a modern UNIX(R)-like OS: multiprocessing, multiuser support, virtual memory, networking, basically everything is there. There are also highly advanced features like virtualization etc. As of 2006 Linux(R) seems to be the most widely used open source operating system with support from independent software vendors like Oracle, RealNetworks, Adobe, etc. Most of the commercial software distributed for Linux(R) can only be obtained in a binary form so recompilation for other operating systems is impossible. Most of the Linux(R) development happens in a Git version control system. Git is a distributed system so there is no central source of the Linux(R) code, but some branches are considered prominent and official. The version number scheme implemented by Linux(R) consists of four numbers A.B.C.D. Currently development happens in 2.6.C.D, where C represents major version, where new features are added or changed while D is a minor version for bugfixes only. More information can be obtained from [3]. [[linux-tech-details]] ==== Technical details Linux(R) follows the traditional UNIX(R) scheme of dividing the run of a process in two halves: the kernel and user space. The kernel can be entered in two ways: via a trap or via a syscall. The return is handled only in one way. The further description applies to Linux(R) 2.6 on the i386(TM) architecture. This information was taken from [2]. [[linux-syscalls]] ===== Syscalls Syscalls in Linux(R) are performed (in userspace) using `syscallX` macros where X substitutes a number representing the number of parameters of the given syscall. This macro translates to a code that loads `%eax` register with a number of the syscall and executes interrupt `0x80`. After this syscall return is called, which translates negative return values to positive `errno` values and sets `res` to `-1` in case of an error. Whenever the interrupt `0x80` is called the process enters the kernel in system call trap handler. This routine saves all registers on the stack and calls the selected syscall entry. Note that the Linux(R) calling convention expects parameters to the syscall to be passed via registers as shown here: . parameter -> `%ebx` . parameter -> `%ecx` . parameter -> `%edx` . parameter -> `%esi` . parameter -> `%edi` . parameter -> `%ebp` There are some exceptions to this, where Linux(R) uses different calling convention (most notably the `clone` syscall). [[linux-traps]] ===== Traps The trap handlers are introduced in [.filename]#arch/i386/kernel/traps.c# and most of these handlers live in [.filename]#arch/i386/kernel/entry.S#, where handling of the traps happens. [[linux-exits]] ===== Exits Return from the syscall is managed by syscall man:exit[3], which checks for the process having unfinished work, then checks whether we used user-supplied selectors. If this happens stack fixing is applied and finally the registers are restored from the stack and the process returns to the userspace. [[linux-unix-primitives]] ===== UNIX(R) primitives In the 2.6 version, the Linux(R) operating system redefined some of the traditional UNIX(R) primitives, notably PID, TID and thread. PID is defined not to be unique for every process, so for some processes (threads) man:getppid[2] returns the same value. Unique identification of process is provided by TID. This is because _NPTL_ (New POSIX(R) Thread Library) defines threads to be normal processes (so called 1:1 threading). Spawning a new process in Linux(R) 2.6 happens using the `clone` syscall (fork variants are reimplemented using it). This clone syscall defines a set of flags that affect behavior of the cloning process regarding thread implementation. The semantic is a bit fuzzy as there is no single flag telling the syscall to create a thread. Implemented clone flags are: * `CLONE_VM` - processes share their memory space * `CLONE_FS` - share umask, cwd and namespace * `CLONE_FILES` - share open files * `CLONE_SIGHAND` - share signal handlers and blocked signals * `CLONE_PARENT` - share parent * `CLONE_THREAD` - be thread (further explanation below) * `CLONE_NEWNS` - new namespace * `CLONE_SYSVSEM` - share SysV undo structures * `CLONE_SETTLS` - setup TLS at supplied address * `CLONE_PARENT_SETTID` - set TID in the parent * `CLONE_CHILD_CLEARTID` - clear TID in the child * `CLONE_CHILD_SETTID` - set TID in the child `CLONE_PARENT` sets the real parent to the parent of the caller. This is useful for threads because if thread A creates thread B we want thread B to be parented to the parent of the whole thread group. `CLONE_THREAD` does exactly the same thing as `CLONE_PARENT`, `CLONE_VM` and `CLONE_SIGHAND`, rewrites PID to be the same as PID of the caller, sets exit signal to be none and enters the thread group. `CLONE_SETTLS` sets up GDT entries for TLS handling. The `CLONE_*_*TID` set of flags sets/clears user supplied address to TID or 0. As you can see the `CLONE_THREAD` does most of the work and does not seem to fit the scheme very well. The original intention is unclear (even for authors, according to comments in the code) but I think originally there was one threading flag, which was then parcelled among many other flags but this separation was never fully finished. It is also unclear what this partition is good for as glibc does not use that so only hand-written use of the clone permits a programmer to access this features. For non-threaded programs the PID and TID are the same. For threaded programs the first thread PID and TID are the same and every created thread shares the same PID and gets assigned a unique TID (because `CLONE_THREAD` is passed in) also parent is shared for all processes forming this threaded program. The code that implements man:pthread_create[3] in NPTL defines the clone flags like this: [.programlisting] .... int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGNAL | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID | CLONE_SYSVSEM #if __ASSUME_NO_CLONE_DETACHED == 0 | CLONE_DETACHED #endif | 0); .... The `CLONE_SIGNAL` is defined like [.programlisting] .... #define CLONE_SIGNAL (CLONE_SIGHAND | CLONE_THREAD) .... the last 0 means no signal is sent when any of the threads exits. [[what-is-emu]] === What is emulation According to a dictionary definition, emulation is the ability of a program or device to imitate another program or device. This is achieved by providing the same reaction to a given stimulus as the emulated object. In practice, the software world mostly sees three types of emulation - a program used to emulate a machine (QEMU, various game console emulators etc.), software emulation of a hardware facility (OpenGL emulators, floating point units emulation etc.) and operating system emulation (either in kernel of the operating system or as a userspace program). Emulation is usually used in a place, where using the original component is not feasible nor possible at all. For example someone might want to use a program developed for a different operating system than they use. Then emulation comes in handy. Sometimes there is no other way but to use emulation - e.g. when the hardware device you try to use does not exist (yet/anymore) then there is no other way but emulation. This happens often when porting an operating system to a new (non-existent) platform. Sometimes it is just cheaper to emulate. Looking from an implementation point of view, there are two main approaches to the implementation of emulation. You can either emulate the whole thing - accepting possible inputs of the original object, maintaining inner state and emitting correct output based on the state and/or input. This kind of emulation does not require any special conditions and basically can be implemented anywhere for any device/program. The drawback is that implementing such emulation is quite difficult, time-consuming and error-prone. In some cases we can use a simpler approach. Imagine you want to emulate a printer that prints from left to right on a printer that prints from right to left. It is obvious that there is no need for a complex emulation layer but simply reversing of the printed text is sufficient. Sometimes the emulating environment is very similar to the emulated one so just a thin layer of some translation is necessary to provide fully working emulation! As you can see this is much less demanding to implement, so less time-consuming and error-prone than the previous approach. But the necessary condition is that the two environments must be similar enough. The third approach combines the two previous. Most of the time the objects do not provide the same capabilities so in a case of emulating the more powerful one on the less powerful we have to emulate the missing features with full emulation described above. This master thesis deals with emulation of UNIX(R) on UNIX(R), which is exactly the case, where only a thin layer of translation is sufficient to provide full emulation. The UNIX(R) API consists of a set of syscalls, which are usually self contained and do not affect some global kernel state. There are a few syscalls that affect inner state but this can be dealt with by providing some structures that maintain the extra state. No emulation is perfect and emulations tend to lack some parts but this usually does not cause any serious drawbacks. Imagine a game console emulator that emulates everything but music output. No doubt that the games are playable and one can use the emulator. It might not be that comfortable as the original game console but its an acceptable compromise between price and comfort. The same goes with the UNIX(R) API. Most programs can live with a very limited set of syscalls working. Those syscalls tend to be the oldest ones (man:read[2]/man:write[2], man:fork[2] family, man:signal[3] handling, man:exit[3], man:socket[2] API) hence it is easy to emulate because their semantics is shared among all UNIX(R)es, which exist todays. [[freebsd-emulation]] == Emulation === How emulation works in FreeBSD As stated earlier, FreeBSD supports running binaries from several other UNIX(R)es. This works because FreeBSD has an abstraction called the execution class loader. This wedges into the man:execve[2] syscall, so when man:execve[2] is about to execute a binary it examines its type. There are basically two types of binaries in FreeBSD. Shell-like text scripts which are identified by `#!` as their first two characters and normal (typically _ELF_) binaries, which are a representation of a compiled executable object. The vast majority (one could say all of them) of binaries in FreeBSD are from type ELF. ELF files contain a header, which specifies the OS ABI for this ELF file. By reading this information, the operating system can accurately determine what type of binary the given file is. Every OS ABI must be registered in the FreeBSD kernel. This applies to the FreeBSD native OS ABI, as well. So when man:execve[2] executes a binary it iterates through the list of registered APIs and when it finds the right one it starts to use the information contained in the OS ABI description (its syscall table, `errno` translation table, etc.). So every time the process calls a syscall, it uses its own set of syscalls instead of some global one. This effectively provides a very elegant and easy way of supporting execution of various binary formats. The nature of emulation of different OSes (and also some other subsystems) led developers to invite a handler event mechanism. There are various places in the kernel, where a list of event handlers are called. Every subsystem can register an event handler and they are called accordingly. For example, when a process exits there is a handler called that possibly cleans up whatever the subsystem needs to be cleaned. Those simple facilities provide basically everything that is needed for the emulation infrastructure and in fact these are basically the only things necessary to implement the Linux(R) emulation layer. [[freebsd-common-primitives]] === Common primitives in the FreeBSD kernel Emulation layers need some support from the operating system. I am going to describe some of the supported primitives in the FreeBSD operating system. [[freebsd-locking-primitives]] ==== Locking primitives Contributed by: `{attilio}` The FreeBSD synchronization primitive set is based on the idea to supply a rather huge number of different primitives in a way that the better one can be used for every particular, appropriate situation. To a high level point of view you can consider three kinds of synchronization primitives in the FreeBSD kernel: * atomic operations and memory barriers * locks * scheduling barriers Below there are descriptions for the 3 families. For every lock, you should really check the linked manpage (where possible) for more detailed explanations. [[freebsd-atomic-op]] ===== Atomic operations and memory barriers Atomic operations are implemented through a set of functions performing simple arithmetics on memory operands in an atomic way with respect to external events (interrupts, preemption, etc.). Atomic operations can guarantee atomicity just on small data types (in the magnitude order of the `.long.` architecture C data type), so should be rarely used directly in the end-level code, if not only for very simple operations (like flag setting in a bitmap, for example). In fact, it is rather simple and common to write down a wrong semantic based on just atomic operations (usually referred as lock-less). The FreeBSD kernel offers a way to perform atomic operations in conjunction with a memory barrier. The memory barriers will guarantee that an atomic operation will happen following some specified ordering with respect to other memory accesses. For example, if we need that an atomic operation happen just after all other pending writes (in terms of instructions reordering buffers activities) are completed, we need to explicitly use a memory barrier in conjunction to this atomic operation. So it is simple to understand why memory barriers play a key role for higher-level locks building (just as refcounts, mutexes, etc.). For a detailed explanatory on atomic operations, please refer to man:atomic[9]. It is far, however, noting that atomic operations (and memory barriers as well) should ideally only be used for building front-ending locks (as mutexes). [[freebsd-refcounts]] ===== Refcounts Refcounts are interfaces for handling reference counters. They are implemented through atomic operations and are intended to be used just for cases, where the reference counter is the only one thing to be protected, so even something like a spin-mutex is deprecated. Using the refcount interface for structures, where a mutex is already used is often wrong since we should probably close the reference counter in some already protected paths. A manpage discussing refcount does not exist currently, just check [.filename]#sys/refcount.h# for an overview of the existing API. [[freebsd-locks]] ===== Locks FreeBSD kernel has huge classes of locks. Every lock is defined by some peculiar properties, but probably the most important is the event linked to contesting holders (or in other terms, the behavior of threads unable to acquire the lock). FreeBSD's locking scheme presents three different behaviors for contenders: . spinning . blocking . sleeping [NOTE] ==== numbers are not casual ==== [[freebsd-spinlocks]] ===== Spinning locks Spin locks let waiters to spin until they cannot acquire the lock. An important matter do deal with is when a thread contests on a spin lock if it is not descheduled. -Since the FreeBSD kernel is preemptive, this exposes spin lock at the risk of deadlocks that can be solved just disabling interrupts while they are acquired. +Since the FreeBSD kernel is preemptive, this exposes spin lock at the risk of deadlocks that can be solved just disabling interrupts while they are acquired. For this and other reasons (like lack of priority propagation support, poorness in load balancing schemes between CPUs, etc.), spin locks are intended to protect very small paths of code, or ideally not to be used at all if not explicitly requested (explained later). [[freebsd-blocking]] ===== Blocking Block locks let waiters to be descheduled and blocked until the lock owner does not drop it and wakes up one or more contenders. -In order to avoid starvation issues, blocking locks do priority propagation from the waiters to the owner. +To avoid starvation issues, blocking locks do priority propagation from the waiters to the owner. Block locks must be implemented through the turnstile interface and are intended to be the most used kind of locks in the kernel, if no particular conditions are met. [[freebsd-sleeping]] ===== Sleeping Sleep locks let waiters to be descheduled and fall asleep until the lock holder does not drop it and wakes up one or more waiters. Since sleep locks are intended to protect large paths of code and to cater asynchronous events, they do not do any form of priority propagation. They must be implemented through the man:sleepqueue[9] interface. The order used to acquire locks is very important, not only for the possibility to deadlock due at lock order reversals, but even because lock acquisition should follow specific rules linked to locks natures. If you give a look at the table above, the practical rule is that if a thread holds a lock of level n (where the level is the number listed close to the kind of lock) it is not allowed to acquire a lock of superior levels, since this would break the specified semantic for a path. For example, if a thread holds a block lock (level 2), it is allowed to acquire a spin lock (level 1) but not a sleep lock (level 3), since block locks are intended to protect smaller paths than sleep lock (these rules are not about atomic operations or scheduling barriers, however). This is a list of lock with their respective behaviors: * spin mutex - spinning - man:mutex[9] * sleep mutex - blocking - man:mutex[9] * pool mutex - blocking - man:mtx[pool] * sleep family - sleeping - man:sleep[9] pause tsleep msleep msleep spin msleep rw msleep sx * condvar - sleeping - man:condvar[9] * rwlock - blocking - man:rwlock[9] * sxlock - sleeping - man:sx[9] * lockmgr - sleeping - man:lockmgr[9] * semaphores - sleeping - man:sema[9] Among these locks only mutexes, sxlocks, rwlocks and lockmgrs are intended to handle recursion, but currently recursion is only supported by mutexes and lockmgrs. [[freebsd-scheduling]] ===== Scheduling barriers -Scheduling barriers are intended to be used in order to drive scheduling of threading. +Scheduling barriers are intended to be used to drive scheduling of threading. They consist mainly of three different stubs: * critical sections (and preemption) * sched_bind * sched_pin Generally, these should be used only in a particular context and even if they can often replace locks, they should be avoided because they do not let the diagnose of simple eventual problems with locking debugging tools (as man:witness[4]). [[freebsd-critical]] ===== Critical sections The FreeBSD kernel has been made preemptive basically to deal with interrupt threads. -In fact, in order to avoid high interrupt latency, time-sharing priority threads can be preempted by interrupt threads (in this way, they do not need to wait to be scheduled as the normal path previews). +In fact, to avoid high interrupt latency, time-sharing priority threads can be preempted by interrupt threads (in this way, they do not need to wait to be scheduled as the normal path previews). Preemption, however, introduces new racing points that need to be handled, as well. -Often, in order to deal with preemption, the simplest thing to do is to completely disable it. +Often, to deal with preemption, the simplest thing to do is to completely disable it. A critical section defines a piece of code (borderlined by the pair of functions man:critical_enter[9] and man:critical_exit[9], where preemption is guaranteed to not happen (until the protected code is fully executed). -This can often replace a lock effectively but should be used carefully in order to not lose the whole advantage that preemption brings. +This can often replace a lock effectively but should be used carefully to not lose the whole advantage that preemption brings. [[freebsd-schedpin]] ===== sched_pin/sched_unpin Another way to deal with preemption is the `sched_pin()` interface. If a piece of code is closed in the `sched_pin()` and `sched_unpin()` pair of functions it is guaranteed that the respective thread, even if it can be preempted, it will always be executed on the same CPU. Pinning is very effective in the particular case when we have to access at per-cpu datas and we assume other threads will not change those data. The latter condition will determine a critical section as a too strong condition for our code. [[freebsd-schedbind]] ===== sched_bind/sched_unbind -`sched_bind` is an API used in order to bind a thread to a particular CPU for all the time it executes the code, until a `sched_unbind` function call does not unbind it. +`sched_bind` is an API used to bind a thread to a particular CPU for all the time it executes the code, until a `sched_unbind` function call does not unbind it. This feature has a key role in situations where you cannot trust the current state of CPUs (for example, at very early stages of boot), as you want to avoid your thread to migrate on inactive CPUs. Since `sched_bind` and `sched_unbind` manipulate internal scheduler structures, they need to be enclosed in `sched_lock` acquisition/releasing when used. [[freebsd-proc]] ==== Proc structure Various emulation layers sometimes require some additional per-process data. It can manage separate structures (a list, a tree etc.) containing these data for every process but this tends to be slow and memory consuming. To solve this problem the FreeBSD `proc` structure contains `p_emuldata`, which is a void pointer to some emulation layer specific data. This `proc` entry is protected by the proc mutex. The FreeBSD `proc` structure contains a `p_sysent` entry that identifies, which ABI this process is running. In fact, it is a pointer to the `sysentvec` described above. So by comparing this pointer to the address where the `sysentvec` structure for the given ABI is stored we can effectively determine whether the process belongs to our emulation layer. The code typically looks like: [.programlisting] .... if (__predict_true(p->p_sysent != &elf_Linux(R)_sysvec)) return; .... As you can see, we effectively use the `__predict_true` modifier to collapse the most common case (FreeBSD process) to a simple return operation thus preserving high performance. This code should be turned into a macro because currently it is not very flexible, i.e. we do not support Linux(R)64 emulation nor A.OUT Linux(R) processes on i386. [[freebsd-vfs]] ==== VFS The FreeBSD VFS subsystem is very complex but the Linux(R) emulation layer uses just a small subset via a well defined API. It can either operate on vnodes or file handlers. Vnode represents a virtual vnode, i.e. representation of a node in VFS. Another representation is a file handler, which represents an opened file from the perspective of a process. A file handler can represent a socket or an ordinary file. A file handler contains a pointer to its vnode. More then one file handler can point to the same vnode. [[freebsd-namei]] ===== namei The man:namei[9] routine is a central entry point to pathname lookup and translation. It traverses the path point by point from the starting point to the end point using lookup function, which is internal to VFS. The man:namei[9] syscall can cope with symlinks, absolute and relative paths. When a path is looked up using man:namei[9] it is inputed to the name cache. This behavior can be suppressed. This routine is used all over the kernel and its performance is very critical. [[freebsd-vn]] ===== vn_fullpath The man:vn_fullpath[9] function takes the best effort to traverse VFS name cache and returns a path for a given (locked) vnode. This process is unreliable but works just fine for the most common cases. The unreliability is because it relies on VFS cache (it does not traverse the on medium structures), it does not work with hardlinks, etc. This routine is used in several places in the Linuxulator. [[freebsd-vnode]] ===== Vnode operations * `fgetvp` - given a thread and a file descriptor number it returns the associated vnode * man:vn_lock[9] - locks a vnode * `vn_unlock` - unlocks a vnode * man:VOP_READDIR[9] - reads a directory referenced by a vnode * man:VOP_GETATTR[9] - gets attributes of a file or a directory referenced by a vnode * man:VOP_LOOKUP[9] - looks up a path to a given directory * man:VOP_OPEN[9] - opens a file referenced by a vnode * man:VOP_CLOSE[9] - closes a file referenced by a vnode * man:vput[9] - decrements the use count for a vnode and unlocks it * man:vrele[9] - decrements the use count for a vnode * man:vref[9] - increments the use count for a vnode [[freebsd-file-handler]] ===== File handler operations * `fget` - given a thread and a file descriptor number it returns associated file handler and references it * `fdrop` - drops a reference to a file handler * `fhold` - references a file handler [[md]] == Linux(R) emulation layer -MD part This section deals with implementation of Linux(R) emulation layer in FreeBSD operating system. It first describes the machine dependent part talking about how and where interaction between userland and kernel is implemented. It talks about syscalls, signals, ptrace, traps, stack fixup. This part discusses i386 but it is written generally so other architectures should not differ very much. The next part is the machine independent part of the Linuxulator. This section only covers i386 and ELF handling. A.OUT is obsolete and untested. [[syscall-handling]] === Syscall handling Syscall handling is mostly written in [.filename]#linux_sysvec.c#, which covers most of the routines pointed out in the `sysentvec` structure. When a Linux(R) process running on FreeBSD issues a syscall, the general syscall routine calls linux prepsyscall routine for the Linux(R) ABI. [[linux-prepsyscall]] ==== Linux(R) prepsyscall Linux(R) passes arguments to syscalls via registers (that is why it is limited to 6 parameters on i386) while FreeBSD uses the stack. The Linux(R) prepsyscall routine must copy parameters from registers to the stack. The order of the registers is: `%ebx`, `%ecx`, `%edx`, `%esi`, `%edi`, `%ebp`. The catch is that this is true for only _most_ of the syscalls. Some (most notably `clone`) uses a different order but it is luckily easy to fix by inserting a dummy parameter in the `linux_clone` prototype. [[syscall-writing]] ==== Syscall writing Every syscall implemented in the Linuxulator must have its prototype with various flags in [.filename]#syscalls.master#. The form of the file is: [.programlisting] .... ... AUE_FORK STD { int linux_fork(void); } ... AUE_CLOSE NOPROTO { int close(int fd); } ... .... The first column represents the syscall number. The second column is for auditing support. The third column represents the syscall type. It is either `STD`, `OBSOL`, `NOPROTO` and `UNIMPL`. `STD` is a standard syscall with full prototype and implementation. `OBSOL` is obsolete and defines just the prototype. `NOPROTO` means that the syscall is implemented elsewhere so do not prepend ABI prefix, etc. `UNIMPL` means that the syscall will be substituted with the `nosys` syscall (a syscall just printing out a message about the syscall not being implemented and returning `ENOSYS`). From [.filename]#syscalls.master# a script generates three files: [.filename]#linux_syscall.h#, [.filename]#linux_proto.h# and [.filename]#linux_sysent.c#. The [.filename]#linux_syscall.h# contains definitions of syscall names and their numerical value, e.g.: [.programlisting] .... ... #define LINUX_SYS_linux_fork 2 ... #define LINUX_SYS_close 6 ... .... The [.filename]#linux_proto.h# contains structure definitions of arguments to every syscall, e.g.: [.programlisting] .... struct linux_fork_args { register_t dummy; }; .... And finally, [.filename]#linux_sysent.c# contains structure describing the system entry table, used to actually dispatch a syscall, e.g.: [.programlisting] .... { 0, (sy_call_t *)linux_fork, AUE_FORK, NULL, 0, 0 }, /* 2 = linux_fork */ { AS(close_args), (sy_call_t *)close, AUE_CLOSE, NULL, 0, 0 }, /* 6 = close */ .... As you can see `linux_fork` is implemented in Linuxulator itself so the definition is of `STD` type and has no argument, which is exhibited by the dummy argument structure. On the other hand `close` is just an alias for real FreeBSD man:close[2] so it has no linux arguments structure associated and in the system entry table it is not prefixed with linux as it calls the real man:close[2] in the kernel. [[dummy-syscalls]] ==== Dummy syscalls The Linux(R) emulation layer is not complete, as some syscalls are not implemented properly and some are not implemented at all. The emulation layer employs a facility to mark unimplemented syscalls with the `DUMMY` macro. These dummy definitions reside in [.filename]#linux_dummy.c# in a form of `DUMMY(syscall);`, which is then translated to various syscall auxiliary files and the implementation consists of printing a message saying that this syscall is not implemented. -The `UNIMPL` prototype is not used because we want to be able to identify the name of the syscall that was called in order to know what syscalls are more important to implement. +The `UNIMPL` prototype is not used because we want to be able to identify the name of the syscall that was called to know what syscalls are more important to implement. [[signal-handling]] === Signal handling Signal handling is done generally in the FreeBSD kernel for all binary compatibilities with a call to a compat-dependent layer. Linux(R) compatibility layer defines `linux_sendsig` routine for this purpose. [[linux-sendsig]] ==== Linux(R) sendsig This routine first checks whether the signal has been installed with a `SA_SIGINFO` in which case it calls `linux_rt_sendsig` routine instead. Furthermore, it allocates (or reuses an already existing) signal handle context, then it builds a list of arguments for the signal handler. It translates the signal number based on the signal translation table, assigns a handler, translates sigset. Then it saves context for the `sigreturn` routine (various registers, translated trap number and signal mask). Finally, it copies out the signal context to the userspace and prepares context for the actual signal handler to run. [[linux-rt-sendsig]] ==== linux_rt_sendsig This routine is similar to `linux_sendsig` just the signal context preparation is different. It adds `siginfo`, `ucontext`, and some POSIX(R) parts. It might be worth considering whether those two functions could not be merged with a benefit of less code duplication and possibly even faster execution. [[linux-sigreturn]] ==== linux_sigreturn This syscall is used for return from the signal handler. It does some security checks and restores the original process context. It also unmasks the signal in process signal mask. [[ptrace]] === Ptrace -Many UNIX(R) derivates implement the man:ptrace[2] syscall in order to allow various tracking and debugging features. +Many UNIX(R) derivates implement the man:ptrace[2] syscall to allow various tracking and debugging features. This facility enables the tracing process to obtain various information about the traced process, like register dumps, any memory from the process address space, etc. and also to trace the process like in stepping an instruction or between system entries (syscalls and traps). man:ptrace[2] also lets you set various information in the traced process (registers etc.). man:ptrace[2] is a UNIX(R)-wide standard implemented in most UNIX(R)es around the world. Linux(R) emulation in FreeBSD implements the man:ptrace[2] facility in [.filename]#linux_ptrace.c#. The routines for converting registers between Linux(R) and FreeBSD and the actual man:ptrace[2] syscall emulation syscall. The syscall is a long switch block that implements its counterpart in FreeBSD for every man:ptrace[2] command. The man:ptrace[2] commands are mostly equal between Linux(R) and FreeBSD so usually just a small modification is needed. For example, `PT_GETREGS` in Linux(R) operates on direct data while FreeBSD uses a pointer to the data so after performing a (native) man:ptrace[2] syscall, a copyout must be done to preserve Linux(R) semantics. The man:ptrace[2] implementation in Linuxulator has some known weaknesses. There have been panics seen when using `strace` (which is a man:ptrace[2] consumer) in the Linuxulator environment. Also `PT_SYSCALL` is not implemented. [[traps]] === Traps Whenever a Linux(R) process running in the emulation layer traps the trap itself is handled transparently with the only exception of the trap translation. Linux(R) and FreeBSD differs in opinion on what a trap is so this is dealt with here. The code is actually very short: [.programlisting] .... static int translate_traps(int signal, int trap_code) { if (signal != SIGBUS) return signal; switch (trap_code) { case T_PROTFLT: case T_TSSFLT: case T_DOUBLEFLT: case T_PAGEFLT: return SIGSEGV; default: return signal; } } .... [[stack-fixup]] === Stack fixup The RTLD run-time link-editor expects so called AUX tags on stack during an `execve` so a fixup must be done to ensure this. Of course, every RTLD system is different so the emulation layer must provide its own stack fixup routine to do this. So does Linuxulator. The `elf_linux_fixup` simply copies out AUX tags to the stack and adjusts the stack of the user space process to point right after those tags. So RTLD works in a smart way. [[aout-support]] === A.OUT support The Linux(R) emulation layer on i386 also supports Linux(R) A.OUT binaries. Pretty much everything described in the previous sections must be implemented for A.OUT support (beside traps translation and signals sending). The support for A.OUT binaries is no longer maintained, especially the 2.6 emulation does not work with it but this does not cause any problem, as the linux-base in ports probably do not support A.OUT binaries at all. This support will probably be removed in future. Most of the stuff necessary for loading Linux(R) A.OUT binaries is in [.filename]#imgact_linux.c# file. [[mi]] == Linux(R) emulation layer -MI part This section talks about machine independent part of the Linuxulator. It covers the emulation infrastructure needed for Linux(R) 2.6 emulation, the thread local storage (TLS) implementation (on i386) and futexes. Then we talk briefly about some syscalls. [[nptl-desc]] === Description of NPTL One of the major areas of progress in development of Linux(R) 2.6 was threading. Prior to 2.6, the Linux(R) threading support was implemented in the linuxthreads library. The library was a partial implementation of POSIX(R) threading. The threading was implemented using separate processes for each thread using the `clone` syscall to let them share the address space (and other things). The main weaknesses of this approach was that every thread had a different PID, signal handling was broken (from the pthreads perspective), etc. Also the performance was not very good (use of `SIGUSR` signals for threads synchronization, kernel resource consumption, etc.) so to overcome these problems a new threading system was developed and named NPTL. The NPTL library focused on two things but a third thing came along so it is usually considered a part of NPTL. Those two things were embedding of threads into a process structure and futexes. The additional third thing was TLS, which is not directly required by NPTL but the whole NPTL userland library depends on it. Those improvements yielded in much improved performance and standards conformance. NPTL is a standard threading library in Linux(R) systems these days. The FreeBSD Linuxulator implementation approaches the NPTL in three main areas. The TLS, futexes and PID mangling, which is meant to simulate the Linux(R) threads. Further sections describe each of these areas. [[linux26-emu]] === Linux(R) 2.6 emulation infrastructure These sections deal with the way Linux(R) threads are managed and how we simulate that in FreeBSD. [[linux26-runtime]] ==== Runtime determining of 2.6 emulation The Linux(R) emulation layer in FreeBSD supports runtime setting of the emulated version. This is done via man:sysctl[8], namely `compat.linux.osrelease`. Setting this man:sysctl[8] affects runtime behavior of the emulation layer. When set to 2.6.x it sets the value of `linux_use_linux26` while setting to something else keeps it unset. This variable (plus per-prison variables of the very same kind) determines whether 2.6 infrastructure (mainly PID mangling) is used in the code or not. The version setting is done system-wide and this affects all Linux(R) processes. The man:sysctl[8] should not be changed when running any Linux(R) binary as it might harm things. [[linux-proc-thread]] ==== Linux(R) processes and thread identifiers The semantics of Linux(R) threading are a little confusing and uses entirely different nomenclature to FreeBSD. A process in Linux(R) consists of a `struct task` embedding two identifier fields - PID and TGID. PID is _not_ a process ID but it is a thread ID. The TGID identifies a thread group in other words a process. For single-threaded process the PID equals the TGID. The thread in NPTL is just an ordinary process that happens to have TGID not equal to PID and have a group leader not equal to itself (and shared VM etc. of course). Everything else happens in the same way as to an ordinary process. There is no separation of a shared status to some external structure like in FreeBSD. This creates some duplication of information and possible data inconsistency. The Linux(R) kernel seems to use task -> group information in some places and task information elsewhere and it is really not very consistent and looks error-prone. Every NPTL thread is created by a call to the `clone` syscall with a specific set of flags (more in the next subsection). The NPTL implements strict 1:1 threading. In FreeBSD we emulate NPTL threads with ordinary FreeBSD processes that share VM space, etc. and the PID gymnastic is just mimicked in the emulation specific structure attached to the process. The structure attached to the process looks like: [.programlisting] .... struct linux_emuldata { pid_t pid; int *child_set_tid; /* in clone(): Child.s TID to set on clone */ int *child_clear_tid;/* in clone(): Child.s TID to clear on exit */ struct linux_emuldata_shared *shared; int pdeath_signal; /* parent death signal */ LIST_ENTRY(linux_emuldata) threads; /* list of linux threads */ }; .... The PID is used to identify the FreeBSD process that attaches this structure. The `child_se_tid` and `child_clear_tid` are used for TID address copyout when a process exits and is created. The `shared` pointer points to a structure shared among threads. The `pdeath_signal` variable identifies the parent death signal and the `threads` pointer is used to link this structure to the list of threads. The `linux_emuldata_shared` structure looks like: [.programlisting] .... struct linux_emuldata_shared { int refs; pid_t group_pid; LIST_HEAD(, linux_emuldata) threads; /* head of list of linux threads */ }; .... The `refs` is a reference counter being used to determine when we can free the structure to avoid memory leaks. The `group_pid` is to identify PID ( = TGID) of the whole process ( = thread group). The `threads` pointer is the head of the list of threads in the process. The `linux_emuldata` structure can be obtained from the process using `em_find`. The prototype of the function is: [.programlisting] .... struct linux_emuldata *em_find(struct proc *, int locked); .... Here, `proc` is the process we want the emuldata structure from and the locked parameter determines whether we want to lock or not. The accepted values are `EMUL_DOLOCK` and `EMUL_DOUNLOCK`. More about locking later. [[pid-mangling]] ==== PID mangling As there is a difference in view as what to the idea of a process ID and thread ID is between FreeBSD and Linux(R) we have to translate the view somehow. We do it by PID mangling. This means that we fake what a PID (=TGID) and TID (=PID) is between kernel and userland. The rule of thumb is that in kernel (in Linuxulator) PID = PID and TGID = shared -> group pid and to userland we present `PID = shared -> group_pid` and `TID = proc -> p_pid`. The PID member of `linux_emuldata structure` is a FreeBSD PID. The above affects mainly getpid, getppid, gettid syscalls. Where we use PID/TGID respectively. In copyout of TIDs in `child_clear_tid` and `child_set_tid` we copy out FreeBSD PID. [[clone-syscall]] ==== Clone syscall The `clone` syscall is the way threads are created in Linux(R). The syscall prototype looks like this: [.programlisting] .... int linux_clone(l_int flags, void *stack, void *parent_tidptr, int dummy, void * child_tidptr); .... The `flags` parameter tells the syscall how exactly the processes should be cloned. As described above, Linux(R) can create processes sharing various things independently, for example two processes can share file descriptors but not VM, etc. Last byte of the `flags` parameter is the exit signal of the newly created process. The `stack` parameter if non-`NULL` tells, where the thread stack is and if it is `NULL` we are supposed to copy-on-write the calling process stack (i.e. do what normal man:fork[2] routine does). The `parent_tidptr` parameter is used as an address for copying out process PID (i.e. thread id) once the process is sufficiently instantiated but is not runnable yet. The `dummy` parameter is here because of the very strange calling convention of this syscall on i386. It uses the registers directly and does not let the compiler do it what results in the need of a dummy syscall. The `child_tidptr` parameter is used as an address for copying out PID once the process has finished forking and when the process exits. The syscall itself proceeds by setting corresponding flags depending on the flags passed in. For example, `CLONE_VM` maps to RFMEM (sharing of VM), etc. The only nit here is `CLONE_FS` and `CLONE_FILES` because FreeBSD does not allow setting this separately so we fake it by not setting RFFDG (copying of fd table and other fs information) if either of these is defined. This does not cause any problems, because those flags are always set together. After setting the flags the process is forked using the internal `fork1` routine, the process is instrumented not to be put on a run queue, i.e. not to be set runnable. After the forking is done we possibly reparent the newly created process to emulate `CLONE_PARENT` semantics. Next part is creating the emulation data. Threads in Linux(R) does not signal their parents so we set exit signal to be 0 to disable this. After that setting of `child_set_tid` and `child_clear_tid` is performed enabling the functionality later in the code. At this point we copy out the PID to the address specified by `parent_tidptr`. The setting of process stack is done by simply rewriting thread frame `%esp` register (`%rsp` on amd64). Next part is setting up TLS for the newly created process. After this man:vfork[2] semantics might be emulated and finally the newly created process is put on a run queue and copying out its PID to the parent process via `clone` return value is done. The `clone` syscall is able and in fact is used for emulating classic man:fork[2] and man:vfork[2] syscalls. Newer glibc in a case of 2.6 kernel uses `clone` to implement man:fork[2] and man:vfork[2] syscalls. [[locking]] ==== Locking The locking is implemented to be per-subsystem because we do not expect a lot of contention on these. There are two locks: `emul_lock` used to protect manipulating of `linux_emuldata` and `emul_shared_lock` used to manipulate `linux_emuldata_shared`. The `emul_lock` is a nonsleepable blocking mutex while `emul_shared_lock` is a sleepable blocking `sx_lock`. Due to of the per-subsystem locking we can coalesce some locks and that is why the em find offers the non-locking access. [[tls]] === TLS This section deals with TLS also known as thread local storage. [[trheading-intro]] ==== Introduction to threading Threads in computer science are entities within a process that can be scheduled independently from each other. The threads in the process share process wide data (file descriptors, etc.) but also have their own stack for their own data. Sometimes there is a need for process-wide data specific to a given thread. Imagine a name of the thread in execution or something like that. The traditional UNIX(R) threading API, pthreads provides a way to do it via man:pthread_key_create[3], man:pthread_setspecific[3] and man:pthread_getspecific[3] where a thread can create a key to the thread local data and using man:pthread_getspecific[3] or man:pthread_getspecific[3] to manipulate those data. You can easily see that this is not the most comfortable way this could be accomplished. So various producers of C/C++ compilers introduced a better way. They defined a new modifier keyword thread that specifies that a variable is thread specific. A new method of accessing such variables was developed as well (at least on i386). The pthreads method tends to be implemented in userspace as a trivial lookup table. The performance of such a solution is not very good. So the new method uses (on i386) segment registers to address a segment, where TLS area is stored so the actual accessing of a thread variable is just appending the segment register to the address thus addressing via it. The segment registers are usually `%gs` and `%fs` acting like segment selectors. Every thread has its own area where the thread local data are stored and the segment must be loaded on every context switch. This method is very fast and used almost exclusively in the whole i386 UNIX(R) world. Both FreeBSD and Linux(R) implement this approach and it yields very good results. The only drawback is the need to reload the segment on every context switch which can slowdown context switches. FreeBSD tries to avoid this overhead by using only 1 segment descriptor for this while Linux(R) uses 3. Interesting thing is that almost nothing uses more than 1 descriptor (only Wine seems to use 2) so Linux(R) pays this unnecessary price for context switches. [[i386-segs]] ==== Segments on i386 The i386 architecture implements the so called segments. A segment is a description of an area of memory. The base address (bottom) of the memory area, the end of it (ceiling), type, protection, etc. The memory described by a segment can be accessed using segment selector registers (`%cs`, `%ds`, `%ss`, `%es`, `%fs`, `%gs`). For example let us suppose we have a segment which base address is 0x1234 and length and this code: [.programlisting] .... mov %edx,%gs:0x10 .... This will load the content of the `%edx` register into memory location 0x1244. Some segment registers have a special use, for example `%cs` is used for code segment and `%ss` is used for stack segment but `%fs` and `%gs` are generally unused. Segments are either stored in a global GDT table or in a local LDT table. LDT is accessed via an entry in the GDT. The LDT can store more types of segments. LDT can be per process. Both tables define up to 8191 entries. [[linux-i386]] ==== Implementation on Linux(R) i386 There are two main ways of setting up TLS in Linux(R). It can be set when cloning a process using the `clone` syscall or it can call `set_thread_area`. When a process passes `CLONE_SETTLS` flag to `clone`, the kernel expects the memory pointed to by the `%esi` register a Linux(R) user space representation of a segment, which gets translated to the machine representation of a segment and loaded into a GDT slot. The GDT slot can be specified with a number or -1 can be used meaning that the system itself should choose the first free slot. In practice, the vast majority of programs use only one TLS entry and does not care about the number of the entry. We exploit this in the emulation and in fact depend on it. [[tls-emu]] ==== Emulation of Linux(R) TLS [[tls-i386]] ===== i386 Loading of TLS for the current thread happens by calling `set_thread_area` while loading TLS for a second process in `clone` is done in the separate block in `clone`. Those two functions are very similar. The only difference being the actual loading of the GDT segment, which happens on the next context switch for the newly created process while `set_thread_area` must load this directly. The code basically does this. It copies the Linux(R) form segment descriptor from the userland. The code checks for the number of the descriptor but because this differs between FreeBSD and Linux(R) we fake it a little. We only support indexes of 6, 3 and -1. The 6 is genuine Linux(R) number, 3 is genuine FreeBSD one and -1 means autoselection. Then we set the descriptor number to constant 3 and copy out this to the userspace. We rely on the userspace process using the number from the descriptor but this works most of the time (have never seen a case where this did not work) as the userspace process typically passes in 1. Then we convert the descriptor from the Linux(R) form to a machine dependant form (i.e. operating system independent form) and copy this to the FreeBSD defined segment descriptor. Finally we can load it. We assign the descriptor to threads PCB (process control block) and load the `%gs` segment using `load_gs`. This loading must be done in a critical section so that nothing can interrupt us. The `CLONE_SETTLS` case works exactly like this just the loading using `load_gs` is not performed. The segment used for this (segment number 3) is shared for this use between FreeBSD processes and Linux(R) processes so the Linux(R) emulation layer does not add any overhead over plain FreeBSD. [[tls-amd64]] ===== amd64 The amd64 implementation is similar to the i386 one but there was initially no 32bit segment descriptor used for this purpose (hence not even native 32bit TLS users worked) so we had to add such a segment and implement its loading on every context switch (when a flag signaling use of 32bit is set). Apart from this the TLS loading is exactly the same just the segment numbers are different and the descriptor format and the loading differs slightly. [[futexes]] === Futexes [[sync-intro]] ==== Introduction to synchronization Threads need some kind of synchronization and POSIX(R) provides some of them: mutexes for mutual exclusion, read-write locks for mutual exclusion with biased ratio of reads and writes and condition variables for signaling a status change. It is interesting to note that POSIX(R) threading API lacks support for semaphores. Those synchronization routines implementations are heavily dependant on the type threading support we have. In pure 1:M (userspace) model the implementation can be solely done in userspace and thus be very fast (the condition variables will probably end up being implemented using signals, i.e. not fast) and simple. In 1:1 model, the situation is also quite clear - the threads must be synchronized using kernel facilities (which is very slow because a syscall must be performed). The mixed M:N scenario just combines the first and second approach or rely solely on kernel. Threads synchronization is a vital part of thread-enabled programming and its performance can affect resulting program a lot. Recent benchmarks on FreeBSD operating system showed that an improved sx_lock implementation yielded 40% speedup in _ZFS_ (a heavy sx user), this is in-kernel stuff but it shows clearly how important the performance of synchronization primitives is. Threaded programs should be written with as little contention on locks as possible. Otherwise, instead of doing useful work the thread just waits on a lock. As a result of this, the most well written threaded programs show little locks contention. [[futex-intro]] ==== Futexes introduction Linux(R) implements 1:1 threading, i.e. it has to use in-kernel synchronization primitives. As stated earlier, well written threaded programs have little lock contention. So a typical sequence could be performed as two atomic increase/decrease mutex reference counter, which is very fast, as presented by the following example: [.programlisting] .... pthread_mutex_lock(&mutex); ... pthread_mutex_unlock(&mutex); .... 1:1 threading forces us to perform two syscalls for those mutex calls, which is very slow. The solution Linux(R) 2.6 implements is called futexes. Futexes implement the check for contention in userspace and call kernel primitives only in a case of contention. Thus the typical case takes place without any kernel intervention. This yields reasonably fast and flexible synchronization primitives implementation. [[futex-api]] ==== Futex API The futex syscall looks like this: [.programlisting] .... int futex(void *uaddr, int op, int val, struct timespec *timeout, void *uaddr2, int val3); .... In this example `uaddr` is an address of the mutex in userspace, `op` is an operation we are about to perform and the other parameters have per-operation meaning. Futexes implement the following operations: * `FUTEX_WAIT` * `FUTEX_WAKE` * `FUTEX_FD` * `FUTEX_REQUEUE` * `FUTEX_CMP_REQUEUE` * `FUTEX_WAKE_OP` [[futex-wait]] ===== FUTEX_WAIT This operation verifies that on address `uaddr` the value `val` is written. If not, `EWOULDBLOCK` is returned, otherwise the thread is queued on the futex and gets suspended. If the argument `timeout` is non-zero it specifies the maximum time for the sleeping, otherwise the sleeping is infinite. [[futex-wake]] ===== FUTEX_WAKE This operation takes a futex at `uaddr` and wakes up `val` first futexes queued on this futex. [[futex-fd]] ===== FUTEX_FD This operations associates a file descriptor with a given futex. [[futex-requeue]] ===== FUTEX_REQUEUE This operation takes `val` threads queued on futex at `uaddr`, wakes them up, and takes `val2` next threads and requeues them on futex at `uaddr2`. [[futex-cmp-requeue]] ===== FUTEX_CMP_REQUEUE This operation does the same as `FUTEX_REQUEUE` but it checks that `val3` equals to `val` first. [[futex-wake-op]] ===== FUTEX_WAKE_OP This operation performs an atomic operation on `val3` (which contains coded some other value) and `uaddr`. Then it wakes up `val` threads on futex at `uaddr` and if the atomic operation returned a positive number it wakes up `val2` threads on futex at `uaddr2`. The operations implemented in `FUTEX_WAKE_OP`: * `FUTEX_OP_SET` * `FUTEX_OP_ADD` * `FUTEX_OP_OR` * `FUTEX_OP_AND` * `FUTEX_OP_XOR` [NOTE] ==== There is no `val2` parameter in the futex prototype. The `val2` is taken from the `struct timespec *timeout` parameter for operations `FUTEX_REQUEUE`, `FUTEX_CMP_REQUEUE` and `FUTEX_WAKE_OP`. ==== [[futex-emu]] ==== Futex emulation in FreeBSD The futex emulation in FreeBSD is taken from NetBSD and further extended by us. It is placed in `linux_futex.c` and [.filename]#linux_futex.h# files. The `futex` structure looks like: [.programlisting] .... struct futex { void *f_uaddr; int f_refcount; LIST_ENTRY(futex) f_list; TAILQ_HEAD(lf_waiting_paroc, waiting_proc) f_waiting_proc; }; .... And the structure `waiting_proc` is: [.programlisting] .... struct waiting_proc { struct thread *wp_t; struct futex *wp_new_futex; TAILQ_ENTRY(waiting_proc) wp_list; }; .... [[futex-get]] ===== futex_get / futex_put A futex is obtained using the `futex_get` function, which searches a linear list of futexes and returns the found one or creates a new futex. When releasing a futex from the use we call the `futex_put` function, which decreases a reference counter of the futex and if the refcount reaches zero it is released. [[futex-sleep]] ===== futex_sleep When a futex queues a thread for sleeping it creates a `working_proc` structure and puts this structure to the list inside the futex structure then it just performs a man:tsleep[9] to suspend the thread. The sleep can be timed out. After man:tsleep[9] returns (the thread was woken up or it timed out) the `working_proc` structure is removed from the list and is destroyed. All this is done in the `futex_sleep` function. If we got woken up from `futex_wake` we have `wp_new_futex` set so we sleep on it. This way the actual requeueing is done in this function. [[futex-wake-2]] ===== futex_wake Waking up a thread sleeping on a futex is performed in the `futex_wake` function. First in this function we mimic the strange Linux(R) behavior, where it wakes up N threads for all operations, the only exception is that the REQUEUE operations are performed on N+1 threads. But this usually does not make any difference as we are waking up all threads. Next in the function in the loop we wake up n threads, after this we check if there is a new futex for requeueing. If so, we requeue up to n2 threads on the new futex. This cooperates with `futex_sleep`. [[futex-wake-op-2]] ===== futex_wake_op The `FUTEX_WAKE_OP` operation is quite complicated. First we obtain two futexes at addresses `uaddr` and `uaddr2` then we perform the atomic operation using `val3` and `uaddr2`. Then `val` waiters on the first futex is woken up and if the atomic operation condition holds we wake up `val2` (i.e. `timeout`) waiter on the second futex. [[futex-atomic-op]] ===== futex atomic operation The atomic operation takes two parameters `encoded_op` and `uaddr`. The encoded operation encodes the operation itself, comparing value, operation argument, and comparing argument. The pseudocode for the operation is like this one: [.programlisting] .... oldval = *uaddr2 *uaddr2 = oldval OP oparg .... And this is done atomically. First a copying in of the number at `uaddr` is performed and the operation is done. The code handles page faults and if no page fault occurs `oldval` is compared to `cmparg` argument with cmp comparator. [[futex-locking]] ===== Futex locking Futex implementation uses two lock lists protecting `sx_lock` and global locks (either Giant or another `sx_lock`). Every operation is performed locked from the start to the very end. [[syscall-impl]] === Various syscalls implementation In this section I am going to describe some smaller syscalls that are worth mentioning because their implementation is not obvious or those syscalls are interesting from other point of view. [[syscall-at]] ==== *at family of syscalls During development of Linux(R) 2.6.16 kernel, the *at syscalls were added. Those syscalls (`openat` for example) work exactly like their at-less counterparts with the slight exception of the `dirfd` parameter. This parameter changes where the given file, on which the syscall is to be performed, is. When the `filename` parameter is absolute `dirfd` is ignored but when the path to the file is relative, it comes to the play. The `dirfd` parameter is a directory relative to which the relative pathname is checked. The `dirfd` parameter is a file descriptor of some directory or `AT_FDCWD`. So for example the `openat` syscall can be like this: [.programlisting] .... file descriptor 123 = /tmp/foo/, current working directory = /tmp/ openat(123, /tmp/bah\, flags, mode) /* opens /tmp/bah */ openat(123, bah\, flags, mode) /* opens /tmp/foo/bah */ openat(AT_FDWCWD, bah\, flags, mode) /* opens /tmp/bah */ openat(stdio, bah\, flags, mode) /* returns error because stdio is not a directory */ .... This infrastructure is necessary to avoid races when opening files outside the working directory. Imagine that a process consists of two threads, thread A and thread B. Thread A issues `open(./tmp/foo/bah., flags, mode)` and before returning it gets preempted and thread B runs. Thread B does not care about the needs of thread A and renames or removes [.filename]#/tmp/foo/#. We got a race. To avoid this we can open [.filename]#/tmp/foo# and use it as `dirfd` for `openat` syscall. This also enables user to implement per-thread working directories. Linux(R) family of *at syscalls contains: `linux_openat`, `linux_mkdirat`, `linux_mknodat`, `linux_fchownat`, `linux_futimesat`, `linux_fstatat64`, `linux_unlinkat`, `linux_renameat`, `linux_linkat`, `linux_symlinkat`, `linux_readlinkat`, `linux_fchmodat` and `linux_faccessat`. All these are implemented using the modified man:namei[9] routine and simple wrapping layer. [[implementation]] ===== Implementation The implementation is done by altering the man:namei[9] routine (described above) to take additional parameter `dirfd` in its `nameidata` structure, which specifies the starting point of the pathname lookup instead of using the current working directory every time. The resolution of `dirfd` from file descriptor number to a vnode is done in native *at syscalls. When `dirfd` is `AT_FDCWD` the `dvp` entry in `nameidata` structure is `NULL` but when `dirfd` is a different number we obtain a file for this file descriptor, check whether this file is valid and if there is vnode attached to it then we get a vnode. Then we check this vnode for being a directory. In the actual man:namei[9] routine we simply substitute the `dvp` vnode for `dp` variable in the man:namei[9] function, which determines the starting point. The man:namei[9] is not used directly but via a trace of different functions on various levels. For example the `openat` goes like this: [.programlisting] .... openat() --> kern_openat() --> vn_open() -> namei() .... For this reason `kern_open` and `vn_open` must be altered to incorporate the additional `dirfd` parameter. No compat layer is created for those because there are not many users of this and the users can be easily converted. This general implementation enables FreeBSD to implement their own *at syscalls. This is being discussed right now. [[ioctl]] ==== Ioctl The ioctl interface is quite fragile due to its generality. We have to bear in mind that devices differ between Linux(R) and FreeBSD so some care must be applied to do ioctl emulation work right. The ioctl handling is implemented in [.filename]#linux_ioctl.c#, where `linux_ioctl` function is defined. This function simply iterates over sets of ioctl handlers to find a handler that implements a given command. The ioctl syscall has three parameters, the file descriptor, command and an argument. The command is a 16-bit number, which in theory is divided into high 8 bits determining class of the ioctl command and low 8 bits, which are the actual command within the given set. The emulation takes advantage of this division. We implement handlers for each set, like `sound_handler` or `disk_handler`. Each handler has a maximum command and a minimum command defined, which is used for determining what handler is used. There are slight problems with this approach because Linux(R) does not use the set division consistently so sometimes ioctls for a different set are inside a set they should not belong to (SCSI generic ioctls inside cdrom set, etc.). FreeBSD currently does not implement many Linux(R) ioctls (compared to NetBSD, for example) but the plan is to port those from NetBSD. The trend is to use Linux(R) ioctls even in the native FreeBSD drivers because of the easy porting of applications. [[debugging]] ==== Debugging Every syscall should be debuggable. For this purpose we introduce a small infrastructure. We have the ldebug facility, which tells whether a given syscall should be debugged (settable via a sysctl). For printing we have LMSG and ARGS macros. Those are used for altering a printable string for uniform debugging messages. [[conclusion]] == Conclusion [[results]] === Results As of April 2007 the Linux(R) emulation layer is capable of emulating the Linux(R) 2.6.16 kernel quite well. The remaining problems concern futexes, unfinished *at family of syscalls, problematic signals delivery, missing `epoll` and `inotify` and probably some bugs we have not discovered yet. Despite this we are capable of running basically all the Linux(R) programs included in FreeBSD Ports Collection with Fedora Core 4 at 2.6.16 and there are some rudimentary reports of success with Fedora Core 6 at 2.6.16. The Fedora Core 6 linux_base was recently committed enabling some further testing of the emulation layer and giving us some more hints where we should put our effort in implementing missing stuff. We are able to run the most used applications like package:www/linux-firefox[], package:net-im/skype[] and some games from the Ports Collection. Some of the programs exhibit bad behavior under 2.6 emulation but this is currently under investigation and hopefully will be fixed soon. The only big application that is known not to work is the Linux(R) Java(TM) Development Kit and this is because of the requirement of `epoll` facility which is not directly related to the Linux(R) kernel 2.6. We hope to enable 2.6.16 emulation by default some time after FreeBSD 7.0 is released at least to expose the 2.6 emulation parts for some wider testing. Once this is done we can switch to Fedora Core 6 linux_base, which is the ultimate plan. [[future-work]] === Future work Future work should focus on fixing the remaining issues with futexes, implement the rest of the *at family of syscalls, fix the signal delivery and possibly implement the `epoll` and `inotify` facilities. We hope to be able to run the most important programs flawlessly soon, so we will be able to switch to the 2.6 emulation by default and make the Fedora Core 6 the default linux_base because our currently used Fedora Core 4 is not supported any more. The other possible goal is to share our code with NetBSD and DragonflyBSD. NetBSD has some support for 2.6 emulation but its far from finished and not really tested. DragonflyBSD has expressed some interest in porting the 2.6 improvements. Generally, as Linux(R) develops we would like to keep up with their development, implementing newly added syscalls. Splice comes to mind first. Some already implemented syscalls are also suboptimal, for example `mremap` and others. Some performance improvements can also be made, finer grained locking and others. [[team]] === Team I cooperated on this project with (in alphabetical order): * `{jhb}` * `{kib}` * Emmanuel Dreyfus * Scot Hetzel * `{jkim}` * `{netchild}` * `{ssouhlal}` * Li Xiao * `{davidxu}` I would like to thank all those people for their advice, code reviews and general support. [[literatures]] == Literatures . Marshall Kirk McKusick - George V. Nevile-Neil. Design and Implementation of the FreeBSD operating system. Addison-Wesley, 2005. . https://tldp.org[https://tldp.org] . https://www.kernel.org[https://www.kernel.org] diff --git a/documentation/content/en/articles/rc-scripting/_index.adoc b/documentation/content/en/articles/rc-scripting/_index.adoc index ae68c31c28..cdfa656e53 100644 --- a/documentation/content/en/articles/rc-scripting/_index.adoc +++ b/documentation/content/en/articles/rc-scripting/_index.adoc @@ -1,846 +1,846 @@ --- title: Practical rc.d scripting in BSD authors: - author: Yar Tikhiy email: yar@FreeBSD.org copyright: 2005-2006, 2012 The FreeBSD Project description: A guide to writing new rc.d scripts and understanding those already written trademarks: ["freebsd", "netbsd", "general"] tags: ["rc.d", "scripting", "guide", "tutorial", "FreeBSD"] --- = Practical rc.d scripting in BSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/rc-scripting/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract Beginners may find it difficult to relate the facts from the formal documentation on the BSD [.filename]#rc.d# framework with the practical tasks of [.filename]#rc.d# scripting. In this article, we consider a few typical cases of increasing complexity, show [.filename]#rc.d# features suited for each case, and discuss how they work. Such an examination should provide reference points for further study of the design and efficient application of [.filename]#rc.d#. ''' toc::[] [[rcng-intro]] == Introduction The historical BSD had a monolithic startup script, [.filename]#/etc/rc#. It was invoked by man:init[8] at system boot time and performed all userland tasks required for multi-user operation: checking and mounting file systems, setting up the network, starting daemons, and so on. The precise list of tasks was not the same in every system; admins needed to customize it. With few exceptions, [.filename]#/etc/rc# had to be modified, and true hackers liked it. The real problem with the monolithic approach was that it provided no control over the individual components started from [.filename]#/etc/rc#. For instance, [.filename]#/etc/rc# could not restart a single daemon. The system admin had to find the daemon process by hand, kill it, wait until it actually exited, then browse through [.filename]#/etc/rc# for the flags, and finally type the full command line to start the daemon again. The task would become even more difficult and prone to errors if the service to restart consisted of more than one daemon or demanded additional actions. In a few words, the single script failed to fulfil what scripts are for: to make the system admin's life easier. Later there was an attempt to split out some parts of [.filename]#/etc/rc# for the sake of starting the most important subsystems separately. The notorious example was [.filename]#/etc/netstart# to bring up networking. It did allow for accessing the network from single-user mode, but it did not integrate well into the automatic startup process because parts of its code needed to interleave with actions essentially unrelated to networking. That was why [.filename]#/etc/netstart# mutated into [.filename]#/etc/rc.network#. The latter was no longer an ordinary script; it comprised of large, tangled man:sh[1] functions called from [.filename]#/etc/rc# at different stages of system startup. However, as the startup tasks grew diverse and sophisticated, the "quasi-modular" approach became even more of a drag than the monolithic [.filename]#/etc/rc# had been. Without a clean and well-designed framework, the startup scripts had to bend over backwards to satisfy the needs of rapidly developing BSD-based operating systems. It became obvious at last that more steps are necessary on the way to a fine-grained and extensible [.filename]#rc# system. Thus BSD [.filename]#rc.d# was born. Its acknowledged fathers were Luke Mewburn and the NetBSD community. Later it was imported into FreeBSD. Its name refers to the location of system scripts for individual services, which is in [.filename]#/etc/rc.d#. Soon we will learn about more components of the [.filename]#rc.d# system and see how the individual scripts are invoked. The basic ideas behind BSD [.filename]#rc.d# are _fine modularity_ and __code reuse__. _Fine modularity_ means that each basic "service" such as a system daemon or primitive startup task gets its own man:sh[1] script able to start the service, stop it, reload it, check its status. A particular action is chosen by the command-line argument to the script. The [.filename]#/etc/rc# script still drives system startup, but now it merely invokes the smaller scripts one by one with the `start` argument. It is easy to perform shutdown tasks as well by running the same set of scripts with the `stop` argument, which is done by [.filename]#/etc/rc.shutdown#. Note how closely this follows the Unix way of having a set of small specialized tools, each fulfilling its task as well as possible. _Code reuse_ means that common operations are implemented as man:sh[1] functions and collected in [.filename]#/etc/rc.subr#. Now a typical script can be just a few lines' worth of man:sh[1] code. Finally, an important part of the [.filename]#rc.d# framework is man:rcorder[8], which helps [.filename]#/etc/rc# to run the small scripts orderly with respect to dependencies between them. It can help [.filename]#/etc/rc.shutdown#, too, because the proper order for the shutdown sequence is opposite to that of startup. The BSD [.filename]#rc.d# design is described in <>, and the [.filename]#rc.d# components are documented in great detail in <>. -However, it might not appear obvious to an [.filename]#rc.d# newbie how to tie the numerous bits and pieces together in order to create a well-styled script for a particular task. +However, it might not appear obvious to an [.filename]#rc.d# newbie how to tie the numerous bits and pieces together to create a well-styled script for a particular task. Therefore this article will try a different approach to describe [.filename]#rc.d#. It will show which features should be used in a number of typical cases, and why. Note that this is not a how-to document because our aim is not at giving ready-made recipes, but at showing a few easy entrances into the [.filename]#rc.d# realm. Neither is this article a replacement for the relevant manual pages. Do not hesitate to refer to them for more formal and complete documentation while reading this article. There are prerequisites to understanding this article. -First of all, you should be familiar with the man:sh[1] scripting language in order to master [.filename]#rc.d#. +First of all, you should be familiar with the man:sh[1] scripting language to master [.filename]#rc.d#. In addition, you should know how the system performs userland startup and shutdown tasks, which is described in man:rc[8]. This article focuses on the FreeBSD branch of [.filename]#rc.d#. Nevertheless, it may be useful to NetBSD developers, too, because the two branches of BSD [.filename]#rc.d# not only share the same design but also stay similar in their aspects visible to script authors. [[rcng-task]] == Outlining the task A little consideration before starting `$EDITOR` will not hurt. -In order to write a well-tempered [.filename]#rc.d# script for a system service, we should be able to answer the following questions first: +To write a well-tempered [.filename]#rc.d# script for a system service, we should be able to answer the following questions first: * Is the service mandatory or optional? * Will the script serve a single program, e.g., a daemon, or perform more complex actions? * Which other services will our service depend on, and vice versa? From the examples that follow we will see why it is important to know the answers to these questions. [[rcng-dummy]] == A dummy script The following script just emits a message each time the system boots up: [.programlisting] .... #!/bin/sh <.> . /etc/rc.subr <.> name="dummy" <.> start_cmd="${name}_start" <.> stop_cmd=":" <.> dummy_start() <.> { echo "Nothing started." } load_rc_config $name <.> run_rc_command "$1" <.> .... Things to note are: ➊ An interpreted script should begin with the magic "shebang" line. That line specifies the interpreter program for the script. Due to the shebang line, the script can be invoked exactly like a binary program provided that it has the execute bit set. (See man:chmod[1].) For example, a system admin can run our script manually, from the command line: [source,shell] .... # /etc/rc.d/dummy start .... [NOTE] ==== -In order to be properly managed by the [.filename]#rc.d# framework, its scripts need to be written in the man:sh[1] language. +To be properly managed by the [.filename]#rc.d# framework, its scripts need to be written in the man:sh[1] language. If you have a service or port that uses a binary control utility or a startup routine written in another language, install that element in [.filename]#/usr/sbin# (for the system) or [.filename]#/usr/local/sbin# (for ports) and call it from a man:sh[1] script in the appropriate [.filename]#rc.d# directory. ==== [TIP] ==== If you would like to learn the details of why [.filename]#rc.d# scripts must be written in the man:sh[1] language, see how [.filename]#/etc/rc# invokes them by means of `run_rc_script`, then study the implementation of `run_rc_script` in [.filename]#/etc/rc.subr#. ==== ➋ In [.filename]#/etc/rc.subr#, a number of man:sh[1] functions are defined for an [.filename]#rc.d# script to use. The functions are documented in man:rc.subr[8]. While it is theoretically possible to write an [.filename]#rc.d# script without ever using man:rc.subr[8], its functions prove extremely handy and make the job an order of magnitude easier. So it is no surprise that everybody resorts to man:rc.subr[8] in [.filename]#rc.d# scripts. We are not going to be an exception. An [.filename]#rc.d# script must "source"[.filename]#/etc/rc.subr# (include it using "`.`") _before_ it calls man:rc.subr[8] functions so that man:sh[1] has an opportunity to learn the functions. The preferred style is to source [.filename]#/etc/rc.subr# first of all. [NOTE] ==== Some useful functions related to networking are provided by another include file, [.filename]#/etc/network.subr#. ==== ➌ [[name-var]]The mandatory variable `name` specifies the name of our script. It is required by man:rc.subr[8]. That is, each [.filename]#rc.d# script _must_ set `name` before it calls man:rc.subr[8] functions. Now it is the right time to choose a unique name for our script once and for all. We will use it in a number of places while developing the script. For a start, let us give the same name to the script file, too. [NOTE] ==== The current style of [.filename]#rc.d# scripting is to enclose values assigned to variables in double quotes. Keep in mind that it is just a style issue that may not always be applicable. You can safely omit quotes from around simple words without man:sh[1] metacharacters in them, while in certain cases you will need single quotes to prevent any interpretation of the value by man:sh[1]. A programmer should be able to tell the language syntax from style conventions and use both of them wisely. ==== ➍ The main idea behind man:rc.subr[8] is that an [.filename]#rc.d# script provides handlers, or methods, for man:rc.subr[8] to invoke. In particular, `start`, `stop`, and other arguments to an [.filename]#rc.d# script are handled this way. A method is a man:sh[1] expression stored in a variable named `argument_cmd`, where _argument_ corresponds to what can be specified on the script's command line. We will see later how man:rc.subr[8] provides default methods for the standard arguments. [NOTE] ==== To make the code in [.filename]#rc.d# more uniform, it is common to use `${name}` wherever appropriate. Thus a number of lines can be just copied from one script to another. ==== ➎ We should keep in mind that man:rc.subr[8] provides default methods for the standard arguments. Consequently, we must override a standard method with a no-op man:sh[1] expression if we want it to do nothing. ➏ The body of a sophisticated method can be implemented as a function. It is a good idea to make the function name meaningful. [IMPORTANT] ==== It is strongly recommended to add the prefix `${name}` to the names of all functions defined in our script so they never clash with the functions from man:rc.subr[8] or another common include file. ==== ➐ This call to man:rc.subr[8] loads man:rc.conf[5] variables. Our script makes no use of them yet, but it still is recommended to load man:rc.conf[5] because there can be man:rc.conf[5] variables controlling man:rc.subr[8] itself. ➑ Usually this is the last command in an [.filename]#rc.d# script. It invokes the man:rc.subr[8] machinery to perform the requested action using the variables and methods our script has provided. [[rcng-confdummy]] == A configurable dummy script Now let us add some controls to our dummy script. As you may know, [.filename]#rc.d# scripts are controlled with man:rc.conf[5]. Fortunately, man:rc.subr[8] hides all the complications from us. The following script uses man:rc.conf[5] via man:rc.subr[8] to see whether it is enabled in the first place, and to fetch a message to show at boot time. These two tasks in fact are independent. On the one hand, an [.filename]#rc.d# script can just support enabling and disabling its service. On the other hand, a mandatory [.filename]#rc.d# script can have configuration variables. We will do both things in the same script though: [.programlisting] .... #!/bin/sh . /etc/rc.subr name=dummy rcvar=dummy_enable <.> start_cmd="${name}_start" stop_cmd=":" load_rc_config $name <.> : ${dummy_enable:=no} <.> : ${dummy_msg="Nothing started."} <.> dummy_start() { echo "$dummy_msg" <.> } run_rc_command "$1" .... What changed in this example? ➊ The variable `rcvar` specifies the name of the ON/OFF knob variable. ➋ Now `load_rc_config` is invoked earlier in the script, before any man:rc.conf[5] variables are accessed. [NOTE] ==== While examining [.filename]#rc.d# scripts, keep in mind that man:sh[1] defers the evaluation of expressions in a function until the latter is called. Therefore it is not an error to invoke `load_rc_config` as late as just before `run_rc_command` and still access man:rc.conf[5] variables from the method functions exported to `run_rc_command`. This is because the method functions are to be called by `run_rc_command`, which is invoked _after_ `load_rc_config`. ==== ➌ A warning will be emitted by `run_rc_command` if `rcvar` itself is set, but the indicated knob variable is unset. If your [.filename]#rc.d# script is for the base system, you should add a default setting for the knob to [.filename]#/etc/defaults/rc.conf# and document it in man:rc.conf[5]. Otherwise it is your script that should provide a default setting for the knob. The canonical approach to the latter case is shown in the example. [NOTE] ==== You can make man:rc.subr[8] act as though the knob is set to `ON`, irrespective of its current setting, by prefixing the argument to the script with `one` or `force`, as in `onestart` or `forcestop`. Keep in mind though that `force` has other dangerous effects we will touch upon below, while `one` just overrides the ON/OFF knob. E.g., assume that `dummy_enable` is `OFF`. The following command will run the `start` method in spite of the setting: [source,shell] .... # /etc/rc.d/dummy onestart .... ==== ➍ Now the message to be shown at boot time is no longer hard-coded in the script. It is specified by an man:rc.conf[5] variable named `dummy_msg`. This is a trivial example of how man:rc.conf[5] variables can control an [.filename]#rc.d# script. [IMPORTANT] ==== The names of all man:rc.conf[5] variables used exclusively by our script _must_ have the same prefix: `${name}_`. For example: `dummy_mode`, `dummy_state_file`, and so on. ==== [NOTE] ==== While it is possible to use a shorter name internally, e.g., just `msg`, adding the unique prefix `${name}_` to all global names introduced by our script will save us from possible collisions with the man:rc.subr[8] namespace. As a rule, [.filename]#rc.d# scripts of the base system need not provide defaults for their man:rc.conf[5] variables because the defaults should be set in [.filename]#/etc/defaults/rc.conf# instead. On the other hand, [.filename]#rc.d# scripts for ports should provide the defaults as shown in the example. ==== ➎ Here we use `dummy_msg` to actually control our script, i.e., to emit a variable message. Use of a shell function is overkill here, since it only runs a single command; an equally valid alternative is: [.programlisting] .... start_cmd="echo \"$dummy_msg\"" .... [[rcng-daemon]] == Startup and shutdown of a simple daemon We said earlier that man:rc.subr[8] could provide default methods. Obviously, such defaults cannot be too general. They are suited for the common case of starting and shutting down a simple daemon program. Let us assume now that we need to write an [.filename]#rc.d# script for such a daemon called `mumbled`. Here it is: [.programlisting] .... #!/bin/sh . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" <.> load_rc_config $name run_rc_command "$1" .... Pleasingly simple, isn't it? Let us examine our little script. The only new thing to note is as follows: ➊ The `command` variable is meaningful to man:rc.subr[8]. If it is set, man:rc.subr[8] will act according to the scenario of serving a conventional daemon. In particular, the default methods will be provided for such arguments: `start`, `stop`, `restart`, `poll`, and `status`. The daemon will be started by running `$command` with command-line flags specified by `$mumbled_flags`. Thus all the input data for the default `start` method are available in the variables set by our script. Unlike `start`, other methods may require additional information about the process started. For instance, `stop` must know the PID of the process to terminate it. In the present case, man:rc.subr[8] will scan through the list of all processes, looking for a process with its name equal to `procname`. The latter is another variable of meaning to man:rc.subr[8], and its value defaults to that of `command`. In other words, when we set `command`, `procname` is effectively set to the same value. This enables our script to kill the daemon and to check if it is running in the first place. [NOTE] ==== Some programs are in fact executable scripts. The system runs such a script by starting its interpreter and passing the name of the script to it as a command-line argument. This is reflected in the list of processes, which can confuse man:rc.subr[8]. You should additionally set `command_interpreter` to let man:rc.subr[8] know the actual name of the process if `$command` is a script. For each [.filename]#rc.d# script, there is an optional man:rc.conf[5] variable that takes precedence over `command`. Its name is constructed as follows: `${name}_program`, where `name` is the mandatory variable we discussed <>. E.g., in this case it will be `mumbled_program`. It is man:rc.subr[8] that arranges `${name}_program` to override `command`. Of course, man:sh[1] will permit you to set `${name}_program` from man:rc.conf[5] or the script itself even if `command` is unset. In that case, the special properties of `${name}_program` are lost, and it becomes an ordinary variable your script can use for its own purposes. However, the sole use of `${name}_program` is discouraged because using it together with `command` became an idiom of [.filename]#rc.d# scripting. ==== For more detailed information on default methods, refer to man:rc.subr[8]. [[rcng-daemon-adv]] == Startup and shutdown of an advanced daemon Let us add some meat onto the bones of the previous script and make it more complex and featureful. The default methods can do a good job for us, but we may need some of their aspects tweaked. Now we will learn how to tune the default methods to our needs. [.programlisting] .... #!/bin/sh . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" command_args="mock arguments > /dev/null 2>&1" <.> pidfile="/var/run/${name}.pid" <.> required_files="/etc/${name}.conf /usr/share/misc/${name}.rules" <.> sig_reload="USR1" <.> start_precmd="${name}_prestart" <.> stop_postcmd="echo Bye-bye" <.> extra_commands="reload plugh xyzzy" <.> plugh_cmd="mumbled_plugh" <.> xyzzy_cmd="echo 'Nothing happens.'" mumbled_prestart() { if checkyesno mumbled_smart; then <.> rc_flags="-o smart ${rc_flags}" <.> fi case "$mumbled_mode" in foo) rc_flags="-frotz ${rc_flags}" ;; bar) rc_flags="-baz ${rc_flags}" ;; *) warn "Invalid value for mumbled_mode" <.> return 1 <.> ;; esac run_rc_command xyzzy <.> return 0 } mumbled_plugh() <.> { echo 'A hollow voice says "plugh".' } load_rc_config $name run_rc_command "$1" .... ➊ Additional arguments to `$command` can be passed in `command_args`. They will be added to the command line after `$mumbled_flags`. Since the final command line is passed to `eval` for its actual execution, input and output redirections can be specified in `command_args`. [NOTE] ==== _Never_ include dashed options, like `-X` or `--foo`, in `command_args`. The contents of `command_args` will appear at the end of the final command line, hence they are likely to follow arguments present in `${name}_flags`; but most commands will not recognize dashed options after ordinary arguments. A better way of passing additional options to `$command` is to add them to the beginning of `${name}_flags`. Another way is to modify `rc_flags` <>. ==== ➋ A good-mannered daemon should create a _pidfile_ so that its process can be found more easily and reliably. The variable `pidfile`, if set, tells man:rc.subr[8] where it can find the pidfile for its default methods to use. [NOTE] ==== In fact, man:rc.subr[8] will also use the pidfile to see if the daemon is already running before starting it. This check can be skipped by using the `faststart` argument. ==== ➌ If the daemon cannot run unless certain files exist, just list them in `required_files`, and man:rc.subr[8] will check that those files do exist before starting the daemon. There also are `required_dirs` and `required_vars` for directories and environment variables, respectively. They all are described in detail in man:rc.subr[8]. [NOTE] ==== The default method from man:rc.subr[8] can be forced to skip the prerequisite checks by using `forcestart` as the argument to the script. ==== ➍ We can customize signals to send to the daemon in case they differ from the well-known ones. In particular, `sig_reload` specifies the signal that makes the daemon reload its configuration; it is SIGHUP by default. Another signal is sent to stop the daemon process; the default is SIGTERM, but this can be changed by setting `sig_stop` appropriately. [NOTE] ==== The signal names should be specified to man:rc.subr[8] without the `SIG` prefix, as it is shown in the example. The FreeBSD version of man:kill[1] can recognize the `SIG` prefix, but the versions from other OS types may not. ==== ➎➏ Performing additional tasks before or after the default methods is easy. For each command-argument supported by our script, we can define `argument_precmd` and `argument_postcmd`. These man:sh[1] commands are invoked before and after the respective method, as it is evident from their names. [NOTE] ==== Overriding a default method with a custom `argument_cmd` still does not prevent us from making use of `argument_precmd` or `argument_postcmd` if we need to. In particular, the former is good for checking custom, sophisticated conditions that should be met before performing the command itself. Using `argument_precmd` along with `argument_cmd` lets us logically separate the checks from the action. Do not forget that you can cram any valid man:sh[1] expressions into the methods, pre-, and post-commands you define. Just invoking a function that makes the real job is a good style in most cases, but never let style limit your understanding of what is going on behind the curtain. ==== ➐ If we would like to implement custom arguments, which can also be thought of as _commands_ to our script, we need to list them in `extra_commands` and provide methods to handle them. [NOTE] ==== The `reload` command is special. On the one hand, it has a preset method in man:rc.subr[8]. On the other hand, `reload` is not offered by default. The reason is that not all daemons use the same reload mechanism and some have nothing to reload at all. So we need to ask explicitly that the builtin functionality be provided. We can do so via `extra_commands`. What do we get from the default method for `reload`? Quite often daemons reload their configuration upon reception of a signal - typically, SIGHUP. Therefore man:rc.subr[8] attempts to reload the daemon by sending a signal to it. The signal is preset to SIGHUP but can be customized via `sig_reload` if necessary. ==== ➑⓮ Our script supports two non-standard commands, `plugh` and `xyzzy`. We saw them listed in `extra_commands`, and now it is time to provide methods for them. The method for `xyzzy` is just inlined while that for `plugh` is implemented as the `mumbled_plugh` function. Non-standard commands are not invoked during startup or shutdown. Usually they are for the system admin's convenience. They can also be used from other subsystems, e.g., man:devd[8] if specified in man:devd.conf[5]. The full list of available commands can be found in the usage line printed by man:rc.subr[8] when the script is invoked without arguments. For example, here is the usage line from the script under study: [source,shell] .... # /etc/rc.d/mumbled Usage: /etc/rc.d/mumbled [fast|force|one](start|stop|restart|rcvar|reload|plugh|xyzzy|status|poll) .... ⓭ A script can invoke its own standard or non-standard commands if needed. This may look similar to calling functions, but we know that commands and shell functions are not always the same thing. For instance, `xyzzy` is not implemented as a function here. In addition, there can be a pre-command and post-command, which should be invoked orderly. So the proper way for a script to run its own command is by means of man:rc.subr[8], as shown in the example. ➒ A handy function named `checkyesno` is provided by man:rc.subr[8]. It takes a variable name as its argument and returns a zero exit code if and only if the variable is set to `YES`, or `TRUE`, or `ON`, or `1`, case insensitive; a non-zero exit code is returned otherwise. In the latter case, the function tests the variable for being set to `NO`, `FALSE`, `OFF`, or `0`, case insensitive; it prints a warning message if the variable contains anything else, i.e., junk. Keep in mind that for man:sh[1] a zero exit code means true and a non-zero exit code means false. [IMPORTANT] ==== The `checkyesno` function takes a __variable name__. Do not pass the expanded _value_ of a variable to it; it will not work as expected. The following is the correct usage of `checkyesno`: [.programlisting] .... if checkyesno mumbled_enable; then foo fi .... On the contrary, calling `checkyesno` as shown below will not work - at least not as expected: [.programlisting] .... if checkyesno "${mumbled_enable}"; then foo fi .... ==== ➓ [[rc-flags]]We can affect the flags to be passed to `$command` by modifying `rc_flags` in `$start_precmd`. ⓫ In certain cases we may need to emit an important message that should go to `syslog` as well. This can be done easily with the following man:rc.subr[8] functions: `debug`, `info`, `warn`, and `err`. The latter function then exits the script with the code specified. ⓬ The exit codes from methods and their pre-commands are not just ignored by default. If `argument_precmd` returns a non-zero exit code, the main method will not be performed. In turn, `argument_postcmd` will not be invoked unless the main method returns a zero exit code. [NOTE] ==== However, man:rc.subr[8] can be instructed from the command line to ignore those exit codes and invoke all commands anyway by prefixing an argument with `force`, as in `forcestart`. ==== [[rcng-hookup]] == Connecting a script to the rc.d framework After a script has been written, it needs to be integrated into [.filename]#rc.d#. The crucial step is to install the script in [.filename]#/etc/rc.d# (for the base system) or [.filename]#/usr/local/etc/rc.d# (for ports). Both [.filename]#bsd.prog.mk# and [.filename]#bsd.port.mk# provide convenient hooks for that, and usually you do not have to worry about the proper ownership and mode. System scripts should be installed from [.filename]#src/libexec/rc/rc.d# through the [.filename]#Makefile# found there. Port scripts can be installed using `USE_RC_SUBR` as described extref:{porters-handbook}[in the Porter's Handbook, rc-scripts]. However, we should consider beforehand the place of our script in the system startup sequence. The service handled by our script is likely to depend on other services. For instance, a network daemon cannot function without the network interfaces and routing up and running. Even if a service seems to demand nothing, it can hardly start before the basic filesystems have been checked and mounted. We mentioned man:rcorder[8] already. Now it is time to have a close look at it. In a nutshell, man:rcorder[8] takes a set of files, examines their contents, and prints a dependency-ordered list of files from the set to `stdout`. The point is to keep dependency information _inside_ the files so that each file can speak for itself only. A file can specify the following information: * the names of the "conditions" (which means services to us) it __provides__; * the names of the "conditions" it __requires__; * the names of the "conditions" this file should run __before__; * additional _keywords_ that can be used to select a subset from the whole set of files (man:rcorder[8] can be instructed via options to include or omit the files having particular keywords listed.) It is no surprise that man:rcorder[8] can handle only text files with a syntax close to that of man:sh[1]. That is, special lines understood by man:rcorder[8] look like man:sh[1] comments. The syntax of such special lines is rather rigid to simplify their processing. See man:rcorder[8] for details. Besides using man:rcorder[8] special lines, a script can insist on its dependency upon another service by just starting it forcibly. This can be needed when the other service is optional and will not start by itself because the system admin has disabled it mistakenly in man:rc.conf[5]. With this general knowledge in mind, let us consider the simple daemon script enhanced with dependency stuff: [.programlisting] .... #!/bin/sh # PROVIDE: mumbled oldmumble <.> # REQUIRE: DAEMON cleanvar frotz <.> # BEFORE: LOGIN <.> # KEYWORD: nojail shutdown <.> . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" start_precmd="${name}_prestart" mumbled_prestart() { if ! checkyesno frotz_enable && \ ! /etc/rc.d/frotz forcestatus 1>/dev/null 2>&1; then force_depend frotz || return 1 <.> fi return 0 } load_rc_config $name run_rc_command "$1" .... As before, detailed analysis follows: ➊ That line declares the names of "conditions" our script provides. Now other scripts can record a dependency on our script by those names. [NOTE] ==== Usually a script specifies a single condition provided. However, nothing prevents us from listing several conditions there, e.g., for compatibility reasons. In any case, the name of the main, or the only, `PROVIDE:` condition should be the same as `${name}`. ==== ➋➌ So our script indicates which "conditions" provided by other scripts it depends on. According to the lines, our script asks man:rcorder[8] to put it after the script(s) providing [.filename]#DAEMON# and [.filename]#cleanvar#, but before that providing [.filename]#LOGIN#. [NOTE] ==== The `BEFORE:` line should not be abused to work around an incomplete dependency list in the other script. The appropriate case for using `BEFORE:` is when the other script does not care about ours, but our script can do its task better if run before the other one. A typical real-life example is the network interfaces vs. the firewall: While the interfaces do not depend on the firewall in doing their job, the system security will benefit from the firewall being ready before there is any network traffic. Besides conditions corresponding to a single service each, there are meta-conditions and their "placeholder" scripts used to ensure that certain groups of operations are performed before others. These are denoted by [.filename]#UPPERCASE# names. Their list and purposes can be found in man:rc[8]. Keep in mind that putting a service name in the `REQUIRE:` line does not guarantee that the service will actually be running by the time our script starts. The required service may fail to start or just be disabled in man:rc.conf[5]. Obviously, man:rcorder[8] cannot track such details, and man:rc[8] will not do that either. Consequently, the application started by our script should be able to cope with any required services being unavailable. In certain cases, we can help it as discussed <> ==== [[keywords]]➍ As we remember from the above text, man:rcorder[8] keywords can be used to select or leave out some scripts. Namely any man:rcorder[8] consumer can specify through `-k` and `-s` options which keywords are on the "keep list" and "skip list", respectively. From all the files to be dependency sorted, man:rcorder[8] will pick only those having a keyword from the keep list (unless empty) and not having a keyword from the skip list. In FreeBSD, man:rcorder[8] is used by [.filename]#/etc/rc# and [.filename]#/etc/rc.shutdown#. These two scripts define the standard list of FreeBSD [.filename]#rc.d# keywords and their meanings as follows: nojail:: The service is not for man:jail[8] environment. The automatic startup and shutdown procedures will ignore the script if inside a jail. nostart:: The service is to be started manually or not started at all. The automatic startup procedure will ignore the script. In conjunction with the [.filename]#shutdown# keyword, this can be used to write scripts that do something only at system shutdown. shutdown:: This keyword is to be listed __explicitly__ if the service needs to be stopped before system shutdown. [NOTE] ==== When the system is going to shut down, [.filename]#/etc/rc.shutdown# runs. It assumes that most [.filename]#rc.d# scripts have nothing to do at that time. Therefore [.filename]#/etc/rc.shutdown# selectively invokes [.filename]#rc.d# scripts with the [.filename]#shutdown# keyword, effectively ignoring the rest of the scripts. For even faster shutdown, [.filename]#/etc/rc.shutdown# passes the [.filename]#faststop# command to the scripts it runs so that they skip preliminary checks, e.g., the pidfile check. As dependent services should be stopped before their prerequisites, [.filename]#/etc/rc.shutdown# runs the scripts in reverse dependency order. If writing a real [.filename]#rc.d# script, you should consider whether it is relevant at system shutdown time. E.g., if your script does its work in response to the [.filename]#start# command only, then you need not to include this keyword. However, if your script manages a service, it is probably a good idea to stop it before the system proceeds to the final stage of its shutdown sequence described in man:halt[8]. In particular, a service should be stopped explicitly if it needs considerable time or special actions to shut down cleanly. A typical example of such a service is a database engine. ==== [[forcedep]]➎ To begin with, `force_depend` should be used with much care. It is generally better to revise the hierarchy of configuration variables for your [.filename]#rc.d# scripts if they are interdependent. If you still cannot do without `force_depend`, the example offers an idiom of how to invoke it conditionally. In the example, our `mumbled` daemon requires that another one, `frotz`, be started in advance. However, `frotz` is optional, too; and man:rcorder[8] knows nothing about such details. Fortunately, our script has access to all man:rc.conf[5] variables. If `frotz_enable` is true, we hope for the best and rely on [.filename]#rc.d# to have started `frotz`. Otherwise we forcibly check the status of `frotz`. Finally, we enforce our dependency on `frotz` if it is found to be not running. A warning message will be emitted by `force_depend` because it should be invoked only if a misconfiguration has been detected. [[rcng-args]] == Giving more flexibility to an rc.d script When invoked during startup or shutdown, an [.filename]#rc.d# script is supposed to act on the entire subsystem it is responsible for. E.g., [.filename]#/etc/rc.d/netif# should start or stop all network interfaces described by man:rc.conf[5]. Either task can be uniquely indicated by a single command argument such as `start` or `stop`. Between startup and shutdown, [.filename]#rc.d# scripts help the admin to control the running system, and it is when the need for more flexibility and precision arises. For instance, the admin may want to add the settings of a new network interface to man:rc.conf[5] and then to start it without interfering with the operation of the existing interfaces. Next time the admin may need to shut down a single network interface. In the spirit of the command line, the respective [.filename]#rc.d# script calls for an extra argument, the interface name. Fortunately, man:rc.subr[8] allows for passing any number of arguments to script's methods (within the system limits). Due to that, the changes in the script itself can be minimal. How can man:rc.subr[8] gain access to the extra command-line arguments. Should it just grab them directly? Not by any means. Firstly, an man:sh[1] function has no access to the positional parameters of its caller, but man:rc.subr[8] is just a sack of such functions. Secondly, the good manner of [.filename]#rc.d# dictates that it is for the main script to decide which arguments are to be passed to its methods. So the approach adopted by man:rc.subr[8] is as follows: `run_rc_command` passes on all its arguments but the first one to the respective method verbatim. The first, omitted, argument is the name of the method itself: `start`, `stop`, etc. It will be shifted out by `run_rc_command`, so what is `$2` in the original command line will be presented as `$1` to the method, and so on. To illustrate this opportunity, let us modify the primitive dummy script so that its messages depend on the additional arguments supplied. Here we go: [.programlisting] .... #!/bin/sh . /etc/rc.subr name="dummy" start_cmd="${name}_start" stop_cmd=":" kiss_cmd="${name}_kiss" extra_commands="kiss" dummy_start() { if [ $# -gt 0 ]; then <.> echo "Greeting message: $*" else echo "Nothing started." fi } dummy_kiss() { echo -n "A ghost gives you a kiss" if [ $# -gt 0 ]; then <.> echo -n " and whispers: $*" fi case "$*" in *[.!?]) echo ;; *) echo . ;; esac } load_rc_config $name run_rc_command "$@" <.> .... What essential changes can we notice in the script? ➊ All arguments you type after `start` can end up as positional parameters to the respective method. We can use them in any way according to our task, skills, and fancy. In the current example, we just pass all of them to man:echo[1] as one string in the next line - note `$*` within the double quotes. Here is how the script can be invoked now: [source,shell] .... # /etc/rc.d/dummy start Nothing started. # /etc/rc.d/dummy start Hello world! Greeting message: Hello world! .... ➋ The same applies to any method our script provides, not only to a standard one. We have added a custom method named `kiss`, and it can take advantage of the extra arguments not less than `start` does. E.g.: [source,shell] .... # /etc/rc.d/dummy kiss A ghost gives you a kiss. # /etc/rc.d/dummy kiss Once I was Etaoin Shrdlu... A ghost gives you a kiss and whispers: Once I was Etaoin Shrdlu... .... ➌ If we want just to pass all extra arguments to any method, we can merely substitute `"$@"` for `"$1"` in the last line of our script, where we invoke `run_rc_command`. [IMPORTANT] ==== An man:sh[1] programmer ought to understand the subtle difference between `$*` and `$@` as the ways to designate all positional parameters. For its in-depth discussion, refer to a good handbook on man:sh[1] scripting. _Do not_ use the expressions until you fully understand them because their misuse will result in buggy and insecure scripts. ==== [NOTE] ==== Currently `run_rc_command` may have a bug that prevents it from keeping the original boundaries between arguments. That is, arguments with embedded whitespace may not be processed correctly. The bug stems from `$*` misuse. ==== [[rcng-furthur]] == Further reading [[lukem]]http://www.mewburn.net/luke/papers/rc.d.pdf[The original article by Luke Mewburn] offers a general overview of [.filename]#rc.d# and detailed rationale for its design decisions. It provides insight on the whole [.filename]#rc.d# framework and its place in a modern BSD operating system. [[manpages]]The manual pages man:rc[8], man:rc.subr[8], and man:rcorder[8] document the [.filename]#rc.d# components in great detail. You cannot fully use the [.filename]#rc.d# power without studying the manual pages and referring to them while writing your own scripts. The major source of working, real-life examples is [.filename]#/etc/rc.d# in a live system. Its contents are easy and pleasant to read because most rough corners are hidden deep in man:rc.subr[8]. Keep in mind though that the [.filename]#/etc/rc.d# scripts were not written by angels, so they might suffer from bugs and suboptimal design decisions. Now you can improve them! diff --git a/documentation/content/en/articles/remote-install/_index.adoc b/documentation/content/en/articles/remote-install/_index.adoc index f1bd16bb37..9efc5e4165 100644 --- a/documentation/content/en/articles/remote-install/_index.adoc +++ b/documentation/content/en/articles/remote-install/_index.adoc @@ -1,395 +1,395 @@ --- title: Remote Installation of the FreeBSD Operating System Without a Remote Console authors: - author: Daniel Gerzo email: danger@FreeBSD.org copyright: 2008-2021 The FreeBSD Documentation Project description: Describes the remote installation of the FreeBSD operating system when the console of the remote system is unavailable trademarks: ["freebsd", "general"] tags: ["Remote", "Installation", "FreeBSD"] --- = Remote Installation of the FreeBSD Operating System Without a Remote Console :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/remote-install/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract This article documents the remote installation of the FreeBSD operating system when the console of the remote system is unavailable. The main idea behind this article is the result of a collaboration with `{mm}` with valuable input provided by `{pjd}`. ''' toc::[] [[background]] == Background There are many server hosting providers in the world, but very few of them are officially supporting FreeBSD. They usually provide support for a Linux(R) distribution to be installed on the servers they offer. In some cases, these companies will install your preferred Linux(R) distribution if you request it. Using this option, we will attempt to install FreeBSD. In other cases, they may offer a rescue system which would be used in an emergency. It is possible to use this for our purposes as well. This article covers the basic installation and configuration steps required to bootstrap a remote installation of FreeBSD with RAID-1 and ZFS capabilities. [[intro]] == Introduction This section will summarize the purpose of this article and better explain what is covered herein. The instructions included in this article will benefit those using services provided by colocation facilities not supporting FreeBSD. [.procedure] ==== -. As we have mentioned in the <> section, many of the reputable server hosting companies provide some kind of rescue system, which is booted from their LAN and accessible over SSH. They usually provide this support in order to help their customers fix broken operating systems. As this article will explain, it is possible to install FreeBSD with the help of these rescue systems. +. As we have mentioned in the <> section, many of the reputable server hosting companies provide some kind of rescue system, which is booted from their LAN and accessible over SSH. They usually provide this support to help their customers fix broken operating systems. As this article will explain, it is possible to install FreeBSD with the help of these rescue systems. + . The next section of this article will describe how to configure, and build minimalistic FreeBSD on the local machine. That version will eventually be running on the remote machine from a ramdisk, which will allow us to install a complete FreeBSD operating system from an FTP mirror using the sysinstall utility. . The rest of this article will describe the installation procedure itself, as well as the configuration of the ZFS file system. ==== [[requirements]] === Requirements To continue successfully, you must: * Have a network accessible operating system with SSH access * Understand the FreeBSD installation process * Be familiar with the man:sysinstall[8] utility * Have the FreeBSD installation SO image or CD handy [[preparation]] == Preparation - mfsBSD Before FreeBSD may be installed on the target system, it is necessary to build the minimal FreeBSD operating system image which will boot from the hard drive. This way the new system can be accessed from the network, and the rest of the installation can be done without remote access to the system console. The mfsBSD tool-set can be used to build a tiny FreeBSD image. As the name of mfsBSD suggests ("mfs" means "memory file system"), the resulting image runs entirely from a ramdisk. Thanks to this feature, the manipulation of hard drives will not be limited, therefore it will be possible to install a complete FreeBSD operating system. The mfsBSD http://mfsbsd.vx.sk/[home page] includes pointers to the latest release of the toolset. Please note that the internals of mfsBSD and how it all fits together is beyond the scope of this article. The interested reader should consult the original documentation of mfsBSD for more details. Download and extract the latest mfsBSD release and change your working directory to the directory where the mfsBSD scripts will reside: [source,shell] .... # fetch http://mfsbsd.vx.sk/release/mfsbsd-2.1.tar.gz # tar xvzf mfsbsd-2.1.tar.gz # cd mfsbsd-2.1/ .... [[mfsbsd-config]] === Configuration of mfsBSD Before booting mfsBSD, a few important configuration options have to be set. The most important that we have to get right is, naturally, the network setup. The most suitable method to configure networking options depends on whether we know beforehand the type of the network interface we will use, and the network interface driver to be loaded for our hardware. We will see how mfsBSD can be configured in either case. Another important thing to set is the `root` password. This can be done by editing [.filename]#conf/loader.conf#. Please see the included comments. ==== The [.filename]#conf/interfaces.conf# method When the installed network interface card is unknown, it is possible to use the auto-detection features of mfsBSD. The startup scripts of mfsBSD can detect the correct driver to use, based on the MAC address of the interface, if we set the following options in [.filename]#conf/interfaces.conf#: [.programlisting] .... mac_interfaces="ext1" ifconfig_ext1_mac="00:00:00:00:00:00" ifconfig_ext1="inet 192.168.0.2/24" .... Do not forget to add the `defaultrouter` information to [.filename]#conf/rc.conf#: [.programlisting] .... defaultrouter="192.168.0.1" .... ==== The [.filename]#conf/rc.conf# Method When the network interface driver is known, it is more convenient to use [.filename]#conf/rc.conf# for networking options. The syntax of this file is the same as the one used in the standard man:rc.conf[5] file of FreeBSD. For example, if you know that a man:re[4] network interface is going to be available, you can set the following options in [.filename]#conf/rc.conf#: [.programlisting] .... defaultrouter="192.168.0.1" ifconfig_re0="inet 192.168.0.2/24" .... [[mfsbsd-build]] === Building an mfsBSD Image The process of building an mfsBSD image is pretty straightforward. The first step is to mount the FreeBSD installation CD, or the installation ISO image to [.filename]#/cdrom#. For the sake of example, in this article we will assume that you have downloaded the FreeBSD 10.1-RELEASE ISO. Mounting this ISO image to the [.filename]#/cdrom# directory is easy with the man:mdconfig[8] utility: [source,shell] .... # mdconfig -a -t vnode -u 10 -f FreeBSD-10.1-RELEASE-amd64-disc1.iso # mount_cd9660 /dev/md10 /cdrom .... Since the recent FreeBSD releases do not contain regular distribution sets, it is required to extract the FreeBSD distribution files from the distribution archives located on the ISO image: [source,shell] .... # mkdir DIST # tar -xvf /cdrom/usr/freebsd-dist/base.txz -C DIST # tar -xvf /cdrom/usr/freebsd-dist/kernel.txz -C DIST .... Next, build the bootable mfsBSD image: [source,shell] .... # make BASE=DIST .... [NOTE] ==== The above `make` has to be run from the top level of the mfsBSD directory tree, for example [.filename]#~/mfsbsd-2.1/#. ==== === Booting mfsBSD Now that the mfsBSD image is ready, it must be uploaded to the remote system running a live rescue system or pre-installed Linux(R) distribution. The most suitable tool for this task is scp: [source,shell] .... # scp disk.img root@192.168.0.2:. .... To boot mfsBSD image properly, it must be placed on the first (bootable) device of the given machine. This may be accomplished using this example providing that [.filename]#sda# is the first bootable disk device: [source,shell] .... # dd if=/root/disk.img of=/dev/sda bs=1m .... If all went well, the image should now be in the MBR of the first device and the machine can be rebooted. Watch for the machine to boot up properly with the man:ping[8] tool. Once it has came back on-line, it should be possible to access it over man:ssh[1] as user `root` with the configured password. [[installation]] == Installation of the FreeBSD Operating System The mfsBSD has been successfully booted and it should be possible to log in through man:ssh[1]. This section will describe how to create and label slices, set up `gmirror` for RAID-1, and how to use `sysinstall` to install a minimal distribution of the FreeBSD operating system. === Preparation of Hard Drives The first task is to allocate disk space for FreeBSD, i.e.: to create slices and partitions. Obviously, the currently running system is fully loaded in system memory and therefore there will be no problems with manipulating hard drives. To complete this task, it is possible to use either `sysinstall` or man:fdisk[8] in conjunction to man:bsdlabel[8]. At the start, mark all system disks as empty. Repeat the following command for each hard drive: [source,shell] .... # dd if=/dev/zero of=/dev/ad0 count=2 .... Next, create slices and label them with your preferred tool. While it is considered easier to use `sysinstall`, a powerful and also probably less buggy method will be to use standard text-based UNIX(R) tools, such as man:fdisk[8] and man:bsdlabel[8], which will also be covered in this section. The former option is well documented in the extref:{handbook}[Installing FreeBSD, install-steps] chapter of the FreeBSD Handbook. As it was mentioned in the introduction, this article will present how to set up a system with RAID-1 and ZFS capabilities. Our set up will consist of a small man:gmirror[8] mirrored [.filename]#/# (root), [.filename]#/usr# and [.filename]#/var# dataset, and the rest of the disk space will be allocated for a man:zpool[8] mirrored ZFS file system. Please note, that the ZFS file system will be configured after the FreeBSD operating system is successfully installed and booted. The following example will describe how to create slices and labels, initialize man:gmirror[8] on each partition and how to create a UFS2 file system in each mirrored partition: [source,shell] .... # fdisk -BI /dev/ad0 <.> # fdisk -BI /dev/ad1 # bsdlabel -wB /dev/ad0s1 <.> # bsdlabel -wB /dev/ad1s1 # bsdlabel -e /dev/ad0s1 <.> # bsdlabel /dev/ad0s1 > /tmp/bsdlabel.txt && bsdlabel -R /dev/ad1s1 /tmp/bsdlabel.txt <.> # gmirror label root /dev/ad[01]s1a <.> # gmirror label var /dev/ad[01]s1d # gmirror label usr /dev/ad[01]s1e # gmirror label -F swap /dev/ad[01]s1b <.> # newfs /dev/mirror/root <.> # newfs /dev/mirror/var # newfs /dev/mirror/usr .... <.> Create a slice covering the entire disk and initialize the boot code contained in sector 0 of the given disk. Repeat this command for all hard drives in the system. <.> Write a standard label for each disk including the bootstrap code. -<.> Now, manually edit the label of the given disk. Refer to the man:bsdlabel[8] manual page in order to find out how to create partitions. Create partitions `a` for [.filename]#/# (root) file system, `b` for swap, `d` for [.filename]#/var#, `e` for [.filename]#/usr# and finally `f` which will later be used for ZFS. +<.> Now, manually edit the label of the given disk. Refer to the man:bsdlabel[8] manual page to find out how to create partitions. Create partitions `a` for [.filename]#/# (root) file system, `b` for swap, `d` for [.filename]#/var#, `e` for [.filename]#/usr# and finally `f` which will later be used for ZFS. <.> Import the recently created label for the second hard drive, so both hard drives will be labeled in the same way. <.> Initialize man:gmirror[8] on each partition. <.> Note that `-F` is used for the swap partition. This instructs man:gmirror[8] to assume that the device is in the consistent state after the power/system failure. <.> Create a UFS2 file system on each mirrored partition. === System Installation This is the most important part. This section will describe how to actually install the minimal distribution of FreeBSD on the hard drives that we have prepared in the previous section. To accomplish this goal, all file systems need to be mounted so `sysinstall` may write the contents of FreeBSD to the hard drives: [source,shell] .... # mount /dev/mirror/root /mnt # mkdir /mnt/var /mnt/usr # mount /dev/mirror/var /mnt/var # mount /dev/mirror/usr /mnt/usr .... When you are done, start man:sysinstall[8]. Select the [.guimenuitem]#Custom# installation from the main menu. Select [.guimenuitem]#Options# and press kbd:[Enter]. With the help of arrow keys, move the cursor on the `Install Root` item, press kbd:[Space] and change it to [.filename]#/mnt#. Press kbd:[Enter] to submit your changes and exit the [.guimenuitem]#Options# menu by pressing kbd:[q]. [WARNING] ==== Note that this step is very important and if skipped, `sysinstall` will be unable to install FreeBSD. ==== Go to the [.guimenuitem]#Distributions# menu, move the cursor with the arrow keys to `Minimal`, and check it by pressing kbd:[Space]. -This article uses the Minimal distribution in order to save network traffic, because the system itself will be installed over ftp. +This article uses the Minimal distribution to save network traffic, because the system itself will be installed over ftp. Exit this menu by choosing `Exit`. [NOTE] ==== The [.guimenuitem]#Partition# and [.guimenuitem]#Label# menus will be skipped, as these are useless now. ==== In the [.guimenuitem]#Media# menu, select `FTP`. Select the nearest mirror and let `sysinstall` assume that the network is already configured. You will be returned back to the [.guimenuitem]#Custom# menu. Finally, perform the system installation by selecting the last option, [.guimenuitem]#Commit#. Exit `sysinstall` when it finishes the installation. === Post Installation Steps The FreeBSD operating system should be installed now; however, the process is not finished yet. -It is necessary to perform some post installation steps in order to allow FreeBSD to boot in the future and to be able to log in to the system. +It is necessary to perform some post installation steps to allow FreeBSD to boot in the future and to be able to log in to the system. -You must now man:chroot[8] into the freshly installed system in order to finish the installation. +You must now man:chroot[8] into the freshly installed system to finish the installation. Use the following command: [source,shell] .... # chroot /mnt .... To complete our goal, perform these steps: * Copy the `GENERIC` kernel to the [.filename]#/boot/kernel# directory: + [source,shell] .... # cp -Rp /boot/GENERIC/* /boot/kernel .... * Create the [.filename]#/etc/rc.conf#, [.filename]#/etc/resolv.conf# and [.filename]#/etc/fstab# files. Do not forget to properly set the network information and to enable sshd in [.filename]#/etc/rc.conf#. The contents of [.filename]#/etc/fstab# will be similar to the following: + [.programlisting] .... # Device Mountpoint FStype Options Dump Pass# /dev/mirror/swap none swap sw 0 0 /dev/mirror/root / ufs rw 1 1 /dev/mirror/usr /usr ufs rw 2 2 /dev/mirror/var /var ufs rw 2 2 /dev/cd0 /cdrom cd9660 ro,noauto 0 0 .... * Create [.filename]#/boot/loader.conf# with the following contents: + [.programlisting] .... geom_mirror_load="YES" zfs_load="YES" .... * Perform the following command, which will make ZFS available on the next boot: + [source,shell] .... # sysrc zfs_enable="YES" .... * Add additional users to the system using the man:adduser[8] tool. Do not forget to add a user to the `wheel` group so you may obtain root access after the reboot. * Double-check all your settings. The system should now be ready for the next boot. Use the man:reboot[8] command to reboot your system. [[zfs]] == ZFS If your system survived the reboot, it should now be possible to log in. Welcome to the fresh FreeBSD installation, performed remotely without the use of a remote console! The only remaining step is to configure man:zpool[8] and create some man:zfs[8] file systems. Creating and administering ZFS is very straightforward. First, create a mirrored pool: [source,shell] .... # zpool create tank mirror /dev/ad[01]s1f .... Next, create some file systems: [source,shell] .... # zfs create tank/ports # zfs create tank/src # zfs set compression=gzip tank/ports # zfs set compression=on tank/src # zfs set mountpoint=/usr/ports tank/ports # zfs set mountpoint=/usr/src tank/src .... That is all. If you are interested in more details about ZFS on FreeBSD, please refer to the https://wiki.freebsd.org/ZFS[ZFS] section of the FreeBSD Wiki. diff --git a/documentation/content/en/articles/vinum/_index.adoc b/documentation/content/en/articles/vinum/_index.adoc index f6b0baea11..0583ae94a7 100644 --- a/documentation/content/en/articles/vinum/_index.adoc +++ b/documentation/content/en/articles/vinum/_index.adoc @@ -1,716 +1,716 @@ --- title: The vinum Volume Manager authors: - author: Greg Lehey description: The vinum Volume Manager in FreeBSD tags: ["vinum", "Volume Manager", "FreeBSD"] --- //// The Vinum Volume Manager By Greg Lehey (grog at lemis dot com) Added to the Handbook by Hiten Pandya and Tom Rhodes For the FreeBSD Documentation Project //// = The vinum Volume Manager :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/vinum/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] ''' toc::[] [[vinum-synopsis]] == Synopsis No matter the type of disks, there are always potential problems. The disks can be too small, too slow, or too unreliable to meet the system's requirements. While disks are getting bigger, so are data storage requirements. Often a file system is needed that is bigger than a disk's capacity. Various solutions to these problems have been proposed and implemented. One method is through the use of multiple, and sometimes redundant, disks. In addition to supporting various cards and controllers for hardware Redundant Array of Independent Disks RAID systems, the base FreeBSD system includes the [.filename]#vinum# volume manager, a block device driver that implements virtual disk drives and addresses these three problems. [.filename]#vinum# provides more flexibility, performance, and reliability than traditional disk storage and implements `RAID`-0, `RAID`-1, and `RAID`-5 models, both individually and in combination. This chapter provides an overview of potential problems with traditional disk storage, and an introduction to the [.filename]#vinum# volume manager. [NOTE] ==== -Starting with FreeBSD 5, [.filename]#vinum# has been rewritten in order to fit into the extref:{handbook}[GEOM architecture, geom], while retaining the original ideas, terminology, and on-disk metadata. +Starting with FreeBSD 5, [.filename]#vinum# has been rewritten to fit into the extref:{handbook}[GEOM architecture, geom], while retaining the original ideas, terminology, and on-disk metadata. This rewrite is called _gvinum_ (for _GEOM vinum_). While this chapter uses the term [.filename]#vinum#, any command invocations should be performed with `gvinum`. The name of the kernel module has changed from the original [.filename]#vinum.ko# to [.filename]#geom_vinum.ko#, and all device nodes reside under [.filename]#/dev/gvinum# instead of [.filename]#/dev/vinum#. As of FreeBSD 6, the original [.filename]#vinum# implementation is no longer available in the code base. ==== [[vinum-access-bottlenecks]] == Access Bottlenecks Modern systems frequently need to access data in a highly concurrent manner. For example, large FTP or HTTP servers can maintain thousands of concurrent sessions and have multiple 100 Mbit/s connections to the outside world, well beyond the sustained transfer rate of most disks. Current disk drives can transfer data sequentially at up to 70 MB/s, but this value is of little importance in an environment where many independent processes access a drive, and where they may achieve only a fraction of these values. In such cases, it is more interesting to view the problem from the viewpoint of the disk subsystem. The important parameter is the load that a transfer places on the subsystem, or the time for which a transfer occupies the drives involved in the transfer. In any disk transfer, the drive must first position the heads, wait for the first sector to pass under the read head, and then perform the transfer. These actions can be considered to be atomic as it does not make any sense to interrupt them. [[vinum-latency]] Consider a typical transfer of about 10 kB: the current generation of high-performance disks can position the heads in an average of 3.5 ms. The fastest drives spin at 15,000 rpm, so the average rotational latency (half a revolution) is 2 ms. At 70 MB/s, the transfer itself takes about 150 μs, almost nothing compared to the positioning time. In such a case, the effective transfer rate drops to a little over 1 MB/s and is clearly highly dependent on the transfer size. The traditional and obvious solution to this bottleneck is "more spindles": rather than using one large disk, use several smaller disks with the same aggregate storage space. Each disk is capable of positioning and transferring independently, so the effective throughput increases by a factor close to the number of disks used. The actual throughput improvement is smaller than the number of disks involved. Although each drive is capable of transferring in parallel, there is no way to ensure that the requests are evenly distributed across the drives. Inevitably the load on one drive will be higher than on another. The evenness of the load on the disks is strongly dependent on the way the data is shared across the drives. In the following discussion, it is convenient to think of the disk storage as a large number of data sectors which are addressable by number, rather like the pages in a book. The most obvious method is to divide the virtual disk into groups of consecutive sectors the size of the individual physical disks and store them in this manner, rather like taking a large book and tearing it into smaller sections. This method is called _concatenation_ and has the advantage that the disks are not required to have any specific size relationships. It works well when the access to the virtual disk is spread evenly about its address space. When access is concentrated on a smaller area, the improvement is less marked. <> illustrates the sequence in which storage units are allocated in a concatenated organization. [[vinum-concat]] .Concatenated Organization image::vinum-concat.png[] An alternative mapping is to divide the address space into smaller, equal-sized components and store them sequentially on different devices. For example, the first 256 sectors may be stored on the first disk, the next 256 sectors on the next disk and so on. After filling the last disk, the process repeats until the disks are full. This mapping is called _striping_ or RAID-0. `RAID` offers various forms of fault tolerance, though RAID-0 is somewhat misleading as it provides no redundancy. Striping requires somewhat more effort to locate the data, and it can cause additional I/O load where a transfer is spread over multiple disks, but it can also provide a more constant load across the disks. <> illustrates the sequence in which storage units are allocated in a striped organization. [[vinum-striped]] .Striped Organization image::vinum-striped.png[] [[vinum-data-integrity]] == Data Integrity The final problem with disks is that they are unreliable. Although reliability has increased tremendously over the last few years, disk drives are still the most likely core component of a server to fail. When they do, the results can be catastrophic and replacing a failed disk drive and restoring data can result in server downtime. One approach to this problem is _mirroring_, or `RAID-1`, which keeps two copies of the data on different physical hardware. Any write to the volume writes to both disks; a read can be satisfied from either, so if one drive fails, the data is still available on the other drive. Mirroring has two problems: * It requires twice as much disk storage as a non-redundant solution. * Writes must be performed to both drives, so they take up twice the bandwidth of a non-mirrored volume. Reads do not suffer from a performance penalty and can even be faster. An alternative solution is _parity_, implemented in `RAID` levels 2, 3, 4 and 5. Of these, `RAID-5` is the most interesting. As implemented in [.filename]#vinum#, it is a variant on a striped organization which dedicates one block of each stripe to parity one of the other blocks. As implemented by [.filename]#vinum#, a `RAID-5` plex is similar to a striped plex, except that it implements `RAID-5` by including a parity block in each stripe. As required by `RAID-5`, the location of this parity block changes from one stripe to the next. The numbers in the data blocks indicate the relative block numbers. [[vinum-raid5-org]] .`RAID`-5 Organization image::vinum-raid5-org.png[] Compared to mirroring, `RAID-5` has the advantage of requiring significantly less storage space. Read access is similar to that of striped organizations, but write access is significantly slower, approximately 25% of the read performance. If one drive fails, the array can continue to operate in degraded mode where a read from one of the remaining accessible drives continues normally, but a read from the failed drive is recalculated from the corresponding block from all the remaining drives. [[vinum-objects]] == [.filename]#vinum# Objects -In order to address these problems, [.filename]#vinum# implements a four-level hierarchy of objects: +To address these problems, [.filename]#vinum# implements a four-level hierarchy of objects: * The most visible object is the virtual disk, called a _volume_. Volumes have essentially the same properties as a UNIX(R) disk drive, though there are some minor differences. For one, they have no size limitations. * Volumes are composed of _plexes_, each of which represent the total address space of a volume. This level in the hierarchy provides redundancy. Think of plexes as individual disks in a mirrored array, each containing the same data. * Since [.filename]#vinum# exists within the UNIX(R) disk storage framework, it would be possible to use UNIX(R) partitions as the building block for multi-disk plexes. In fact, this turns out to be too inflexible as UNIX(R) disks can have only a limited number of partitions. Instead, [.filename]#vinum# subdivides a single UNIX(R) partition, the _drive_, into contiguous areas called _subdisks_, which are used as building blocks for plexes. * Subdisks reside on [.filename]#vinum#_drives_, currently UNIX(R) partitions. [.filename]#vinum# drives can contain any number of subdisks. With the exception of a small area at the beginning of the drive, which is used for storing configuration and state information, the entire drive is available for data storage. The following sections describe the way these objects provide the functionality required of [.filename]#vinum#. === Volume Size Considerations Plexes can include multiple subdisks spread over all drives in the [.filename]#vinum# configuration. As a result, the size of an individual drive does not limit the size of a plex or a volume. === Redundant Data Storage [.filename]#vinum# implements mirroring by attaching multiple plexes to a volume. Each plex is a representation of the data in a volume. A volume may contain between one and eight plexes. Although a plex represents the complete data of a volume, it is possible for parts of the representation to be physically missing, either by design (by not defining a subdisk for parts of the plex) or by accident (as a result of the failure of a drive). As long as at least one plex can provide the data for the complete address range of the volume, the volume is fully functional. === Which Plex Organization? [.filename]#vinum# implements both concatenation and striping at the plex level: * A _concatenated plex_ uses the address space of each subdisk in turn. Concatenated plexes are the most flexible as they can contain any number of subdisks, and the subdisks may be of different length. The plex may be extended by adding additional subdisks. They require less CPU time than striped plexes, though the difference in CPU overhead is not measurable. On the other hand, they are most susceptible to hot spots, where one disk is very active and others are idle. -* A _striped plex_ stripes the data across each subdisk. The subdisks must all be the same size and there must be at least two subdisks in order to distinguish it from a concatenated plex. The greatest advantage of striped plexes is that they reduce hot spots. By choosing an optimum sized stripe, about 256 kB, the load can be evened out on the component drives. Extending a plex by adding new subdisks is so complicated that [.filename]#vinum# does not implement it. +* A _striped plex_ stripes the data across each subdisk. The subdisks must all be the same size and there must be at least two subdisks to distinguish it from a concatenated plex. The greatest advantage of striped plexes is that they reduce hot spots. By choosing an optimum sized stripe, about 256 kB, the load can be evened out on the component drives. Extending a plex by adding new subdisks is so complicated that [.filename]#vinum# does not implement it. <> summarizes the advantages and disadvantages of each plex organization. [[vinum-comparison]] .[.filename]#vinum# Plex Organizations [cols="1,1,1,1,1", frame="none", options="header"] |=== | Plex type | Minimum subdisks | Can add subdisks | Must be equal size | Application |concatenated |1 |yes |no |Large data storage with maximum placement flexibility and moderate performance |striped |2 |no |yes |High performance in combination with highly concurrent access |=== [[vinum-examples]] == Some Examples [.filename]#vinum# maintains a _configuration database_ which describes the objects known to an individual system. Initially, the user creates the configuration database from one or more configuration files using man:gvinum[8]. [.filename]#vinum# stores a copy of its configuration database on each disk _device_ under its control. This database is updated on each state change, so that a restart accurately restores the state of each [.filename]#vinum# object. === The Configuration File The configuration file describes individual [.filename]#vinum# objects. The definition of a simple volume might be: [.programlisting] .... drive a device /dev/da3h volume myvol plex org concat sd length 512m drive a .... This file describes four [.filename]#vinum# objects: * The _drive_ line describes a disk partition (_drive_) and its location relative to the underlying hardware. It is given the symbolic name _a_. This separation of symbolic names from device names allows disks to be moved from one location to another without confusion. * The _volume_ line describes a volume. The only required attribute is the name, in this case _myvol_. * The _plex_ line defines a plex. The only required parameter is the organization, in this case _concat_. No name is necessary as the system automatically generates a name from the volume name by adding the suffix _.px_, where _x_ is the number of the plex in the volume. Thus this plex will be called _myvol.p0_. * The _sd_ line describes a subdisk. The minimum specifications are the name of a drive on which to store it, and the length of the subdisk. No name is necessary as the system automatically assigns names derived from the plex name by adding the suffix _.sx_, where _x_ is the number of the subdisk in the plex. Thus [.filename]#vinum# gives this subdisk the name _myvol.p0.s0_. After processing this file, man:gvinum[8] produces the following output: [.programlisting] .... # gvinum -> create config1 Configuration summary Drives: 1 (4 configured) Volumes: 1 (4 configured) Plexes: 1 (8 configured) Subdisks: 1 (16 configured) D a State: up Device /dev/da3h Avail: 2061/2573 MB (80%) V myvol State: up Plexes: 1 Size: 512 MB P myvol.p0 C State: up Subdisks: 1 Size: 512 MB S myvol.p0.s0 State: up PO: 0 B Size: 512 MB .... This output shows the brief listing format of man:gvinum[8]. It is represented graphically in <>. [[vinum-simple-vol]] .A Simple [.filename]#vinum# Volume image::vinum-simple-vol.png[] This figure, and the ones which follow, represent a volume, which contains the plexes, which in turn contains the subdisks. In this example, the volume contains one plex, and the plex contains one subdisk. This particular volume has no specific advantage over a conventional disk partition. It contains a single plex, so it is not redundant. The plex contains a single subdisk, so there is no difference in storage allocation from a conventional disk partition. The following sections illustrate various more interesting configuration methods. === Increased Resilience: Mirroring The resilience of a volume can be increased by mirroring. When laying out a mirrored volume, it is important to ensure that the subdisks of each plex are on different drives, so that a drive failure will not take down both plexes. The following configuration mirrors a volume: [.programlisting] .... drive b device /dev/da4h volume mirror plex org concat sd length 512m drive a plex org concat sd length 512m drive b .... In this example, it was not necessary to specify a definition of drive _a_ again, since [.filename]#vinum# keeps track of all objects in its configuration database. After processing this definition, the configuration looks like: [.programlisting] .... Drives: 2 (4 configured) Volumes: 2 (4 configured) Plexes: 3 (8 configured) Subdisks: 3 (16 configured) D a State: up Device /dev/da3h Avail: 1549/2573 MB (60%) D b State: up Device /dev/da4h Avail: 2061/2573 MB (80%) V myvol State: up Plexes: 1 Size: 512 MB V mirror State: up Plexes: 2 Size: 512 MB P myvol.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p1 C State: initializing Subdisks: 1 Size: 512 MB S myvol.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p1.s0 State: empty PO: 0 B Size: 512 MB .... <> shows the structure graphically. [[vinum-mirrored-vol]] .A Mirrored [.filename]#vinum# Volume image::vinum-mirrored-vol.png[] In this example, each plex contains the full 512 MB of address space. As in the previous example, each plex contains only a single subdisk. === Optimizing Performance The mirrored volume in the previous example is more resistant to failure than an unmirrored volume, but its performance is less as each write to the volume requires a write to both drives, using up a greater proportion of the total disk bandwidth. Performance considerations demand a different approach: instead of mirroring, the data is striped across as many disk drives as possible. The following configuration shows a volume with a plex striped across four disk drives: [.programlisting] .... drive c device /dev/da5h drive d device /dev/da6h volume stripe plex org striped 512k sd length 128m drive a sd length 128m drive b sd length 128m drive c sd length 128m drive d .... As before, it is not necessary to define the drives which are already known to [.filename]#vinum#. After processing this definition, the configuration looks like: [.programlisting] .... Drives: 4 (4 configured) Volumes: 3 (4 configured) Plexes: 4 (8 configured) Subdisks: 7 (16 configured) D a State: up Device /dev/da3h Avail: 1421/2573 MB (55%) D b State: up Device /dev/da4h Avail: 1933/2573 MB (75%) D c State: up Device /dev/da5h Avail: 2445/2573 MB (95%) D d State: up Device /dev/da6h Avail: 2445/2573 MB (95%) V myvol State: up Plexes: 1 Size: 512 MB V mirror State: up Plexes: 2 Size: 512 MB V striped State: up Plexes: 1 Size: 512 MB P myvol.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p0 C State: up Subdisks: 1 Size: 512 MB P mirror.p1 C State: initializing Subdisks: 1 Size: 512 MB P striped.p1 State: up Subdisks: 1 Size: 512 MB S myvol.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p0.s0 State: up PO: 0 B Size: 512 MB S mirror.p1.s0 State: empty PO: 0 B Size: 512 MB S striped.p0.s0 State: up PO: 0 B Size: 128 MB S striped.p0.s1 State: up PO: 512 kB Size: 128 MB S striped.p0.s2 State: up PO: 1024 kB Size: 128 MB S striped.p0.s3 State: up PO: 1536 kB Size: 128 MB .... [[vinum-striped-vol]] .A Striped [.filename]#vinum# Volume image::vinum-striped-vol.png[] This volume is represented in <>. The darkness of the stripes indicates the position within the plex address space, where the lightest stripes come first and the darkest last. === Resilience and Performance [[vinum-resilience]]With sufficient hardware, it is possible to build volumes which show both increased resilience and increased performance compared to standard UNIX(R) partitions. A typical configuration file might be: [.programlisting] .... volume raid10 plex org striped 512k sd length 102480k drive a sd length 102480k drive b sd length 102480k drive c sd length 102480k drive d sd length 102480k drive e plex org striped 512k sd length 102480k drive c sd length 102480k drive d sd length 102480k drive e sd length 102480k drive a sd length 102480k drive b .... The subdisks of the second plex are offset by two drives from those of the first plex. This helps to ensure that writes do not go to the same subdisks even if a transfer goes over two drives. <> represents the structure of this volume. [[vinum-raid10-vol]] .A Mirrored, Striped [.filename]#vinum# Volume image::vinum-raid10-vol.png[] [[vinum-object-naming]] == Object Naming [.filename]#vinum# assigns default names to plexes and subdisks, although they may be overridden. Overriding the default names is not recommended as it does not bring a significant advantage and it can cause confusion. Names may contain any non-blank character, but it is recommended to restrict them to letters, digits and the underscore characters. The names of volumes, plexes, and subdisks may be up to 64 characters long, and the names of drives may be up to 32 characters long. [.filename]#vinum# objects are assigned device nodes in the hierarchy [.filename]#/dev/gvinum#. The configuration shown above would cause [.filename]#vinum# to create the following device nodes: * Device entries for each volume. These are the main devices used by [.filename]#vinum#. The configuration above would include the devices [.filename]#/dev/gvinum/myvol#, [.filename]#/dev/gvinum/mirror#, [.filename]#/dev/gvinum/striped#, [.filename]#/dev/gvinum/raid5# and [.filename]#/dev/gvinum/raid10#. * All volumes get direct entries under [.filename]#/dev/gvinum/#. * The directories [.filename]#/dev/gvinum/plex#, and [.filename]#/dev/gvinum/sd#, which contain device nodes for each plex and for each subdisk, respectively. For example, consider the following configuration file: [.programlisting] .... drive drive1 device /dev/sd1h drive drive2 device /dev/sd2h drive drive3 device /dev/sd3h drive drive4 device /dev/sd4h volume s64 setupstate plex org striped 64k sd length 100m drive drive1 sd length 100m drive drive2 sd length 100m drive drive3 sd length 100m drive drive4 .... After processing this file, man:gvinum[8] creates the following structure in [.filename]#/dev/gvinum#: [.programlisting] .... drwxr-xr-x 2 root wheel 512 Apr 13 16:46 plex crwxr-xr-- 1 root wheel 91, 2 Apr 13 16:46 s64 drwxr-xr-x 2 root wheel 512 Apr 13 16:46 sd /dev/vinum/plex: total 0 crwxr-xr-- 1 root wheel 25, 0x10000002 Apr 13 16:46 s64.p0 /dev/vinum/sd: total 0 crwxr-xr-- 1 root wheel 91, 0x20000002 Apr 13 16:46 s64.p0.s0 crwxr-xr-- 1 root wheel 91, 0x20100002 Apr 13 16:46 s64.p0.s1 crwxr-xr-- 1 root wheel 91, 0x20200002 Apr 13 16:46 s64.p0.s2 crwxr-xr-- 1 root wheel 91, 0x20300002 Apr 13 16:46 s64.p0.s3 .... Although it is recommended that plexes and subdisks should not be allocated specific names, [.filename]#vinum# drives must be named. This makes it possible to move a drive to a different location and still recognize it automatically. Drive names may be up to 32 characters long. === Creating File Systems Volumes appear to the system to be identical to disks, with one exception. Unlike UNIX(R) drives, [.filename]#vinum# does not partition volumes, which thus do not contain a partition table. This has required modification to some disk utilities, notably man:newfs[8], so that it does not try to interpret the last letter of a [.filename]#vinum# volume name as a partition identifier. For example, a disk drive may have a name like [.filename]#/dev/ad0a# or [.filename]#/dev/da2h#. These names represent the first partition ([.filename]#a#) on the first (0) IDE disk ([.filename]#ad#) and the eighth partition ([.filename]#h#) on the third (2) SCSI disk ([.filename]#da#) respectively. By contrast, a [.filename]#vinum# volume might be called [.filename]#/dev/gvinum/concat#, which has no relationship with a partition name. -In order to create a file system on this volume, use man:newfs[8]: +To create a file system on this volume, use man:newfs[8]: [source,shell] .... # newfs /dev/gvinum/concat .... [[vinum-config]] == Configuring [.filename]#vinum# The [.filename]#GENERIC# kernel does not contain [.filename]#vinum#. It is possible to build a custom kernel which includes [.filename]#vinum#, but this is not recommended. The standard way to start [.filename]#vinum# is as a kernel module. man:kldload[8] is not needed because when man:gvinum[8] starts, it checks whether the module has been loaded, and if it is not, it loads it automatically. === Startup [.filename]#vinum# stores configuration information on the disk slices in essentially the same form as in the configuration files. When reading from the configuration database, [.filename]#vinum# recognizes a number of keywords which are not allowed in the configuration files. For example, a disk configuration might contain the following text: [.programlisting] .... volume myvol state up volume bigraid state down plex name myvol.p0 state up org concat vol myvol plex name myvol.p1 state up org concat vol myvol plex name myvol.p2 state init org striped 512b vol myvol plex name bigraid.p0 state initializing org raid5 512b vol bigraid sd name myvol.p0.s0 drive a plex myvol.p0 state up len 1048576b driveoffset 265b plexoffset 0b sd name myvol.p0.s1 drive b plex myvol.p0 state up len 1048576b driveoffset 265b plexoffset 1048576b sd name myvol.p1.s0 drive c plex myvol.p1 state up len 1048576b driveoffset 265b plexoffset 0b sd name myvol.p1.s1 drive d plex myvol.p1 state up len 1048576b driveoffset 265b plexoffset 1048576b sd name myvol.p2.s0 drive a plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 0b sd name myvol.p2.s1 drive b plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 524288b sd name myvol.p2.s2 drive c plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 1048576b sd name myvol.p2.s3 drive d plex myvol.p2 state init len 524288b driveoffset 1048841b plexoffset 1572864b sd name bigraid.p0.s0 drive a plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 0b sd name bigraid.p0.s1 drive b plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 4194304b sd name bigraid.p0.s2 drive c plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 8388608b sd name bigraid.p0.s3 drive d plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 12582912b sd name bigraid.p0.s4 drive e plex bigraid.p0 state initializing len 4194304b driveoff set 1573129b plexoffset 16777216b .... The obvious differences here are the presence of explicit location information and naming, both of which are allowed but discouraged, and the information on the states. [.filename]#vinum# does not store information about drives in the configuration information. It finds the drives by scanning the configured disk drives for partitions with a [.filename]#vinum# label. This enables [.filename]#vinum# to identify drives correctly even if they have been assigned different UNIX(R) drive IDs. [[vinum-rc-startup]] ==== Automatic Startup _Gvinum_ always features an automatic startup once the kernel module is loaded, via man:loader.conf[5]. To load the _Gvinum_ module at boot time, add `geom_vinum_load="YES"` to [.filename]#/boot/loader.conf#. When [.filename]#vinum# is started with `gvinum start`, [.filename]#vinum# reads the configuration database from one of the [.filename]#vinum# drives. Under normal circumstances, each drive contains an identical copy of the configuration database, so it does not matter which drive is read. After a crash, however, [.filename]#vinum# must determine which drive was updated most recently and read the configuration from this drive. It then updates the configuration, if necessary, from progressively older drives. [[vinum-root]] == Using [.filename]#vinum# for the Root File System For a machine that has fully-mirrored file systems using [.filename]#vinum#, it is desirable to also mirror the root file system. Setting up such a configuration is less trivial than mirroring an arbitrary file system because: * The root file system must be available very early during the boot process, so the [.filename]#vinum# infrastructure must already be available at this time. * The volume containing the root file system also contains the system bootstrap and the kernel. These must be read using the host system's native utilities, such as the BIOS, which often cannot be taught about the details of [.filename]#vinum#. In the following sections, the term "root volume" is generally used to describe the [.filename]#vinum# volume that contains the root file system. === Starting up [.filename]#vinum# Early Enough for the Root File System [.filename]#vinum# must be available early in the system boot as man:loader[8] must be able to load the vinum kernel module before starting the kernel. This can be accomplished by putting this line in [.filename]#/boot/loader.conf#: [.programlisting] .... geom_vinum_load="YES" .... === Making a [.filename]#vinum#-based Root Volume Accessible to the Bootstrap The current FreeBSD bootstrap is only 7.5 KB of code and does not understand the internal [.filename]#vinum# structures. This means that it cannot parse the [.filename]#vinum# configuration data or figure out the elements of a boot volume. Thus, some workarounds are necessary to provide the bootstrap code with the illusion of a standard `a` partition that contains the root file system. For this to be possible, the following requirements must be met for the root volume: * The root volume must not be a stripe or `RAID`-5. * The root volume must not contain more than one concatenated subdisk per plex. Note that it is desirable and possible to use multiple plexes, each containing one replica of the root file system. The bootstrap process will only use one replica for finding the bootstrap and all boot files, until the kernel mounts the root file system. Each single subdisk within these plexes needs its own `a` partition illusion, for the respective device to be bootable. It is not strictly needed that each of these faked `a` partitions is located at the same offset within its device, compared with other devices containing plexes of the root volume. However, it is probably a good idea to create the [.filename]#vinum# volumes that way so the resulting mirrored devices are symmetric, to avoid confusion. -In order to set up these `a` partitions for each device containing part of the root volume, the following is required: +To set up these `a` partitions for each device containing part of the root volume, the following is required: [.procedure] ==== . The location, offset from the beginning of the device, and size of this device's subdisk that is part of the root volume needs to be examined, using the command: + [source,shell] .... # gvinum l -rv root .... + [.filename]#vinum# offsets and sizes are measured in bytes. -They must be divided by 512 in order to obtain the block numbers that are to be used by `bsdlabel`. +They must be divided by 512 to obtain the block numbers that are to be used by `bsdlabel`. . Run this command for each device that participates in the root volume: + [source,shell] .... # bsdlabel -e devname .... + _devname_ must be either the name of the disk, like [.filename]#da0# for disks without a slice table, or the name of the slice, like [.filename]#ad0s1#. + If there is already an `a` partition on the device from a pre-[.filename]#vinum# root file system, it should be renamed to something else so that it remains accessible (just in case), but will no longer be used by default to bootstrap the system. A currently mounted root file system cannot be renamed, so this must be executed either when being booted from a "Fixit" media, or in a two-step process where, in a mirror, the disk that is not been currently booted is manipulated first. + The offset of the [.filename]#vinum# partition on this device (if any) must be added to the offset of the respective root volume subdisk on this device. The resulting value will become the `offset` value for the new `a` partition. The `size` value for this partition can be taken verbatim from the calculation above. The `fstype` should be `4.2BSD`. The `fsize`, `bsize`, and `cpg` values should be chosen to match the actual file system, though they are fairly unimportant within this context. + That way, a new `a` partition will be established that overlaps the [.filename]#vinum# partition on this device. `bsdlabel` will only allow for this overlap if the [.filename]#vinum# partition has properly been marked using the `vinum` fstype. . A faked `a` partition now exists on each device that has one replica of the root volume. It is highly recommendable to verify the result using a command like: + [source,shell] .... # fsck -n /dev/devnamea .... ==== It should be remembered that all files containing control information must be relative to the root file system in the [.filename]#vinum# volume which, when setting up a new [.filename]#vinum# root volume, might not match the root file system that is currently active. So in particular, [.filename]#/etc/fstab# and [.filename]#/boot/loader.conf# need to be taken care of. At next reboot, the bootstrap should figure out the appropriate control information from the new [.filename]#vinum#-based root file system, and act accordingly. At the end of the kernel initialization process, after all devices have been announced, the prominent notice that shows the success of this setup is a message like: [source,shell] .... Mounting root from ufs:/dev/gvinum/root .... === Example of a [.filename]#vinum#-based Root Setup After the [.filename]#vinum# root volume has been set up, the output of `gvinum l -rv root` could look like: [source,shell] .... ... Subdisk root.p0.s0: Size: 125829120 bytes (120 MB) State: up Plex root.p0 at offset 0 (0 B) Drive disk0 (/dev/da0h) at offset 135680 (132 kB) Subdisk root.p1.s0: Size: 125829120 bytes (120 MB) State: up Plex root.p1 at offset 0 (0 B) Drive disk1 (/dev/da1h) at offset 135680 (132 kB) .... The values to note are `135680` for the offset, relative to partition [.filename]#/dev/da0h#. This translates to 265 512-byte disk blocks in `bsdlabel`'s terms. Likewise, the size of this root volume is 245760 512-byte blocks. [.filename]#/dev/da1h#, containing the second replica of this root volume, has a symmetric setup. The bsdlabel for these devices might look like: [source,shell] .... ... 8 partitions: # size offset fstype [fsize bsize bps/cpg] a: 245760 281 4.2BSD 2048 16384 0 # (Cyl. 0*- 15*) c: 71771688 0 unused 0 0 # (Cyl. 0 - 4467*) h: 71771672 16 vinum # (Cyl. 0*- 4467*) .... It can be observed that the `size` parameter for the faked `a` partition matches the value outlined above, while the `offset` parameter is the sum of the offset within the [.filename]#vinum# partition `h`, and the offset of this partition within the device or slice. This is a typical setup that is necessary to avoid the problem described in <>. The entire `a` partition is completely within the `h` partition containing all the [.filename]#vinum# data for this device. In the above example, the entire device is dedicated to [.filename]#vinum# and there is no leftover pre-[.filename]#vinum# root partition. === Troubleshooting The following list contains a few known pitfalls and solutions. ==== System Bootstrap Loads, but System Does Not Boot If for any reason the system does not continue to boot, the bootstrap can be interrupted by pressing kbd:[space] at the 10-seconds warning. The loader variable `vinum.autostart` can be examined by typing `show` and manipulated using `set` or `unset`. If the [.filename]#vinum# kernel module was not yet in the list of modules to load automatically, type `load geom_vinum`. When ready, the boot process can be continued by typing `boot -as` which `-as` requests the kernel to ask for the root file system to mount (`-a`) and make the boot process stop in single-user mode (`-s`), where the root file system is mounted read-only. That way, even if only one plex of a multi-plex volume has been mounted, no data inconsistency between plexes is being risked. At the prompt asking for a root file system to mount, any device that contains a valid root file system can be entered. If [.filename]#/etc/fstab# is set up correctly, the default should be something like `ufs:/dev/gvinum/root`. A typical alternate choice would be something like `ufs:da0d` which could be a hypothetical partition containing the pre-[.filename]#vinum# root file system. Care should be taken if one of the alias `a` partitions is entered here, that it actually references the subdisks of the [.filename]#vinum# root device, because in a mirrored setup, this would only mount one piece of a mirrored root device. If this file system is to be mounted read-write later on, it is necessary to remove the other plex(es) of the [.filename]#vinum# root volume since these plexes would otherwise carry inconsistent data. ==== Only Primary Bootstrap Loads If [.filename]#/boot/loader# fails to load, but the primary bootstrap still loads (visible by a single dash in the left column of the screen right after the boot process starts), an attempt can be made to interrupt the primary bootstrap by pressing kbd:[space]. This will make the bootstrap stop in extref:{handbook}[stage two, boot-boot1]. An attempt can be made here to boot off an alternate partition, like the partition containing the previous root file system that has been moved away from `a`. [[vinum-root-panic]] ==== Nothing Boots, the Bootstrap Panics This situation will happen if the bootstrap had been destroyed by the [.filename]#vinum# installation. Unfortunately, [.filename]#vinum# accidentally leaves only 4 KB at the beginning of its partition free before starting to write its [.filename]#vinum# header information. However, the stage one and two bootstraps plus the bsdlabel require 8 KB. So if a [.filename]#vinum# partition was started at offset 0 within a slice or disk that was meant to be bootable, the [.filename]#vinum# setup will trash the bootstrap. Similarly, if the above situation has been recovered, by booting from a "Fixit" media, and the bootstrap has been re-installed using `bsdlabel -B` as described in extref:{handbook}[stage two, boot-boot1], the bootstrap will trash the [.filename]#vinum# header, and [.filename]#vinum# will no longer find its disk(s). Though no actual [.filename]#vinum# configuration data or data in [.filename]#vinum# volumes will be trashed, and it would be possible to recover all the data by entering exactly the same [.filename]#vinum# configuration data again, the situation is hard to fix. -It is necessary to move the entire [.filename]#vinum# partition by at least 4 KB, in order to have the [.filename]#vinum# header and the system bootstrap no longer collide. +It is necessary to move the entire [.filename]#vinum# partition by at least 4 KB, to have the [.filename]#vinum# header and the system bootstrap no longer collide. diff --git a/documentation/content/en/articles/vm-design/_index.adoc b/documentation/content/en/articles/vm-design/_index.adoc index 7499c1e362..1a6431a489 100644 --- a/documentation/content/en/articles/vm-design/_index.adoc +++ b/documentation/content/en/articles/vm-design/_index.adoc @@ -1,421 +1,421 @@ --- title: Design elements of the FreeBSD VM system authors: - author: Matthew Dillon email: dillon@apollo.backplane.com description: An easy to follow description of the design of the FreeBSD virtual memory system trademarks: ["freebsd", "linux", "microsoft", "opengroup", "daemon-news", "general"] tags: ["Design", "virtual machine", "FreeBSD"] --- = Design elements of the FreeBSD VM system :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :images-path: articles/vm-design/ ifdef::env-beastie[] ifdef::backend-html5[] include::shared/authors.adoc[] include::shared/mirrors.adoc[] include::shared/releases.adoc[] include::shared/attributes/attributes-{{% lang %}}.adoc[] include::shared/{{% lang %}}/teams.adoc[] include::shared/{{% lang %}}/mailing-lists.adoc[] include::shared/{{% lang %}}/urls.adoc[] :imagesdir: ../../../images/{images-path} endif::[] ifdef::backend-pdf,backend-epub3[] include::../../../../shared/asciidoctor.adoc[] endif::[] endif::[] ifndef::env-beastie[] include::../../../../../shared/asciidoctor.adoc[] endif::[] [.abstract-title] Abstract The title is really just a fancy way of saying that I am going to attempt to describe the whole VM enchilada, hopefully in a way that everyone can follow. For the last year I have concentrated on a number of major kernel subsystems within FreeBSD, with the VM and Swap subsystems being the most interesting and NFS being "a necessary chore". I rewrote only small portions of the code. In the VM arena the only major rewrite I have done is to the swap subsystem. Most of my work was cleanup and maintenance, with only moderate code rewriting and no major algorithmic adjustments within the VM subsystem. The bulk of the VM subsystem's theoretical base remains unchanged and a lot of the credit for the modernization effort in the last few years belongs to John Dyson and David Greenman. Not being a historian like Kirk I will not attempt to tag all the various features with peoples names, since I will invariably get it wrong. ''' toc::[] [[introduction]] == Introduction Before moving along to the actual design let's spend a little time on the necessity of maintaining and modernizing any long-living codebase. In the programming world, algorithms tend to be more important than code and it is precisely due to BSD's academic roots that a great deal of attention was paid to algorithm design from the beginning. More attention paid to the design generally leads to a clean and flexible codebase that can be fairly easily modified, extended, or replaced over time. While BSD is considered an "old" operating system by some people, those of us who work on it tend to view it more as a "mature" codebase which has various components modified, extended, or replaced with modern code. It has evolved, and FreeBSD is at the bleeding edge no matter how old some of the code might be. This is an important distinction to make and one that is unfortunately lost to many people. The biggest error a programmer can make is to not learn from history, and this is precisely the error that many other modern operating systems have made. Windows NT(R) is the best example of this, and the consequences have been dire. Linux also makes this mistake to some degree-enough that we BSD folk can make small jokes about it every once in a while, anyway. Linux's problem is simply one of a lack of experience and history to compare ideas against, a problem that is easily and rapidly being addressed by the Linux community in the same way it has been addressed in the BSD community-by continuous code development. The Windows NT(R) folk, on the other hand, repeatedly make the same mistakes solved by UNIX(R) decades ago and then spend years fixing them. Over and over again. They have a severe case of "not designed here" and "we are always right because our marketing department says so". I have little tolerance for anyone who cannot learn from history. Much of the apparent complexity of the FreeBSD design, especially in the VM/Swap subsystem, is a direct result of having to solve serious performance issues that occur under various conditions. These issues are not due to bad algorithmic design but instead rise from environmental factors. In any direct comparison between platforms, these issues become most apparent when system resources begin to get stressed. As I describe FreeBSD's VM/Swap subsystem the reader should always keep two points in mind: -. The most important aspect of performance design is what is known as "Optimizing the Critical Path". It is often the case that performance optimizations add a little bloat to the code in order to make the critical path perform better. +. The most important aspect of performance design is what is known as "Optimizing the Critical Path". It is often the case that performance optimizations add a little bloat to the code to make the critical path perform better. . A solid, generalized design outperforms a heavily-optimized design over the long run. While a generalized design may end up being slower than an heavily-optimized design when they are first implemented, the generalized design tends to be easier to adapt to changing conditions and the heavily-optimized design winds up having to be thrown away. Any codebase that will survive and be maintainable for years must therefore be designed properly from the beginning even if it costs some performance. Twenty years ago people were still arguing that programming in assembly was better than programming in a high-level language because it produced code that was ten times as fast. Today, the fallibility of that argument is obvious - as are the parallels to algorithmic design and code generalization. [[vm-objects]] == VM Objects The best way to begin describing the FreeBSD VM system is to look at it from the perspective of a user-level process. Each user process sees a single, private, contiguous VM address space containing several types of memory objects. These objects have various characteristics. Program code and program data are effectively a single memory-mapped file (the binary file being run), but program code is read-only while program data is copy-on-write. Program BSS is just memory allocated and filled with zeros on demand, called demand zero page fill. Arbitrary files can be memory-mapped into the address space as well, which is how the shared library mechanism works. Such mappings can require modifications to remain private to the process making them. The fork system call adds an entirely new dimension to the VM management problem on top of the complexity already given. A program binary data page (which is a basic copy-on-write page) illustrates the complexity. A program binary contains a preinitialized data section which is initially mapped directly from the program file. When a program is loaded into a process's VM space, this area is initially memory-mapped and backed by the program binary itself, allowing the VM system to free/reuse the page and later load it back in from the binary. The moment a process modifies this data, however, the VM system must make a private copy of the page for that process. Since the private copy has been modified, the VM system may no longer free it, because there is no longer any way to restore it later on. You will notice immediately that what was originally a simple file mapping has become much more complex. Data may be modified on a page-by-page basis whereas the file mapping encompasses many pages at once. The complexity further increases when a process forks. When a process forks, the result is two processes-each with their own private address spaces, including any modifications made by the original process prior to the call to `fork()`. It would be silly for the VM system to make a complete copy of the data at the time of the `fork()` because it is quite possible that at least one of the two processes will only need to read from that page from then on, allowing the original page to continue to be used. What was a private page is made copy-on-write again, since each process (parent and child) expects their own personal post-fork modifications to remain private to themselves and not affect the other. FreeBSD manages all of this with a layered VM Object model. The original binary program file winds up being the lowest VM Object layer. A copy-on-write layer is pushed on top of that to hold those pages which had to be copied from the original file. If the program modifies a data page belonging to the original file the VM system takes a fault and makes a copy of the page in the higher layer. When a process forks, additional VM Object layers are pushed on. This might make a little more sense with a fairly basic example. A `fork()` is a common operation for any *BSD system, so this example will consider a program that starts up, and forks. When the process starts, the VM system creates an object layer, let's call this A: image::fig1.png[A picture] A represents the file-pages may be paged in and out of the file's physical media as necessary. Paging in from the disk is reasonable for a program, but we really do not want to page back out and overwrite the executable. The VM system therefore creates a second layer, B, that will be physically backed by swap space: image::fig2.png[] On the first write to a page after this, a new page is created in B, and its contents are initialized from A. All pages in B can be paged in or out to a swap device. When the program forks, the VM system creates two new object layers-C1 for the parent, and C2 for the child-that rest on top of B: image::fig3.png[] In this case, let's say a page in B is modified by the original parent process. The process will take a copy-on-write fault and duplicate the page in C1, leaving the original page in B untouched. Now, let's say the same page in B is modified by the child process. The process will take a copy-on-write fault and duplicate the page in C2. The original page in B is now completely hidden since both C1 and C2 have a copy and B could theoretically be destroyed if it does not represent a "real" file; however, this sort of optimization is not trivial to make because it is so fine-grained. FreeBSD does not make this optimization. Now, suppose (as is often the case) that the child process does an `exec()`. Its current address space is usually replaced by a new address space representing a new file. In this case, the C2 layer is destroyed: image::fig4.png[] In this case, the number of children of B drops to one, and all accesses to B now go through C1. This means that B and C1 can be collapsed together. Any pages in B that also exist in C1 are deleted from B during the collapse. Thus, even though the optimization in the previous step could not be made, we can recover the dead pages when either of the processes exit or `exec()`. This model creates a number of potential problems. The first is that you can wind up with a relatively deep stack of layered VM Objects which can cost scanning time and memory when you take a fault. Deep layering can occur when processes fork and then fork again (either parent or child). The second problem is that you can wind up with dead, inaccessible pages deep in the stack of VM Objects. In our last example if both the parent and child processes modify the same page, they both get their own private copies of the page and the original page in B is no longer accessible by anyone. That page in B can be freed. FreeBSD solves the deep layering problem with a special optimization called the "All Shadowed Case". This case occurs if either C1 or C2 take sufficient COW faults to completely shadow all pages in B. Lets say that C1 achieves this. C1 can now bypass B entirely, so rather then have C1->B->A and C2->B->A we now have C1->A and C2->B->A. But look what also happened-now B has only one reference (C2), so we can collapse B and C2 together. The end result is that B is deleted entirely and we have C1->A and C2->A. It is often the case that B will contain a large number of pages and neither C1 nor C2 will be able to completely overshadow it. If we fork again and create a set of D layers, however, it is much more likely that one of the D layers will eventually be able to completely overshadow the much smaller dataset represented by C1 or C2. The same optimization will work at any point in the graph and the grand result of this is that even on a heavily forked machine VM Object stacks tend to not get much deeper then 4. This is true of both the parent and the children and true whether the parent is doing the forking or whether the children cascade forks. The dead page problem still exists in the case where C1 or C2 do not completely overshadow B. Due to our other optimizations this case does not represent much of a problem and we simply allow the pages to be dead. If the system runs low on memory it will swap them out, eating a little swap, but that is it. The advantage to the VM Object model is that `fork()` is extremely fast, since no real data copying need take place. The disadvantage is that you can build a relatively complex VM Object layering that slows page fault handling down a little, and you spend memory managing the VM Object structures. The optimizations FreeBSD makes proves to reduce the problems enough that they can be ignored, leaving no real disadvantage. [[swap-layers]] == SWAP Layers Private data pages are initially either copy-on-write or zero-fill pages. When a change, and therefore a copy, is made, the original backing object (usually a file) can no longer be used to save a copy of the page when the VM system needs to reuse it for other purposes. This is where SWAP comes in. SWAP is allocated to create backing store for memory that does not otherwise have it. FreeBSD allocates the swap management structure for a VM Object only when it is actually needed. However, the swap management structure has had problems historically: * Under FreeBSD 3.X the swap management structure preallocates an array that encompasses the entire object requiring swap backing store-even if only a few pages of that object are swap-backed. This creates a kernel memory fragmentation problem when large objects are mapped, or processes with large runsizes (RSS) fork. -* Also, in order to keep track of swap space, a "list of holes" is kept in kernel memory, and this tends to get severely fragmented as well. Since the "list of holes" is a linear list, the swap allocation and freeing performance is a non-optimal O(n)-per-page. +* Also, to keep track of swap space, a "list of holes" is kept in kernel memory, and this tends to get severely fragmented as well. Since the "list of holes" is a linear list, the swap allocation and freeing performance is a non-optimal O(n)-per-page. * It requires kernel memory allocations to take place during the swap freeing process, and that creates low memory deadlock problems. * The problem is further exacerbated by holes created due to the interleaving algorithm. * Also, the swap block map can become fragmented fairly easily resulting in non-contiguous allocations. * Kernel memory must also be allocated on the fly for additional swap management structures when a swapout occurs. It is evident from that list that there was plenty of room for improvement. For FreeBSD 4.X, I completely rewrote the swap subsystem: * Swap management structures are allocated through a hash table rather than a linear array giving them a fixed allocation size and much finer granularity. * Rather then using a linearly linked list to keep track of swap space reservations, it now uses a bitmap of swap blocks arranged in a radix tree structure with free-space hinting in the radix node structures. This effectively makes swap allocation and freeing an O(1) operation. -* The entire radix tree bitmap is also preallocated in order to avoid having to allocate kernel memory during critical low memory swapping operations. After all, the system tends to swap when it is low on memory so we should avoid allocating kernel memory at such times in order to avoid potential deadlocks. +* The entire radix tree bitmap is also preallocated to avoid having to allocate kernel memory during critical low memory swapping operations. After all, the system tends to swap when it is low on memory so we should avoid allocating kernel memory at such times to avoid potential deadlocks. * To reduce fragmentation the radix tree is capable of allocating large contiguous chunks at once, skipping over smaller fragmented chunks. -I did not take the final step of having an "allocating hint pointer" that would trundle through a portion of swap as allocations were made in order to further guarantee contiguous allocations or at least locality of reference, but I ensured that such an addition could be made. +I did not take the final step of having an "allocating hint pointer" that would trundle through a portion of swap as allocations were made to further guarantee contiguous allocations or at least locality of reference, but I ensured that such an addition could be made. [[freeing-pages]] == When to free a page Since the VM system uses all available memory for disk caching, there are usually very few truly-free pages. The VM system depends on being able to properly choose pages which are not in use to reuse for new allocations. Selecting the optimal pages to free is possibly the single-most important function any VM system can perform because if it makes a poor selection, the VM system may be forced to unnecessarily retrieve pages from disk, seriously degrading system performance. -How much overhead are we willing to suffer in the critical path to avoid freeing the wrong page? Each wrong choice we make will cost us hundreds of thousands of CPU cycles and a noticeable stall of the affected processes, so we are willing to endure a significant amount of overhead in order to be sure that the right page is chosen. +How much overhead are we willing to suffer in the critical path to avoid freeing the wrong page? Each wrong choice we make will cost us hundreds of thousands of CPU cycles and a noticeable stall of the affected processes, so we are willing to endure a significant amount of overhead to be sure that the right page is chosen. This is why FreeBSD tends to outperform other systems when memory resources become stressed. The free page determination algorithm is built upon a history of the use of memory pages. To acquire this history, the system takes advantage of a page-used bit feature that most hardware page tables have. In any case, the page-used bit is cleared and at some later point the VM system comes across the page again and sees that the page-used bit has been set. This indicates that the page is still being actively used. If the bit is still clear it is an indication that the page is not being actively used. By testing this bit periodically, a use history (in the form of a counter) for the physical page is developed. When the VM system later needs to free up some pages, checking this history becomes the cornerstone of determining the best candidate page to reuse. For those platforms that do not have this feature, the system actually emulates a page-used bit. It unmaps or protects a page, forcing a page fault if the page is accessed again. When the page fault is taken, the system simply marks the page as having been used and unprotects the page so that it may be used. While taking such page faults just to determine if a page is being used appears to be an expensive proposition, it is much less expensive than reusing the page for some other purpose only to find that a process needs it back and then have to go to disk. FreeBSD makes use of several page queues to further refine the selection of pages to reuse as well as to determine when dirty pages must be flushed to their backing store. Since page tables are dynamic entities under FreeBSD, it costs virtually nothing to unmap a page from the address space of any processes using it. When a page candidate has been chosen based on the page-use counter, this is precisely what is done. The system must make a distinction between clean pages which can theoretically be freed up at any time, and dirty pages which must first be written to their backing store before being reusable. When a page candidate has been found it is moved to the inactive queue if it is dirty, or the cache queue if it is clean. A separate algorithm based on the dirty-to-clean page ratio determines when dirty pages in the inactive queue must be flushed to disk. Once this is accomplished, the flushed pages are moved from the inactive queue to the cache queue. At this point, pages in the cache queue can still be reactivated by a VM fault at relatively low cost. However, pages in the cache queue are considered to be "immediately freeable" and will be reused in an LRU (least-recently used) fashion when the system needs to allocate new memory. It is important to note that the FreeBSD VM system attempts to separate clean and dirty pages for the express reason of avoiding unnecessary flushes of dirty pages (which eats I/O bandwidth), nor does it move pages between the various page queues gratuitously when the memory subsystem is not being stressed. This is why you will see some systems with very low cache queue counts and high active queue counts when doing a `systat -vm` command. As the VM system becomes more stressed, it makes a greater effort to maintain the various page queues at the levels determined to be the most effective. An urban myth has circulated for years that Linux did a better job avoiding swapouts than FreeBSD, but this in fact is not true. -What was actually occurring was that FreeBSD was proactively paging out unused pages in order to make room for more disk cache while Linux was keeping unused pages in core and leaving less memory available for cache and process pages. +What was actually occurring was that FreeBSD was proactively paging out unused pages to make room for more disk cache while Linux was keeping unused pages in core and leaving less memory available for cache and process pages. I do not know whether this is still true today. [[prefault-optimizations]] == Pre-Faulting and Zeroing Optimizations Taking a VM fault is not expensive if the underlying page is already in core and can simply be mapped into the process, but it can become expensive if you take a whole lot of them on a regular basis. A good example of this is running a program such as man:ls[1] or man:ps[1] over and over again. If the program binary is mapped into memory but not mapped into the page table, then all the pages that will be accessed by the program will have to be faulted in every time the program is run. This is unnecessary when the pages in question are already in the VM Cache, so FreeBSD will attempt to pre-populate a process's page tables with those pages that are already in the VM Cache. One thing that FreeBSD does not yet do is pre-copy-on-write certain pages on exec. For example, if you run the man:ls[1] program while running `vmstat 1` you will notice that it always takes a certain number of page faults, even when you run it over and over again. These are zero-fill faults, not program code faults (which were pre-faulted in already). Pre-copying pages on exec or fork is an area that could use more study. A large percentage of page faults that occur are zero-fill faults. You can usually see this by observing the `vmstat -s` output. These occur when a process accesses pages in its BSS area. The BSS area is expected to be initially zero but the VM system does not bother to allocate any memory at all until the process actually accesses it. When a fault occurs the VM system must not only allocate a new page, it must zero it as well. To optimize the zeroing operation the VM system has the ability to pre-zero pages and mark them as such, and to request pre-zeroed pages when zero-fill faults occur. -The pre-zeroing occurs whenever the CPU is idle but the number of pages the system pre-zeros is limited in order to avoid blowing away the memory caches. -This is an excellent example of adding complexity to the VM system in order to optimize the critical path. +The pre-zeroing occurs whenever the CPU is idle but the number of pages the system pre-zeros is limited to avoid blowing away the memory caches. +This is an excellent example of adding complexity to the VM system to optimize the critical path. [[page-table-optimizations]] == Page Table Optimizations The page table optimizations make up the most contentious part of the FreeBSD VM design and they have shown some strain with the advent of serious use of `mmap()`. I think this is actually a feature of most BSDs though I am not sure when it was first introduced. There are two major optimizations. The first is that hardware page tables do not contain persistent state but instead can be thrown away at any time with only a minor amount of management overhead. The second is that every active page table entry in the system has a governing `pv_entry` structure which is tied into the `vm_page` structure. FreeBSD can simply iterate through those mappings that are known to exist while Linux must check all page tables that _might_ contain a specific mapping to see if it does, which can achieve O(n^2) overhead in certain situations. It is because of this that FreeBSD tends to make better choices on which pages to reuse or swap when memory is stressed, giving it better performance under load. However, FreeBSD requires kernel tuning to accommodate large-shared-address-space situations such as those that can occur in a news system because it may run out of `pv_entry` structures. Both Linux and FreeBSD need work in this area. FreeBSD is trying to maximize the advantage of a potentially sparse active-mapping model (not all processes need to map all pages of a shared library, for example), whereas Linux is trying to simplify its algorithms. FreeBSD generally has the performance advantage here at the cost of wasting a little extra memory, but FreeBSD breaks down in the case where a large file is massively shared across hundreds of processes. Linux, on the other hand, breaks down in the case where many processes are sparsely-mapping the same shared library and also runs non-optimally when trying to determine whether a page can be reused or not. [[page-coloring-optimizations]] == Page Coloring We will end with the page coloring optimizations. Page coloring is a performance optimization designed to ensure that accesses to contiguous pages in virtual memory make the best use of the processor cache. In ancient times (i.e. 10+ years ago) processor caches tended to map virtual memory rather than physical memory. This led to a huge number of problems including having to clear the cache on every context switch in some cases, and problems with data aliasing in the cache. Modern processor caches map physical memory precisely to solve those problems. This means that two side-by-side pages in a processes address space may not correspond to two side-by-side pages in the cache. In fact, if you are not careful side-by-side pages in virtual memory could wind up using the same page in the processor cache-leading to cacheable data being thrown away prematurely and reducing CPU performance. This is true even with multi-way set-associative caches (though the effect is mitigated somewhat). FreeBSD's memory allocation code implements page coloring optimizations, which means that the memory allocation code will attempt to locate free pages that are contiguous from the point of view of the cache. For example, if page 16 of physical memory is assigned to page 0 of a process's virtual memory and the cache can hold 4 pages, the page coloring code will not assign page 20 of physical memory to page 1 of a process's virtual memory. It would, instead, assign page 21 of physical memory. The page coloring code attempts to avoid assigning page 20 because this maps over the same cache memory as page 16 and would result in non-optimal caching. This code adds a significant amount of complexity to the VM memory allocation subsystem as you can well imagine, but the result is well worth the effort. Page Coloring makes VM memory as deterministic as physical memory in regards to cache performance. [[conclusion]] == Conclusion Virtual memory in modern operating systems must address a number of different issues efficiently and for many different usage patterns. The modular and algorithmic approach that BSD has historically taken allows us to study and understand the current implementation as well as relatively cleanly replace large sections of the code. There have been a number of improvements to the FreeBSD VM system in the last several years, and work is ongoing. [[allen-briggs-qa]] == Bonus QA session by Allen Briggs === What is the interleaving algorithm that you refer to in your listing of the ills of the FreeBSD 3.X swap arrangements? FreeBSD uses a fixed swap interleave which defaults to 4. This means that FreeBSD reserves space for four swap areas even if you only have one, two, or three. Since swap is interleaved the linear address space representing the "four swap areas" will be fragmented if you do not actually have four swap areas. For example, if you have two swap areas A and B FreeBSD's address space representation for that swap area will be interleaved in blocks of 16 pages: .... A B C D A B C D A B C D A B C D .... FreeBSD 3.X uses a "sequential list of free regions" approach to accounting for the free swap areas. The idea is that large blocks of free linear space can be represented with a single list node ([.filename]#kern/subr_rlist.c#). But due to the fragmentation the sequential list winds up being insanely fragmented. In the above example, completely unused swap will have A and B shown as "free" and C and D shown as "all allocated". Each A-B sequence requires a list node to account for because C and D are holes, so the list node cannot be combined with the next A-B sequence. Why do we interleave our swap space instead of just tack swap areas onto the end and do something fancier? It is a whole lot easier to allocate linear swaths of an address space and have the result automatically be interleaved across multiple disks than it is to try to put that sophistication elsewhere. The fragmentation causes other problems. Being a linear list under 3.X, and having such a huge amount of inherent fragmentation, allocating and freeing swap winds up being an O(N) algorithm instead of an O(1) algorithm. Combined with other factors (heavy swapping) and you start getting into O(N^2) and O(N^3) levels of overhead, which is bad. The 3.X system may also need to allocate KVM during a swap operation to create a new list node which can lead to a deadlock if the system is trying to pageout pages in a low-memory situation. Under 4.X we do not use a sequential list. Instead we use a radix tree and bitmaps of swap blocks rather than ranged list nodes. We take the hit of preallocating all the bitmaps required for the entire swap area up front but it winds up wasting less memory due to the use of a bitmap (one bit per block) instead of a linked list of nodes. The use of a radix tree instead of a sequential list gives us nearly O(1) performance no matter how fragmented the tree becomes. === How is the separation of clean and dirty (inactive) pages related to the situation where you see low cache queue counts and high active queue counts in systat -vm? Do the systat stats roll the active and dirty pages together for the active queue count? Yes, that is confusing. The relationship is "goal" verses "reality". Our goal is to separate the pages but the reality is that if we are not in a memory crunch, we do not really have to. What this means is that FreeBSD will not try very hard to separate out dirty pages (inactive queue) from clean pages (cache queue) when the system is not being stressed, nor will it try to deactivate pages (active queue -> inactive queue) when the system is not being stressed, even if they are not being used. === In man:ls[1] the / vmstat 1 example, would not some of the page faults be data page faults (COW from executable file to private page)? I.e., I would expect the page faults to be some zero-fill and some program data. Or are you implying that FreeBSD does do pre-COW for the program data? A COW fault can be either zero-fill or program-data. The mechanism is the same either way because the backing program-data is almost certainly already in the cache. I am indeed lumping the two together. FreeBSD does not pre-COW program data or zero-fill, but it _does_ pre-map pages that exist in its cache. === In your section on page table optimizations, can you give a little more detail about pv_entry and vm_page (or should vm_page be vm_pmap-as in 4.4, cf. pp. 180-181 of McKusick, Bostic, Karel, Quarterman)? Specifically, what kind of operation/reaction would require scanning the mappings? A `vm_page` represents an (object,index#) tuple. A `pv_entry` represents a hardware page table entry (pte). If you have five processes sharing the same physical page, and three of those processes's page tables actually map the page, that page will be represented by a single `vm_page` structure and three `pv_entry` structures. `pv_entry` structures only represent pages mapped by the MMU (one `pv_entry` represents one pte). -This means that when we need to remove all hardware references to a `vm_page` (in order to reuse the page for something else, page it out, clear it, dirty it, and so forth) we can simply scan the linked list of pv_entry's associated with that vm_page to remove or modify the pte's from their page tables. +This means that when we need to remove all hardware references to a `vm_page` (to reuse the page for something else, page it out, clear it, dirty it, and so forth) we can simply scan the linked list of pv_entry's associated with that vm_page to remove or modify the pte's from their page tables. Under Linux there is no such linked list. -In order to remove all the hardware page table mappings for a `vm_page` linux must index into every VM object that _might_ have mapped the page. +To remove all the hardware page table mappings for a `vm_page` linux must index into every VM object that _might_ have mapped the page. For example, if you have 50 processes all mapping the same shared library and want to get rid of page X in that library, you need to index into the page table for each of those 50 processes even if only 10 of them have actually mapped the page. So Linux is trading off the simplicity of its design against performance. Many VM algorithms which are O(1) or (small N) under FreeBSD wind up being O(N), O(N^2), or worse under Linux. Since the pte's representing a particular page in an object tend to be at the same offset in all the page tables they are mapped in, reducing the number of accesses into the page tables at the same pte offset will often avoid blowing away the L1 cache line for that offset, which can lead to better performance. -FreeBSD has added complexity (the `pv_entry` scheme) in order to increase performance (to limit page table accesses to _only_ those pte's that need to be modified). +FreeBSD has added complexity (the `pv_entry` scheme) to increase performance (to limit page table accesses to _only_ those pte's that need to be modified). But FreeBSD has a scaling problem that Linux does not in that there are a limited number of `pv_entry` structures and this causes problems when you have massive sharing of data. In this case you may run out of `pv_entry` structures even though there is plenty of free memory available. This can be fixed easily enough by bumping up the number of `pv_entry` structures in the kernel config, but we really need to find a better way to do it. In regards to the memory overhead of a page table verses the `pv_entry` scheme: Linux uses "permanent" page tables that are not throw away, but does not need a `pv_entry` for each potentially mapped pte. FreeBSD uses "throw away" page tables but adds in a `pv_entry` structure for each actually-mapped pte. I think memory utilization winds up being about the same, giving FreeBSD an algorithmic advantage with its ability to throw away page tables at will with very low overhead. === Finally, in the page coloring section, it might help to have a little more description of what you mean here. I did not quite follow it. Do you know how an L1 hardware memory cache works? I will explain: Consider a machine with 16MB of main memory but only 128K of L1 cache. Generally the way this cache works is that each 128K block of main memory uses the _same_ 128K of cache. If you access offset 0 in main memory and then offset 128K in main memory you can wind up throwing away the cached data you read from offset 0! Now, I am simplifying things greatly. What I just described is what is called a "direct mapped" hardware memory cache. Most modern caches are what are called 2-way-set-associative or 4-way-set-associative caches. The set-associatively allows you to access up to N different memory regions that overlap the same cache memory without destroying the previously cached data. But only N. So if I have a 4-way set associative cache I can access offset 0, offset 128K, 256K and offset 384K and still be able to access offset 0 again and have it come from the L1 cache. If I then access offset 512K, however, one of the four previously cached data objects will be thrown away by the cache. It is extremely important... _extremely_ important for most of a processor's memory accesses to be able to come from the L1 cache, because the L1 cache operates at the processor frequency. The moment you have an L1 cache miss and have to go to the L2 cache or to main memory, the processor will stall and potentially sit twiddling its fingers for _hundreds_ of instructions worth of time waiting for a read from main memory to complete. Main memory (the dynamic ram you stuff into a computer) is __slow__, when compared to the speed of a modern processor core. Ok, so now onto page coloring: All modern memory caches are what are known as _physical_ caches. They cache physical memory addresses, not virtual memory addresses. This allows the cache to be left alone across a process context switch, which is very important. But in the UNIX(R) world you are dealing with virtual address spaces, not physical address spaces. Any program you write will see the virtual address space given to it. The actual _physical_ pages underlying that virtual address space are not necessarily physically contiguous! In fact, you might have two pages that are side by side in a processes address space which wind up being at offset 0 and offset 128K in _physical_ memory. A program normally assumes that two side-by-side pages will be optimally cached. That is, that you can access data objects in both pages without having them blow away each other's cache entry. But this is only true if the physical pages underlying the virtual address space are contiguous (insofar as the cache is concerned). This is what Page coloring does. Instead of assigning _random_ physical pages to virtual addresses, which may result in non-optimal cache performance, Page coloring assigns _reasonably-contiguous_ physical pages to virtual addresses. Thus programs can be written under the assumption that the characteristics of the underlying hardware cache are the same for their virtual address space as they would be if the program had been run directly in a physical address space. Note that I say "reasonably" contiguous rather than simply "contiguous". From the point of view of a 128K direct mapped cache, the physical address 0 is the same as the physical address 128K. So two side-by-side pages in your virtual address space may wind up being offset 128K and offset 132K in physical memory, but could also easily be offset 128K and offset 4K in physical memory and still retain the same cache performance characteristics. So page-coloring does _not_ have to assign truly contiguous pages of physical memory to contiguous pages of virtual memory, it just needs to make sure it assigns contiguous pages from the point of view of cache performance and operation.