diff --git a/documentation/content/en/articles/building-products/_index.adoc b/documentation/content/en/articles/building-products/_index.adoc index 577b1d3d42..6c9a001278 100644 --- a/documentation/content/en/articles/building-products/_index.adoc +++ b/documentation/content/en/articles/building-products/_index.adoc @@ -1,374 +1,374 @@ --- title: Building Products with FreeBSD authors: - author: Joseph Koshy email: jkoshy@FreeBSD.org organizations: - organization: The FreeBSD Project -description: Building Products with FreeBSD +description: How FreeBSD can help you build a better product trademarks: ["freebsd", "general"] --- = Building Products with FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] ifeval::["{backend}" == "html5"] :imagesdir: ../../../images/articles/building-products/ endif::[] ifeval::["{backend}" == "pdf"] :imagesdir: ../../../../static/images/articles/building-products/ endif::[] ifeval::["{backend}" == "epub3"] :imagesdir: ../../../../static/images/articles/building-products/ endif::[] [.abstract-title] Abstract The FreeBSD project is a worldwide, volunteer based, and collaborative project, which develops a portable and high-quality operating system. The FreeBSD project distributes the source code for its product under a liberal license, with the intention of encouraging the use of its code. Collaborating with the FreeBSD project can help organizations reduce their time to market, reduce engineering costs and improve their product quality. This article examines the issues in using FreeBSD code in appliances and software products. It highlights the characteristics of FreeBSD that make it an excellent substrate for product development. The article concludes by suggesting a few "best practices" for organizations collaborating with the FreeBSD project. ''' toc::[] [[introduction]] == Introduction FreeBSD today is well-known as a high-performance server operating system. It is deployed on millions of web servers and internet-facing hosts worldwide. FreeBSD code also forms an integral part of many products, ranging from appliances such as network routers, firewalls, and storage devices, to personal computers. Portions of FreeBSD have also been used in commercial shrink-wrapped software (see <>). In this article we look at the link:https://www.FreeBSD.org/[FreeBSD project] as a software engineering resource-as a collection of building blocks and processes which you can use to build products. While FreeBSD's source is distributed freely to the public, to fully enjoy the benefits of the project's work, organizations need to _collaborate_ with the project. In subsequent sections of this article we discuss effective means of collaboration with the project and the pitfalls that need to be avoided while doing so. *Caveat Reader.* The author believes that the characteristics of the FreeBSD Project listed in this article were substantially true at the time the article was conceived and written (2005). However, the reader should keep in mind that the practices and processes used by open-source communities can change over time, and that the information in this article should therefore be taken as indicative rather than normative. === Target Audience This document would be of interest to the following broad groups of people: * Decision makers in product companies looking at ways to improve their product quality, reduce their time to market and lower engineering costs in the long term. * Technology consultants looking for best-practices in leveraging "open-source". * Industry observers interested in understanding the dynamics of open-source projects. * Software developers seeking to use FreeBSD and looking for ways to contribute back. === Article Goals After reading this article you should have: * An understanding of the goals of the FreeBSD Project and its organizational structure. * An understanding of its development model and release engineering processes. * An understanding of how conventional corporate software development processes differ from that used in the FreeBSD project. * Awareness of the communication channels used by the project and the level of transparency you can expect. * Awareness of optimal ways of working with the project-how best to reduce engineering costs, improve time to market, manage security vulnerabilities, and preserve future compatibility with your product as the FreeBSD project evolves. === Article Structure The rest of the article is structured as follows: * <> introduces the FreeBSD project, explores its organizational structure, key technologies and release engineering processes. * <> describes ways to collaborate with the FreeBSD project. It examines common pitfalls encountered by corporates working with voluntary projects like FreeBSD. * <> concludes. [[freebsd-intro]] == FreeBSD as a set of building blocks FreeBSD makes an excellent foundation on which to build products: * FreeBSD source code is distributed under a liberal BSD license facilitating its adoption in commercial products <> with minimum hassle. * The FreeBSD project has excellent engineering practices that can be leveraged. * The project offers exceptional transparency into its workings, allowing organizations using its code to plan effectively for the future. * The culture of the FreeBSD project, carried over from the Computer Science Research Group at The University of California, Berkeley <>, fosters high-quality work. Some features in FreeBSD define the state of the art. <> examines the business reasons for using open-source in greater detail. For organizations, the benefits of using FreeBSD components in their products include a shorter time to market, lower development costs and lower development risks. === Building with FreeBSD Here are a few ways organizations have used FreeBSD: * As an upstream source for tested code for libraries and utilities. + By being "downstream" of the project, organizations leverage the new features, bug fixes and testing that the upstream code receives. * As an embedded OS (for example, for an OEM router and firewall device). In this model, organizations use a customized FreeBSD kernel and application program set along with a proprietary management layer for their device. OEMs benefit from new hardware support being added by the FreeBSD project upstream, and from the testing that the base system receives. + FreeBSD ships with a self-hosting development environment that allows easy creation of such configurations. * As a Unix compatible environment for the management functions of high-end storage and networking devices, running on a separate processor "blade". + FreeBSD provides the tools for creating dedicated OS and application program images. Its implementation of a BSD unix API is mature and tested. FreeBSD can also provide a stable cross-development environment for the other components of the high-end device. * As a vehicle to get widespread testing and support from a worldwide team of developers for non-critical "intellectual property". + In this model, organizations contribute useful infrastructural frameworks to the FreeBSD project (for example, see man:netgraph[3]). The widespread exposure that the code gets helps to quickly identify performance issues and bugs. The involvement of top-notch developers also leads to useful extensions to the infrastructure that the contributing organization also benefits from. * As a development environment supporting cross-development for embedded OSes like http://www.rtems.com/[RTEMS] and http://ecos.sourceware.org/[eCOS]. + There are many full-fledged development environments in the {numports}-strong collection of applications ported and packaged with FreeBSD. * As a way to support a Unix-like API in an otherwise proprietary OS, increasing its palatability for application developers. + Here parts of FreeBSD's kernel and application programs are "ported" to run alongside other tasks in the proprietary OS. The availability of a stable and well tested Unix(TM) API implementation can reduce the effort needed to port popular applications to the proprietary OS. As FreeBSD ships with high-quality documentation for its internals and has effective vulnerability management and release engineering processes, the costs of keeping up-to-date are kept low. [[freebsd-technologies]] === Technologies There are a large number of technologies supported by the FreeBSD project. A selection of these are listed below: * A complete system that can cross-host itself for link:https://www.FreeBSD.org/platforms/[many architectures:] * A modular symmetric multiprocessing capable kernel, with loadable kernel modules and a flexible and easy to use configuration system. * Support for emulation of Linux(TM) and SVR4 binaries at near machine speeds. Support for binary Windows(TM) (NDIS) network drivers. * Libraries for many programming tasks: archivers, FTP and HTTP support, thread support, in addition to a full POSIX(TM) like programming environment. * Security features: Mandatory Access Control (man:mac[9]), jails (man:jail[2]), ACLs, and in-kernel cryptographic device support. * Networking features: firewall-ing, QoS management, high-performance TCP/IP networking with support for many extensions. + FreeBSD's in-kernel Netgraph (man:netgraph[4]) framework allows kernel networking modules to be connected together in flexible ways. * Support for storage technologies: Fibre Channel, SCSI, software and hardware RAID, ATA and SATA. + FreeBSD supports a number of filesystems, and its native UFS2 filesystem supports soft updates, snapshots and very large filesystem sizes (16TB per filesystem) <>. + FreeBSD's in-kernel GEOM (man:geom[4]) framework allows kernel storage modules to be composed in flexible ways. * Over {numports} ported applications, both commercial and open-source, managed via the FreeBSD ports collection. === Organizational Structure FreeBSD's organizational structure is non-hierarchical. There are essentially two kinds of contributors to FreeBSD, general users of FreeBSD, and developers with write access (known as _committers_ in the jargon) to the source base. There are many thousands of contributors in the first group; the vast majority of contributions to FreeBSD come from individuals in this group. Commit rights (write access) to the repository are granted to individuals who contribute consistently to the project. Commit rights come with additional responsibilities, and new committers are assigned mentors to help them learn the ropes. .FreeBSD Organization image::freebsd-organization.png[] Conflict resolution is performed by a nine member "Core Team" that is elected from the group of committers. FreeBSD does not have "corporate" committers. Individual committers are required to take responsibility for the changes they introduce to the code. The link:{committers-guide}[FreeBSD Committer's guide] <> documents the rules and responsibilities for committers. FreeBSD's project model is examined in detail in <>. === FreeBSD Release Engineering Processes FreeBSD's release engineering processes play a major role in ensuring that its released versions are of a high quality. At any point of time, FreeBSD's volunteers support multiple code lines (<>): * New features and disruptive code enters on the development branch, also known as the _-CURRENT_ branch. * _-STABLE_ branches are code lines that are branched from HEAD at regular intervals. Only tested code is allowed onto a -STABLE branch. New features are allowed once they have been tested and stabilized in the -CURRENT branch. * _-RELEASE_ branches are maintained by the FreeBSD security team. Only bug fixes for critical issues are permitted onto -RELEASE branches. [[fig-freebsd-branches]] .FreeBSD Release Branches image::freebsd-branches.png[] Code lines are kept alive for as long as there is user and developer interest in them. Machine architectures are grouped into "tiers"; _Tier 1_ architectures are fully supported by the project's release engineering and security teams, _Tier 2_ architectures are supported on a best effort basis, and experimental architectures comprise _Tier 3_. The list of link:{committers-guide}#archs[supported architectures] is part of the FreeBSD documentation collection. The release engineering team publishes a link:https://www.FreeBSD.org/releng/[road map] for future releases of FreeBSD on the project's web site. The dates laid down in the road map are not deadlines; FreeBSD is released when its code and documentation are ready. FreeBSD's release engineering processes are described in <>. [[freebsd-collaboration]] == Collaborating with FreeBSD Open-source projects like FreeBSD offer finished code of a very high quality. While access to quality source code can reduce the cost of initial development, in the long-term the costs of managing change begin to dominate. As computing environments change over the years and new security vulnerabilities are discovered, your product too needs to change and adapt. Using open-source code is best viewed not as a one-off activity, but as an __ongoing process__. The best projects to collaborate with are the ones that are __live__; i.e., with an active community, clear goals and a transparent working style. * FreeBSD has an active developer community around it. At the time of writing there are many thousands of contributors from every populated continent in the world and over 300 individuals with write access to the project's source repositories. * The goals of the FreeBSD project are <>: ** To develop a high-quality operating system for popular computer hardware, and, ** To make our work available to all under a liberal license. * FreeBSD enjoys an open and transparent working culture. Nearly all discussion in the project happens by email, on https://lists.freebsd.org/mailman/listinfo[public mailing lists] that are also archived for posterity. The project's policies are link:https://www.FreeBSD.org/internal/policies/[documented] and maintained under revision control. Participation in the project is open to all. [[freebsd-org]] === Understanding FreeBSD culture To be able to work effectively with the FreeBSD project, you need to understand the project's culture. Volunteer driven projects operate under different rules than for-profit corporates. A common mistake that companies make when venturing into the open-source world is that of underplaying these differences. *Motivation.* Most contributions to FreeBSD are done voluntarily without monetary rewards entering the picture. The factors that motivate individuals are complex, ranging from altruism, to an interest in solving the kinds of problems that FreeBSD attempts to solve. In this environment, "elegance is never optional"<>. *The Long Term View.* FreeBSD traces its roots back nearly twenty years to the work of the Computer Science Research Group at the University of California Berkeley.footnote:[FreeBSD's source repository contains a history of the project since its inception, and there are CDROMs available that contain earlier code from the CSRG.] A number of the original CSRG developers remain associated with the project. The project values long-term perspectives <>. A frequent acronym encountered in the project is DTRT, which stands for "Do The Right Thing". *Development Processes.* Computer programs are tools for communication: at one level programmers communicate their intentions using a precise notation to a tool (a compiler) that translates their instructions to executable code. At another level, the same notation is used for communication of intent between two programmers. Formal specifications and design documents are seldom used in the project. Clear and well-written code and well-written change logs (<>) are used in their place. FreeBSD development happens by "rough consensus and running code"<>. [.programlisting] .... r151864 | bde | 2005-10-29 09:34:50 -0700 (Sat, 29 Oct 2005) | 13 lines Changed paths: M /head/lib/msun/src/e_rem_pio2f.c Use double precision to simplify and optimize arg reduction for small and medium size args too: instead of conditionally subtracting a float 17+24, 17+17+24 or 17+17+17+24 bit approximation to pi/2, always subtract a double 33+53 bit one. The float version is now closer to the double version than to old versions of itself -- it uses the same 33+53 bit approximation as the simplest cases in the double version, and where the float version had to switch to the slow general case at |x| == 2^7*pi/2, it now switches at |x| == 2^19*pi/2 the same as the double version. This speeds up arg reduction by a factor of 2 for |x| between 3*pi/4 and 2^7*pi/4, and by a factor of 7 for |x| between 2^7*pi/4 and 2^19*pi/4. .... .A sample change log entry [[fig-change-log]] Communication between programmers is enhanced by the use of a common coding standard man:style[9]. *Communication Channels.* FreeBSD's contributors are spread across the world. Email (and to a lesser extent, IRC) is the preferred means of communication in the project. === Best Practices for collaborating with the FreeBSD project We now look at a few best practices for making the best use of FreeBSD in product development. Plan for the long term:: Setup processes that help in tracking the development of FreeBSD. For example: + *Track FreeBSD source code.* The project makes it easy to mirror its SVN repository using link:{committers-guide}#svn-advanced-use-setting-up-svnsync[svnsync]. Having the complete history of the source is useful when debugging complex problems and offers valuable insight into the intentions of the original developers. Use a capable source control system that allows you to easily merge changes between the upstream FreeBSD code base and your own in-house code. + <> shows a portion of an annotated listing of the file referenced by the change log in <>. The ancestry of each line of the source is clearly visible. Annotated listings showing the history of every file that is part of FreeBSD are https://svnweb.freebsd.org/[available on the web]. + [.programlisting] .... #REV #WHO #DATE #TEXT 176410 bde 2008-02-19 07:42:46 -0800 (Tue, 19 Feb 2008) #include 176410 bde 2008-02-19 07:42:46 -0800 (Tue, 19 Feb 2008) __FBSDID("$FreeBSD$"); 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) /* __ieee754_rem_pio2f(x,y) 8870 rgrimes 1995-05-29 22:51:47 -0700 (Mon, 29 May 1995) * 176552 bde 2008-02-25 05:33:20 -0800 (Mon, 25 Feb 2008) * return the remainder of x rem pi/2 in *y 176552 bde 2008-02-25 05:33:20 -0800 (Mon, 25 Feb 2008) * use double precision for everything except passing x 152535 bde 2005-11-16 18:20:04 -0800 (Wed, 16 Nov 2005) * use __kernel_rem_pio2() for large x 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) */ 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) 176465 bde 2008-02-22 07:55:14 -0800 (Fri, 22 Feb 2008) #include 176465 bde 2008-02-22 07:55:14 -0800 (Fri, 22 Feb 2008) 2116 jkh 1994-08-19 02:40:01 -0700 (Fri, 19 Aug 1994) #include "math.h" .... .An annotated source listing generated using `svn blame` [[fig-svn-blame]] + *Use a gatekeeper.* Appoint a _gatekeeper_ to monitor FreeBSD development, to keep an eye out for changes that could potentially impact your products. + *Report bugs upstream.* If you notice bug in the FreeBSD code that you are using, file a https://www.FreeBSD.org/support/bugreports/[bug report]. This step helps ensure that you do not have to fix the bug the next time you take a code drop from upstream. Leverage FreeBSD's release engineering efforts:: Use code from a -STABLE development branch of FreeBSD. These development branches are formally supported by FreeBSD's release engineering and security teams and comprise of tested code. Donate code to reduce costs:: A major proportion of the costs associated with developing products is that of doing maintenance. By donating non-critical code to the project, you benefit by having your code see much wider exposure than it would otherwise get. This in turn leads to more bugs and security vulnerabilities being flushed out and performance anomalies being identified and fixed. Get support effectively:: For products with tight deadlines, it is recommended that you hire or enter into a consulting agreement with a developer or firm with FreeBSD experience. The {freebsd-jobs} is a useful communication channel to find talent. The FreeBSD project maintains a link:https://www.FreeBSD.org/commercial/consult_bycat/[gallery of consultants and consulting firms] undertaking FreeBSD work. The http://www.bsdcertification.org/[BSD Certification Group] offers certification for all the major BSD derived OSes. + For less critical needs, you can ask for help on the http://lists.FreeBSD.org/mailman/listinfo[project mailing lists]. A useful guide to follow when asking for help is given in <>. Publicize your involvement:: You are not required to publicize your use of FreeBSD, but doing so helps both your effort as well as that of the project. + Letting the FreeBSD community know that your company uses FreeBSD helps improve your chances of attracting high quality talent. A large roster of support for FreeBSD also means more mind share for it among developers. This in turn yields a healthier foundation for your future. Support FreeBSD developers:: Sometimes the most direct way to get a desired feature into FreeBSD is to support a developer who is already looking at a related problem. Help can range from hardware donations to direct financial assistance. In some countries, donations to the FreeBSD project enjoy tax benefits. The project has a dedicated link:https://www.FreeBSD.org/donations/[donations liaison] to assist donors. The project also maintains a web page where developers link:https://www.FreeBSD.org/donations/wantlist/[list their needs]. + As a policy the FreeBSD project link:{contributors}[acknowledges] all contributions received on its web site. [[conclusion]] == Conclusion The FreeBSD project's goals are to create and give away the source code for a high-quality operating system. By working with the FreeBSD project you can reduce development costs and improve your time to market in a number of product development scenarios. We examined the characteristics of the FreeBSD project that make it an excellent choice for being part of an organization's product strategy. We then looked at the prevailing culture of the project and examined effective ways of interacting with its developers. The article concluded with a list of best-practices that could help organizations collaborating with the project. :sectnums!: [bibliography] == Bibliography [[Carp1996]] [Carp1996] http://www.ietf.org/rfc/rfc1958.txt[The Architectural Principles of the Internet] B. Carpenter. The Internet Architecture Board.The Internet Architecture Board. Copyright(R) 1996. [[ComGuide]] [ComGuide] link:{committers-guide}[Committer's Guide] The FreeBSD Project. Copyright(R) 2005. [[GoldGab2005]] [GoldGab2005] http://dreamsongs.com/IHE/IHE.html[Innovation Happens Elsewhere: Open Source as Business Strategy] Ron Goldman. Richard Gabriel. Copyright(R) 2005. Morgan-Kaufmann. [[Hub1994]] [Hub1994] link:{contributing}[Contributing to the FreeBSD Project] Jordan Hubbard. Copyright(R) 1994-2005. The FreeBSD Project. [[McKu1999]] [McKu1999] http://www.usenix.org/publications/library/proceedings/usenix99/mckusick.html[Soft Updates: A Technique for Eliminating Most Synchronous Writes in the Fast Filesystem] Kirk McKusick. Gregory Ganger. Copyright(R) 1999. [[McKu1999-1]] [McKu1999-1] http://www.oreilly.com/catalog/opensources/book/kirkmck.html[Twenty Years of Berkeley Unix: From AT&T-Owned to Freely Redistributable] Marshall Kirk McKusick. http://www.oreilly.com/catalog/opensources/book/toc.html[Open Sources: Voices from the Open Source Revolution] O'Reilly Inc.. Copyright(R) 1993. [[Mon2005]] [Mon2005] link:{bsdl-gpl}[Why you should use a BSD style license for your Open Source Project] Bruce Montague. The FreeBSD Project. Copyright(R) 2005. [[Nik2005]] [Nik2005] link:{dev-model}[A project model for the FreeBSD Project] Niklas Saers. Copyright(R) 2005. The FreeBSD Project. [[Nor1993]] [Nor1993] http://www.norvig.com/luv-slides.ps[Tutorial on Good Lisp Programming Style] Peter Norvig. Kent Pitman. Copyright(R) 1993. [[Nor2001]] [Nor2001] http://www.norvig.com/21-days.html[Teach Yourself Programming in Ten Years] Peter Norvig. Copyright(R) 2001. [[Ray2004]] [Ray2004] http://www.catb.org/~esr/faqs/smart-questions.html[How to ask questions the smart way] Eric Steven Raymond. Copyright(R) 2004. [[RelEngDoc]] [RelEngDoc] link:{releng}[FreeBSD Release Engineering] Murray Stokely. Copyright(R) 2001. The FreeBSD Project. diff --git a/documentation/content/en/articles/committers-guide/_index.adoc b/documentation/content/en/articles/committers-guide/_index.adoc index 7aad6a68ef..13cbb0c6a2 100644 --- a/documentation/content/en/articles/committers-guide/_index.adoc +++ b/documentation/content/en/articles/committers-guide/_index.adoc @@ -1,3774 +1,3775 @@ --- title: Committer's Guide authors: - author: The FreeBSD Documentation Project copyright: 1999-2021 The FreeBSD Documentation Project -description: FreeBSD Committer's Guide +description: Introductory information for FreeBSD committers trademarks: ["freebsd", "coverity", "ibm", "intel", "general"] +weight: 25 --- = Committer's Guide :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/authors.adoc[] include::shared/en/teams.adoc[lines=16..-1] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This document provides information for the FreeBSD committer community. All new committers should read this document before they start, and existing committers are strongly encouraged to review it from time to time. Almost all FreeBSD developers have commit rights to one or more repositories. However, a few developers do not, and some of the information here applies to them as well. (For instance, some people only have rights to work with the Problem Report database). Please see <> for more information. This document may also be of interest to members of the FreeBSD community who want to learn more about how the project works. ''' toc::[] [[admin]] == Administrative Details [.informaltable] [cols="1,1", frame="none"] |=== |_Login Methods_ |man:ssh[1], protocol 2 only |_Main Shell Host_ |`freefall.FreeBSD.org` |_SMTP Host_ |`smtp.FreeBSD.org:587` (see also <>). |`_src/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/src.git` (see also <>). |`_doc/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/doc.git` (see also <>). |`_ports/_` Git Repository |`ssh://git@gitrepo.FreeBSD.org/ports.git` (see also <>). |_Internal Mailing Lists_ |developers (technically called all-developers), doc-developers, doc-committers, ports-developers, ports-committers, src-developers, src-committers. (Each project repository has its own -developers and -committers mailing lists. Archives for these lists can be found in the files [.filename]#/local/mail/repository-name-developers-archive# and [.filename]#/local/mail/repository-name-committers-archive# on the `FreeBSD.org` cluster.) |_Core Team monthly reports_ |[.filename]#/home/core/public/monthly-reports# on the `FreeBSD.org` cluster. |_Ports Management Team monthly reports_ |[.filename]#/home/portmgr/public/monthly-reports# on the `FreeBSD.org` cluster. |_Noteworthy `src/` Git Branches:_ |`stable/n` (`n`-STABLE), `main` (-CURRENT) |=== man:ssh[1] is required to connect to the project hosts. For more information, see <>. Useful links: * link:https://www.FreeBSD.org/internal/[FreeBSD Project Internal Pages] * link:https://www.FreeBSD.org/internal/machines/[FreeBSD Project Hosts] * link:https://www.FreeBSD.org/administration/[FreeBSD Project Administrative Groups] [[pgpkeys]] == OpenPGP Keys for FreeBSD Cryptographic keys conforming to the OpenPGP (__Pretty Good Privacy__) standard are used by the FreeBSD project to authenticate committers. Messages carrying important information like public SSH keys can be signed with the OpenPGP key to prove that they are really from the committer. See http://www.nostarch.com/pgp_ml.htm[PGP & GPG: Email for the Practical Paranoid by Michael Lucas] and http://en.wikipedia.org/wiki/Pretty_Good_Privacy[] for more information. [[pgpkeys-creating]] === Creating a Key Existing keys can be used, but should be checked with [.filename]#documentation/tools/checkkey.sh# first. In this case, make sure the key has a FreeBSD user ID. For those who do not yet have an OpenPGP key, or need a new key to meet FreeBSD security requirements, here we show how to generate one. [[pgpkeys-create-steps]] [.procedure] ==== . Install [.filename]#security/gnupg#. Enter these lines in [.filename]#~/.gnupg/gpg.conf# to set minimum acceptable defaults: + [.programlisting] .... fixed-list-mode keyid-format 0xlong personal-digest-preferences SHA512 SHA384 SHA256 SHA224 default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP Uncompressed use-agent verify-options show-uid-validity list-options show-uid-validity sig-notation issuer-fpr@notations.openpgp.fifthhorseman.net=%g cert-digest-algo SHA512 .... . Generate a key: + [source,shell] .... % gpg --full-gen-key gpg (GnuPG) 2.1.8; Copyright (C) 2015 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Warning: using insecure memory! Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 2048 <.> Requested keysize is 2048 bits Please specify how long the key should be valid. 0 = key does not expire = key expires in n days w = key expires in n weeks m = key expires in n months y = key expires in n years Key is valid for? (0) 3y <.> Key expires at Wed Nov 4 17:20:20 2015 MST Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Chucky Daemon <.> Email address: notreal@example.com Comment: You selected this USER-ID: "Chucky Daemon " Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o You need a Passphrase to protect your secret key. .... <.> 2048-bit keys with a three-year expiration provide adequate protection at present (2013-12). http://danielpocock.com/rsa-key-sizes-2048-or-4096-bits[] describes the situation in more detail. <.> A three year key lifespan is short enough to obsolete keys weakened by advancing computer power, but long enough to reduce key management problems. <.> Use your real name here, preferably matching that shown on government-issued ID to make it easier for others to verify your identity. Text that may help others identify you can be entered in the `Comment` section. + After the email address is entered, a passphrase is requested. Methods of creating a secure passphrase are contentious. Rather than suggest a single way, here are some links to sites that describe various methods: http://world.std.com/~reinhold/diceware.html[], http://www.iusmentis.com/security/passphrasefaq/[], http://xkcd.com/936/[], http://en.wikipedia.org/wiki/Passphrase[]. ==== Protect the private key and passphrase. If either the private key or passphrase may have been compromised or disclosed, immediately notify mailto:accounts@FreeBSD.org[accounts@FreeBSD.org] and revoke the key. Committing the new key is shown in <>. [[kerberos-ldap]] == Kerberos and LDAP web Password for FreeBSD Cluster The FreeBSD cluster requires a Kerberos password to access certain services. The Kerberos password also serves as the LDAP web password, since LDAP is proxying to Kerberos in the cluster. Some of the services which require this include: * https://bugs.freebsd.org/bugzilla[Bugzilla] * https://ci.freebsd.org[Jenkins] To create a new Kerberos account in the FreeBSD cluster, or to reset a Kerberos password for an existing account using a random password generator: [source,shell] .... % ssh kpasswd.freebsd.org .... [NOTE] ==== This must be done from a machine outside of the FreeBSD.org cluster. ==== A Kerberos password can also be set manually by logging into `freefall.FreeBSD.org` and running: [source,shell] .... % kpasswd .... [NOTE] ==== Unless the Kerberos-authenticated services of the FreeBSD.org cluster have been used previously, `Client unknown` will be shown. This error means that the `ssh kpasswd.freebsd.org` method shown above must be used first to initialize the Kerberos account. ==== [[committer.types]] == Commit Bit Types The FreeBSD repository has a number of components which, when combined, support the basic operating system source, documentation, third party application ports infrastructure, and various maintained utilities. When FreeBSD commit bits are allocated, the areas of the tree where the bit may be used are specified. Generally, the areas associated with a bit reflect who authorized the allocation of the commit bit. Additional areas of authority may be added at a later date: when this occurs, the committer should follow normal commit bit allocation procedures for that area of the tree, seeking approval from the appropriate entity and possibly getting a mentor for that area for some period of time. [.informaltable] [cols="1,1,1", frame="none"] |=== |__Committer Type__ |__Responsible__ |__Tree Components__ |src |core@ |src/ |doc |doceng@ |doc/, ports/, src/ documentation |ports |portmgr@ |ports/ |=== Commit bits allocated prior to the development of the notion of areas of authority may be appropriate for use in many parts of the tree. However, common sense dictates that a committer who has not previously worked in an area of the tree seek review prior to committing, seek approval from the appropriate responsible party, and/or work with a mentor. Since the rules regarding code maintenance differ by area of the tree, this is as much for the benefit of the committer working in an area of less familiarity as it is for others working on the tree. Committers are encouraged to seek review for their work as part of the normal development process, regardless of the area of the tree where the work is occurring. === Policy for Committer Activity in Other Trees * All committers may modify [.filename]#src/share/misc/committers-*.dot#, [.filename]#src/usr.bin/calendar/calendars/calendar.freebsd#, and [.filename]#ports/astro/xearth/files#. * doc committers may commit documentation changes to [.filename]#src# files, such as man pages, READMEs, fortune databases, calendar files, and comment fixes without approval from a src committer, subject to the normal care and tending of commits. * Any committer may make changes to any other tree with an "Approved by" from a non-mentored committer with the appropriate bit. Mentored committers can provide a "Reviewed by" but not an "Approved by". * Committers can acquire an additional bit by the usual process of finding a mentor who will propose them to core, doceng, or portmgr, as appropriate. When approved, they will be added to 'access' and the normal mentoring period will ensue, which will involve a continuing of "Approved by" for some period. [[git-primer]] == Git Primer [NOTE] ==== this section is a work in progress... ==== [[git-basics]] === Git basics There are many primers on how to use Git on the web. There's a lot of them (search for "Git primer"). https://danielmiessler.com/study/git/ and https://gist.github.com/williewillus/068e9a8543de3a7ef80adb2938657b6b are good overviews. The Git book is also complete, but much longer https://git-scm.com/book/en/v2. There is also this website https://ohshitgit.com/ for common traps and pitfalls of Git, in case you need guidance to fix things up. This document will assume that you've read through it and will try not to belabor the basics (though it will cover them briefly). [[git-mini-primer]] === Git Mini Primer This primer is less ambitiously scoped than the old Subversion Primer, but should cover the basics. ==== Scope If you want to download FreeBSD, compile it from sources, and generally keep up to date that way, this primer is for you. It covers getting the sources, updating the sources, bisecting and touches briefly on how to cope with a few local changes. It covers the basics, and tries to give good pointers to more in-depth treatment for when the reader finds the basics insufficient. Other sections of this guide cover more advanced topics related to contributing to the project. The goal of this section is to highlight those bits of Git needed to track sources. They assume a basic understanding of Git. There are many primers for Git on the web, but the https://git-scm.com/book/en/v2[Git Book] provides one of the better treatments. ==== Keeping Current With The FreeBSD src Tree [[keeping_current]] First step: cloning a tree. This downloads the entire tree. There are two ways to download. Most people will want to do a deep clone of the repository. However, there are times when you may wish to do a shallow clone. ===== Branch names The branch names in the new Git repository are similar to the old names. For the stable branches, they are stable/X where X is the major release (like 11 or 12). The main branch in the new repository is 'main'. The main branch in the old GitHub mirror was 'master', but is now 'main'. Both reflect the defaults of Git at the time they were created. The 'main' branch is the default branch if you omit the '-b branch' or '--branch branch' options below. ===== Repositories Please see the <> for the latest information on where to get FreeBSD sources. $URL below can be obtained from that page. Note: The project doesn't use submodules as they are a poor fit for our workflows and development model. How we track changes in third-party applications is discussed elsewhere and generally of little concern to the casual user. ===== Deep Clone A deep clone pulls in the entire tree, as well as all the history and branches. It is the easiest to do. It also allows you to use Git's worktree feature to have all your active branches checked out into separate directories but with only one copy of the repository. [source,shell] .... % git clone -o freebsd $URL -b branch [dir] .... is how you make a deep clone. 'branch' should be one of the branches listed in the previous section. It is optional if it is the main branch. 'dir' is an optional directory to place it in (the default will be the name of the repo you are cloning (src, doc, etc)). You will want a deep clone if you are interested in the history, plan on making local changes, or plan on working on more than one branch. It is the easiest to keep up to date as well. If you are interested in the history, but are working with only one branch and are short on space, you can also use --single-branch to only download the one branch (though some merge commits will not reference the merged-from branch which may be important for some users who are interested in detailed versions of history). ===== Shallow Clone A shallow clone copies just the most current code, but none or little of the history. This can be useful when you need to build a specific revision of FreeBSD, or when you are just starting out and plan to track the tree more fully. You can also use it to limit history to only so many revisions. However, see below for a significant limitation of this approach. [source,shell] .... % git clone -o freebsd -b branch --depth 1 $URL [dir] .... This clones the repository, but only has the most recent version in the repository. The rest of the history is not downloaded. Should you change your mind later, you can do 'git fetch --unshallow' to get the old history. [WARNING] ==== When you make a shallow clone, you will lose the commit count in your uname output. This can make it more difficult to determine if your system needs to be updated when a security advisory is issued. ==== ===== Building Once you've downloaded, building is done as described in the handbook, eg: [source,shell] .... % cd src % make buildworld % make buildkernel % make installkernel % make installworld .... so that won't be covered in depth here. If you want to build a custom kernel, link:{handbook}#kernelconfig[the kernel config section] of the FreeBSD Handbook recommends creating a file MYKERNEL under sys/${ARCH}/conf with your changes against GENERIC. To have MYKERNEL disregarded by Git, it can be added to .git/info/exclude. ===== Updating To update both types of trees uses the same commands. This pulls in all the revisions since your last update. [source,shell] .... % git pull --ff-only .... will update the tree. In Git, a 'fast forward' merge is one that only needs to set a new branch pointer and doesn't need to re-create the commits. By always doing a 'fast forward' merge/pull, you'll ensure that you have an exact copy of the FreeBSD tree. This will be important if you want to maintain local patches. See below for how to manage local changes. The simplest is to use --autostash on the 'git pull' command, but more sophisticated options are available. ==== Selecting a Specific Version In Git, the 'git checkout' checks out both branches and specific versions. Git's versions are the long hashes rather than a sequential number. When you checkout a specific version, just specify the hash you want on the command line (the git log command can help you decide which hash you might want): [source,shell] .... % git checkout 08b8197a74 .... and you have that checked out. You will be greeted with a message similar to the following: [source,shell] .... Note: checking out '08b8197a742a96964d2924391bf9fdfeb788865d'. You are in a 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 08b8197a742a hook gpiokeys.4 to the build .... where the last line is generated from the hash you are checking out and the first line of the commit message from that revision. The hash can be abbreviated to the shortest unique length. Git itself is inconsistent about how many digits it displays. ==== Bisecting Sometimes, things go wrong. The last version worked, but the one you just updated to does not. A developer may ask you to bisect the problem to track down which commit caused the regression. Git makes bisecting changes easy with a powerful 'git bisect' command. Here's a brief outline of how to use it. For more information, you can view https://www.metaltoad.com/blog/beginners-guide-git-bisect-process-elimination or https://git-scm.com/docs/git-bisect for more details. The man git-bisect page is good at describing what can go wrong, what to do when versions won't build, when you want to use terms other than 'good' and 'bad', etc, none of which will be covered here. `git bisect start` will start the bisection process. Next, you need to tell a range to go through. 'git bisect good XXXXXX' will tell it the working version and 'git bisect bad XXXXX' will tell it the bad version. The bad version will almost always be HEAD (a special tag for what you have checked out). The good version will be the last one you checked out. [TIP] ==== If you want to know the last version you checked out, you should use 'git reflog': [source,shell] .... 5ef0bd68b515 (HEAD -> main, freebsd/main, freebsd/HEAD) HEAD@{0}: pull --ff-only: Fast-forward a8163e165c5b (upstream/main) HEAD@{1}: checkout: moving from b6fb97efb682994f59b21fe4efb3fcfc0e5b9eeb to main ... .... shows me moving the working tree to the main branch (a816...) and then updating from upstream (to 5ef0...). In this case, bad would be HEAD (or 5rf0bd68) and good would be a8163e165. As you can see from the output, HEAD@{1} also often works, but isn't foolproof if you have done other things to your Git tree after updating, but before you discover the need to bisect. ==== Set the 'good' version first, then set the bad (though the order doesn't matter). When you set the bad version, it will give you some statistics on the process: [source,shell] .... % git bisect start % git bisect good a8163e165c5b % git bisect bad HEAD Bisecting: 1722 revisions left to test after this (roughly 11 steps) [c427b3158fd8225f6afc09e7e6f62326f9e4de7e] Fixup r361997 by balancing parens. Duh. .... You would then build/install that version. If it's good you'd type 'git bisect good' otherwise 'git bisect bad'. If the version doesn't compile, type 'git bisect skip'. You will get a similar message to the above after each step. When you are done, report the bad version to the developer (or fix the bug yourself and send a patch). 'git bisect reset' will end the process and return you back to where you started (usually tip of main). Again, the git-bisect manual (linked above) is a good resource for when things go wrong or for unusual cases. [[git-gpg-signing]] ==== Signing the commits, tags, and pushes, with GnuPG Git knows how to sign commits, tags, and pushes. When you sign a Git commit or a tag, you can prove that the code you submitted came from you and wasn't altered while you were transferring it. You also can prove that you submitted the code and not someone else. A more in-depth documentation on signing commits and tags can be found in the https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work[Git Tools - Signing Your Work] chapter of the Git's book. The rationale behind signing pushes can be found in the https://github.com/git/git/commit/a85b377d0419a9dfaca8af2320cc33b051cbed04[commit that introduced the feature]. The best way is to simply tell Git you always want to sign commits, tags, and pushes. You can do this by setting a few configuration variables: [source,shell] .... % git config --add user.signingKey=LONG-KEY-ID % git config --add commit.gpgSign=true % git config --add tag.gpgSign=true % git config --add push.gpgSign=if-asked .... // push.gpgSign should probably be set to `yes` once we enable it, or be set with --global, so that it is enabled for all repositories. [NOTE] ====== To avoid possible collisions, make sure you give a long key id to Git. You can get the long id with: `gpg --list-secret-keys --keyid-format LONG`. ====== [TIP] ====== To use specific subkeys, and not have GnuPG to resolve the subkey to a primary key, attach `!` to the key. For example, to encrypt for the subkey `DEADBEEF`, use `DEADBEEF!`. ====== ===== Verifying signatures Commit signatures can be verified by running either `git verify-commit `, or `git log --show-signature`. Tag signatures can be verifed with `git verity-tag `, or `git tag -v `. //// Commented out for now until we decide what to do. Git pushes are a bit different, they live in a special ref in the repository. TODO: write how to verify them //// ==== Ports Considerations The ports tree operates the same way. The branch names are different and the repositories are in different locations. The cgit repository web interface for use with web browsers is at https://cgit.FreeBSD.org/ports/ . The production Git repository is at https://git.FreeBSD.org/ports.git and at ssh://anongit@git.FreeBSD.org/ports.git (or anongit@git.FreeBSD.org:ports.git). There is also a mirror on GitHub, see link:{handbook}mirrors/#mirrors[External mirrors] for an overview. The 'current' branch is 'main' . The quarterly branches are named 'yyyyQn' for year 'yyyy' and quarter 'n'. ===== Commit message formats A hook is available in the ports repository to help you write up your commit messages in https://cgit.freebsd.org/ports/tree/.hooks/prepare-commit-msg[.hooks/prepare-commit-message]. It can be enabled by running ``git config --add core.hooksPath .hooks``. The main point being that a commit message should be formatted in the following way: .... category/port: Summary. Description of why the changes where made. PR: 12345 .... [IMPORTANT] ==== The first line is the subject of the commit, it contains what port was changed, and a summary of the commit. It should contain 50 characters or less. A blank line should separate it from the rest of the commit message. The rest of the commit message should be wrapped at the 72 characters boundary. Another blank line should be added if there are any metadata fields, so that they are easily distinguishable from the commit message. ==== ==== Managing Local Changes This section addresses tracking local changes. If you have no local changes, you can stop reading now (it is the last section and OK to skip). One item that is important for all of them: all changes are local until pushed. Unlike Subversion, Git uses a distributed model. For users, for most things, there is very little difference. However, if you have local changes, you can use the same tool to manage them as you use to pull in changes from FreeBSD. All changes that you have not pushed are local and can easily be modified (git rebase, discussed below does this). ===== Keeping local changes The simplest way to keep local changes (especially trivial ones) is to use 'git stash'. In its simplest form, you use 'git stash' to record the changes (which pushes them onto the stash stack). Most people use this to save changes before updating the tree as described above. They then use 'git stash apply' to re-apply them to the tree. The stash is a stack of changes that can be examined with 'git stash list'. The git-stash man page (https://git-scm.com/docs/git-stash) has all the details. This method is suitable when you have tiny tweaks to the tree. When you have anything non trivial, you'll likely be better off keeping a local branch and rebasing. Stashing is also integrated with the 'git pull' command: just add '--autostash' to the command line. ===== Keeping a local branch [[keeping_a_local_branch]] It is much easier to keep a local branch with Git than Subversion. In Subversion you need to merge the commit, and resolve the conflicts. This is manageable, but can lead to a convoluted history that's hard to upstream should that ever be necessary, or hard to replicate if you need to do so. Git also allows one to merge, along with the same problems. That's one way to manage the branch, but it's the least flexible. In addition to merging, Git supports the concept of 'rebasing' which avoids these issues. The 'git rebase' command replays all the commits of a branch at a newer location on the parent branch. We will cover the most common scenarios that arise using it. ====== Create a branch Let's say you want to make a change to FreeBSD's ls command to never, ever do color. There are many reasons to do this, but this example will use that as a baseline. The FreeBSD ls command changes from time to time, and you'll need to cope with those changes. Fortunately, with Git rebase it usually is automatic. [source,shell] .... % cd src % git checkout main % git checkout -b no-color-ls % cd bin/ls % vi ls.c # hack the changes in % git diff # check the changes diff --git a/bin/ls/ls.c b/bin/ls/ls.c index 7378268867ef..cfc3f4342531 100644 --- a/bin/ls/ls.c +++ b/bin/ls/ls.c @@ -66,6 +66,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#undef COLORLS #ifdef COLORLS #include #include % # these look good, make the commit... % git commit ls.c .... The commit will pop you into an editor to describe what you've done. Once you enter that, you have your own **local** branch in the Git repo. Build and install it like you normally would, following the directions in the handbook. Git differs from other version control systems in that you have to tell it explicitly which files to commit. I have opted to do it on the commit command line, but you can also do it with 'git add' which many of the more in depth tutorials cover. ====== Time to update When it is time to bring in a new version, it is almost the same as w/o the branches. You would update like you would above, but there is one extra command before you update, and one after. The following assumes you are starting with an unmodified tree. It is important to start rebasing operations with a clean tree (Git usually requires this). [source,shell] .... % git checkout main % git pull --no-ff % git rebase -i main no-color-ls .... This will bring up an editor that lists all the commits in it. For this example, do not change it at all. This is typically what you are doing while updating the baseline (though you also use the Git rebase command to curate the commits you have in the branch). Once you are done with the above, you have to move the commits to ls.c forward from the old version of FreeBSD to the newer one. Sometimes there are merge conflicts. That is OK. Do not panic. Instead, handle them the same as any other merge conflicts. To keep it simple, I will just describe a common issue that may arise. A pointer to a more complete treatment can be found at the end of this section. Let's say the includes changes upstream in a radical shift to terminfo as well as a name change for the option. When you updated, you might see something like this: [source,shell] .... Auto-merging bin/ls/ls.c CONFLICT (content): Merge conflict in bin/ls/ls.c error: could not apply 646e0f9cda11... no color ls Resolve all conflicts manually, mark them as resolved with "git add/rm ", then run "git rebase --continue". You can instead skip this commit: run "git rebase --skip". To abort and get back to the state before "git rebase", run "git rebase --abort". Could not apply 646e0f9cda11... no color ls .... which looks scary. If you bring up an editor, you will see it is a typical 3-way merge conflict resolution that you may be familiar with from other source code systems (the rest of ls.c has been omitted): [source,shell] .... <<<<<<< HEAD #ifdef COLORLS_NEW #include ======= #undef COLORLS #ifdef COLORLS #include >>>>>>> 646e0f9cda11... no color ls .... The new code is first, and your code is second. The right fix here is to just add a #undef COLORLS_NEW before #ifdef and then delete the old changes: [source,shell] .... #undef COLORLS_NEW #ifdef COLORLS_NEW #include .... save the file. The rebase was interrupted, so you have to complete it: [source,shell] .... % git add ls.c % git rebase --continue .... which tells Git that ls.c has been fixed and to continue the rebase operation. Since there was a conflict, you will get kicked into the editor to update the commit message if necessary. If the commit message is still accurate, just exit the editor. If you get stuck during the rebase, do not panic. git rebase --abort will take you back to a clean slate. It is important, though, to start with an unmodified tree. An aside: The above mentioned 'git reflog' comes in handy here, as it will have a list of all the (intermediate) commits that you can view or inspect or cherry-pick. For more on this topic, https://www.freecodecamp.org/news/the-ultimate-guide-to-git-merge-and-git-rebase/ provides a rather extensive treatment. It is a good resource for issues that arise occasionally but are too obscure for this guide. ===== Switching to a Different FreeBSD Branch If you wish to shift from stable/12 to the current branch. If you have a deep clone, the following will suffice: [source,shell] .... % git checkout main % # build and install here... .... If you have a local branch, though, there are one or two caveats. First, rebase will rewrite history, so you will likely want to do something to save it. Second, jumping branches tends to cause more conflicts. If we pretend the example above was relative to stable/12, then to move to main, I'd suggest the following: [source,shell] .... % git checkout no-color-ls % git checkout -b no-color-ls-stable-12 # create another name for this branch % git rebase -i stable/12 no-color-ls --onto main .... What the above does is checkout no-color-ls. Then create a new name for it (no-color-ls-stable-12) in case you need to get back to it. Then you rebase onto the main branch. This will find all the commits to the current no-color-ls branch (back to where it meets up with the stable/12 branch) and then it will replay them onto the main branch creating a new no-color-ls branch there (which is why I had you create a place holder name). ===== Migrating from an existing Git clone If you have work based on a previous Git conversion or a locally running git-svn conversion, migrating to new repository can encounter problems because Git has no knowledge about the connection between the two. When you have only a few local changes, the easiest way would be to cherry-pick those changes to the new base: [source,shell] .... % git checkout main % git cherry-pick old_branch..your_branch .... Or alternatively, do the same thing with rebase: [source,shell] .... % git rebase --onto main master your_branch .... If you do have a lot of changes, you would probably want to perform a merge instead. The idea is to create a merge point that consolidates the history of the old_branch, and the new FreeBSD repository (main). You can find out by looking up the same commit that are found on both parents: [source,shell] .... % git show old_branch .... You will see a commit message, now search for that in the new branch: [source,shell] .... % git log --grep="commit message on old_branch" freebsd/main .... You would help locate the commit hash on the new main branch, create a helper branch (in the example we call it 'stage') from that hash: [source,shell] .... % git checkout -b stage _hash_found_from_git_log_ .... Then perform a merge of the old branch: [source,shell] .... % git merge -s ours -m "Mark old branch as merged" old_branch .... With that, it's possible to merge your work branch or the main branch in any order without problem. Eventually, when you are ready to commit your work back to main, you can perform a rebase to main, or do a squash commit by combining everything into one commit. [[mfc-with-git]] === MFC (Merge From Current) Procedures ==== Summary MFC workflow can be summarized as `git cherry-pick -x` plus `git commit --amend` to adjust the commit message. For multiple commits, use `git rebase -i` to squash them together and edit the commit message. ==== Single commit MFC [source,shell] .... % git checkout stable/X % git cherry-pick -x $HASH --edit .... For MFC commits, for example a vendor import, you would need to specify one parent for cherry-pick purposes. Normally, that would be the "first parent" of the branch you are cherry-picking from, so: [source,shell] .... % git checkout stable/X % git cherry-pick -x $HASH -m 1 --edit .... If things go wrong, you'll either need to abort the cherry-pick with `git cherry-pick --abort` or fix it up and do a `git cherry-pick --continue`. Once the cherry-pick is finished, push with `git push`. If you get an error due to losing the commit race, use `git pull --rebase` and try to push again. ==== MFC to RELENG branch MFCs to branches that require approval require a bit more care. The process is the same for either a typical merge or an exceptional direct commit. * Merge or direct commit to the appropriate `stable/X` branch first before merging to the `releng/X.Y` branch. * Use the hash that's in the `stable/X` branch for the MFC to `releng/X.Y` branch. * Leave both "cherry picked from" lines in the commit message. * Be sure to add the `Approved by:` line when you are in the editor. [source,shell] .... % git checkout releng/13.0 % git cherry-pick -x $HASH --edit .... If you forget to to add the `Approved by:` line, you can do a `git commit --amend` to edit the commit message before you push the change. ==== Multiple commit MFC [source,shell] .... % git checkout -b tmp-branch stable/X % for h in $HASH_LIST; do git cherry-pick -x $h; done % git rebase -i stable/X # mark each of the commits after the first as 'squash' # Update the commit message to reflect all elements of commit, if necessary. # Be sure to retain the "cherry picked from" lines. % git push freebsd HEAD:stable/X .... If the push fails due to losing the commit race, rebase and try again: [source,shell] .... % git checkout stable/X % git pull % git checkout tmp-branch % git rebase stable/X % git push freebsd HEAD:stable/X .... Once the MFC is complete, you can delete the temporary branch: [source,shell] .... % git checkout stable/X % git branch -d tmp-branch .... ==== MFC a vendor import Vendor imports are the only thing in the tree that creates a merge commit in the main line. Cherry picking merge commits into stable/XX presents an additional difficulty because there are two parents for a merge commit. Generally, you'll want the first parent's diff since that's the diff to mainline (though there may be some exceptions). [source,shell] .... % git cherry-pick -x -m 1 $HASH .... is typically what you want. This will tell cherry-pick to apply the correct diff. There are some, hopefully, rare cases where it's possible that the mainline was merged backwards by the conversion script. Should that be the case (and we've not found any yet), you'd change the above to '-m 2' to pickup the proper parent. Just do [source,shell] .... % git cherry-pick --abort % git cherry-pick -x -m 2 $HASH .... to do that. The `--aboort` will cleanup the failed first attempt. ==== Redoing a MFC If you do a MFC, and it goes horribly wrong and you want to start over, then the easiest way is to use `git reset --hard` like so: [source,shell] .... % git reset --hard freebsd/stable/12 .... though if you have some revs you want to keep, and others you don't, using 'git rebase -i' is better. ==== Considerations when MFCing When committing source commits to stable and releng branches, we have the following goals: * Clearly mark direct commits distinct from commits that land a change from another branch. * Avoid introducing known breakage into stable and releng branches. * Allow developers to determine which changes have or have not been landed from one branch to another. With Subversion, we used the following practices to achieve these goals: * Using 'MFC' and 'MFS' tags to mark commits that merged changes from another branch. * Squashing fixup commits into the main commit when merging a change. * Recording mergeinfo so that `svn mergeinfo --show-revs` worked. With Git, we will need to use different strategies to achieve the same goals. This document aims to define best practices when merging source commits using Git that achieve these goals. In general, we aim to use Git's native support to achieve these goals rather than enforcing practices built on Subversion's model. One general note: due to technical differences with Git, we will not be using Git "merge commits" (created via `git merge`) in stable or releng branches. Instead, when this document refers to "merge commits", it means a commit originally made to `main` that is replicated or "landed" to a stable branch, or a commit from a stable branch that is replicated to a releng branch with some variation of `git cherry-pick`. ==== Finding Eligible Hashes to MFC Git provides some built-in support for this via the `git cherry` and `git log --cherry` commands. These commands compare the raw diffs of commits (but not other metadata such as log messages) to determine if two commits are identical. This works well when each commit from head is landed as a single commit to a stable branch, but it falls over if multiple commits from main are squashed together as a single commit to a stable branch. There are a few options for resolving this: 1. We could ban squashing of commits and instead require that committers stage all of the fixup / follow-up commits to stable into a single push. This would still achieve the goal of stability in stable and releng branches since pushes are atomic and users doing a simple pull will never end up with a tree that has the main commit without the fixup(s). `git bisect` is also able to cope with this model via `git bisect skip`. 2. We could adopt a consistent style for describing MFCs and write our own tooling to wrap around `git cherry` to determine the list of eligible commits. A simple approach here might be to use the syntax from `git cherry-pick -x`, but require that a squashed commit list all of the hashes (one line per hash) at the end of the commit message. Developers could do this by using `git cherry-pick -x` of each individual commit into a branch and then use `git rebase` to squash the commits down into a single commit, but collecting the `-x` annotations at the end of the landed commit log. ==== Commit message standards ===== Marking MFCs The project has adopted the following practice for marking MFCs: * Use the `-x` flag with `git cherry-pick`. This adds a line to the commit message that includes the hash of the original commit when merging. Since it is added by Git directly, committers do not have to manually edit the commit log when merging. When merging multiple commits, keep all the "cherry picked from" lines. ===== Trim Metadata? One area that was not clearly documented with Subversion (or even CVS) is how to format metadata in log messages for MFC commits. Should it include the metadata from the original commit unchanged, or should it be altered to reflect information about the MFC commit itself? Historical practice has varied, though some of the variance is by field. For example, MFCs that are relevant to a PR generally include the PR field in the MFC so that MFC commits are included in the bug tracker's audit trail. Other fields are less clear. For example, Phabricator shows the diff of the last commit tagged to a review, so including Phabricator URLs replaces the `main` commit with the landed commits. The list of reviewers is also not clear. If a reviewer has approved a change to `main`, does that mean they have approved the MFC commit? Is that true if it's identical code only, or with merely trivial reworkes? It's clearly not true for more extensive reworks. Even for identical code what if the commit doesn't conflict but introduces an ABI change? A reviewer may have ok'd a commit for `main` due to the ABI breakage but may not approve of merging the same commit as-is. One will have to use one's best judgement until clear guidelines can be agreed upon. For MFCs regulated by re@, new metadata fields are added, such as the Approved by tag for approved commits. This new metadata will have to be added via `git commit --amend` or similar after the original commit has been reviewed and approved. We may also want to reserve some metadata fields in MFC commits such as Phabricator URLs for use by re@ in the future. Preserving existing metadata provides a very simple workflow. Developers can just use `git cherry-pick -x` without having to edit the log message. If instead we choose to adjust metadata in MFCs, developers will have to edit log messages explicitly via the use of `git cherry-pick --edit` or `git commit --amend`. However, as compared to svn, at least the existing commit message can be pre-populated and metadata fields can be added or removed without having to re-enter the entire commit message. The bottom line is that developers will likely need to curate their commit message for MFCs that are non-trivial. ==== Examples ===== Merging a Single Subversion Commit This walks through the process of merging a commit to stable/12 that was originally committed to head in Subversion. In this case, the original commit is r368685. The first step is to map the Subversion commit to a Git hash. Once you have fetched refs/notes/commits, you can pass the revision number to `git log --grep`: [source,shell] .... % git log main --grep 368685 commit ce8395ecfda2c8e332a2adf9a9432c2e7f35ea81 Author: John Baldwin Date: Wed Dec 16 00:11:30 2020 +0000 Use the 't' modifier to print a ptrdiff_t. Reviewed by: imp Obtained from: CheriBSD Sponsored by: DARPA Differential Revision: https://reviews.freebsd.org/D27576 Notes: svn path=/head/; revision=368685 .... Next, MFC the commit to a `stable/12` checkout: [source,shell] .... git checkout stable/12 git cherry-pick -x ce8395ecfda2c8e332a2adf9a9432c2e7f35ea81 --edit .... Git will invoke the editor. Use this to remove the metadata that only applied to the original commit (Phabricator URL and Reviewed by). After the editor saves the updated log message, Git completes the commit: [source,shell] .... [stable/12 3e3a548c4874] Use the 't' modifier to print a ptrdiff_t. Date: Wed Dec 16 00:11:30 2020 +0000 1 file changed, 1 insertion(+), 1 deletion(-) .... The contents of the MFCd commit can be examined via `git show`: [source,shell] .... % git show commit 3e3a548c487450825679e4bd63d8d1a67fd8bd2d (HEAD -> stable/12) Author: John Baldwin Date: Wed Dec 16 00:11:30 2020 +0000 Use the 't' modifier to print a ptrdiff_t. Obtained from: CheriBSD Sponsored by: DARPA (cherry picked from commit ce8395ecfda2c8e332a2adf9a9432c2e7f35ea81) diff --git a/sys/compat/linuxkpi/common/include/linux/printk.h b/sys/compat/linuxkpi/common/include/linux/printk.h index 31802bdd2c99..e6510e9e9834 100644 --- a/sys/compat/linuxkpi/common/include/linux/printk.h +++ b/sys/compat/linuxkpi/common/include/linux/printk.h @@ -68,7 +68,7 @@ print_hex_dump(const char *level, const char *prefix_str, printf("[%p] ", buf); break; case DUMP_PREFIX_OFFSET: - printf("[%p] ", (const char *)((const char *)buf - + printf("[%#tx] ", ((const char *)buf - (const char *)buf_old)); break; default: .... The MFC commit can now be published via `git push` [source,shell] .... % git push freebsd Enumerating objects: 17, done. Counting objects: 100% (17/17), done. Delta compression using up to 4 threads Compressing objects: 100% (7/7), done. Writing objects: 100% (9/9), 817 bytes | 204.00 KiB/s, done. Total 9 (delta 5), reused 1 (delta 1), pack-reused 0 To gitrepo-dev.FreeBSD.org:src.git 525bd9c9dda7..3e3a548c4874 stable/12 -> stable/12 .... ===== Merging a Single Subversion Commit with a Conflict This example is similar to the previous example except that the commit in question encounters a merge conflict. In this case, the original commit is r368314. As above, the first step is to map the Subversion commit to a Git hash: [source,shell] .... % git log main --grep 368314 commit 99963f5343a017e934e4d8ea2371a86789a46ff9 Author: John Baldwin Date: Thu Dec 3 22:01:13 2020 +0000 Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. Reviewed by: np MFC after: 2 weeks Sponsored by: Chelsio Communications Differential Revision: https://reviews.freebsd.org/D27469 Notes: svn path=/head/; revision=368314 .... Next, MFC the commit to a `stable/12` checkout: [source,shell] .... % git checkout stable/12 % git cherry-pick -x 99963f5343a017e934e4d8ea2371a86789a46ff9 --edit Auto-merging sys/dev/cxgbe/tom/t4_cpl_io.c CONFLICT (content): Merge conflict in sys/dev/cxgbe/tom/t4_cpl_io.c warning: inexact rename detection was skipped due to too many files. warning: you may want to set your merge.renamelimit variable to at least 7123 and retry the command. error: could not apply 99963f5343a0... Don't transmit mbufs that aren't yet ready on TOE sockets. hint: after resolving the conflicts, mark the corrected paths hint: with 'git add ' or 'git rm ' hint: and commit the result with 'git commit' .... In this case, the commit encountered a merge conflict in sys/dev/cxge/tom/t4_cpl_io.c as kernel TLS is not present in stable/12. Note that Git does not invoke an editor to adjust the commit message due to the conflict. `git status` confirms that this file has merge conflicts: [source,shell] .... % git status On branch stable/12 Your branch is up to date with 'upstream/stable/12'. You are currently cherry-picking commit 99963f5343a0. (fix conflicts and run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Unmerged paths: (use "git add ..." to mark resolution) both modified: sys/dev/cxgbe/tom/t4_cpl_io.c no changes added to commit (use "git add" and/or "git commit -a") .... After editing the file to resolve the conflict, `git status` shows the conflict as resolved: [source,shell] .... % git status On branch stable/12 Your branch is up to date with 'upstream/stable/12'. You are currently cherry-picking commit 99963f5343a0. (all conflicts fixed: run "git cherry-pick --continue") (use "git cherry-pick --skip" to skip this patch) (use "git cherry-pick --abort" to cancel the cherry-pick operation) Changes to be committed: modified: sys/dev/cxgbe/tom/t4_cpl_io.c .... The cherry-pick can now be completed: [source,shell] .... % git cherry-pick --continue .... Since there was a merge conflict, Git invokes the editor to adjust the commit message. Trim the metadata fields from the commit log from the original commit to head and save the updated log message. The contents of the MFC commit can be examined via `git show`: [source,shell] .... % git show commit 525bd9c9dda7e7c7efad2d4570c7fd8e1a8ffabc (HEAD -> stable/12) Author: John Baldwin Date: Thu Dec 3 22:01:13 2020 +0000 Don't transmit mbufs that aren't yet ready on TOE sockets. This includes mbufs waiting for data from sendfile() I/O requests, or mbufs awaiting encryption for KTLS. Sponsored by: Chelsio Communications (cherry picked from commit 99963f5343a017e934e4d8ea2371a86789a46ff9) diff --git a/sys/dev/cxgbe/tom/t4_cpl_io.c b/sys/dev/cxgbe/tom/t4_cpl_io.c index 8e8c2b8639e6..43861f10b689 100644 --- a/sys/dev/cxgbe/tom/t4_cpl_io.c +++ b/sys/dev/cxgbe/tom/t4_cpl_io.c @@ -746,6 +746,8 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop) for (m = sndptr; m != NULL; m = m->m_next) { int n; + if ((m->m_flags & M_NOTAVAIL) != 0) + break; if (IS_AIOTX_MBUF(m)) n = sglist_count_vmpages(aiotx_mbuf_pages(m), aiotx_mbuf_pgoff(m), m->m_len); @@ -821,8 +823,9 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop) /* nothing to send */ if (plen == 0) { - KASSERT(m == NULL, - ("%s: nothing to send, but m != NULL", __func__)); + KASSERT(m == NULL || (m->m_flags & M_NOTAVAIL) != 0, + ("%s: nothing to send, but m != NULL is ready", + __func__)); break; } @@ -910,7 +913,7 @@ t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop) toep->txsd_avail--; t4_l2t_send(sc, wr, toep->l2te); - } while (m != NULL); + } while (m != NULL && (m->m_flags & M_NOTAVAIL) == 0); /* Send a FIN if requested, but only if there's no more data to send */ if (m == NULL && toep->flags & TPF_SEND_FIN) .... The MFC commit can now be published via `git push` [source,shell] .... git push freebsd Enumerating objects: 13, done. Counting objects: 100% (13/13), done. Delta compression using up to 4 threads Compressing objects: 100% (7/7), done. Writing objects: 100% (7/7), 819 bytes | 117.00 KiB/s, done. Total 7 (delta 6), reused 0 (delta 0), pack-reused 0 To gitrepo.FreeBSD.org:src.git f4d0bc6aa6b9..525bd9c9dda7 stable/12 -> stable/12 .... [[vendor-import-git]] === Vendor Imports with Git This section describes the vendor import procedure with Git in detail. ==== Branch naming convention All vendor branches and tags start with `vendor/`. These branches and tags are visible by default. [NOTE] ==== This chapter follows the convention that the `freebsd` origin is the origin name for the official FreeBSD Git repository. If you use a different convention, replace `freebsd` with the name you use instead in the examples below. ==== We will explore an example for updating NetBSD's mtree that is in our tree. The vendor branch for this is `vendor/NetBSD/mtree`. ==== Updating an old vendor import The vendor trees usually have only the subset of the third-party software that is appropriate to FreeBSD. These trees are usually tiny in comparison to the FreeBSD tree. Git worktrees are thus quite small and fast and the preferred method to use. Make sure that whatever directory you choose below (the `../mtree`) does not currently exist. [source,shell] .... % git worktree add ../mtree vendor/NetBSD/mtree .... ==== Update the Sources in the Vendor Branch Prepare a full, clean tree of the vendor sources. Import everything but merge only what is needed. This example assumes the NetBSD checked out from their GitHub mirror in `~/git/NetBSD`. Note that "upstream" might have added or removed files, so we want to make sure deletions are propagated as well. rsync(1) is commonly installed, so I'll use that. [source,shell] .... % cd ../mtree % rsync -va --del --exclude=".git" ~/git/NetBSD/usr.sbin/mtree/ . % git add -A % git status ... % git diff --staged ... % git commit -m"Vendor import of NetBSD's mtree at 2020-12-11" [vendor/NetBSD/mtree 8e7aa25fcf1] Vendor import of NetBSD's mtree at 2020-12-11 7 files changed, 114 insertions(+), 82 deletions(-) % git tag -a vendor/NetBSD/mtree/20201211 .... Note: I run the `git diff` and `git status` commands to make sure nothing weird was present. Also I used `-m` to illustrate, but you should compose a proper message in an editor (using a commit message template). It is also important to create an annotated tag, otherwise the push will be rejected. Only annotated tags are allowed to be pushed. The annotated tag gives you a chance to enter a commit message. Enter the version you are importing, along with any salient new features or fixes in that version. ==== Updating the FreeBSD Copy At this point you can push the import to vendor into our repo. [source,shell] .... % git push --follow-tags freebsd vendor/NetBSD/mtree .... `--follow-tags` tells `git push` to also push tags associated with the locally committed revision. ==== Updating the FreeBSD source tree Now you need to update the mtree in FreeBSD. The sources live in `contrib/mtree` since it is upstream software. [source,shell] .... % cd ../src % git subtree merge -P contrib/mtree vendor/NetBSD/mtree .... This would generate a subtree merge commit of `contrib/mtree` against the local `vendor/NetBSD/mtree` branch. If there were conflicts, you would need to fix them before committing. Include details about the changes being merged in the merge commit message. ==== Rebasing your change against latest FreeBSD source tree Because the current policy recommends against using merges, if the upstream FreeBSD `main` moved forward before you get a chance to push, you would have to redo the merge. Regular `git rebase` or `git pull --rebase` doesn't know how to rebase a merge commit **as a merge commit**, so instead of that you would have to recreate the commit. The easiest way to do this would be to create a side branch with the **contents** of the merged tree: [source,shell] .... % cd ../src % git fetch freebsd % git checkout -b merge_result % git merge freebsd/main .... Typically, there would be no merge conflicts here (because developers tend to work on different components). In the worst case scenario, you would still have to resolve merge conflicts, if there was any, but this should be really rare. Now, checkout `freebsd/main` again as `new_merge`, and redo the merge: [source,shell] .... % git checkout -b new_merge freebsd/main % git subtree merge -P contrib/mtree vendor/NetBSD/mtree .... Instead of resolving the conflicts, perform this instead: [source,shell] .... % git checkout merge_result . .... Which will overwrite the files with conflicts with the version found in `merge_result`. Examine the tree against `merge_result` to make sure that you haven't missed deleted files: [source,shell] .... % git diff merge_result .... ==== Pushing the changes Once you are sure that you have a set of deltas you think is good, you can push it to a fork off GitHub or GitLab for others to review. One nice thing about Git is that it allows you to publish rough drafts of your work for others to review. While phabricator is good for content review, publishing the updated vendor branch and merge commits lets others check the details as they will eventually appear in the repository. After review, when you are sure it is a good change, you can push it to the FreeBSD repo: [source,shell] .... % git push freebsd main .... === Creating a new vendor branch There are a number of ways to create a new vendor branch. The recommended way is to create a new repository and then merge that with FreeBSD. If one is importing `glorbnitz` into the FreeBSD tree, release 3.1415. For the sake of simplicity, we will not trim this release. It is a simple user command that puts the nitz device into different magical glorb states and is small enough trimming will not save much. ==== Create the repo [source,shell] .... % cd /some/where % mkdir glorbnitz % cd glorbnitz % git init % git checkout -b vendor/glorbnitz .... At this point, you have a new repo, where all new commits will go on the `vendor/glorbnitz` branch. Git experts can also do this right in their FreeBSD clone, using `git checkout --orphan vendor/glorbnitz` if they are more comfortable with that. ==== Copy the sources in Since this is a new import, you can just cp the sources in, or use tar or even rsync as shown above. And we will add everything, assuming no dot files. [source,shell] .... % cp -r ~/glorbnitz/* . % git add * .... At this point, you should have a pristine copy of glorbnitz ready to commit. [source,shell] .... % git commit -m"Import GlorbNitz frobnosticator revision 3.1415" .... As above, I used `-m` for simplicity, but you should likely create a commit message that explains what a Glorb is and why you'd use a Nitz to get it. Not everybody will know. But for your actual commit, you should follow the <> section instead of emulating the brief style used here. ==== Now import it into our repository Now you need to import the branch into our repository. [source,shell] .... % cd /path/to/freebsd/repo/src % git remote add glorbnitz /some/where/glorbnitz % git fetch glorbnitz vendor/glorbnitz .... Note the vendor/glorbnitz branch is in the repo. At this point the `/some/where/glorbnitz` can be deleted, if you like. It was only a means to an end. ==== Tag and push Steps from here on out are much the same as they are in the case of updating a vendor branch, though without the updating the vendor branch step. [source,shell] .... % git worktree add ../glorbnitz vendor/glorbnitz % cd ../glorbnitz % git tag --annotate vendor/glorbnitz/3.1415 # Make sure the commit is good with "git show" % git push --follow-tags freebsd vendor/glorbnitz .... By 'good' we mean: . All the right files are present . None of the wrong files are present . The vendor branch points at something sensible . The tag looks good, and is annotated . The commit message for the tag has a quick summary of what's new since the last tag ==== Time to finally merge it into the base tree [source,shell] .... % cd ../src % git subtree add -P contrib/glorbnitz vendor/glorbnitz # Make sure the commit is good with "git show" % git commit --amend # one last sanity check on commit message % git push freebsd .... Here 'good' means: . All the right files, and none of the wrong ones, were merged into contrib/glorbnitz. . No other changes are in the tree. . The commit messages look <>. It should contain a summary of what's changed since the last merge to the FreeBSD main line and any caveats. . UPDATING should be updated if there is anything of note, such as user visible changes, important upgrade concerns, etc. [NOTE] ==== This hasn't connected `glorbnitz` to the build yet. How so do that is specific to the software being imported and is beyond the scope of this tutorial. ==== === FreeBSD Src Committer Transition Guide This section is designed to walk people through the conversion process from Subversion to Git, written from the source committer's point of view. ==== Migrating from a Subversion tree This section will cover a couple of common scenarios for migrating from using the FreeBSD Subversion repo to the FreeBSD source Git repo. The FreeBSD Git conversion is still in beta status, so some minor things may change between this and going into production. The first thing to do is install Git. Any version of Git will do, though the latest one in ports / packages generally will be good. Either build it from ports, or install it using pkg (though some folks might use `su` or `doas` instead of `sudo`): [source,shell] .... % sudo pkg install git .... ===== No staged changes migration If you have no changes pending, the migration is straightforward. In this, you abandon the Subversion tree and clone the Git repository. It's likely best to retain your Subversion tree, in case there's something you've forgotten about there. First, let's clone the repository: [source,shell] .... % git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' https://git.freebsd.org/src.git freebsd-src .... will create a clone of the FreeBSD src repository into a subdirectory called `freebsd-src` and include the 'notes' about the revisions. We are currently mirroring the source repository to https://github.com/freebsd/freebsd-src.git as well. https://github.com/freebsd/freebsd-legacy.git has the old GitHub mirror with the old hashes should you need that for your migration. The GitHub `master` branch has been frozen. As the default in Git has changed, we've shifted from `master` to `main`; the new repository uses `main`. We also mirror the repository to GitLab at https://gitlab.com/FreeBSD/src.git . It's useful to have the old Subversion revisions available. This data is stored using Git notes, but Git doesn't fetch those by default. The --config and the argument above changed the default to fetch the notes. If you've cloned the repository without this, or wish to add notes to a previously cloned repository, use the following commands: [source,shell] .... % git config --add remote.freebsd.fetch "+refs/notes/*:refs/notes/*" % git fetch .... At this point you have the src checked out into a Git tree, ready to do other things. ===== But I have changes that I've not committed If you are migrating from a tree that has changes you've not yet committed to FreeBSD, you'll need to follow the steps from the previous section first, and then follow these. [source,shell] .... % cd path-to-svn-checkout-tree % svn diff > /tmp/src.diff % cd _mumble_/freebsd-src % git checkout -b working .... This will create a diff of your current changes. The last command creates a branch called `working` though you can call it whatever you want. [source,shell] .... % git apply /tmp/src.diff .... this will apply all your pending changes to the working tree. This doesn't commit the change, so you'll need to make this permanent: [source,shell] .... % git add _files_ % git commit .... The last command will commit these changes to the branch. The editor will prompt you for a commit message. Enter one as if you were committing to FreeBSD. At this point, your work is preserved, and in the Git repository. ===== Keeping current So, time passes. It's time now to update the tree for the latest changes upstream. When you checkout `main` make sure that you have no diffs. It's a lot easier to commit those to a branch (or use `git stash`) before doing the following. If you are used to `git pull`, we strongly recommend using the `--ff-only` option, and further setting it as the default option. Alternatively, `git pull --rebase` is useful if you have changes staged in the main branch. [source,shell] .... % git config --global pull.ff only .... You may need to omit the --global if you want this setting to apply to only this repository. [source,shell] .... % cd freebsd-src % git checkout main % git pull (--ff-only|--rebase) .... There is a common trap, that the combination command `git pull` will try to perform a merge, which would sometimes creates a merge commit that didn't exist before. This can be harder to recover from. The longer form is also recommended. [source,shell] .... % cd freebsd-src % git checkout main % git fetch freebsd % git merge --ff-only freebsd/main .... These commands reset your tree to the main branch, and then update it from where you pulled the tree from originally. It's important to switch to `main` before doing this so it moves forward. Now, it's time to move the changes forward: [source,shell] .... % git rebase -i main working .... This will bring up an interactive screen to change the defaults. For now, just exit the editor. Everything should just apply. If not, then you'll need to resolve the diffs. https://docs.github.com/en/free-pro-team@latest/github/using-git/resolving-merge-conflicts-after-a-git-rebase[This github document] can help you navigate this process. ===== Time to push changes upstream First, ensure that the push URL is properly configured for the upstream repository. [source,shell] .... % git remote set-url --push freebsd ssh://git@gitrepo.freebsd.org/src.git .... Then, verify that user name and email are configured right. We require that they exactly match the passwd entry in FreeBSD cluster. Use [source,shell] .... freefall% gen-gitconfig.sh .... on freefall.freebsd.org to get a recipe that you can use directly, assuming /usr/local/bin is in the PATH. The below command merges the 'working' branch into the upstream main line. It's important that you curate your changes to be just like you want them in the FreeBSD source repo before doing this. [source,shell] .... % git push freebsd working:main .... If your push is rejected due to losing a commit race, rebase your branch before trying again: [source,shell] .... % git checkout working % git fetch freebsd % git rebase freebsd/main % git push freebsd working:main .... ===== Finding the Subversion Revision You'll need to make sure that you've fetched the notes (see the `No staged changes migration` section above for details. Once you have these, notes will show up in the git log command like so: [source,shell] .... % git log .... If you have a specific version in mind, you can use this construct: [source,shell] .... % git log --grep revision=XXXX .... to find the specific revision. The hex number after 'commit' is the hash you can use to refer to this commit. ==== Migrating from GitHub fork Note: as of this writing, https://github.com/freebsd/freebsd-src is mirroring all official branches, along with a `master` branch which is the legacy svn2git result. The `master` branch will not be updated anymore, and the link:https://github.com/freebsd/freebsd-src/commit/de1aa3dab23c06fec962a14da3e7b4755c5880cf[last commit] contains the instructions for migrating to the new `main` branch. We'll retain the `master` branch for a certain time, but in the future it will only be kept in the link:https://github.com/freebsd/freebsd-legacy[freebsd-legacy] repository. When migrating branches from a GitHub fork from the old GitHub mirror to the official repo, the process is straight forward. This assumes that you have a `freebsd` upstream pointing to GitHub, adjust if necessary. This also assumes a clean tree before starting... ===== Add the new `freebsd` upstream repository: [source,shell] .... % git remote add freebsd https://git.freebsd.org/src.git % git fetch freebsd % git checkout --track freebsd/main .... ===== Rebase all your WIP branches. For each branch FOO, do the following after fetching the `freebsd` sources and creating a local `main` branch with the above checkout: [source,shell] .... % git rebase -i freebsd/master FOO --onto main .... And you'll now be tracking the official repository. You can then follow the `Keeping Current` section above to stay up to date. If you need to then commit work to FreeBSD, you can do so following the `Time to push changes upstream` instructions. You'll need to do the following once to update the push URL if you are a FreeBSD committer: [source,shell] .... % git remote set-url --push freebsd ssh://git@gitrepo.freebsd.org/src.git .... (note that gitrepo.freebsd.org will be change to repo.freebsd.org in the future.) You will also need to add `freebsd` as the location to push to. The author recommends that your upstream GitHub repository remain the default push location so that you only push things into FreeBSD you intend to by making it explicit. [[git-faq]] === Git FAQ This section provides a number of targeted answers to questions that are likely to come up often for users and developers. [NOTE] ==== We use the common convention of having the origin for the FreeBSD repository being 'freebsd' rather than the default 'origin' to allow people to use that for their own development and to minimize "whoopse" pushes to the wrong repository. ==== ==== Users ===== How do I track -current and -stable with only one copy of the repository? **Q:** Although disk space is not a huge issue, it's more efficient to use only one copy of the repository. With SVN mirroring, I could checkout multiple trees from the same repository. How do I do this with Git? **A:** You can use Git worktrees. There's a number of ways to do this, but the simplest way is to use a clone to track -current, and a worktree to track stable releases. While using a 'bare repository' has been put forward as a way to cope, it's more complicated and will not be documented here. First, you need to clone the FreeBSD repository, shown here cloning into `freebsd-current` to reduce confusion. $URL is whatever mirror works best for you: [source,shell] .... % git clone -o freebsd --config remote.freebsd.fetch='+refs/notes/*:refs/notes/*' $URL freebsd-current .... then once that's cloned, you can simply create a worktree from it: [source,shell] .... % cd freebsd-current % git worktree add ../freebsd-stable-12 stable/12 .... this will checkout `stable/12` into a directory named `freebsd-stable-12` that's a peer to the `freebsd-current` directory. Once created, it's updated very similarly to how you might expect: [source,shell] .... % cd freebsd-current % git checkout main % git pull --ff-only # changes from upstream now local and current tree updated % cd ../freebsd-stable-12 % git merge --ff-only freebsd/stable/12 # now your stable/12 is up to date too .... I recommend using `--ff-only` because it's safer and you avoid accidentally getting into a 'merge nightmare' where you have an extra change in your tree, forcing a complicated merge rather than a simple one. Here's https://adventurist.me/posts/00296[a good writeup] that goes into more detail. ==== Developers ===== Ooops! I committed to `main` instead of a branch. **Q:** From time to time, I goof up and commit to main instead of to a branch. What do I do? **A:** First, don't panic. Second, don't push. In fact, you can fix almost anything if you haven't pushed. All the answers in this section assume no push has happened. The following answer assumes you committed to `main` and want to create a branch called `issue`: [source,shell] .... % git branch issue # Create the 'issue' branch % git reset --hard freebsd/main # Reset 'main' back to the official tip % git checkout issue # Back to where you were .... ===== Ooops! I committed something to the wrong branch! **Q:** I was working on feature on the `wilma` branch, but accidentally committed a change relevant to the `fred` branch in 'wilma'. What do I do? **A:** The answer is similar to the previous one, but with cherry picking. This assumes there's only one commit on wilma, but will generalize to more complicated situations. It also assumes that it's the last commit on wilma (hence using wilma in the `git cherry-pick` command), but that too can be generalized. [source,shell] .... # We're on branch wilma % git checkout fred # move to fred branch % git cherry-pick wilma # copy the misplaced commit % git checkout wilma # go back to wilma branch % git reset --hard HEAD^ # move what wilma refers to back 1 commit .... Git experts would first rewind the wilma branch by 1 commit, switch over to fred and then use `git reflog` to see what that 1 deleted commit was and cherry-pick it over. **Q:** But what if I want to commit a few changes to `main`, but keep the rest in `wilma` for some reason? **A:** The same technique above also works if you are wanting to 'land' parts of the branch you are working on into `main` before the rest of the branch is ready (say you noticed an unrelated typo, or fixed an incidental bug). You can cherry pick those changes into main, then push to the parent repository. Once you've done that, cleanup couldn't be simpler: just `git rebase -i`. Git will notice you've done this and skip the common changes automatically (even if you had to change the commit message or tweak the commit slightly). There's no need to switch back to wilma to adjust it: just rebase! **Q:** I want to split off some changes from branch `wilma` into branch `fred` **A:** The more general answer would be the same as the previous. You'd checkout/create the `fred` branch, cherry pick the changes you want from `wilma` one at a time, then rebase `wilma` to remove those changes you cherry picked. `git rebase -i main wilma` will toss you into an editor, and remove the `pick` lines that correspond to the commits you copied to `fred`. If all goes well, and there are no conflicts, you're done. If not, you'll need to resolve the conflicts as you go. The other way to do this would be to checkout `wilma` and then create the branch `fred` to point to the same point in the tree. You can then `git rebase -i` both these branches, selecting the changes you want in `fred` or `wilma` by retaining the pick likes, and deleting the rest from the editor. Some people would create a tag/branch called `pre-split` before starting in case something goes wrong in the split. You can undo it with the following sequence: [source,shell] .... % git checkout pre-split # Go back % git branch -D fred # delete the fred branch % git checkout -B wilma # reset the wilma branch % git branch -d pre-split # Pretend it didn't happen .... The last step is optional. If you are going to try again to split, you'd omit it. **Q:** But I did things as I read along and didn't see your advice at the end to create a branch, and now `fred` and `wilma` are all screwed up. How do I find what `wilma` was before I started. I don't know how many times I moved things around. **A:** All is not lost. You can figure out it, so long as it hasn't been too long, or too many commits (hundreds). So I created a wilma branch and committed a couple of things to it, then decided I wanted to split it into fred and wilma. Nothing weird happened when I did that, but let's say it did. The way to look at what you've done is with the `git reflog`: [source,shell] .... % git reflog 6ff9c25 (HEAD -> wilma) HEAD@{0}: rebase -i (finish): returning to refs/heads/wilma 6ff9c25 (HEAD -> wilma) HEAD@{1}: rebase -i (start): checkout main 869cbd3 HEAD@{2}: rebase -i (start): checkout wilma a6a5094 (fred) HEAD@{3}: rebase -i (finish): returning to refs/heads/fred a6a5094 (fred) HEAD@{4}: rebase -i (pick): Encourage contributions 1ccd109 (freebsd/main, main) HEAD@{5}: rebase -i (start): checkout main 869cbd3 HEAD@{6}: rebase -i (start): checkout fred 869cbd3 HEAD@{7}: checkout: moving from wilma to fred 869cbd3 HEAD@{8}: commit: Encourage contributions ... % .... Here we see the changes I've made. You can use it to figure out where things went wrong. I'll just point out a few things here. The first one is that HEAD@{X} is a 'commitish' thing, so you can use that as an argument to a command. Although if that command commits anything to the repository, the X numbers change. You can also use the hash (first column). Next, 'Encourage contributions' was the last commit I made to `wilma` before I decided to split things up. You can also see the same hash is there when I created the `fred` branch to do that. I started by rebasing `fred` and you see the 'start', each step, and the 'finish' for that process. While we don't need it here, you can figure out exactly what happened. Fortunately, to fix this, you can follow the prior answer's steps, but with the hash `869cbd3` instead of `pre-split`. While that seems a bit verbose, it's easy to remember since you're doing one thing at a time. You can also stack: [source,shell] .... % git checkout -B wilma 869cbd3 % git branch -D fred .... and you are ready to try again. The 'checkout -B' with the hash combines checking out and creating a branch for it. The -B instead of -b forces the movement of a pre-existing branch. Either way works, which is what's great (and awful) about Git. One reason I tend to use `git checkout -B xxxx hash` instead of checking out the hash, and then creating / moving the branch is purely to avoid the slightly distressing message about detached heads: [source,shell] .... % git checkout 869cbd3 M faq.md Note: checking out '869cbd3'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example: git checkout -b HEAD is now at 869cbd3 Encourage contributions % git checkout -B wilma .... this produces the same effect, but I have to read a lot more and severed heads aren't an image I like to contemplate. ===== Ooops! I did a `git pull` and it created a merge commit, what do I do? **Q:** I was on autopilot and did a `git pull` for my development tree and that created a merge commit on the mainline. How do I recover? **A:** This can happen when you invoke the pull with your development branch checked out. Right after the pull, you will have the new merge commit checked out. Git supports a `HEAD^#` syntax to examine the parents of a merge commit: [source,shell] .... git log --oneline HEAD^1 # Look at the first parent's commits git log --oneline HEAD^2 # Look at the second parent's commits .... From those logs, you can easily identify which commit is your development work. Then you simply reset your branch to the corresponding `HEAD^#`: [source,shell] .... git reset --hard HEAD^2 .... **Q:** But I also need to fix my `main` branch. How do I do that? **A:** Git keeps track of the remote repository branches in a `freebsd/` namespace. To fix your `main` branch, just make it point to the remote's `main`: [source,shell] .... git branch -f main freebsd/main .... There's nothing magical about branches in Git: they are just labels on a graph that are automatically moved forward by making commits. So the above works because you're just moving a label. There's no metadata about the branch that needs to be preserved due to this. ===== Mixing and matching branches **Q:** So I have two branches `worker` and `async` that I'd like to combine into one branch called `feature` while maintaining the commits in both. **A:** This is a job for cherry pick. [source,shell] .... % git checkout worker % git checkout -b feature # create a new branch % git cherry-pick main..async # bring in the changes .... You now have a new branch called `feature`. This branch combines commits from both branches. You can further curate it with `git rebase`. **Q:** I have a branch called `driver` and I'd like to break it up into `kernel` and `userland` so I can evolve them separately and commit each branch as it becomes ready. **A:** This takes a little bit of prep work, but `git rebase` will do the heavy lifting here. [source,shell] .... % git checkout driver # Checkout the driver % git checkout -b kernel # Create kernel branch % git checkout -b userland # Create userland branch .... Now you have two identical branches. So, it's time to separate out the commits. We'll assume first that all the commits in `driver` go into either the `kernel` or the `userland` branch, but not both. [source,shell] .... % git rebase -i main kernel .... and just include the changes you want (with a 'p' or 'pick' line) and just delete the commits you don't (this sounds scary, but if worse comes to worse, you can throw this all away and start over with the `driver` branch since you've not yet moved it). [source,shell] .... % git rebase -i main userland .... and do the same thing you did with the `kernel` branch. **Q:** Oh great! I followed the above and forgot a commit in the `kernel` branch. How do I recover? **A:** You can use the `driver` branch to find the hash of the commit is missing and cherry pick it. [source,shell] .... % git checkout kernel % git log driver % git cherry-pick $HASH .... **Q:** OK. I have the same situation as the above, but my commits are all mixed up. I need parts of one commit to go to one branch and the rest to go to the other. In fact, I have several. Your rebase method to select sounds tricky. **A:** In this situation, you'd be better off to curate the original branch to separate out the commits, and then use the above method to split the branch. So let's assume that there's just one commit with a clean tree. You can either use `git rebase` with an `edit` line, or you can use this with the commit on the tip. The steps are the same either way. The first thing we need to do is to back up one commit while leaving the changes uncommitted in the tree: [source,shell] .... % git reset HEAD^ .... Note: Do not, repeat do not, add `--hard` here since that also removes the changes from your tree. Now, if you are lucky, the change needing to be split up falls entirely along file lines. In that case you can just do the usual `git add` for the files in each group than do a `git commit`. Note: when you do this, you'll lose the commit message when you do the reset, so if you need it for some reason, you should save a copy (though `git log $HASH` can recover it). If you are not lucky, you'll need to split apart files. There's another tool to do that which you can apply one file at a time. [source,shell] .... git add -i foo/bar.c .... will step through the diffs, prompting you, one at time, whether to include or exclude the hunk. Once you're done, `git commit` and you'll have the remainder in your tree. You can run it multiple times as well, and even over multiple files (though I find it easier to do one file at a time and use the `git rebase -i` to fold the related commits together). ==== Cloning and Mirroring **Q:** I'd like to mirror the entire Git repository, how do I do that? **A:** If all you want to do is mirror, then [source,shell] .... % git clone --mirror $URL .... will do the trick. However, there are two disadvantages to this if you want to use it for anything other than a mirror you'll reclone. First, this is a 'bare repository' which has the repository database, but no checked out worktree. This is great for mirroring, but terrible for day to day work. There's a number of ways around this with 'git worktree': [source,shell] .... % git clone --mirror https://git.freebsd.org/ports.git ports.git % cd ports.git % git worktree add ../ports main % git worktree add ../quarterly branches/2020Q4 % cd ../ports .... But if you aren't using your mirror for further local clones, then it's a poor match. The second disadvantage is that Git normally rewrites the refs (branch name, tags, etc) from upstream so that your local refs can evolve independently of upstream. This means that you'll lose changes if you are committing to this repository on anything other than private project branches. **Q:** So what can I do instead? **A:** Well, you can stuff all of the upstream repository's refs into a private namespace in your local repository. Git clones everything via a 'refspec' and the default refspec is: [source,shell] .... fetch = +refs/heads/*:refs/remotes/freebsd/* .... which says just fetch the branch refs. However, the FreeBSD repository has a number of other things in it. To see those, you can add explicit refspecs for each ref namespace, or you can fetch everything. To setup your repository to do that: [source,shell] .... git config --add remote.freebsd.fetch '+refs/*:refs/freebsd/*' .... which will put everything in the upstream repository into your local repository's 'refs/freebsd/' namespace. Please note, that this also grabs all the unconverted vendor branches and the number of refs associated with them is quite large. You'll need to refer to these 'refs' with their full name because they aren't in and of Git's regular namespaces. [source,shell] .... git log refs/freebsd/vendor/zlib/1.2.10 .... would look at the log for the vendor branch for zlib starting at 1.2.10. === Collaborating with others One of the keys to good software development on a project as large as FreeBSD is the ability to collaborate with others before you push your changes to the tree. The FreeBSD project's Git repositories do not, yet, allow user created branches to be pushed to the repository, and therefore if you wish to share your changes with others you must use another mechanism, such as a hosted GitLab or GitHub, in order to share changes in a user generated branch. The following instructions show how to set up a user generated branch, based on the FreeBSD main branch, and push it to GitHub. Before you begin, make sure that your local Git repo is up to date and has the correct origins set <> [source,shell] ```` % git remote -v freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) ```` The first step is to create a fork of https://github.com/freebsd/freebsd-src[FreeBSD] on GitHub following these https://docs.github.com/en/github/getting-started-with-github/fork-a-repo[guidelines]. The destination of the fork should be your own, personal, GitHub account (gvnn3 in my case). Now add a remote on your local system that points to your fork: [source,shell] .... % git remote add github git@github.com:gvnn3/freebsd-src.git % git remote -v github git@github.com:gvnn3/freebsd-src.git (fetch) github git@github.com:gvnn3/freebsd-src.git (push) freebsd https://git.freebsd.org/src.git (fetch) freebsd ssh://git@gitrepo.freebsd.org/src.git (push) .... With this in place you can create a branch <> [source,shell] .... % git checkout -b gnn-github .... Make whatever modifications you wish in your branch. Build, test, and once you're ready to collaborate with others it's time to push your changes into your hosted branch. Before you can push you'll have to set the appropriate upstream, as Git will tell you the first time you try to push to your +github+ remote: [source,shell] .... % git push github fatal: The current branch gnn-github has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream github gnn-github .... Setting the push as +git+ advises allows it to succeed: [source,shell] .... % git push --set-upstream github gnn-feature Enumerating objects: 20486, done. Counting objects: 100% (20486/20486), done. Delta compression using up to 8 threads Compressing objects: 100% (12202/12202), done. Writing objects: 100% (20180/20180), 56.25 MiB | 13.15 MiB/s, done. Total 20180 (delta 11316), reused 12972 (delta 7770), pack-reused 0 remote: Resolving deltas: 100% (11316/11316), completed with 247 local objects. remote: remote: Create a pull request for 'gnn-feature' on GitHub by visiting: remote: https://github.com/gvnn3/freebsd-src/pull/new/gnn-feature remote: To github.com:gvnn3/freebsd-src.git * [new branch] gnn-feature -> gnn-feature Branch 'gnn-feature' set up to track remote branch 'gnn-feature' from 'github'. .... Subsequent changes to the same branch will push correctly by default: [source,shell] .... % git push Enumerating objects: 4, done. Counting objects: 100% (4/4), done. Delta compression using up to 8 threads Compressing objects: 100% (2/2), done. Writing objects: 100% (3/3), 314 bytes | 1024 bytes/s, done. Total 3 (delta 1), reused 1 (delta 0), pack-reused 0 remote: Resolving deltas: 100% (1/1), completed with 1 local object. To github.com:gvnn3/freebsd-src.git 9e5243d7b659..cf6aeb8d7dda gnn-feature -> gnn-feature .... At this point your work is now in your branch on +GitHub+ and you can share the link with other collaborators. [[vcs-history]] == Version Control History The project has moved to <>. The FreeBSD source repository switched from CVS to Subversion on May 31st, 2008. The first real SVN commit is __r179447__. The source repository switched from Subversion to Git on December 23rd, 2020. The last real svn commit is __r368820__. The first real git commit hash is __5ef5f51d2bef80b0ede9b10ad5b0e9440b60518c__ The FreeBSD `doc/www` repository switched from CVS to Subversion on May 19th, 2012. The first real SVN commit is __r38821__. The documentation repository switched from Subversion to Git on December 8th, 2020. The last SVN commit is __r54737__. The first real git commit hash is __3be01a475855e7511ad755b2defd2e0da5d58bbe__. The FreeBSD `ports` repository switched from CVS to Subversion on July 14th, 2012. The first real SVN commit is __r300894__. The ports repository switched from Subversion to Git on April 6, 2021. The last SVN commit is __r569609__ The first real git commit hash is __ed8d3eda309dd863fb66e04bccaa513eee255cbf__. [[conventions]] == Setup, Conventions, and Traditions There are a number of things to do as a new developer. The first set of steps is specific to committers only. These steps must be done by a mentor for those who are not committers. [[conventions-committers]] === For New Committers Those who have been given commit rights to the FreeBSD repositories must follow these steps. * Get mentor approval before committing each of these changes! * The [.filename]#.ent# and [.filename]#.xml# files mentioned below exist in the FreeBSD Documentation Project SVN repository at `svn+ssh://repo.FreeBSD.org/doc/`. * All [.filename]#src# commits go to FreeBSD-CURRENT first before being merged to FreeBSD-STABLE. The FreeBSD-STABLE branch must maintain ABI and API compatibility with earlier versions of that branch. Do not merge changes that break this compatibility. [[commit-steps]] [.procedure] ==== *Procedure 1. Steps for New Committers* . Add an Author Entity + [.filename]#doc/shared/authors.adoc# - Add an author entity. Later steps depend on this entity, and missing this step will cause the [.filename]#doc/# build to fail. This is a relatively easy task, but remains a good first test of version control skills. . Update the List of Developers and Contributors + [.filename]#doc/en/articles/contributors/contrib-committers.adoc# - Add an entry to the "Developers" section of the link:{contributors}#staff-committers[Contributors List]. Entries are sorted by last name. + [.filename]#doc/en/articles/contributors/contrib-additional.adoc# - _Remove_ the entry from the "Additional Contributors" section. Entries are sorted by first name. . Add a News Item + [.filename]#website/data/en/news/news.toml# - Add an entry. Look for the other entries that announce new committers and follow the format. Use the date from the commit bit approval email from mailto:core@FreeBSD.org[core@FreeBSD.org]. . Add a PGP Key + `{des}` has written a shell script ([.filename]#documentation/tools/addkey.sh#) to make this easier. See the https://cgit.freebsd.org/doc/plain/documentation/static/pgpkeys/README[README] file for more information. + Use [.filename]#documentation/tools/checkkey.sh# to verify that keys meet minimal best-practices standards. + After adding and checking a key, add both updated files to source control and then commit them. Entries in this file are sorted by last name. + [NOTE] ====== It is very important to have a current PGP/GnuPG key in the repository. The key may be required for positive identification of a committer. For example, the `{admins}` might need it for account recovery. A complete keyring of `FreeBSD.org` users is available for download from link:https://www.FreeBSD.org/doc/pgpkeyring.txt[https://www.FreeBSD.org/doc/pgpkeyring.txt]. ====== . Update Mentor and Mentee Information + [.filename]#base/head/share/misc/committers-repository.dot# - Add an entry to the current committers section, where _repository_ is `doc`, `ports`, or `src`, depending on the commit privileges granted. + Add an entry for each additional mentor/mentee relationship in the bottom section. . Generate a Kerberos Password + See <> to generate or set a Kerberos for use with other FreeBSD services like the bug tracking database. . Optional: Enable Wiki Account + https://wiki.freebsd.org[FreeBSD Wiki] Account - A wiki account allows sharing projects and ideas. Those who do not yet have an account can follow instructions on the https://wiki.freebsd.org/AboutWiki[AboutWiki Page] to obtain one. Contact mailto:wiki-admin@FreeBSD.org[wiki-admin@FreeBSD.org] if you need help with your Wiki account. . Optional: Update Wiki Information + Wiki Information - After gaining access to the wiki, some people add entries to the https://wiki.freebsd.org/HowWeGotHere[How We Got Here], https://wiki.freebsd.org/IRC/Nicknames[IRC Nicks], and https://wiki.freebsd.org/Community/Dogs[Dogs of FreeBSD] pages. . Optional: Update Ports with Personal Information + [.filename]#ports/astro/xearth/files/freebsd.committers.markers# and [.filename]#src/usr.bin/calendar/calendars/calendar.freebsd# - Some people add entries for themselves to these files to show where they are located or the date of their birthday. . Optional: Prevent Duplicate Mailings + Subscribers to {dev-commits-doc-all}, {dev-commits-ports-all} or {dev-commits-src-all} might wish to unsubscribe to avoid receiving duplicate copies of commit messages and followups. ==== [[conventions-everyone]] === For Everyone [[conventions-everyone-steps]] [.procedure] ==== . Introduce yourself to the other developers, otherwise no one will have any idea who you are or what you are working on. The introduction need not be a comprehensive biography, just write a paragraph or two about who you are, what you plan to be working on as a developer in FreeBSD, and who will be your mentor. Email this to the {developers-name} and you will be on your way! . Log into `freefall.FreeBSD.org` and create a [.filename]#/var/forward/user# (where _user_ is your username) file containing the e-mail address where you want mail addressed to _yourusername_@FreeBSD.org to be forwarded. This includes all of the commit messages as well as any other mail addressed to the {committers-name} and the {developers-name}. Really large mailboxes which have taken up permanent residence on `freefall` may get truncated without warning if space needs to be freed, so forward it or save it elsewhere. + [NOTE] ====== If your e-mail system uses SPF with strict rules, you should whitelist `mx2.FreeBSD.org` from SPF checks. ====== + Due to the severe load dealing with SPAM places on the central mail servers that do the mailing list processing, the front-end server does do some basic checks and will drop some messages based on these checks. At the moment proper DNS information for the connecting host is the only check in place but that may change. Some people blame these checks for bouncing valid email. To have these checks turned off for your email, create a file named [.filename]#~/.spam_lover# on `freefall.FreeBSD.org`. + [NOTE] ====== Those who are developers but not committers will not be subscribed to the committers or developers mailing lists. The subscriptions are derived from the access rights. ====== ==== [[smtp-setup]] ==== SMTP Access Setup For those willing to send e-mail messages through the FreeBSD.org infrastructure, follow the instructions below: [.procedure] ==== . Point your mail client at `smtp.FreeBSD.org:587`. . Enable STARTTLS. . Ensure your `From:` address is set to `_yourusername_@FreeBSD.org`. . For authentication, you can use your FreeBSD Kerberos username and password (see <>). The `_yourusername_/mail` principal is preferred, as it is only valid for authenticating to mail resources. + [NOTE] ====== Do not include `@FreeBSD.org` when entering in your username. ====== + .Additional Notes [NOTE] ====== * Will only accept mail from `_yourusername_@FreeBSD.org`. If you are authenticated as one user, you are not permitted to send mail from another. * A header will be appended with the SASL username: (`Authenticated sender: _username_`). * Host has various rate limits in place to cut down on brute force attempts. ====== ==== [[smtp-setup-local-mta]] ===== Using a Local MTA to Forward Emails to the FreeBSD.org SMTP Service It is also possible to use a local MTA to forward locally sent emails to the FreeBSD.org SMTP servers. [[smtp-setup-local-postfix]] .Using Postfix [example] ==== To tell a local Postfix instance that anything from `_yourusername_@FreeBSD.org` should be forwarded to the FreeBSD.org servers, add this to your [.filename]#main.cf#: [.programlisting] .... sender_dependent_relayhost_maps = hash:/usr/local/etc/postfix/relayhost_maps smtp_sasl_auth_enable = yes smtp_sasl_security_options = noanonymous smtp_sasl_password_maps = hash:/usr/local/etc/postfix/sasl_passwd smtp_use_tls = yes .... Create [.filename]#/usr/local/etc/postfix/relayhost_maps# with the following content: [.programlisting] .... yourusername@FreeBSD.org [smtp.freebsd.org]:587 .... Create [.filename]#/usr/local/etc/postfix/sasl_passwd# with the following content: [.programlisting] .... [smtp.freebsd.org]:587 yourusername:yourpassword .... If the email server is used by other people, you may want to prevent them from sending e-mails from your address. To achieve this, add this to your [.filename]#main.cf#: [.programlisting] .... smtpd_sender_login_maps = hash:/usr/local/etc/postfix/sender_login_maps smtpd_sender_restrictions = reject_known_sender_login_mismatch .... Create [.filename]#/usr/local/etc/postfix/sender_login_maps# with the following content: [.programlisting] .... yourusername@FreeBSD.org yourlocalusername .... Where _yourlocalusername_ is the SASL username used to connect to the local instance of Postfix. ==== [[smtp-setup-local-opensmtpd]] .Using OpenSMTPD [example] ==== To tell a local OpenSMTPD instance that anything from `_yourusername_@FreeBSD.org` should be forwarded to the FreeBSD.org servers, add this to your [.filename]#smtpd.conf#: [.programlisting] .... action "freebsd" relay host smtp+tls://freebsd@smtp.freebsd.org:587 auth match from any auth yourlocalusername mail-from "_yourusername_@freebsd.org" for any action "freebsd" .... Where _yourlocalusername_ is the SASL username used to connect to the local instance of OpenSMTPD. Create [.filename]#/usr/local/etc/mail/secrets# with the following content: [.programlisting] .... freebsd yourusername:yourpassword .... ==== [[mentors]] === Mentors All new developers have a mentor assigned to them for the first few months. A mentor is responsible for teaching the mentee the rules and conventions of the project and guiding their first steps in the developer community. The mentor is also personally responsible for the mentee's actions during this initial period. For committers: do not commit anything without first getting mentor approval. Document that approval with an `Approved by:` line in the commit message. When the mentor decides that a mentee has learned the ropes and is ready to commit on their own, the mentor announces it with a commit to [.filename]#conf/mentors#. This file is in the [.filename]#svnadmin# branch of each repository: [.informaltable] [cols="1,1", frame="none"] |=== |`src` |[.filename]#base/svnadmin/conf/mentors# |`doc` |[.filename]#doc/svnadmin/conf/mentors# |`ports` |[.filename]#ports/svnadmin/conf/mentors# |=== [[pre-commit-review]] == Pre-Commit Review Code review is one way to increase the quality of software. The following guidelines apply to commits to the `head` (-CURRENT) branch of the `src` repository. Other branches and the `ports` and `docs` trees have their own review policies, but these guidelines generally apply to commits requiring review: * All non-trivial changes should be reviewed before they are committed to the repository. * Reviews may be conducted by email, in Bugzilla, in Phabricator, or by another mechanism. Where possible, reviews should be public. * The developer responsible for a code change is also responsible for making all necessary review-related changes. * Code review can be an iterative process, which continues until the patch is ready to be committed. Specifically, once a patch is sent out for review, it should receive an explicit "looks good" before it is committed. So long as it is explicit, this can take whatever form makes sense for the review method. * Timeouts are not a substitute for review. Sometimes code reviews will take longer than you would hope for, especially for larger features. Accepted ways to speed up review times for your patches are: * Review other people's patches. If you help out, everybody will be more willing to do the same for you; goodwill is our currency. * Ping the patch. If it is urgent, provide reasons why it is important to you to get this patch landed and ping it every couple of days. If it is not urgent, the common courtesy ping rate is one week. Remember that you are asking for valuable time from other professional developers. * Ask for help on mailing lists, IRC, etc. Others may be able to either help you directly, or suggest a reviewer. * Split your patch into multiple smaller patches that build on each other. The smaller your patch, the higher the probability that somebody will take a quick look at it. + When making large changes, it is helpful to keep this in mind from the beginning of the effort as breaking large changes into smaller ones is often difficult after the fact. Developers should participate in code reviews as both reviewers and reviewees. If someone is kind enough to review your code, you should return the favor for someone else. Note that while anyone is welcome to review and give feedback on a patch, only an appropriate subject-matter expert can approve a change. This will usually be a committer who works with the code in question on a regular basis. In some cases, no subject-matter expert may be available. In those cases, a review by an experienced developer is sufficient when coupled with appropriate testing. [[commit-log-message]] == Commit Log Messages This section contains some suggestions and traditions for how commit logs are formatted. === Why are commit messages important? When you commit a change in Git, Subversion, or another version control system (VCS), you're prompted to write some text describing the commit -- a commit message. How important is this commit message? Should you spend some significant effort writing it? Does it really matter if you write simply fixed a bug? Most projects have more than one developer and last for some length of time. Commit messages are a very important method of communicating with other developers, in the present and for the future. FreeBSD has hundreds of active developers and hundreds of thousands of commits spanning decades of history. Over that time the developer community has learned how valuable good commit messages are; sometimes these are hard-learned lessons. Commit messages serve at least three purposes: * Communicating with other developers + FreeBSD commits generate email to various mailing lists. These include the commit message along with a copy of the patch itself. Commit messages are also viewed through commands like git log. These serve to make other developers aware of changes that are ongoing; that other developer may want to test the change, may have an interest in the topic and will want to review in more detail, or may have their own projects underway that would benefit from interaction. * Making Changes Discoverable + In a large project with a long history it may be difficult to find changes of interest when investigating an issue or change in behaviour. Verbose, detailed commit messages allow searches for changes that might be relevant. For example, `git log --since 1year --grep 'USB timeout'`. * Providing historical documentation + Commit messages serve to document changes for future developers, perhaps years or decades later. This future developer may even be you, the original author. A change that seems obvious today may be decidedly not so much later on. The `git blame` command annotates each line of a source file with the change (hash and subject line) that brought it in. Having established the importance, here are elements of a good FreeBSD commit message: === Start with a subject line Commit messages should start with a single-line subject that briefly summarizes the change. The subject should, by itself, allow the reader to quickly determine if the change is of interest or not. === Keep subject lines short The subject line should be as short as possible while still retaining the required information. This is to make browsing Git log more efficient, and so that git log --oneline can display the short hash and subject on a single 80-column line. A good rule of thumb is to stay below 63 characters, and aim for about 50 or fewer if possible. === Prefix the subject line with a component, if applicable If the change relates to a specific component the subject line may be prefixed with that component name and a colon (:). ✓ `foo: Add -k option to keep temporary data` Include the prefix in the 63-character limit suggested above, so that `git log --oneline` avoids wrapping. === Capitalize the first letter of the subject Capitalize the first letter of the subject itself. The prefix, if any, is not capitalized unless necessary (e.g., `USB:` is capitalized). === Do not end the subject line with punctuation Do not end with a period or other punctuation. In this regard the subject line is like a newspaper headline. === Separate the subject and body with a blank line Separate the body from the subject with a blank line. Some trivial commits do not require a body, and will have only a subject. ✓ `ls: Fix typo in usage text` === Limit messages to 72 columns `git log` and `git format-patch` indent the commit message by four spaces. Wrapping at 72 columns provides a matching margin on the right edge. Limiting messages to 72 characters also keeps the commit message in formatted patches below RFC 2822's suggested email line length limit of 78 characters. This limit works well with a variety of tools that may render commit messages; line wrapping might be inconsistent with longer line length. === Use the present tense, imperative mood This facilitates short subject lines and provides consistency, including with automatically generated commit messages (e.g., as generated by git revert). This is important when reading a list of commit subjects. Think of the subject as finishing the sentence "when applied, this change will ...". ✓ `foo: Implement the -k (keep) option` + ✗ `foo: Implemented the -k option` + ✗ `This change implements the -k option in foo` + ✗ `-k option added` === Focus on what and why, not how Explain what the change accomplishes and why it is being done, rather than how. Do not assume that the reader is familiar with the issue. Explain the background and motivation for the change. Include benchmark data if you have it. If there are limitations or incomplete aspects of the change, describe them in the commit message. === Consider whether parts of the commit message could be code comments instead Sometimes while writing a commit message you may find yourself writing a sentence or two explaining some tricky or confusing aspect of the change. When this happens consider whether it would be valuable to have that explanation as a comment in the code itself. === Write commit messages for your future self While writing the commit message for a change you have all of the context in mind - what prompted the change, alternate approaches that were considered and rejected, limitations of the change, and so on. Imagine yourself revisiting the change a year or two in the future, and write the commit message in a way that would provide that necessary context. === Commit messages should stand alone You may include references to mailing list postings, benchmark result web sites, or code review links. However, the commit message should contain all of the relevant information in case these references are no longer available in the future. Similarly, a commit may refer to a previous commit, for example in the case of a bug fix or revert. In addition to the commit identifier (revision or hash), include the subject line from the referenced commit (or another suitable brief reference). With each VCS migration (from CVS to Subversion to Git) revision identifiers from previous systems may become difficult to follow. === Include appropriate metadata in a footer As well as including an informative message with each commit, some additional information may be needed. This information consists of one or more lines containing the key word or phrase, a colon, tabs for formatting, and then the additional information. The key words or phrases are: [.informaltable] [cols="20%,80%", frame="none"] |=== |`PR:` |The problem report (if any) which is affected (typically, by being closed) by this commit. Multiple PRs may be specified on one line, separated by commas or spaces. |`Reported by:` |The name and e-mail address of the person that reported the issue; for developers, just the username on the FreeBSD cluster. Typically used when there is no PR, for example if the issue was reported on a mailing list. |`Submitted by:` |The name and e-mail address of the person that submitted the fix; for developers, just the username on the FreeBSD cluster. Typically not used with Git; submitted patches should have the author set by using `git commit --author`. If the submitter is the maintainer of the port being committed, include "(maintainer)" after the email address. Avoid obfuscating the email address of the submitter as this adds additional work when searching logs. |`Reviewed by:` |The name and e-mail address of the person or people that reviewed the change; for developers, just the username on the FreeBSD cluster. If a patch was submitted to a mailing list for review, and the review was favorable, then just include the list name. |`Tested by:` |The name and e-mail address of the person or people that tested the change; for developers, just the username on the FreeBSD cluster. |`Approved by:` a| The name and e-mail address of the person or people that approved the change; for developers, just the username on the FreeBSD cluster. There are several cases where approval is customary: * while a new committer is under mentorship * commits to an area of the tree to which you do not usually commit * during a release cycle * committing to a repo where you do not hold a commit bit (e.g. src committer committing to docs) While under mentorship, get mentor approval before the commit. Enter the mentor's username in this field, and note that they are a mentor: [source,shell] .... Approved by: username-of-mentor (mentor) .... If a team approved these commits then include the team name followed by the username of the approver in parentheses. For example: [source,shell] .... Approved by: re (username) .... |`Obtained from:` |The name of the project (if any) from which the code was obtained. Do not use this line for the name of an individual person. |`Fixes:` |The Git short hash and the title line of a commit that is fixed by this change as returned by `git log -n 1 --oneline GIT-COMMIT-HASH`. |`MFC after:` |To receive an e-mail reminder to MFC at a later date, specify the number of days, weeks, or months after which an MFC is planned. |`MFC to:` |If the commit should be merged to a subset of stable branches, specify the branch names. |`MFC with:` |If the commit should be merged together with a previous one in a single MFC commit (for example, where this commit corrects a bug in the previous change), specify the corresponding Git hash. |`MFH:` |If the commit is to be merged into a ports quarterly branch name, specify the quarterly branch. For example `2021Q2`. |`Relnotes:` |If the change is a candidate for inclusion in the release notes for the next release from the branch, set to `yes`. |`Security:` |If the change is related to a security vulnerability or security exposure, include one or more references or a description of the issue. If possible, include a VuXML URL or a CVE ID. |`Event:` |The description for the event where this commit was made. If this is a recurring event, add the year or even the month to it. For example, this could be `FooBSDcon 2019`. The idea behind this line is to put recognition to conferences, gatherings, and other types of meetups and to show that these are useful to have. Please do not use the `Sponsored by:` line for this as that is meant for organizations sponsoring certain features or developers working on them. |`Sponsored by:` |Sponsoring organizations for this change, if any. Separate multiple organizations with commas. If only a portion of the work was sponsored, or different amounts of sponsorship were provided to different authors, please give appropriate credit in parentheses after each sponsor name. For example, `Example.com (alice, code refactoring), Wormulon (bob), Momcorp (cindy)` shows that Alice was sponsored by Example.com to do code refactoring, while Wormulon sponsored Bob's work and Momcorp sponsored Cindy's work. Other authors were either not sponsored or chose not to list sponsorship. |`Differential Revision:` |The full URL of the Phabricator review. This line __must be the last line__. For example: `https://reviews.freebsd.org/D1708`. |`Signed-off-by:` |ID certifies compliance with https://developercertificate.org/ |=== .Commit Log for a Commit Based on a PR [example] ==== The commit is based on a patch from a PR submitted by John Smith. The commit message "PR" field is filled. [.programlisting] .... ... PR: 12345 .... The committer sets the author of the patch with `git commit --author "John Smith "`. ==== .Commit Log for a Commit Needing Review [example] ==== The virtual memory system is being changed. After posting patches to the appropriate mailing list (in this case, `freebsd-arch`) and the changes have been approved. [.programlisting] .... ... Reviewed by: -arch .... ==== .Commit Log for a Commit Needing Approval [example] ==== Commit a port, after working with the listed MAINTAINER, who said to go ahead and commit. [.programlisting] .... ... Approved by: abc (maintainer) .... Where _abc_ is the account name of the person who approved. ==== .Commit Log for a Commit Bringing in Code from OpenBSD [example] ==== Committing some code based on work done in the OpenBSD project. [.programlisting] .... ... Obtained from: OpenBSD .... ==== .Commit Log for a Change to FreeBSD-CURRENT with a Planned Commit to FreeBSD-STABLE to Follow at a Later Date. [example] ==== Committing some code which will be merged from FreeBSD-CURRENT into the FreeBSD-STABLE branch after two weeks. [.programlisting] .... ... MFC after: 2 weeks .... Where _2_ is the number of days, weeks, or months after which an MFC is planned. The _weeks_ option may be `day`, `days`, `week`, `weeks`, `month`, `months`. ==== It is often necessary to combine these. Consider the situation where a user has submitted a PR containing code from the NetBSD project. Looking at the PR, the developer sees it is not an area of the tree they normally work in, so they have the change reviewed by the `arch` mailing list. Since the change is complex, the developer opts to MFC after one month to allow adequate testing. The extra information to include in the commit would look something like .Example Combined Commit Log [example] ==== [.programlisting] .... PR: 54321 Reviewed by: -arch Obtained from: NetBSD MFC after: 1 month Relnotes: yes .... ==== [[pref-license]] == Preferred License for New Files The FreeBSD Project's full license policy can be found at link:https://www.FreeBSD.org/internal/software-license/[https://www.FreeBSD.org/internal/software-license]. The rest of this section is intended to help you get started. As a rule, when in doubt, ask. It is much easier to give advice than to fix the source tree. The FreeBSD Project suggests and uses this text as the preferred license scheme: [.programlisting] .... /*- * SPDX-License-Identifier: BSD-2-Clause-FreeBSD * * Copyright (c) [year] [your name] * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * [id for your version control system, if any] */ .... The FreeBSD project strongly discourages the so-called "advertising clause" in new code. Due to the large number of contributors to the FreeBSD project, complying with this clause for many commercial vendors has become difficult. If you have code in the tree with the advertising clause, please consider removing it. In fact, please consider using the above license for your code. The FreeBSD project discourages completely new licenses and variations on the standard licenses. New licenses require the approval of the {core-email} to reside in the main repository. The more different licenses that are used in the tree, the more problems that this causes to those wishing to utilize this code, typically from unintended consequences from a poorly worded license. Project policy dictates that code under some non-BSD licenses must be placed only in specific sections of the repository, and in some cases, compilation must be conditional or even disabled by default. For example, the GENERIC kernel must be compiled under only licenses identical to or substantially similar to the BSD license. GPL, APSL, CDDL, etc, licensed software must not be compiled into GENERIC. Developers are reminded that in open source, getting "open" right is just as important as getting "source" right, as improper handling of intellectual property has serious consequences. Any questions or concerns should immediately be brought to the attention of the core team. [[tracking.license.grants]] == Keeping Track of Licenses Granted to the FreeBSD Project Various software or data exist in the repositories where the FreeBSD project has been granted a special licence to be able to use them. A case in point are the Terminus fonts for use with man:vt[4]. Here the author Dimitar Zhekov has allowed us to use the "Terminus BSD Console" font under a 2-clause BSD license rather than the regular Open Font License he normally uses. It is clearly sensible to keep a record of any such license grants. To that end, the {core-email} has decided to keep an archive of them. Whenever the FreeBSD project is granted a special license we require the {core-email} to be notified. Any developers involved in arranging such a license grant, please send details to the {core-email} including: * Contact details for people or organizations granting the special license. * What files, directories etc. in the repositories are covered by the license grant including the revision numbers where any specially licensed material was committed. * The date the license comes into effect from. Unless otherwise agreed, this will be the date the license was issued by the authors of the software in question. * The license text. * A note of any restrictions, limitations or exceptions that apply specifically to FreeBSD's usage of the licensed material. * Any other relevant information. Once the {core-email} is satisfied that all the necessary details have been gathered and are correct, the secretary will send a PGP-signed acknowledgement of receipt including the license details. This receipt will be persistently archived and serve as our permanent record of the license grant. The license archive should contain only details of license grants; this is not the place for any discussions around licensing or other subjects. Access to data within the license archive will be available on request to the {core-email}. [[spdx.tags]] == SPDX Tags in the tree The project uses https://spdx.dev[SPDX] tags in our source base. At present, these tags are indented to help automated tools reconstruct license requirements mechanically. All _SPDX-License-Identifier_ tags in the tree should be considered to be informative. All files in the FreeBSD source tree with these tags also have a copy of the license which governs use of that file. In the event of a discrepency, the verbatim license is controlling. The project tries to follow the https://spdx.github.io/spdx-spec/[SPDX Specification, Version 2.2]. How to mark source files and valid algebraic expressions are found in https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/[Appendix IV] and https://spdx.github.io/spdx-spec/appendix-V-using-SPDX-short-identifiers-in-source-files/[Appendix V]. The project draws identifiers from SPDX's list of valid https://spdx.org/licenses/[short license identifiers]. The project uses only the _SPDX-License-Identifier_ tag. As of March 2021, approximately 25,000 out of 90,000 files in the tree have been marked. [[developer.relations]] == Developer Relations When working directly on your own code or on code which is already well established as your responsibility, then there is probably little need to check with other committers before jumping in with a commit. Working on a bug in an area of the system which is clearly orphaned (and there are a few such areas, to our shame), the same applies. When modifying parts of the system which are maintained, formally, or informally, consider asking for review just as a developer would have before becoming a committer. For ports, contact the listed `MAINTAINER` in the [.filename]#Makefile#. To determine if an area of the tree is maintained, check the MAINTAINERS file at the root of the tree. If nobody is listed, scan the revision history to see who has committed changes in the past. An example script that lists each person who has committed to a given file along with the number of commits each person has made can be found at on `freefall` at [.filename]#~eadler/bin/whodid#. If queries go unanswered or the committer otherwise indicates a lack of interest in the area affected, go ahead and commit it. [IMPORTANT] ==== Avoid sending private emails to maintainers. Other people might be interested in the conversation, not just the final output. ==== If there is any doubt about a commit for any reason at all, have it reviewed before committing. Better to have it flamed then and there rather than when it is part of the repository. If a commit does results in controversy erupting, it may be advisable to consider backing the change out again until the matter is settled. Remember, with a version control system we can always change it back. Do not impugn the intentions of others. If they see a different solution to a problem, or even a different problem, it is probably not because they are stupid, because they have questionable parentage, or because they are trying to destroy hard work, personal image, or FreeBSD, but basically because they have a different outlook on the world. Different is good. Disagree honestly. Argue your position from its merits, be honest about any shortcomings it may have, and be open to seeing their solution, or even their vision of the problem, with an open mind. Accept correction. We are all fallible. When you have made a mistake, apologize and get on with life. Do not beat up yourself, and certainly do not beat up others for your mistake. Do not waste time on embarrassment or recrimination, just fix the problem and move on. Ask for help. Seek out (and give) peer reviews. One of the ways open source software is supposed to excel is in the number of eyeballs applied to it; this does not apply if nobody will review code. [[if-in-doubt]] == If in Doubt... When unsure about something, whether it be a technical issue or a project convention be sure to ask. If you stay silent you will never make progress. If it relates to a technical issue ask on the public mailing lists. Avoid the temptation to email the individual person that knows the answer. This way everyone will be able to learn from the question and the answer. For project specific or administrative questions ask, in order: * Your mentor or former mentor. * An experienced committer on IRC, email, etc. * Any team with a "hat", as they can give you a definitive answer. * If still not sure, ask on {developers-name}. Once your question is answered, if no one pointed you to documentation that spelled out the answer to your question, document it, as others will have the same question. [[bugzilla]] == Bugzilla The FreeBSD Project utilizes Bugzilla for tracking bugs and change requests. Be sure that if you commit a fix or suggestion found in the PR database to close it. It is also considered nice if you take time to close any PRs associated with your commits, if appropriate. Committers with non-``FreeBSD.org`` Bugzilla accounts can have the old account merged with the `FreeBSD.org` account by following these steps: [.procedure] ==== . Log in using your old account. . Open new bug. Choose `Services` as the Product, and `Bug Tracker` as the Component. In bug description list accounts you wish to be merged. . Log in using `FreeBSD.org` account and post comment to newly opened bug to confirm ownership. See <> for more details on how to generate or set a password for your `FreeBSD.org` account. . If there are more than two accounts to merge, post comments from each of them. ==== You can find out more about Bugzilla at: * link:{pr-guidelines}[FreeBSD Problem Report Handling Guidelines] * link:https://www.FreeBSD.org/support/[https://www.FreeBSD.org/support] [[phabricator]] == Phabricator The FreeBSD Project utilizes https://reviews.freebsd.org[Phabricator] for code review requests. See the https://wiki.freebsd.org/CodeReview[CodeReview] wiki page for details. Committers with non-``FreeBSD.org`` Phabricator accounts can have the old account renamed to the ``FreeBSD.org`` account by following these steps: [.procedure] ==== . Change your Phabricator account email to your `FreeBSD.org` email. . Open new bug on our bug tracker using your `FreeBSD.org` account, see <> for more information. Choose `Services` as the Product, and `Code Review` as the Component. In bug description request that your Phabricator account be renamed, and provide a link to your Phabricator user. For example, `https://reviews.freebsd.org/p/bob_example.com/` ==== [IMPORTANT] ==== Phabricator accounts cannot be merged, please do not open a new account. ==== [[people]] == Who's Who Besides the repository meisters, there are other FreeBSD project members and teams whom you will probably get to know in your role as a committer. Briefly, and by no means all-inclusively, these are: `{doceng}`:: doceng is the group responsible for the documentation build infrastructure, approving new documentation committers, and ensuring that the FreeBSD website and documentation on the FTP site is up to date with respect to the Subversion tree. It is not a conflict resolution body. The vast majority of documentation related discussion takes place on the {freebsd-doc}. More details regarding the doceng team can be found in its https://www.FreeBSD.org/internal/doceng/[charter]. Committers interested in contributing to the documentation should familiarize themselves with the link:{fdp-primer}[Documentation Project Primer]. `{re-members}`:: These are the members of the `{re}`. This team is responsible for setting release deadlines and controlling the release process. During code freezes, the release engineers have final authority on all changes to the system for whichever branch is pending release status. If there is something you want merged from FreeBSD-CURRENT to FreeBSD-STABLE (whatever values those may have at any given time), these are the people to talk to about it. `{so}`:: `{so-name}` is the link:https://www.FreeBSD.org/security/[FreeBSD Security Officer] and oversees the `{security-officer}`. `{wollman}`:: If you need advice on obscure network internals or are not sure of some potential change to the networking subsystem you have in mind, Garrett is someone to talk to. Garrett is also very knowledgeable on the various standards applicable to FreeBSD. {committers-name}:: {svn-src-all}, {svn-ports-all} and {svn-doc-all} are the mailing lists that the version control system uses to send commit messages to. _Never_ send email directly to these lists. Only send replies to this list when they are short and are directly related to a commit. {developers-name}:: All committers are subscribed to -developers. This list was created to be a forum for the committers "community" issues. Examples are Core voting, announcements, etc. + The {developers-name} is for the exclusive use of FreeBSD committers. To develop FreeBSD, committers must have the ability to openly discuss matters that will be resolved before they are publicly announced. Frank discussions of work in progress are not suitable for open publication and may harm FreeBSD. + All FreeBSD committers are expected not to not publish or forward messages from the {developers-name} outside the list membership without permission of all of the authors. Violators will be removed from the {developers-name}, resulting in a suspension of commit privileges. Repeated or flagrant violations may result in permanent revocation of commit privileges. + This list is _not_ intended as a place for code reviews or for any technical discussion. In fact using it as such hurts the FreeBSD Project as it gives a sense of a closed list where general decisions affecting all of the FreeBSD using community are made without being "open". Last, but not least __never, never ever, email the {developers-name} and CC:/BCC: another FreeBSD list__. Never, ever email another FreeBSD email list and CC:/BCC: the {developers-name}. Doing so can greatly diminish the benefits of this list. [[ssh.guide]] == SSH Quick-Start Guide [.procedure] ==== . If you do not wish to type your password in every time you use man:ssh[1], and you use keys to authenticate, man:ssh-agent[1] is there for your convenience. If you want to use man:ssh-agent[1], make sure that you run it before running other applications. X users, for example, usually do this from their [.filename]#.xsession# or [.filename]#.xinitrc#. See man:ssh-agent[1] for details. . Generate a key pair using man:ssh-keygen[1]. The key pair will wind up in your [.filename]#$HOME/.ssh/# directory. + [IMPORTANT] ====== Only ECDSA, Ed25519 or RSA keys are supported. ====== . Send your public key ([.filename]#$HOME/.ssh/id_ecdsa.pub#, [.filename]#$HOME/.ssh/id_ed25519.pub#, or [.filename]#$HOME/.ssh/id_rsa.pub#) to the person setting you up as a committer so it can be put into [.filename]#yourlogin# in [.filename]#/etc/ssh-keys/# on `freefall`. ==== Now man:ssh-add[1] can be used for authentication once per session. It prompts for the private key's pass phrase, and then stores it in the authentication agent (man:ssh-agent[1]). Use `ssh-add -d` to remove keys stored in the agent. Test with a simple remote command: `ssh freefall.FreeBSD.org ls /usr`. For more information, see package:security/openssh-portable[], man:ssh[1], man:ssh-add[1], man:ssh-agent[1], man:ssh-keygen[1], and man:scp[1]. For information on adding, changing, or removing man:ssh[1] keys, see https://wiki.freebsd.org/clusteradm/ssh-keys[this article]. [[coverity]] == Coverity(R) Availability for FreeBSD Committers All FreeBSD developers can obtain access to Coverity analysis results of all FreeBSD Project software. All who are interested in obtaining access to the analysis results of the automated Coverity runs, can sign up at http://scan.coverity.com/[Coverity Scan]. The FreeBSD wiki includes a mini-guide for developers who are interested in working with the Coverity(R) analysis reports: https://wiki.freebsd.org/CoverityPrevent[https://wiki.freebsd.org/CoverityPrevent]. Please note that this mini-guide is only readable by FreeBSD developers, so if you cannot access this page, you will have to ask someone to add you to the appropriate Wiki access list. Finally, all FreeBSD developers who are going to use Coverity(R) are always encouraged to ask for more details and usage information, by posting any questions to the mailing list of the FreeBSD developers. [[rules]] == The FreeBSD Committers' Big List of Rules Everyone involved with the FreeBSD project is expected to abide by the _Code of Conduct_ available from link:https://www.FreeBSD.org/internal/code-of-conduct/[https://www.FreeBSD.org/internal/code-of-conduct]. As committers, you form the public face of the project, and how you behave has a vital impact on the public perception of it. This guide expands on the parts of the _Code of Conduct_ specific to committers. . Respect other committers. . Respect other contributors. . Discuss any significant change _before_ committing. . Respect existing maintainers (if listed in the `MAINTAINER` field in [.filename]#Makefile# or in [.filename]#MAINTAINER# in the top-level directory). . Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. . Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the FreeBSD-STABLE branch as outlined for the maintainer in rule #5. . Do not fight in public with other committers; it looks bad. . Respect all code freezes and read the `committers` and `developers` mailing lists in a timely manner so you know when a code freeze is in effect. . When in doubt on any procedure, ask first! . Test your changes before committing them. . Do not commit to contributed software without _explicit_ approval from the respective maintainers. As noted, breaking some of these rules can be grounds for suspension or, upon repeated offense, permanent removal of commit privileges. Individual members of core have the power to temporarily suspend commit privileges until core as a whole has the chance to review the issue. In case of an "emergency" (a committer doing damage to the repository), a temporary suspension may also be done by the repository meisters. Only a 2/3 majority of core has the authority to suspend commit privileges for longer than a week or to remove them permanently. This rule does not exist to set core up as a bunch of cruel dictators who can dispose of committers as casually as empty soda cans, but to give the project a kind of safety fuse. If someone is out of control, it is important to be able to deal with this immediately rather than be paralyzed by debate. In all cases, a committer whose privileges are suspended or revoked is entitled to a "hearing" by core, the total duration of the suspension being determined at that time. A committer whose privileges are suspended may also request a review of the decision after 30 days and every 30 days thereafter (unless the total suspension period is less than 30 days). A committer whose privileges have been revoked entirely may request a review after a period of 6 months has elapsed. This review policy is _strictly informal_ and, in all cases, core reserves the right to either act on or disregard requests for review if they feel their original decision to be the right one. In all other aspects of project operation, core is a subset of committers and is bound by the __same rules__. Just because someone is in core this does not mean that they have special dispensation to step outside any of the lines painted here; core's "special powers" only kick in when it acts as a group, not on an individual basis. As individuals, the core team members are all committers first and core second. === Details [[respect]] . Respect other committers. + This means that you need to treat other committers as the peer-group developers that they are. Despite our occasional attempts to prove the contrary, one does not get to be a committer by being stupid and nothing rankles more than being treated that way by one of your peers. Whether we always feel respect for one another or not (and everyone has off days), we still have to _treat_ other committers with respect at all times, on public forums and in private email. + Being able to work together long term is this project's greatest asset, one far more important than any set of changes to the code, and turning arguments about code into issues that affect our long-term ability to work harmoniously together is just not worth the trade-off by any conceivable stretch of the imagination. + To comply with this rule, do not send email when you are angry or otherwise behave in a manner which is likely to strike others as needlessly confrontational. First calm down, then think about how to communicate in the most effective fashion for convincing the other persons that your side of the argument is correct, do not just blow off some steam so you can feel better in the short term at the cost of a long-term flame war. Not only is this very bad "energy economics", but repeated displays of public aggression which impair our ability to work well together will be dealt with severely by the project leadership and may result in suspension or termination of your commit privileges. The project leadership will take into account both public and private communications brought before it. It will not seek the disclosure of private communications, but it will take it into account if it is volunteered by the committers involved in the complaint. + All of this is never an option which the project's leadership enjoys in the slightest, but unity comes first. No amount of code or good advice is worth trading that away. . Respect other contributors. + You were not always a committer. At one time you were a contributor. Remember that at all times. Remember what it was like trying to get help and attention. Do not forget that your work as a contributor was very important to you. Remember what it was like. Do not discourage, belittle, or demean contributors. Treat them with respect. They are our committers in waiting. They are every bit as important to the project as committers. Their contributions are as valid and as important as your own. After all, you made many contributions before you became a committer. Always remember that. + Consider the points raised under <> and apply them also to contributors. . Discuss any significant change _before_ committing. + The repository is not where changes are initially submitted for correctness or argued over, that happens first in the mailing lists or by use of the Phabricator service. The commit will only happen once something resembling consensus has been reached. This does not mean that permission is required before correcting every obvious syntax error or manual page misspelling, just that it is good to develop a feel for when a proposed change is not quite such a no-brainer and requires some feedback first. People really do not mind sweeping changes if the result is something clearly better than what they had before, they just do not like being _surprised_ by those changes. The very best way of making sure that things are on the right track is to have code reviewed by one or more other committers. + When in doubt, ask for review! . Respect existing maintainers if listed. + Many parts of FreeBSD are not "owned" in the sense that any specific individual will jump up and yell if you commit a change to "their" area, but it still pays to check first. One convention we use is to put a maintainer line in the [.filename]#Makefile# for any package or subtree which is being actively maintained by one or more people; see link:{developers-handbook}#policies[Source Tree Guidelines and Policies] for documentation on this. Where sections of code have several maintainers, commits to affected areas by one maintainer need to be reviewed by at least one other maintainer. In cases where the "maintainer-ship" of something is not clear, look at the repository logs for the files in question and see if someone has been working recently or predominantly in that area. . Any disputed change must be backed out pending resolution of the dispute if requested by a maintainer. Security related changes may override a maintainer's wishes at the Security Officer's discretion. + This may be hard to swallow in times of conflict (when each side is convinced that they are in the right, of course) but a version control system makes it unnecessary to have an ongoing dispute raging when it is far easier to simply reverse the disputed change, get everyone calmed down again and then try to figure out what is the best way to proceed. If the change turns out to be the best thing after all, it can be easily brought back. If it turns out not to be, then the users did not have to live with the bogus change in the tree while everyone was busily debating its merits. People _very_ rarely call for back-outs in the repository since discussion generally exposes bad or controversial changes before the commit even happens, but on such rare occasions the back-out should be done without argument so that we can get immediately on to the topic of figuring out whether it was bogus or not. . Changes go to FreeBSD-CURRENT before FreeBSD-STABLE unless specifically permitted by the release engineer or unless they are not applicable to FreeBSD-CURRENT. Any non-trivial or non-urgent change which is applicable should also be allowed to sit in FreeBSD-CURRENT for at least 3 days before merging so that it can be given sufficient testing. The release engineer has the same authority over the FreeBSD-STABLE branch as outlined in rule #5. + This is another "do not argue about it" issue since it is the release engineer who is ultimately responsible (and gets beaten up) if a change turns out to be bad. Please respect this and give the release engineer your full cooperation when it comes to the FreeBSD-STABLE branch. The management of FreeBSD-STABLE may frequently seem to be overly conservative to the casual observer, but also bear in mind the fact that conservatism is supposed to be the hallmark of FreeBSD-STABLE and different rules apply there than in FreeBSD-CURRENT. There is also really no point in having FreeBSD-CURRENT be a testing ground if changes are merged over to FreeBSD-STABLE immediately. Changes need a chance to be tested by the FreeBSD-CURRENT developers, so allow some time to elapse before merging unless the FreeBSD-STABLE fix is critical, time sensitive or so obvious as to make further testing unnecessary (spelling fixes to manual pages, obvious bug/typo fixes, etc.) In other words, apply common sense. + Changes to the security branches (for example, `releng/9.3`) must be approved by a member of the `{security-officer}`, or in some cases, by a member of the `{re}`. . Do not fight in public with other committers; it looks bad. + This project has a public image to uphold and that image is very important to all of us, especially if we are to continue to attract new members. There will be occasions when, despite everyone's very best attempts at self-control, tempers are lost and angry words are exchanged. The best thing that can be done in such cases is to minimize the effects of this until everyone has cooled back down. Do not air angry words in public and do not forward private correspondence or other private communications to public mailing lists, mail aliases, instant messaging channels or social media sites. What people say one-to-one is often much less sugar-coated than what they would say in public, and such communications therefore have no place there - they only serve to inflame an already bad situation. If the person sending a flame-o-gram at least had the grace to send it privately, then have the grace to keep it private yourself. If you feel you are being unfairly treated by another developer, and it is causing you anguish, bring the matter up with core rather than taking it public. Core will do its best to play peace makers and get things back to sanity. In cases where the dispute involves a change to the codebase and the participants do not appear to be reaching an amicable agreement, core may appoint a mutually-agreeable third party to resolve the dispute. All parties involved must then agree to be bound by the decision reached by this third party. . Respect all code freezes and read the `committers` and `developers` mailing list on a timely basis so you know when a code freeze is in effect. + Committing unapproved changes during a code freeze is a really big mistake and committers are expected to keep up-to-date on what is going on before jumping in after a long absence and committing 10 megabytes worth of accumulated stuff. People who abuse this on a regular basis will have their commit privileges suspended until they get back from the FreeBSD Happy Reeducation Camp we run in Greenland. . When in doubt on any procedure, ask first! + Many mistakes are made because someone is in a hurry and just assumes they know the right way of doing something. If you have not done it before, chances are good that you do not actually know the way we do things and really need to ask first or you are going to completely embarrass yourself in public. There is no shame in asking "how in the heck do I do this?" We already know you are an intelligent person; otherwise, you would not be a committer. . Test your changes before committing them. + This may sound obvious, but if it really were so obvious then we probably would not see so many cases of people clearly not doing this. If your changes are to the kernel, make sure you can still compile both GENERIC and LINT. If your changes are anywhere else, make sure you can still make world. If your changes are to a branch, make sure your testing occurs with a machine which is running that code. If you have a change which also may break another architecture, be sure and test on all supported architectures. Please refer to the https://www.FreeBSD.org/internal/[FreeBSD Internal Page] for a list of available resources. As other architectures are added to the FreeBSD supported platforms list, the appropriate shared testing resources will be made available. . Do not commit to contributed software without _explicit_ approval from the respective maintainers. + Contributed software is anything under the [.filename]#src/contrib#, [.filename]#src/crypto#, or [.filename]#src/sys/contrib# trees. + The trees mentioned above are for contributed software usually imported onto a vendor branch. Committing something there may cause unnecessary headaches when importing newer versions of the software. As a general consider sending patches upstream to the vendor. Patches may be committed to FreeBSD first with permission of the maintainer. + Reasons for modifying upstream software range from wanting strict control over a tightly coupled dependency to lack of portability in the canonical repository's distribution of their code. Regardless of the reason, effort to minimize the maintenance burden of fork is helpful to fellow maintainers. Avoid committing trivial or cosmetic changes to files since it makes every merge thereafter more difficult: such patches need to be manually re-verified every import. + If a particular piece of software lacks a maintainer, you are encouraged to take up ownership. If you are unsure of the current maintainership email {freebsd-arch} and ask. === Policy on Multiple Architectures FreeBSD has added several new architecture ports during recent release cycles and is truly no longer an i386(TM) centric operating system. In an effort to make it easier to keep FreeBSD portable across the platforms we support, core has developed this mandate: [.blockquote] Our 32-bit reference platform is i386, and our 64-bit reference platform is amd64. Major design work (including major API and ABI changes) must prove itself on at least one 32-bit and at least one 64-bit platform, preferably the primary reference platforms, before it may be committed to the source tree. The i386 and amd64 platforms were chosen due to being more readily available to developers and as representatives of more diverse processor and system designs - big versus little endian, register file versus register stack, different DMA and cache implementations, hardware page tables versus software TLB management etc. We will continue to re-evaluate this policy as cost and availability of the 64-bit platforms change. Developers should also be aware of our Tier Policy for the long term support of hardware architectures. The rules here are intended to provide guidance during the development process, and are distinct from the requirements for features and architectures listed in that section. The Tier rules for feature support on architectures at release-time are more strict than the rules for changes during the development process. === Other Suggestions When committing documentation changes, use a spell checker before committing. For all XML docs, verify that the formatting directives are correct by running `make lint` and package:textproc/igor[]. For manual pages, run package:sysutils/manck[] and package:textproc/igor[] over the manual page to verify all of the cross references and file references are correct and that the man page has all of the appropriate `MLINKS` installed. Do not mix style fixes with new functionality. A style fix is any change which does not modify the functionality of the code. Mixing the changes obfuscates the functionality change when asking for differences between revisions, which can hide any new bugs. Do not include whitespace changes with content changes in commits to [.filename]#doc/#. The extra clutter in the diffs makes the translators' job much more difficult. Instead, make any style or whitespace changes in separate commits that are clearly labeled as such in the commit message. === Deprecating Features When it is necessary to remove functionality from software in the base system, follow these guidelines whenever possible: . Mention is made in the manual page and possibly the release notes that the option, utility, or interface is deprecated. Use of the deprecated feature generates a warning. . The option, utility, or interface is preserved until the next major (point zero) release. . The option, utility, or interface is removed and no longer documented. It is now obsolete. It is also generally a good idea to note its removal in the release notes. === Privacy and Confidentiality . Most FreeBSD business is done in public. + FreeBSD is an _open_ project. Which means that not only can anyone use the source code, but that most of the development process is open to public scrutiny. . Certain sensitive matters must remain private or held under embargo. + There unfortunately cannot be complete transparency. As a FreeBSD developer you will have a certain degree of privileged access to information. Consequently you are expected to respect certain requirements for confidentiality. Sometimes the need for confidentiality comes from external collaborators or has a specific time limit. Mostly though, it is a matter of not releasing private communications. . The Security Officer has sole control over the release of security advisories. + Where there are security problems that affect many different operating systems, FreeBSD frequently depends on early access to be able to prepare advisories for coordinated release. Unless FreeBSD developers can be trusted to maintain security, such early access will not be made available. The Security Officer is responsible for controlling pre-release access to information about vulnerabilities, and for timing the release of all advisories. He may request help under condition of confidentiality from any developer with relevant knowledge to prepare security fixes. . Communications with Core are kept confidential for as long as necessary. + Communications to core will initially be treated as confidential. Eventually however, most of Core's business will be summarized into the monthly or quarterly core reports. Care will be taken to avoid publicising any sensitive details. Records of some particularly sensitive subjects may not be reported on at all and will be retained only in Core's private archives. . Non-disclosure Agreements may be required for access to certain commercially sensitive data. + Access to certain commercially sensitive data may only be available under a Non-Disclosure Agreement. The FreeBSD Foundation legal staff must be consulted before any binding agreements are entered into. . Private communications must not be made public without permission. + Beyond the specific requirements above there is a general expectation not to publish private communications between developers without the consent of all parties involved. Ask permission before forwarding a message onto a public mailing list, or posting it to a forum or website that can be accessed by other than the original correspondents. . Communications on project-only or restricted access channels must be kept private. + Similarly to personal communications, certain internal communications channels, including FreeBSD Committer only mailing lists and restricted access IRC channels are considered private communications. Permission is required to publish material from these sources. . Core may approve publication. + Where it is impractical to obtain permission due to the number of correspondents or where permission to publish is unreasonably withheld, Core may approve release of such private matters that merit more general publication. [[archs]] == Support for Multiple Architectures FreeBSD is a highly portable operating system intended to function on many different types of hardware architectures. Maintaining clean separation of Machine Dependent (MD) and Machine Independent (MI) code, as well as minimizing MD code, is an important part of our strategy to remain agile with regards to current hardware trends. Each new hardware architecture supported by FreeBSD adds substantially to the cost of code maintenance, toolchain support, and release engineering. It also dramatically increases the cost of effective testing of kernel changes. As such, there is strong motivation to differentiate between classes of support for various architectures while remaining strong in a few key architectures that are seen as the FreeBSD "target audience". === Statement of General Intent The FreeBSD Project targets "production quality commercial off-the-shelf (COTS) workstation, server, and high-end embedded systems". By retaining a focus on a narrow set of architectures of interest in these environments, the FreeBSD Project is able to maintain high levels of quality, stability, and performance, as well as minimize the load on various support teams on the project, such as the ports team, documentation team, security officer, and release engineering teams. Diversity in hardware support broadens the options for FreeBSD consumers by offering new features and usage opportunities, but these benefits must always be carefully considered in terms of the real-world maintenance cost associated with additional platform support. The FreeBSD Project differentiates platform targets into four tiers. Each tier includes a list of guarantees consumers may rely on as well as obligations by the Project and developers to fulfill those guarantees. These lists define the minimum guarantees for each tier. The Project and developers may provide additional levels of support beyond the minimum guarantees for a given tier, but such additional support is not guaranteed. Each platform target is assigned to a specific tier for each stable branch. As a result, a platform target might be assigned to different tiers on concurrent stable branches. === Platform Targets Support for a hardware platform consists of two components: kernel support and userland Application Binary Interfaces (ABIs). Kernel platform support includes things needed to run a FreeBSD kernel on a hardware platform such as machine-dependent virtual memory management and device drivers. A userland ABI specifies an interface for user processes to interact with a FreeBSD kernel and base system libraries. A userland ABI includes system call interfaces, the layout and semantics of public data structures, and the layout and semantics of arguments passed to subroutines. Some components of an ABI may be defined by specifications such as the layout of C++ exception objects or calling conventions for C functions. A FreeBSD kernel also uses an ABI (sometimes referred to as the Kernel Binary Interface (KBI)) which includes the semantics and layouts of public data structures and the layout and semantics of arguments to public functions within the kernel itself. A FreeBSD kernel may support multiple userland ABIs. For example, FreeBSD's amd64 kernel supports FreeBSD amd64 and i386 userland ABIs as well as Linux x86_64 and i386 userland ABIs. A FreeBSD kernel should support a "native" ABI as the default ABI. The native "ABI" generally shares certain properties with the kernel ABI such as the C calling convention, sizes of basic types, etc. Tiers are defined for both kernels and userland ABIs. In the common case, a platform's kernel and FreeBSD ABIs are assigned to the same tier. === Tier 1: Fully-Supported Architectures Tier 1 platforms are the most mature FreeBSD platforms. They are supported by the security officer, release engineering, and port management teams. Tier 1 architectures are expected to be Production Quality with respect to all aspects of the FreeBSD operating system, including installation and development environments. The FreeBSD Project provides the following guarantees to consumers of Tier 1 platforms: * Official FreeBSD release images will be provided by the release engineering team. * Binary updates and source patches for Security Advisories and Errata Notices will be provided for supported releases. * Source patches for Security Advisories will be provided for supported branches. * Binary updates and source patches for cross-platform Security Advisories will typically be provided at the time of the announcement. * Changes to userland ABIs will generally include compatibility shims to ensure correct operation of binaries compiled against any stable branch where the platform is Tier 1. These shims might not be enabled in the default install. If compatibility shims are not provided for an ABI change, the lack of shims will be clearly documented in the release notes. * Changes to certain portions of the kernel ABI will include compatibility shims to ensure correct operation of kernel modules compiled against the oldest supported release on the branch. Note that not all parts of the kernel ABI are protected. * Official binary packages for third party software will be provided by the ports team. For embedded architectures, these packages may be cross-built from a different architecture. * Most relevant ports should either build or have the appropriate filters to prevent inappropriate ones from building. * New features which are not inherently platform-specific will be fully functional on all Tier 1 architectures. * Features and compatibility shims used by binaries compiled against older stable branches may be removed in newer major versions. Such removals will be clearly documented in the release notes. * Tier 1 platforms should be fully documented. Basic operations will be documented in the FreeBSD Handbook. * Tier 1 platforms will be included in the source tree. * Tier 1 platforms should be self-hosting either via the in-tree toolchain or an external toolchain. If an external toolchain is required, official binary packages for an external toolchain will be provided. To maintain maturity of Tier 1 platforms, the FreeBSD Project will maintain the following resources to support development: * Build and test automation support either in the FreeBSD.org cluster or some other location easily available for all developers. Embedded platforms may substitute an emulator available in the FreeBSD.org cluster for actual hardware. * Inclusion in the `make universe` and `make tinderbox` targets. * Dedicated hardware in one of the FreeBSD clusters for package building (either natively or via qemu-user). Collectively, developers are required to provide the following to maintain the Tier 1 status of a platform: * Changes to the source tree should not knowingly break the build of a Tier 1 platform. * Tier 1 architectures must have a mature, healthy ecosystem of users and active developers. * Developers should be able to build packages on commonly available, non-embedded Tier 1 systems. This can mean either native builds if non-embedded systems are commonly available for the platform in question, or it can mean cross-builds hosted on some other Tier 1 architecture. * Changes cannot break the userland ABI. If an ABI change is required, ABI compatibility for existing binaries should be provided via use of symbol versioning or shared library version bumps. * Changes merged to stable branches cannot break the protected portions of the kernel ABI. If a kernel ABI change is required, the change should be modified to preserve functionality of existing kernel modules. === Tier 2: Developmental and Niche Architectures Tier 2 platforms are functional, but less mature FreeBSD platforms. They are not supported by the security officer, release engineering, and port management teams. Tier 2 platforms may be Tier 1 platform candidates that are still under active development. Architectures reaching end of life may also be moved from Tier 1 status to Tier 2 status as the availability of resources to continue to maintain the system in a Production Quality state diminishes. Well-supported niche architectures may also be Tier 2. The FreeBSD Project provides the following guarantees to consumers of Tier 2 platforms: * The ports infrastructure should include basic support for Tier 2 architectures sufficient to support building ports and packages. This includes support for basic packages such as ports-mgmt/pkg, but there is no guarantee that arbitrary ports will be buildable or functional. * New features which are not inherently platform-specific should be feasible on all Tier 2 architectures if not implemented. * Tier 2 platforms will be included in the source tree. * Tier 2 platforms should be self-hosting either via the in-tree toolchain or an external toolchain. If an external toolchain is required, official binary packages for an external toolchain will be provided. * Tier 2 platforms should provide functional kernels and userlands even if an official release distribution is not provided. To maintain maturity of Tier 2 platforms, the FreeBSD Project will maintain the following resources to support development: * Inclusion in the `make universe` and `make tinderbox` targets. Collectively, developers are required to provide the following to maintain the Tier 2 status of a platform: * Changes to the source tree should not knowingly break the build of a Tier 2 platform. * Tier 2 architectures must have an active ecosystem of users and developers. * While changes are permitted to break the userland ABI, the ABI should not be broken gratuitously. Significant userland ABI changes should be restricted to major versions. * New features that are not yet implemented on Tier 2 architectures should provide a means of disabling them on those architectures. === Tier 3: Experimental Architectures Tier 3 platforms have at least partial FreeBSD support. They are _not_ supported by the security officer, release engineering, and port management teams. Tier 3 platforms are architectures in the early stages of development, for non-mainstream hardware platforms, or which are considered legacy systems unlikely to see broad future use. Initial support for Tier 3 platforms may exist in a separate repository rather than the main source repository. The FreeBSD Project provides no guarantees to consumers of Tier 3 platforms and is not committed to maintaining resources to support development. Tier 3 platforms may not always be buildable, nor are any kernel or userland ABIs considered stable. === Tier 4: Unsupported Architectures Tier 4 platforms are not supported in any form by the project. All systems not otherwise classified are Tier 4 systems. When a platform transitions to Tier 4, all support for the platform is removed from the source and ports trees. Note that ports support should remain as long as the platform is supported in a branch supported by ports. === Policy on Changing the Tier of an Architecture Systems may only be moved from one tier to another by approval of the FreeBSD Core Team, which shall make that decision in collaboration with the Security Officer, Release Engineering, and ports management teams. For a platform to be promoted to a higher tier, any missing support guarantees must be satisfied before the promotion is completed. [[ports]] == Ports Specific FAQ [[ports-qa-adding]] === Adding a New Port [[ports-qa-add-new]] ==== How do I add a new port? First, please read the section about repository copies. The easiest way to add a new port is the `addport` script located in the [.filename]#ports/Tools/scripts# directory. It adds a port from the directory specified, determining the category automatically from the port [.filename]#Makefile#. It also adds an entry to the port's category [.filename]#Makefile#. It was written by `{mharo}`, `{will}`, and `{garga}`. When sending questions about this script to the {freebsd-ports}, please also CC `{crees}`, the current maintainer. [[ports-qa-add-new-extra]] ==== Any other things I need to know when I add a new port? Check the port, preferably to make sure it compiles and packages correctly. This is the recommended sequence: [source,shell] .... # make install # make package # make deinstall # pkg add package you built above # make deinstall # make reinstall # make package .... The link:{porters-handbook}[Porters Handbook] contains more detailed instructions. Use man:portlint[1] to check the syntax of the port. You do not necessarily have to eliminate all warnings but make sure you have fixed the simple ones. If the port came from a submitter who has not contributed to the Project before, add that person's name to the link:{contributors}#contrib-additional[Additional Contributors] section of the FreeBSD Contributors List. Close the PR if the port came in as a PR. To close a PR, change the state to `Issue Resolved` and the resolution as `Fixed`. [[ports-qa-removing]] === Removing an Existing Port [[ports-qa-remove-one]] ==== How do I remove an existing port? First, please read the section about repository copies. Before you remove the port, you have to verify there are no other ports depending on it. * Make sure there is no dependency on the port in the ports collection: ** The port's PKGNAME appears in exactly one line in a recent INDEX file. ** No other ports contains any reference to the port's directory or PKGNAME in their Makefiles + [TIP] ==== When using Git, consider using `git grep`, it is much faster than `grep -r`. ==== + * Then, remove the port: + [.procedure] ==== * Remove the port's files and directory with `git rm`. * Remove the `SUBDIR` listing of the port in the parent directory [.filename]#Makefile#. * Add an entry to [.filename]#ports/MOVED#. * Search for entries in [.filename]#ports/security/vuxml/vuln.xml# and adjust them accordingly. In particular, check for previous packages with the new name which version could include the new port. * Remove the port from [.filename]#ports/LEGAL# if it is there. ==== Alternatively, you can use the rmport script, from [.filename]#ports/Tools/scripts#. This script was written by {vd}. When sending questions about this script to the {freebsd-ports}, please also CC {crees}, the current maintainer. [[ports-qa-move-port]] === How do I move a port to a new location? [.procedure] ==== . Perform a thorough check of the ports collection for any dependencies on the old port location/name, and update them. Running `grep` on [.filename]#INDEX# is not enough because some ports have dependencies enabled by compile-time options. A full `git grep` of the ports collection is recommended. . Remove the `SUBDIR` entry from the old category Makefile and add a `SUBDIR` entry to the new category Makefile. . Add an entry to [.filename]#ports/MOVED#. . Move the port with `git mv`. . Commit the changes. ==== [[ports-qa-freeze]] === Ports Freeze [[ports-qa-freeze-what]] ==== What is a “ports freeze”? A “ports freeze” was a restricted state the ports tree was put in before a release. It was used to ensure a higher quality for the packages shipped with a release. It usually lasted a couple of weeks. During that time, build problems were fixed, and the release packages were built. This practice is no longer used, as the packages for the releases are built from the current stable, quarterly branch. For more information on how to merge commits to the quarterly branch, see <>. [[ports-qa-quarterly]] === Quarterly Branches [[ports-qa-misc-request-mfh]] ==== What is the procedure to request authorization for merging a commit to the quarterly branch? As of November 30, 2020, there is no need to seek explicit approval to commit to the quarterly branch. [[ports-qa-misc-commit-mfh]] ==== What is the procedure for merging commits to the quarterly branch? Merging commits to the quarterly branch is very similar to MFC'ing a commit in the src repository, so basically: [source,shell] .... % git checkout 2021Q2 % git cherry-pick -x $HASH (verify everything is OK, for example by doing a build test) % git push .... where '$HASH' is the hash of the commit you want to copy over to the quarterly branch. The -x parameter ensures the hash '$HASH' of the main branch is included in the new commit message of the quarterly branch. [[ports-qa-new-category]] === Creating a New Category [[ports-qa-new-category-how]] ==== What is the procedure for creating a new category? Please see link:{porters-handbook}#proposing-categories[Proposing a New Category] in the Porter's Handbook. Once that procedure has been followed and the PR has been assigned to the {portmgr}, it is their decision whether or not to approve it. If they do, it is their responsibility to: [.procedure] ==== . Perform any needed moves. (This only applies to physical categories.) . Update the `VALID_CATEGORIES` definition in [.filename]#ports/Mk/bsd.port.mk#. . Assign the PR back to you. ==== [[ports-qa-new-category-physical]] ==== What do I need to do to implement a new physical category? [.procedure] ==== . Upgrade each moved port's [.filename]#Makefile#. Do not connect the new category to the build yet. + To do this, you will need to: + [.procedure] ====== . Change the port's `CATEGORIES` (this was the point of the exercise, remember?) The new category is listed first. This will help to ensure that the PKGORIGIN is correct. . Run a `make describe`. Since the top-level `make index` that you will be running in a few steps is an iteration of `make describe` over the entire ports hierarchy, catching any errors here will save you having to re-run that step later on. . If you want to be really thorough, now might be a good time to run man:portlint[1]. ====== + . Check that the ``PKGORIGIN``s are correct. The ports system uses each port's `CATEGORIES` entry to create its `PKGORIGIN`, which is used to connect installed packages to the port directory they were built from. If this entry is wrong, common port tools like man:pkg_version[1] and man:portupgrade[1] fail. + To do this, use the [.filename]#chkorigin.sh# tool: `env PORTSDIR=/path/to/ports sh -e /path/to/ports/Tools/scripts/chkorigin.sh`. This will check every port in the ports tree, even those not connected to the build, so you can run it directly after the move operation. Hint: do not forget to look at the ``PKGORIGIN``s of any slave ports of the ports you just moved! . On your own local system, test the proposed changes: first, comment out the SUBDIR entries in the old ports' categories' [.filename]##Makefile##s; then enable building the new category in [.filename]#ports/Makefile#. Run make checksubdirs in the affected category directories to check the SUBDIR entries. Next, in the [.filename]#ports/# directory, run make index. This can take over 40 minutes on even modern systems; however, it is a necessary step to prevent problems for other people. . Once this is done, you can commit the updated [.filename]#ports/Makefile# to connect the new category to the build and also commit the [.filename]#Makefile# changes for the old category or categories. . Add appropriate entries to [.filename]#ports/MOVED#. . Update the documentation by modifying: ** the link:{porters-handbook}#PORTING-CATEGORIES[list of categories] in the Porter's Handbook + . Only once all the above have been done, and no one is any longer reporting problems with the new ports, should the old ports be deleted from their previous locations in the repository. ==== ==== What do I need to do to implement a new virtual category? This is much simpler than a physical category. Only a few modifications are needed: * the link:{porters-handbook}#PORTING-CATEGORIES[list of categories] in the Porter's Handbook [[ports-qa-misc-questions]] === Miscellaneous Questions [[ports-qa-misc-blanket-approval]] ==== Are there changes that can be committed without asking the maintainer for approval? Blanket approval for most ports applies to these types of fixes: * Most infrastructure changes to a port (that is, modernizing, but not changing the functionality). For example, the blanket covers converting to new `USES` macros, enabling verbose builds, and switching to new ports system syntaxes. * Trivial and _tested_ build and runtime fixes. * Documentations or metadata changes to ports, like [.filename]#pkg-descr# or `COMMENT`. [IMPORTANT] ==== Exceptions to this are anything maintained by the {portmgr}, or the {security-officer}. No unauthorized commits may ever be made to ports maintained by those groups. ==== [[ports-qa-misc-correctly-building]] ==== How do I know if my port is building correctly or not? The packages are built multiple times each week. If a port fails, the maintainer will receive an email from `pkg-fallout@FreeBSD.org`. Reports for all the package builds (official, experimental, and non-regression) are aggregated at link:pkg-status.FreeBSD.org[pkg-status.FreeBSD.org]. [[ports-qa-misc-INDEX]] ==== I added a new port. Do I need to add it to the [.filename]#INDEX#? No. The file can either be generated by running `make index`, or a pre-generated version can be downloaded with `make fetchindex`. [[ports-qa-misc-no-touch]] ==== Are there any other files I am not allowed to touch? Any file directly under [.filename]#ports/#, or any file under a subdirectory that starts with an uppercase letter ([.filename]#Mk/#, [.filename]#Tools/#, etc.). In particular, the {portmgr} is very protective of [.filename]#ports/Mk/bsd.port*.mk# so do not commit changes to those files unless you want to face their wrath. [[ports-qa-misc-updated-distfile]] ==== What is the proper procedure for updating the checksum for a port distfile when the file changes without a version change? When the checksum for a distribution file is updated due to the author updating the file without changing the port revision, the commit message includes a summary of the relevant diffs between the original and new distfile to ensure that the distfile has not been corrupted or maliciously altered. If the current version of the port has been in the ports tree for a while, a copy of the old distfile will usually be available on the ftp servers; otherwise the author or maintainer should be contacted to find out why the distfile has changed. [[ports-exp-run]] ==== How can an experimental test build of the ports tree (exp-run) be requested? An exp-run must be completed before patches with a significant ports impact are committed. The patch can be against the ports tree or the base system. Full package builds will be done with the patches provided by the submitter, and the submitter is required to fix detected problems _(fallout)_ before commit. [.procedure] ==== . Go to the link:https://bugs.freebsd.org/submit[Bugzilla new PR page]. . Select the product your patch is about. . Fill in the bug report as normal. Remember to attach the patch. . If at the top it says “Show Advanced Fields” click on it. It will now say “Hide Advanced Fields”. Many new fields will be available. If it already says “Hide Advanced Fields”, no need to do anything. . In the “Flags” section, set the “exp-run” one to `?`. As for all other fields, hovering the mouse over any field shows more details. . Submit. Wait for the build to run. . {portmgr} will reply with a possible fallout. . Depending on the fallout: ** If there is no fallout, the procedure stops here, and the change can be committed, pending any other approval required. ... If there is fallout, it _must_ be fixed, either by fixing the ports directly in the ports tree, or adding to the submitted patch. ... When this is done, go back to step 6 saying the fallout was fixed and wait for the exp-run to be run again. Repeat as long as there are broken ports. ==== [[non-committers]] == Issues Specific to Developers Who Are Not Committers A few people who have access to the FreeBSD machines do not have commit bits. Almost all of this document will apply to these developers as well (except things specific to commits and the mailing list memberships that go with them). In particular, we recommend that you read: * <> * <> + [NOTE] ==== Get your mentor to add you to the "Additional Contributors" ([.filename]#~/documentation/content/en/articles/contributors/contrib-additional.adoc#), if you are not already listed there. ==== * <> * <> * <> [[google-analytics]] == Information About Google Analytics As of December 12, 2012, Google Analytics was enabled on the FreeBSD Project website to collect anonymized usage statistics regarding usage of the site. The information collected is valuable to the FreeBSD Documentation Project, to identify various problems on the FreeBSD website. [[google-analytics-policy]] === Google Analytics General Policy The FreeBSD Project takes visitor privacy very seriously. As such, the FreeBSD Project website honors the "Do Not Track" header _before_ fetching the tracking code from Google. For more information, please see the https://www.FreeBSD.org/privacy/[FreeBSD Privacy Policy]. Google Analytics access is _not_ arbitrarily allowed - access must be requested, voted on by the `{doceng}`, and explicitly granted. Requests for Google Analytics data must include a specific purpose. For example, a valid reason for requesting access would be "to see the most frequently used web browsers when viewing FreeBSD web pages to ensure page rendering speeds are acceptable." Conversely, "to see what web browsers are most frequently used" (without stating __why__) would be rejected. All requests must include the timeframe for which the data would be required. For example, it must be explicitly stated if the requested data would be needed for a timeframe covering a span of 3 weeks, or if the request would be one-time only. Any request for Google Analytics data without a clear, reasonable reason beneficial to the FreeBSD Project will be rejected. [[google-analytics-data]] === Data Available Through Google Analytics A few examples of the types of Google Analytics data available include: * Commonly used web browsers * Page load times * Site access by language [[misc]] == Miscellaneous Questions === Are there changes that can be committed without asking the maintainer for approval? Blanket approval for most ports applies to these types of fixes: * Most infrastructure changes to a port (that is, modernizing, but not changing the functionality). For example, the blanket covers converting to new `USES` macros, enabling verbose builds, and switching to new ports system syntaxes. * Trivial and _tested_ build and runtime fixes. * Documentations or metadata changes to ports, like [.filename]#pkg-descr# or `COMMENT`. === How do I access people.FreeBSD.org to put up personal or project information? `people.FreeBSD.org` is the same as `freefall.FreeBSD.org`. Just create a [.filename]#public_html# directory. Anything you place in that directory will automatically be visible under https://people.FreeBSD.org/[https://people.FreeBSD.org/]. === Where are the mailing list archives stored? The mailing lists are archived under [.filename]#/local/mail# on `freefall.FreeBSD.org`. === I would like to mentor a new committer. What process do I need to follow? See the https://www.freebsd.org/internal/new-account/[New Account Creation Procedure] document on the internal pages. [[benefits]] == Benefits and Perks for FreeBSD Committers [[benefits-recognition]] === Recognition Recognition as a competent software engineer is the longest lasting value. In addition, getting a chance to work with some of the best people that every engineer would dream of meeting is a great perk! [[benefits-freebsdmall]] === FreeBSD Mall FreeBSD committers can get a free 4-CD or DVD set at conferences from http://www.freebsdmall.com[FreeBSD Mall, Inc.]. [[benefits-irc]] === IRC In addition, developers may request a cloaked hostmask for their account on the Freenode IRC network in the form of `freebsd/developer/` or `freebsd/developer/`. To request a cloak, send an email to `{irc-email}` with your requested hostmask and NickServ account name. See the https://wiki.freebsd.org/IRC/Cloaks[IRC Cloaks] wiki page for more details. [[benefits-gandi]] === `Gandi.net` Gandi provides website hosting, cloud computing, domain registration, and X.509 certificate services. Gandi offers an E-rate discount to all FreeBSD developers. Send mail to mailto:non-profit@gandi.net[non-profit@gandi.net] using your `@freebsd.org` mail address, and indicate your Gandi handle. [[benefits-rsync]] === `rsync.net` https://rsync.net[rsync.net] provides cloud storage for offsite backup that is optimized for UNIX users. Their service runs entirely on FreeBSD and ZFS. rsync.net offers a free-forever 500 GB account to FreeBSD developers. Simply sign up at https://www.rsync.net/freebsd.html[https://www.rsync.net/freebsd.html] using your `@freebsd.org` address to receive this free account. diff --git a/documentation/content/en/articles/contributing/_index.adoc b/documentation/content/en/articles/contributing/_index.adoc index c7d61004ec..6dab1d5096 100644 --- a/documentation/content/en/articles/contributing/_index.adoc +++ b/documentation/content/en/articles/contributing/_index.adoc @@ -1,573 +1,574 @@ --- title: Contributing to FreeBSD authors: - author: Jordan Hubbard - author: Sam Lawrance - author: Mark Linimon -description: Contributing to the FreeBSD Project +description: How to contribute to the FreeBSD Project trademarks: ["freebsd", "ieee", "general"] +weight: 15 --- = Contributing to FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This article describes the different ways in which an individual or organization may contribute to the FreeBSD Project. ''' toc::[] So you want to contribute to FreeBSD? That is great! FreeBSD _relies_ on the contributions of its user base to survive. Your contributions are not only appreciated, they are vital to FreeBSD's continued growth. A large and growing number of international contributors, of greatly varying ages and areas of technical expertise, develop FreeBSD. There is always more work to be done than there are people available to do it, and more help is always appreciated. As a volunteer, what you do is limited only by what you want to do. However, we do ask that you are aware of what other members of the FreeBSD community will expect of you. You may want to take this into account before deciding to volunteer. The FreeBSD project is responsible for an entire operating system environment, rather than just a kernel or a few scattered utilities. As such, our [.filename]#TODO# lists span a very wide range of tasks: from documentation, beta testing and presentation, to the system installer and highly specialized types of kernel development. People of any skill level, in almost any area, can almost certainly help the project. Commercial entities engaged in FreeBSD-related enterprises are also encouraged to contact us. Do you need a special extension to make your product work? You will find us receptive to your requests, given that they are not too outlandish. Are you working on a value-added product? Please let us know! We may be able to work cooperatively on some aspect of it. The free software world is challenging many existing assumptions about how software is developed, sold, and maintained, and we urge you to at least give it a second look. [[contrib-what]] == What Is Needed The following list of tasks and sub-projects represents something of an amalgam of various [.filename]#TODO# lists and user requests. [[non-programmer-tasks]] === Ongoing Non-Programmer Tasks Many people who are involved in FreeBSD are not programmers. The Project includes documentation writers, Web designers, and support people. All that these people need to contribute is an investment of time and a willingness to learn. . Read through the FAQ and Handbook periodically. If anything is poorly explained, ambiguous, out of date or incorrect, let us know. Even better, send us a fix (Docbook is not difficult to learn, but there is no objection to ASCII submissions). . Help translate FreeBSD documentation into your native language. If documentation already exists for your language, you can help translate additional documents or verify that the translations are up-to-date and correct. First take a look at the link:{fdp-primer}#translations[Translations FAQ] in the FreeBSD Documentation Project Primer. You are not committing yourself to translating every single FreeBSD document by doing this - as a volunteer, you can do as much or as little translation as you desire. Once someone begins translating, others almost always join the effort. If you only have the time or energy to translate one part of the documentation, please translate the installation instructions. . Read the {freebsd-questions} occasionally (or even regularly). It can be very satisfying to share your expertise and help people solve their problems; sometimes you may even learn something new yourself! These forums can also be a source of ideas for things to improve upon. [[ongoing-programmer-tasks]] === Ongoing Programmer Tasks Most of the tasks listed here may require a considerable investment of time, an in-depth knowledge of the FreeBSD kernel, or both. However, there are also many useful tasks which are suitable for "weekend hackers". . If you run FreeBSD-CURRENT and have a good Internet connection, there is a machine `current.FreeBSD.org` which builds a full release once a day-every now and again, try to install the latest release from it and report any failures in the process. . Read the {freebsd-bugs}. There may be a problem you can comment constructively on or with patches you can test. Or you could even try to fix one of the problems yourself. . If you know of any bug fixes which have been successfully applied to -CURRENT but have not been merged into -STABLE after a decent interval (normally a couple of weeks), send the committer a polite reminder. . Move contributed software to [.filename]#src/contrib# in the source tree. . Make sure code in [.filename]#src/contrib# is up to date. . Build the source tree (or just part of it) with extra warnings enabled and clean up the warnings. A list of build warnings can also be found from our https://ci.freebsd.org[CI] by selecting a build and checking "LLVM/Clang Warnings". . Fix warnings for ports which do deprecated things like using `gets()` or including [.filename]#malloc.h#. . If you have contributed any ports and you had to make FreeBSD-specific changes, send your patches back to the original authors (this will make your life easier when they bring out the next version). . Get copies of formal standards like POSIX(R). Compare FreeBSD's behavior to that required by the standard. If the behavior differs, particularly in subtle or obscure corners of the specification, send in a PR about it. If you are able, figure out how to fix it and include a patch in the PR. If you think the standard is wrong, ask the standards body to consider the question. . Suggest further tasks for this list! === Work through the PR Database The https://bugs.FreeBSD.org/search/[FreeBSD PR list] shows all the current active problem reports and requests for enhancement that have been submitted by FreeBSD users. The PR database includes both programmer and non-programmer tasks. Look through the open PRs, and see if anything there takes your interest. Some of these might be very simple tasks that just need an extra pair of eyes to look over them and confirm that the fix in the PR is a good one. Others might be much more complex, or might not even have a fix included at all. Start with the PRs that have not been assigned to anyone else. If a PR is assigned to someone else, but it looks like something you can handle, email the person it is assigned to and ask if you can work on it-they might already have a patch ready to be tested, or further ideas that you can discuss with them. === Ongoing Ports Tasks The Ports Collection is a perpetual work in progress. We want to provide our users with an easy to use, up to date, high quality repository of third party software. We need people to donate some of their time and effort to help us achieve this goal. Anyone can get involved, and there are lots of different ways to do so. Contributing to ports is an excellent way to help "give back" something to the project. Whether you are looking for an ongoing role, or a fun challenge for a rainy day, we would love to have your help! There are a number of easy ways you can contribute to keeping the ports tree up to date and in good working order: * Find some cool or useful software and link:{porters-handbook}[create a port] for it. * There are a large number of ports that have no maintainer. Become a maintainer and <>. * If you have created or adopted a port, be aware of <>. * When you are looking for a quick challenge you could <>. === Pick one of the items from the Ideas page The https://wiki.freebsd.org/IdeasPage[FreeBSD list of projects and ideas for volunteers] is also available for people willing to contribute to the FreeBSD project. The list is being regularly updated and contains items for both programmers and non-programmers with information about each project. [[contrib-how]] == How to Contribute Contributions to the system generally fall into one or more of the following 5 categories: [[contrib-general]] === Bug Reports and General Commentary An idea or suggestion of _general_ technical interest should be mailed to the {freebsd-hackers}. Likewise, people with an interest in such things (and a tolerance for a _high_ volume of mail!) may subscribe to the {freebsd-hackers}. See link:{handbook}#eresources-mail[The FreeBSD Handbook] for more information about this and other mailing lists. If you find a bug or are submitting a specific change, please report it using the https://bugs.FreeBSD.org/submit/[bug submission form]. Try to fill-in each field of the bug report. Unless they exceed 65KB, include any patches directly in the report. If the patch is suitable to be applied to the source tree put `[PATCH]` in the synopsis of the report. When including patches, _do not_ use cut-and-paste because cut-and-paste turns tabs into spaces and makes them unusable. When patches are a lot larger than 20KB, consider compressing them (eg. with man:gzip[1] or man:bzip2[1]) prior to uploading them. After filing a report, you should receive confirmation along with a tracking number. Keep this tracking number so that you can update us with details about the problem. See also link:{problem-reports}[this article] on how to write good problem reports. === Changes to the Documentation Changes to the documentation are overseen by the {freebsd-doc}. Please look at the link:{fdp-primer}[FreeBSD Documentation Project Primer] for complete instructions. Send submissions and changes (even small ones are welcome!) using the same method as any other bug report. === Changes to Existing Source Code An addition or change to the existing source code is a somewhat trickier affair and depends a lot on how far out of date you are with the current state of FreeBSD development. There is a special on-going release of FreeBSD known as "FreeBSD-CURRENT" which is made available in a variety of ways for the convenience of developers working actively on the system. See link:{handbook}#current-stable[The FreeBSD Handbook] for more information about getting and using FreeBSD-CURRENT. Working from older sources unfortunately means that your changes may sometimes be too obsolete or too divergent for easy re-integration into FreeBSD. Chances of this can be minimized somewhat by subscribing to the {freebsd-announce} and the {freebsd-current} lists, where discussions on the current state of the system take place. Assuming that you can manage to secure fairly up-to-date sources to base your changes on, the next step is to produce a set of diffs to send to the FreeBSD maintainers. This is done with the man:diff[1] command. The preferred man:diff[1] format for submitting patches is the unified output format generated by `diff -u`. [source,shell] .... % diff -u oldfile newfile .... or [source,shell] .... % diff -u -r -N olddir newdir .... would generate a set of unified diffs for the given source file or directory hierarchy. See man:diff[1] for more information. Once you have a set of diffs (which you may test with the man:patch[1] command), you should submit them for inclusion with FreeBSD as a bug report. _Do not_ just send the diffs to the {freebsd-hackers} or they will get lost! We greatly appreciate your submission (this is a volunteer project!); because we are busy, we may not be able to address it immediately, but it will remain in the PR database until we do. Indicate your submission by including `[PATCH]` in the synopsis of the report. If you feel it appropriate (e.g. you have added, deleted, or renamed files), bundle your changes into a `tar` file. Archives created with man:shar[1] are also welcome. If your change is of a potentially sensitive nature, such as if you are unsure of copyright issues governing its further distribution then you should send it to {core-email} directly rather than submitting as a bug report. The {core-email} reaches a much smaller group of people who do much of the day-to-day work on FreeBSD. Note that this group is also _very busy_ and so you should only send mail to them where it is truly necessary. Please refer to man:intro[9] and man:style[9] for some information on coding style. We would appreciate it if you were at least aware of this information before submitting code. === New Code or Major Value-Added Packages In the case of a significant contribution of a large body work, or the addition of an important new feature to FreeBSD, it becomes almost always necessary to either send changes as tar files or upload them to a web or FTP site for other people to access. If you do not have access to a web or FTP site, ask on an appropriate FreeBSD mailing list for someone to host the changes for you. When working with large amounts of code, the touchy subject of copyrights also invariably comes up. FreeBSD prefers free software licenses such as BSD or ISC. Copyleft licenses such as GPLv2 are sometimes permitted. The complete listing can be found on the link:https://www.FreeBSD.org/internal/software-license/[core team licensing policy] page. === Money or Hardware We are always very happy to accept donations to further the cause of the FreeBSD Project and, in a volunteer effort like ours, a little can go a long way! Donations of hardware are also very important to expanding our list of supported peripherals since we generally lack the funds to buy such items ourselves. [[donations]] ==== Donating Funds The https://www.freebsdfoundation.org[FreeBSD Foundation] is a non-profit, tax-exempt foundation established to further the goals of the FreeBSD Project. As a 501(c)3 entity, the Foundation is generally exempt from US federal income tax as well as Colorado State income tax. Donations to a tax-exempt entity are often deductible from taxable federal income. Donations may be sent in check form to: [.address] **** The FreeBSD Foundation + P.O. Box 20247, + Boulder, + CO 80308 + USA **** The FreeBSD Foundation is also able to accept https://www.freebsdfoundation.org/donate/[online donations] through various payment options. More information about the FreeBSD Foundation can be found in https://people.FreeBSD.org/~jdp/foundation/announcement.html[The FreeBSD Foundation -- an Introduction]. To contact the Foundation by email, write to mailto:info@FreeBSDFoundation.org[info@FreeBSDFoundation.org]. ==== Donating Hardware The FreeBSD Project happily accepts donations of hardware that it can find good use for. If you are interested in donating hardware, please contact the link:https://www.FreeBSD.org/donations/[Donations Liaison Office]. [[ports-contributing]] == Contributing to ports [[adopt-port]] === Adopting an unmaintained port ==== Choosing an unmaintained port Taking over maintainership of ports that are unmaintained is a great way to get involved. Unmaintained ports are only updated and fixed when somebody volunteers to work on them. There are a large number of unmaintained ports. It is a good idea to start with adopting a port that you use regularly. Unmaintained ports have their `MAINTAINER` set to `ports@FreeBSD.org`. A list of unmaintained ports and their current errors and problem reports can be seen at the http://portsmon.FreeBSD.org/portsconcordanceformaintainer.py?maintainer=ports%40FreeBSD.org[FreeBSD Ports Monitoring System]. On https://portsfallout.com/fallout?port=&maintainer=ports%40FreeBSD.org[PortsFallout] can be seen a list of unmaintained ports with errors. Many unmaintained ports can have pending updates, this can be seen at the https://portscout.freebsd.org/ports@freebsd.org.html[FreeBSD Ports distfile scanner]. Some ports affect a large number of others due to dependencies and slave port relationships. Generally, we want people to have some experience before they maintain such ports. You can find out whether or not a port has dependencies or slave ports by looking at a master index of ports called [.filename]#INDEX#. (The name of the file varies by release of FreeBSD; for instance, [.filename]#INDEX-8#.) Some ports have conditional dependencies that are not included in a default [.filename]#INDEX# build. We expect you to be able to recognize such ports by looking through other ports' [.filename]#Makefile#'s. [NOTE] ====== The FreeBSD Ports Monitoring System (portsmon) is currently not working due to latest Python updates. ====== ==== How to adopt the port First make sure you understand your <>. Also read the link:{porters-handbook}[Porter's Handbook]. _Please do not commit yourself to more than you feel you can comfortably handle._ You may request maintainership of any unmaintained port as soon as you wish. Simply set `MAINTAINER` to your own email address and send a PR (Problem Report) with the change. If the port has build errors or needs updating, you may wish to include any other changes in the same PR. This will help because many committers are less willing to assign maintainership to someone who does not have a known track record with FreeBSD. Submitting PRs that fix build errors or update ports are the best ways to establish one. File your PR with category `ports` and class `change-request`. A committer will examine your PR, commit the changes, and finally close the PR. Sometimes this process can take a little while (committers are volunteers, too :). [[maintain-port]] === The challenge for port maintainers This section will give you an idea of why ports need to be maintained and outline the responsibilities of a port maintainer. [[why-maintenance]] ==== Why ports require maintenance Creating a port is a once-off task. Ensuring that a port is up to date and continues to build and run requires an ongoing maintenance effort. Maintainers are the people who dedicate some of their time to meeting these goals. The foremost reason ports need maintenance is to bring the latest and greatest in third party software to the FreeBSD community. An additional challenge is to keep individual ports working within the Ports Collection framework as it evolves. As a maintainer, you will need to manage the following challenges: * *New software versions and updates.* New versions and updates of existing ported software become available all the time, and these need to be incorporated into the Ports Collection in order to provide up-to-date software. * *Changes to dependencies.* If significant changes are made to the dependencies of your port, it may need to be updated so that it will continue to work correctly. * *Changes affecting dependent ports.* If other ports depend on a port that you maintain, changes to your port may require coordination with other maintainers. * *Interaction with other users, maintainers and developers.* Part of being a maintainer is taking on a support role. You are not expected to provide general support (but we welcome it if you choose to do so). What you should provide is a point of coordination for FreeBSD-specific issues regarding your ports. * *Bug hunting.* A port may be affected by bugs which are specific to FreeBSD. You will need to investigate, find, and fix these bugs when they are reported. Thoroughly testing a port to identify problems before they make their way into the Ports Collection is even better. * *Changes to ports infrastructure and policy.* Occasionally the systems that are used to build ports and packages are updated or a new recommendation affecting the infrastructure is made. You should be aware of these changes in case your ports are affected and require updating. * *Changes to the base system.* FreeBSD is under constant development. Changes to software, libraries, the kernel or even policy changes can cause flow-on change requirements to ports. ==== Maintainer responsibilities ===== Keep your ports up to date This section outlines the process to follow to keep your ports up to date. This is an overview. More information about upgrading a port is available in the link:{porters-handbook}[Porter's Handbook]. [.procedure] ==== . Watch for updates + Monitor the upstream vendor for new versions, updates and security fixes for the software. Announcement mailing lists or news web pages are useful for doing this. Sometimes users will contact you and ask when your port will be updated. If you are busy with other things or for any reason just cannot update it at the moment, ask if they will help you by submitting an update. + You may also receive automated email from the `FreeBSD Ports Version Check` informing you that a newer version of your port's distfile is available. More information about that system (including how to stop future emails) will be provided in the message. . Incorporate changes + When they become available, incorporate the changes into the port. You need to be able to generate a patch between the original port and your updated port. . Review and test + Thoroughly review and test your changes: ** Build, install and test your port on as many platforms and architectures as you can. It is common for a port to work on one branch or platform and fail on another. ** Make sure your port's dependencies are complete. The recommended way of doing this is by installing your own ports tinderbox. See <> for more information. ** Check that the packing list is up to date. This involves adding in any new files and directories and removing unused entries. ** Verify your port using man:portlint[1] as a guide. See <> for important information about using portlint. ** Consider whether changes to your port might cause any other ports to break. If this is the case, coordinate the changes with the maintainers of those ports. This is especially important if your update changes the shared library version; in this case, at the very least, the dependent ports will need to get a `PORTREVISION` bump so that they will automatically be upgraded by automated tools such as portmaster or man:portupgrade[1]. . Submit changes + Send your update by submitting a PR with an explanation of the changes and a patch containing the differences between the original port and the updated one. Please refer to link:{problem-reports}[Writing FreeBSD Problem Reports] for information on how to write a really good PR. + [NOTE] ====== Please do not submit a man:shar[1] archive of the entire port; instead, use man:diff[1] `-ruN`. In this way, committers can much more easily see exactly what changes are being made. The Porter's Handbook section on link:{porters-handbook}#port-upgrading[Upgrading] has more information. ====== . Wait + At some stage a committer will deal with your PR. It may take minutes, or it may take weeks - so please be patient. . Give feedback + If a committer finds a problem with your changes, they will most likely refer it back to you. A prompt response will help get your PR committed faster, and is better for maintaining a thread of conversation when trying to resolve any problems. . And Finally + Your changes will be committed and your port will have been updated. The PR will then be closed by the committer. That's it! ==== ===== Ensure your ports continue to build correctly This section is about discovering and fixing problems that stop your ports from building correctly. FreeBSD only guarantees that the Ports Collection works on the `-STABLE` branches. In theory, you should be able to get by with running the latest release of each stable branch (since the ABIs are not supposed to change) but if you can run the branch, that is even better. Since the majority of FreeBSD installations run on PC-compatible machines (what is termed the `i386` architecture), we expect you to keep the port working on that architecture. We prefer that ports also work on the `amd64` architecture running native. It is completely fair to ask for help if you do not have one of these machines. [NOTE] ==== The usual failure modes for non-`x86` machines are that the original programmers assumed that, for instance, pointers are `int`-s, or that a relatively lax older gcc compiler was being used. More and more, application authors are reworking their code to remove these assumptions - but if the author is not actively maintaining their code, you may need to do this yourself. ==== These are the tasks you need to perform to ensure your port is able to be built: [.procedure] ==== . Watch for build failures + Check your mail for mail from `pkg-fallout@FreeBSD.org` and the http://portscout.FreeBSD.org[distfiles scanner] to see if any of the port which are failing to build are out of date. . Collect information + Once you are aware of a problem, collect information to help you fix it. Build errors reported by `pkg-fallout` are accompanied by logs which will show you where the build failed. If the failure was reported to you by a user, ask them to send you information which may help in diagnosing the problem, such as: ** Build logs ** The commands and options used to build the port (including options set in [.filename]#/etc/make.conf#) ** A list of packages installed on their system as shown by man:pkg-info[8] ** The version of FreeBSD they are running as shown by man:uname[1] `-a` ** When their ports collection was last updated ** When their ports tree and [.filename]#INDEX# was last updated . Investigate and find a solution + Unfortunately there is no straightforward process to follow to do this. Remember, though: if you are stuck, ask for help! The {freebsd-ports} is a good place to start, and the upstream developers are often very helpful. . Submit changes + Just as with updating a port, you should now incorporate changes, review and test, submit your changes in a PR, and provide feedback if required. . Send patches to upstream authors + In some cases, you will have to make patches to the port to make it run on FreeBSD. Some (but not all) upstream authors will accept such patches back into their code for the next release. If so, this may even help their users on other BSD-based systems as well and perhaps save duplicated effort. Please consider sending any applicable patches to the authors as a courtesy. ==== ===== Investigate bug reports and PRs related to your port This section is about discovering and fixing bugs. FreeBSD-specific bugs are generally caused by assumptions about the build and runtime environments that do not apply to FreeBSD. You are less likely to encounter a problem of this type, but it can be more subtle and difficult to diagnose. These are the tasks you need to perform to ensure your port continues to work as intended: [.procedure] ==== . Respond to bug reports + Bugs may be reported to you through email via the https://bugs.FreeBSD.org/search/[Problem Report database]. Bugs may also be reported directly to you by users. + You should respond to PRs and other reports within 14 days, but please try not to take that long. Try to respond as soon as possible, even if it is just to say you need some more time before you can work on the PR. + If you have not responded after 14 days, any committer may commit from a PR that you have not responded to via a `maintainer-timeout`. . Collect information + If the person reporting the bug has not also provided a fix, you need to collect the information that will allow you to generate one. + If the bug is reproducible, you can collect most of the required information yourself. If not, ask the person who reported the bug to collect the information for you, such as: ** A detailed description of their actions, expected program behavior and actual behavior ** Copies of input data used to trigger the bug ** Information about their build and execution environment - for example, a list of installed packages and the output of man:env[1] ** Core dumps ** Stack traces . Eliminate incorrect reports + Some bug reports may be incorrect. For example, the user may have simply misused the program; or their installed packages may be out of date and require updating. Sometimes a reported bug is not specific to FreeBSD. In this case report the bug to the upstream developers. If the bug is within your capabilities to fix, you can also patch the port so that the fix is applied before the next upstream release. . Find a solution + As with build errors, you will need to sort out a fix to the problem. Again, remember to ask if you are stuck! . Submit or approve changes + Just as with updating a port, you should now incorporate changes, review and test, and submit your changes in a PR (or send a follow-up if a PR already exists for the problem). If another user has submitted changes in the PR, you can also send a follow-up saying whether or not you approve the changes. ==== ===== Providing support Part of being a maintainer is providing support - not for the software in general - but for the port and any FreeBSD-specific quirks and problems. Users may contact you with questions, suggestions, problems and patches. Most of the time their correspondence will be specific to FreeBSD. Occasionally you may have to invoke your skills in diplomacy, and kindly point users seeking general support to the appropriate resources. Less frequently you will encounter a person asking why the `RPMS` are not up to date or how can they get the software to run under Foo Linux. Take the opportunity to tell them that your port is up to date (if it is, of course!), and suggest that they try FreeBSD. Sometimes users and developers will decide that you are a busy person whose time is valuable and do some of the work for you. For example, they might: * submit a PR or send you patches to update your port, * investigate and perhaps provide a fix to a PR, or * otherwise submit changes to your port. In these cases your main obligation is to respond in a timely manner. Again, the timeout for non-responsive maintainers is 14 days. After this period changes may be committed unapproved. They have taken the trouble to do this for you; so please try to at least respond promptly. Then review, approve, modify or discuss their changes with them as soon as possible. If you can make them feel that their contribution is appreciated (and it should be) you will have a better chance persuading them to do more things for you in the future :-). [[fix-broken]] === Finding and fixing a broken port There are some really good places to find a port that needs some attention. You can use the https://bugs.freebsd.org/search[web interface] to the Problem Report database to search through and view unresolved PRs. The majority of ports PRs are updates, but with a little searching and skimming over synopses you should be able to find something interesting to work on (the `sw-bug` class is a good place to start). The other place is the http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System]. In particular look for unmaintained ports with build errors and ports that are marked `BROKEN`. https://portsfallout.com/[PortsFallout] shows port issues gathered from the FreeBSD package building. It is OK to send changes for a maintained port as well, but remember to ask the maintainer in case they are already working on the problem. Once you have found a bug or problem, collect information, investigate and fix! If there is an existing PR, follow up to that. Otherwise create a new PR. Your changes will be reviewed and, if everything checks out, committed. [NOTE] ====== The FreeBSD Ports Monitoring System (portsmon) is currently not working due to latest Python updates. ====== [[mortal-coil]] === When to call it quits As your interests and commitments change, you may find that you no longer have time to continue some (or all) of your ports contributions. That is fine! Please let us know if you are no longer using a port or have otherwise lost time or interest in being a maintainer. In this way we can go ahead and allow other people to try to work on existing problems with the port without waiting for your response. Remember, FreeBSD is a volunteer project, so if maintaining a port is no fun any more, it is probably time to let someone else do it! In any case, the Ports Management Team (`portmgr`) reserves the right to reset your maintainership if you have not actively maintained your port in some time. (Currently, this is set to 3 months.) By this, we mean that there are unresolved problems or pending updates that have not been worked on during that time. [[resources]] === Resources for ports maintainers and contributors The link:{porters-handbook}[Porter's Handbook] is your hitchhiker's guide to the ports system. Keep it handy! link:{problem-reports}[Writing FreeBSD Problem Reports] describes how to best formulate and submit a PR. In 2005 more than eleven thousand ports PRs were submitted! Following this article will greatly assist us in reducing the time needed to handle your PRs. The https://bugs.freebsd.org/bugzilla/query.cgi[Problem Report database]. The http://portsmon.FreeBSD.org/[FreeBSD Ports Monitoring System (portsmon)] can show you cross-referenced information about ports such as build errors and problem reports. If you are a maintainer you can use it to check on the build status of your ports. As a contributor you can use it to find broken and unmaintained ports that need to be fixed. The http://portscout.FreeBSD.org[FreeBSD Ports distfile scanner (portscout)] can show you ports for which the distfiles are not fetchable. You can check on your own ports or use it to find ports that need their `MASTER_SITES` updated. package:ports-mgmt/poudriere[] is the most thorough way to test a port through the entire cycle of installation, packaging, and deinstallation. Documentation is located at the https://github.com/freebsd/poudriere[poudriere github repository] man:portlint[1] is an application which can be used to verify that your port conforms to many important stylistic and functional guidelines. portlint is a simple heuristic application, so you should use it __only as a guide__. If portlint suggests changes which seem unreasonable, consult the link:{porters-handbook}[Porter's Handbook] or ask for advice. The {freebsd-ports} is for general ports-related discussion. It is a good place to ask for help. You can https://lists.freebsd.org/mailman/listinfo[subscribe, or read and search the list archives]. Reading the archives of the {freebsd-ports-bugs} and the {svn-ports-head} may also be of interest. https://portsfallout.com/[PortsFallout] is a place to help in searching for the https://lists.freebsd.org/pipermail/freebsd-pkg-fallout/[FreeBSD package-fallout archive]. [[ideas-contributing]] == Getting Started in Other Areas Looking for something interesting to get started that is not mentioned elsewhere in this article? The FreeBSD Project has several Wiki pages containing areas within which new contributors can get ideas on how to get started. The https://wiki.freebsd.org/JuniorJobs[Junior Jobs] page has a list of projects that might be of interest to people just getting started in FreeBSD, and want to work on interesting things to get their feet wet. The https://wiki.freebsd.org/IdeasPage[Ideas Page] contains various "nice to have" or "interesting" things to work on in the Project. diff --git a/documentation/content/en/articles/contributors/_index.adoc b/documentation/content/en/articles/contributors/_index.adoc index afd0f120b4..fecc0e6fe6 100644 --- a/documentation/content/en/articles/contributors/_index.adoc +++ b/documentation/content/en/articles/contributors/_index.adoc @@ -1,166 +1,166 @@ --- title: Contributors to FreeBSD -description: Contributors to FreeBSD +description: A list of organizations and individuals who have contributed to FreeBSD trademarks: ["freebsd", "sun", "general"] --- = Contributors to FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :source-highlighter: rouge :experimental: :sectnumlevels: 6 include::shared/authors.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This article lists individuals and organizations who have made a contribution to FreeBSD. ''' toc::[] [[donors]] == Donors Gallery [NOTE] ==== As of 2010, the following section is several years out-of-date. Donations from the past several years appear https://www.FreeBSD.org/donations/donors/[here]. ==== The FreeBSD Project is indebted to the following donors and would like to publicly thank them here! * _Contributors to the central server project:_ The following individuals and businesses made it possible for the FreeBSD Project to build a new central server machine, which has replaced `freefall.FreeBSD.org` at one point, by donating the following items: ** {mbarkah} and his employer, http://www.hemi.com/[ Hemisphere Online], donated a _Pentium Pro (P6) 200MHz CPU_ ** http://www.asacomputers.com/[ASA Computers] donated a _Tyan 1662 motherboard_. ** Joe McGuckin mailto:joe@via.net[joe@via.net] of http://www.via.net/[ViaNet Communications] donated a _Kingston ethernet controller._ ** Jack O'Neill mailto:jack@diamond.xtalwind.net[jack@diamond.xtalwind.net] donated an _NCR 53C875 SCSI controller card_. ** Ulf Zimmermann mailto:ulf@Alameda.net[ulf@Alameda.net] of http://www.Alameda.net/[Alameda Networks] donated _128MB of memory_, a _4 Gb disk drive and the case._ * _Direct funding:_ + The following individuals and businesses have generously contributed direct funding to the project: ** Annelise Anderson mailto:ANDRSN@HOOVER.STANFORD.EDU[ANDRSN@HOOVER.STANFORD.EDU] ** {dillon} ** http://www.bluemountain.com/[Blue Mountain Arts] ** http://www.epilogue.com/[Epilogue Technology Corporation] ** {sef} ** http://www.gta.com/[Global Technology Associates, Inc] ** Don Scott Wilde ** Gianmarco Giovannelli mailto:gmarco@masternet.it[gmarco@masternet.it] ** Josef C. Grosch mailto:joeg@truenorth.org[joeg@truenorth.org] ** Robert T. Morris ** {chuckr} ** Kenneth P. Stox mailto:ken@stox.sa.enteract.com[ken@stox.sa.enteract.com] of http://www.imagescape.com/[Imaginary Landscape, LLC.] ** Dmitry S. Kohmanyuk mailto:dk@dog.farm.org[dk@dog.farm.org] ** http://www.cdrom.co.jp/[Laser5] of Japan (a portion of the profits from sales of their various FreeBSD CDROMs). ** http://www.mmjp.or.jp/fuki/[Fuki Shuppan Publishing Co.] donated a portion of their profits from _Hajimete no FreeBSD_ (FreeBSD, Getting started) to the FreeBSD and XFree86 projects. ** http://www.ascii.co.jp/[ASCII Corp.] donated a portion of their profits from several FreeBSD-related books to the FreeBSD project. ** http://www.yokogawa.co.jp/[Yokogawa Electric Corp] has generously donated significant funding to the FreeBSD project. ** http://www.buffnet.net/[BuffNET] ** http://www.pacificsolutions.com/[Pacific Solutions] ** http://www.siemens.de/[Siemens AG] via Andre Albsmeier mailto:andre.albsmeier@mchp.siemens.de[andre.albsmeier@mchp.siemens.de] ** Chris Silva mailto:ras@interaccess.com[ras@interaccess.com] * _Hardware contributors:_ + The following individuals and businesses have generously contributed hardware for testing and device driver development/support: ** BSDi for providing the Pentium P5-90 and 486/DX2-66 EISA/VL systems that are being used for our development work, to say nothing of the network access and other donations of hardware resources. ** http://www.compaq.com[Compaq] has donated a variety of Alpha systems to the FreeBSD Project. Among the many generous donations are 4 AlphaStation DS10s, an AlphaServer DS20, AlphaServer 2100s, an AlphaServer 4100, 8 500Mhz Personal Workstations, 4 433Mhz Personal Workstations, and more! These machines are used for release engineering, package building, SMP development, and general development on the Alpha architecture. ** TRW Financial Systems, Inc. provided 130 PCs, three 68 GB file servers, twelve Ethernets, two routers and an ATM switch for debugging the diskless code. ** Dermot McDonnell donated the Toshiba XM3401B CDROM drive currently used in freefall. ** Chuck Robey mailto:chuckr@glue.umd.edu[chuckr@glue.umd.edu] contributed his floppy tape streamer for experimental work. ** Larry Altneu mailto:larry@ALR.COM[larry@ALR.COM], and {wilko}, provided Wangtek and Archive QIC-02 tape drives in order to improve the [.filename]#wt# driver. ** Ernst Winter (http://berklix.org/ewinter/[Deceased]) contributed a 2.88 MB floppy drive to the project. This will hopefully increase the pressure for rewriting the floppy disk driver. ** http://www.tekram.com/[Tekram Technologies] sent one each of their DC-390, DC-390U and DC-390F FAST and ULTRA SCSI host adapter cards for regression testing of the NCR and AMD drivers with their cards. They are also to be applauded for making driver sources for free operating systems available from their FTP server link:ftp://ftp.tekram.com/scsi/FreeBSD/[ftp://ftp.tekram.com/scsi/FreeBSD/]. ** Larry M. Augustin contributed not only a Symbios Sym8751S SCSI card, but also a set of data books, including one about the forthcoming Sym53c895 chip with Ultra-2 and LVD support, and the latest programming manual with information on how to safely use the advanced features of the latest Symbios SCSI chips. Thanks a lot! ** {kuku} donated an FX120 12 speed Mitsumi CDROM drive for IDE CDROM driver development. ** Mike Tancsa mailto:mike@sentex.ca[mike@sentex.ca] donated four various ATM PCI cards in order to help increase support of these cards as well as help support the development effort of the netatm ATM stack. * _Special contributors:_ ** http://www.osd.bsdi.com/[BSDi] (formerly Walnut Creek CDROM) has donated almost more than we can say (see the 'About the FreeBSD Project' section of the link:{handbook}[FreeBSD Handbook] for more details). In particular, we would like to thank them for the original hardware used for `freefall.FreeBSD.org`, our primary development machine, and for `thud.FreeBSD.org`, a testing and build box. We are also indebted to them for funding various contributors over the years and providing us with unrestricted use of their T1 connection to the Internet. ** The http://www.interface-business.de/[interface business GmbH, Dresden] has been patiently supporting {joerg} who has often preferred FreeBSD work over paid work, and used to fall back to their (quite expensive) EUnet Internet connection whenever his private connection became too slow or flaky to work with it... ** http://www.bsdi.com/[Berkeley Software Design, Inc.] has contributed their DOS emulator code to the remaining BSD world, which is used in the _doscmd_ command. [[staff-committers]] == The FreeBSD Developers These are the people who have commit privileges and do the engineering work on the FreeBSD source tree. All core team members are also developers. (in alphabetical order by last name): include::content/en/articles/contributors/contrib-committers.adoc[] [[contrib-corealumni]] == Core Team Alumni The following people were members of the FreeBSD core team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::content/en/articles/contributors/contrib-corealumni.adoc[] [[contrib-develalumni]] == Development Team Alumni The following people were members of the FreeBSD development team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::content/en/articles/contributors/contrib-develalumni.adoc[] [[contrib-portmgralumni]] == Ports Management Team Alumni The following people were members of the FreeBSD portmgr team during the periods indicated. We thank them for their past efforts in the service of the FreeBSD project. _In rough reverse chronological order:_ include::content/en/articles/contributors/contrib-portmgralumni.adoc[] [[contrib-develinmemoriam]] == Development Team: In Memoriam During the many years that the FreeBSD Project has been in existence, sadly, some of our developers have passed away. Here are some remembrances. _In rough reverse chronological order of their passing:_ include::content/en/articles/contributors/contrib-develinmemoriam.adoc[] [[contrib-derived]] == Derived Software Contributors This software was originally derived from William F. Jolitz's 386BSD release 0.1, though almost none of the original 386BSD specific code remains. This software has been essentially re-implemented from the 4.4BSD-Lite release provided by the Computer Science Research Group (CSRG) at the University of California, Berkeley and associated academic contributors. There are also portions of NetBSD and OpenBSD that have been integrated into FreeBSD as well, and we would therefore like to thank all the contributors to NetBSD and OpenBSD for their work. [[contrib-additional]] == Additional FreeBSD Contributors (in alphabetical order by first name): include::content/en/articles/contributors/contrib-additional.adoc[] [[contrib-386bsd]] == 386BSD Patch Kit Patch Contributors (in alphabetical order by first name): include::content/en/articles/contributors/contrib-386bsd.adoc[] diff --git a/documentation/content/en/articles/filtering-bridges/_index.adoc b/documentation/content/en/articles/filtering-bridges/_index.adoc index 7a3647c0fb..9ae472d44e 100644 --- a/documentation/content/en/articles/filtering-bridges/_index.adoc +++ b/documentation/content/en/articles/filtering-bridges/_index.adoc @@ -1,274 +1,274 @@ --- title: Filtering Bridges authors: - author: Alex Dupre email: ale@FreeBSD.org -description: Filtering Bridges in FreeBSD +description: Configuring firewalls and filtering on FreeBSD hosts acting as bridges rather than routers trademarks: ["freebsd", "3com", "intel", "general"] --- = Filtering Bridges :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :source-highlighter: rouge :experimental: :sectnumlevels: 6 include::shared/en/urls.adoc[] [.abstract-title] Abstract Often it is useful to divide one physical network (like an Ethernet) into two separate segments without having to create subnets, and use a router to link them together. The device that connects the two networks in this way is called a bridge. A FreeBSD system with two network interfaces is enough in order to act as a bridge. A bridge works by scanning the addresses of MAC level (Ethernet addresses) of the devices connected to each of its network interfaces and then forwarding the traffic between the two networks only if the source and the destination are on different segments. Under many points of view a bridge is similar to an Ethernet switch with only two ports. ''' toc::[] [[filtering-bridges-why]] == Why use a filtering bridge? More and more frequently, thanks to the lowering costs of broad band Internet connections (xDSL) and also because of the reduction of available IPv4 addresses, many companies are connected to the Internet 24 hours on 24 and with few (sometimes not even a power of 2) IP addresses. In these situations it is often desirable to have a firewall that filters incoming and outgoing traffic from and towards Internet, but a packet filtering solution based on router may not be applicable, either due to subnetting issues, the router is owned by the connectivity supplier (ISP), or because it does not support such functionalities. In these scenarios the use of a filtering bridge is highly advised. A bridge-based firewall can be configured and inserted between the xDSL router and your Ethernet hub/switch without any IP numbering issues. [[filtering-bridges-how]] == How to Install Adding bridge functionalities to a FreeBSD system is not difficult. Since 4.5 release it is possible to load such functionalities as modules instead of having to rebuild the kernel, simplifying the procedure a great deal. In the following subsections I will explain both installation ways. [IMPORTANT] ==== _Do not_ follow both instructions: a procedure _excludes_ the other one. Select the best choice according to your needs and abilities. ==== Before going on, be sure to have at least two Ethernet cards that support the promiscuous mode for both reception and transmission, since they must be able to send Ethernet packets with any address, not just their own. Moreover, to have a good throughput, the cards should be PCI bus mastering cards. The best choices are still the Intel EtherExpress(TM) Pro, followed by the 3Com(R) 3c9xx series. To simplify the firewall configuration it may be useful to have two cards of different manufacturers (using different drivers) in order to distinguish clearly which interface is connected to the router and which to the inner network. [[filtering-bridges-kernel]] === Kernel Configuration So you have decided to use the older but well tested installation method. To begin, you have to add the following rows to your kernel configuration file: [.programlisting] .... options BRIDGE options IPFIREWALL options IPFIREWALL_VERBOSE .... The first line is to compile the bridge support, the second one is the firewall and the third one is the logging functions of the firewall. Now it is necessary to build and install the new kernel. You may find detailed instructions in the link:{handbook}#kernelconfig-building[Building and Installing a Custom Kernel] section of the FreeBSD Handbook. [[filtering-bridges-modules]] === Modules Loading If you have chosen to use the new and simpler installation method, the only thing to do now is add the following row to [.filename]#/boot/loader.conf#: [.programlisting] .... bridge_load="YES" .... In this way, during the system startup, the [.filename]#bridge.ko# module will be loaded together with the kernel. It is not required to add a similar row for the [.filename]#ipfw.ko# module, since it will be loaded automatically after the execution of the steps in the following section. [[filtering-bridges-finalprep]] == Final Preparation Before rebooting in order to load the new kernel or the required modules (according to the previously chosen installation method), you have to make some changes to the [.filename]#/etc/rc.conf# configuration file. The default rule of the firewall is to reject all IP packets. Initially we will set up an `open` firewall, in order to verify its operation without any issue related to packet filtering (in case you are going to execute this procedure remotely, such configuration will avoid you to remain isolated from the network). Put these lines in [.filename]#/etc/rc.conf#: [.programlisting] .... firewall_enable="YES" firewall_type="open" firewall_quiet="YES" firewall_logging="YES" .... The first row will enable the firewall (and will load the module [.filename]#ipfw.ko# if it is not compiled in the kernel), the second one to set up it in `open` mode (as explained in [.filename]#/etc/rc.firewall#), the third one to not show rules loading and the fourth one to enable logging support. About the configuration of the network interfaces, the most used way is to assign an IP to only one of the network cards, but the bridge will work equally even if both interfaces or none has a configured IP. In the last case (IP-less) the bridge machine will be still more hidden, as inaccessible from the network: to configure it, you have to login from console or through a third network interface separated from the bridge. Sometimes, during the system startup, some programs require network access, say for domain resolution: in this case it is necessary to assign an IP to the external interface (the one connected to Internet, where DNS server resides), since the bridge will be activated at the end of the startup procedure. It means that the [.filename]#fxp0# interface (in our case) must be mentioned in the ifconfig section of the [.filename]#/etc/rc.conf# file, while the [.filename]#xl0# is not. Assigning an IP to both the network cards does not make much sense, unless, during the start procedure, applications should access to services on both Ethernet segments. There is another important thing to know. When running IP over Ethernet, there are actually two Ethernet protocols in use: one is IP, the other is ARP. ARP does the conversion of the IP address of a host into its Ethernet address (MAC layer). In order to allow the communication between two hosts separated by the bridge, it is necessary that the bridge will forward ARP packets. Such protocol is not included in the IP layer, since it exists only with IP over Ethernet. The FreeBSD firewall filters exclusively on the IP layer and therefore all non-IP packets (ARP included) will be forwarded without being filtered, even if the firewall is configured to not permit anything. Now it is time to reboot the system and use it as before: there will be some new messages about the bridge and the firewall, but the bridge will not be activated and the firewall, being in `open` mode, will not avoid any operations. If there are any problems, you should sort them out now before proceeding. [[filtering-bridges-enabling]] == Enabling the Bridge At this point, to enable the bridge, you have to execute the following commands (having the shrewdness to replace the names of the two network interfaces [.filename]#fxp0# and [.filename]#xl0# with your own ones): [source,shell] .... # sysctl net.link.ether.bridge.config=fxp0:0,xl0:0 # sysctl net.link.ether.bridge.ipfw=1 # sysctl net.link.ether.bridge.enable=1 .... The first row specifies which interfaces should be activated by the bridge, the second one will enable the firewall on the bridge and finally the third one will enable the bridge. At this point you should be able to insert the machine between two sets of hosts without compromising any communication abilities between them. If so, the next step is to add the `net.link.ether.bridge._[blah]_=_[blah]_` portions of these rows to the [.filename]#/etc/sysctl.conf# file, in order to have them execute at startup. [[filtering-bridges-ipfirewall]] == Configuring The Firewall Now it is time to create your own file with custom firewall rules, in order to secure the inside network. There will be some complication in doing this because not all of the firewall functionalities are available on bridged packets. Furthermore, there is a difference between the packets that are in the process of being forwarded and packets that are being received by the local machine. In general, incoming packets are run through the firewall only once, not twice as is normally the case; in fact they are filtered only upon receipt, so rules that use `out` or `xmit` will never match. Personally, I use `in via` which is an older syntax, but one that has a sense when you read it. Another limitation is that you are restricted to use only `pass` or `drop` commands for packets filtered by a bridge. Sophisticated things like `divert`, `forward` or `reject` are not available. Such options can still be used, but only on traffic to or from the bridge machine itself (if it has an IP address). New in FreeBSD 4.0, is the concept of stateful filtering. This is a big improvement for UDP traffic, which typically is a request going out, followed shortly thereafter by a response with the exact same set of IP addresses and port numbers (but with source and destination reversed, of course). For firewalls that have no statekeeping, there is almost no way to deal with this sort of traffic as a single session. But with a firewall that can "remember" an outgoing UDP packet and, for the next few minutes, allow a response, handling UDP services is trivial. The following example shows how to do it. It is possible to do the same thing with TCP packets. This allows you to avoid some denial of service attacks and other nasty tricks, but it also typically makes your state table grow quickly in size. Let's look at an example setup. Note first that at the top of [.filename]#/etc/rc.firewall# there are already standard rules for the loopback interface [.filename]#lo0#, so we should not have to care for them anymore. Custom rules should be put in a separate file (say [.filename]#/etc/rc.firewall.local#) and loaded at system startup, by modifying the row of [.filename]#/etc/rc.conf# where we defined the `open` firewall: [.programlisting] .... firewall_type="/etc/rc.firewall.local" .... [IMPORTANT] ==== You have to specify the _full_ path, otherwise it will not be loaded with the risk to remain isolated from the network. ==== For our example imagine to have the [.filename]#fxp0# interface connected towards the outside (Internet) and the [.filename]#xl0# towards the inside (LAN). The bridge machine has the IP `1.2.3.4` (it is not possible that your ISP can give you an address quite like this, but for our example it is good). [.programlisting] .... # Things that we have kept state on before get to go through in a hurry add check-state # Throw away RFC 1918 networks add drop all from 10.0.0.0/8 to any in via fxp0 add drop all from 172.16.0.0/12 to any in via fxp0 add drop all from 192.168.0.0/16 to any in via fxp0 # Allow the bridge machine to say anything it wants # (if the machine is IP-less do not include these rows) add pass tcp from 1.2.3.4 to any setup keep-state add pass udp from 1.2.3.4 to any keep-state add pass ip from 1.2.3.4 to any # Allow the inside hosts to say anything they want add pass tcp from any to any in via xl0 setup keep-state add pass udp from any to any in via xl0 keep-state add pass ip from any to any in via xl0 # TCP section # Allow SSH add pass tcp from any to any 22 in via fxp0 setup keep-state # Allow SMTP only towards the mail server add pass tcp from any to relay 25 in via fxp0 setup keep-state # Allow zone transfers only by the slave name server [dns2.nic.it] add pass tcp from 193.205.245.8 to ns 53 in via fxp0 setup keep-state # Pass ident probes. It is better than waiting for them to timeout add pass tcp from any to any 113 in via fxp0 setup keep-state # Pass the "quarantine" range add pass tcp from any to any 49152-65535 in via fxp0 setup keep-state # UDP section # Allow DNS only towards the name server add pass udp from any to ns 53 in via fxp0 keep-state # Pass the "quarantine" range add pass udp from any to any 49152-65535 in via fxp0 keep-state # ICMP section # Pass 'ping' add pass icmp from any to any icmptypes 8 keep-state # Pass error messages generated by 'traceroute' add pass icmp from any to any icmptypes 3 add pass icmp from any to any icmptypes 11 # Everything else is suspect add drop log all from any to any .... Those of you who have set up firewalls before may notice some things missing. In particular, there are no anti-spoofing rules, in fact we did _not_ add: [.programlisting] .... add deny all from 1.2.3.4/8 to any in via fxp0 .... That is, drop packets that are coming in from the outside claiming to be from our network. This is something that you would commonly do to be sure that someone does not try to evade the packet filter, by generating nefarious packets that look like they are from the inside. The problem with that is that there is _at least_ one host on the outside interface that you do not want to ignore: the router. But usually, the ISP anti-spoofs at their router, so we do not need to bother that much. The last rule seems to be an exact duplicate of the default rule, that is, do not let anything pass that is not specifically allowed. But there is a difference: all suspected traffic will be logged. There are two rules for passing SMTP and DNS traffic towards the mail server and the name server, if you have them. Obviously the whole rule set should be flavored to personal taste, this is only a specific example (rule format is described accurately in the man:ipfw[8] man page). Note that in order for "relay" and "ns" to work, name service lookups must work _before_ the bridge is enabled. This is an example of making sure that you set the IP on the correct network card. Alternatively it is possible to specify the IP address instead of the host name (required if the machine is IP-less). People that are used to setting up firewalls are probably also used to either having a `reset` or a `forward` rule for ident packets (TCP port 113). Unfortunately, this is not an applicable option with the bridge, so the best thing is to simply pass them to their destination. As long as that destination machine is not running an ident daemon, this is relatively harmless. The alternative is dropping connections on port 113, which creates some problems with services like IRC (the ident probe must timeout). The only other thing that is a little weird that you may have noticed is that there is a rule to let the bridge machine speak, and another for internal hosts. Remember that this is because the two sets of traffic will take different paths through the kernel and into the packet filter. The inside net will go through the bridge, while the local machine will use the normal IP stack to speak. Thus the two rules to handle the different cases. The `in via fxp0` rules work for both paths. In general, if you use `in via` rules throughout the filter, you will need to make an exception for locally generated packets, because they did not come in via any of our interfaces. [[filtering-bridges-contributors]] == Contributors Many parts of this article have been taken, updated and adapted from an old text about bridging, edited by Nick Sayer. A pair of inspirations are due to an introduction on bridging by Steve Peterson. A big thanks to Luigi Rizzo for the implementation of the bridge code in FreeBSD and for the time he has dedicated to me answering all of my related questions. A thanks goes out also to Tom Rhodes who looked over my job of translation from Italian (the original language of this article) into English. diff --git a/documentation/content/en/articles/fonts/_index.adoc b/documentation/content/en/articles/fonts/_index.adoc index 70e52a1a91..34b12ba906 100644 --- a/documentation/content/en/articles/fonts/_index.adoc +++ b/documentation/content/en/articles/fonts/_index.adoc @@ -1,581 +1,581 @@ --- title: Fonts and FreeBSD subtitle: A Tutorial authors: - author: Dave Bodenstab email: imdave@synet.net -description: Description of the various font files that may be used with FreeBSD +description: A description of the various font technologies in FreeBSD, and how to use them with different programs trademarks: ["freebsd", "adobe", "apple", "linux", "microsoft", "opengroup", "general"] --- = Fonts and FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :source-highlighter: rouge :experimental: :sectnumlevels: 6 [.abstract-title] Abstract This document contains a description of the various font files that may be used with FreeBSD and the syscons driver, X11, Ghostscript and Groff. Cookbook examples are provided for switching the syscons display to 80x60 mode, and for using type 1 fonts with the above application programs. ''' toc::[] [[intro]] == Introduction There are many sources of fonts available, and one might ask how they might be used with FreeBSD. The answer can be found by carefully searching the documentation for the component that one would like to use. This is very time consuming, so this tutorial is an attempt to provide a shortcut for others who might be interested. [[terminology]] == Basic Terminology There are many different font formats and associated font file suffixes. A few that will be addressed here are: [.filename]#.pfa#, [.filename]#.pfb#:: PostScript(R) type 1 fonts. The [.filename]#.pfa# is the __A__scii form and [.filename]#.pfb# the __B__inary form. [.filename]#.afm#:: The font metrics associated with a type 1 font. [.filename]#.pfm#:: The printer font metrics associated with a type 1 font. [.filename]#.ttf#:: A TrueType(R) font [.filename]#.fot#:: An indirect reference to a TrueType font (not an actual font) [.filename]#.fon#, [.filename]#.fnt#:: Bitmapped screen fonts The [.filename]#.fot# is used by Windows(R) as sort of a symbolic link to the actual TrueType(R) font ([.filename]#.ttf#) file. The [.filename]#.fon# font files are also used by Windows. I know of no way to use this font format with FreeBSD. [[font-formats]] == What Font Formats Can I Use? Which font file format is useful depends on the application being used. FreeBSD by itself uses no fonts. Application programs and/or drivers may make use of the font files. Here is a small cross reference of application/driver to the font type suffixes: Driver:: vt::: [.filename]#.hex# syscons::: [.filename]#.fnt# Application:: Ghostscript::: [.filename]#.pfa#, [.filename]#.pfb#, [.filename]#.ttf# X11::: [.filename]#.pfa#, [.filename]#.pfb# Groff::: [.filename]#.pfa#, [.filename]#.afm# Povray::: [.filename]#.ttf# The [.filename]#.fnt# suffix is used quite frequently. I suspect that whenever someone wanted to create a specialized font file for their application, more often than not they chose this suffix. Therefore, it is likely that files with this suffix are not all the same format; specifically, the [.filename]#.fnt# files used by syscons under FreeBSD may not be the same format as a [.filename]#.fnt# one encounters in the MS-DOS(R)/Windows(R) environment. I have not made any attempt at using other [.filename]#.fnt# files other than those provided with FreeBSD. [[virtual-console]] == Setting a Virtual Console to 80x60 Line Mode First, an 8x8 font must be loaded. To do this, [.filename]#/etc/rc.conf# should contain the line (change the font name to an appropriate one for your locale): [.programlisting] .... font8x8="iso-8x8" # font 8x8 from /usr/share/syscons/fonts/* (or NO). .... The command to actually switch the mode is man:vidcontrol[1]: [source,shell] .... % vidcontrol VGA_80x60 .... Various screen-oriented programs, such as man:vi[1], must be able to determine the current screen dimensions. As this is achieved this through `ioctl` calls to the console driver (such as man:syscons[4]) they will correctly determine the new screen dimensions. To make this more seamless, one can embed these commands in the startup scripts so it takes place when the system boots. To do this is add this line to [.filename]#/etc/rc.conf#. [.programlisting] .... allscreens_flags="VGA_80x60" # Set this vidcontrol mode for all virtual screens .... References: man:rc.conf[5], man:vidcontrol[1]. [[type1-fonts-x11]] == Using Type 1 Fonts with X11 X11 can use either the [.filename]#.pfa# or the [.filename]#.pfb# format fonts. The X11 fonts are located in various subdirectories under [.filename]#/usr/X11R6/lib/X11/fonts#. Each font file is cross referenced to its X11 name by the contents of [.filename]#fonts.dir# in each directory. There is already a directory named [.filename]#Type1#. The most straight forward way to add a new font is to put it into this directory. A better way is to keep all new fonts in a separate directory and use a symbolic link to the additional font. This allows one to more easily keep track of ones fonts without confusing them with the fonts that were originally provided. For example: [source,shell] .... Create a directory to contain the font files % mkdir -p /usr/local/share/fonts/type1 % cd /usr/local/share/fonts/type1 Place the .pfa, .pfb and .afm files here One might want to keep readme files, and other documentation for the fonts here also % cp /cdrom/fonts/atm/showboat/showboat.pfb . % cp /cdrom/fonts/atm/showboat/showboat.afm . Maintain an index to cross reference the fonts % echo showboat - InfoMagic CICA, Dec 1994, /fonts/atm/showboat >>INDEX .... Now, to use a new font with X11, one must make the font file available and update the font name files. The X11 font names look like: [.programlisting] .... -bitstream-charter-medium-r-normal-xxx-0-0-0-0-p-0-iso8859-1 | | | | | | | | | | | | \ \ | | | | | \ \ \ \ \ \ \ +----+- character set | | | | \ \ \ \ \ \ \ +- average width | | | | \ \ \ \ \ \ +- spacing | | | \ \ \ \ \ \ +- vertical res. | | | \ \ \ \ \ +- horizontal res. | | | \ \ \ \ +- points | | | \ \ \ +- pixels | | | \ \ \ foundry family weight slant width additional style .... A new name needs to be created for each new font. If you have some information from the documentation that accompanied the font, then it could serve as the basis for creating the name. If there is no information, then you can get some idea by using man:strings[1] on the font file. For example: [source,shell] .... % strings showboat.pfb | more %!FontType1-1.0: Showboat 001.001 %%CreationDate: 1/15/91 5:16:03 PM %%VMusage: 1024 45747 % Generated by Fontographer 3.1 % Showboat 1991 by David Rakowski. Alle Rechte Vorbehalten. FontDirectory/Showboat known{/Showboat findfont dup/UniqueID known{dup /UniqueID get 4962377 eq exch/FontType get 1 eq and}{pop false}ifelse {save true}{false}ifelse}{false}ifelse 12 dict begin /FontInfo 9 dict dup begin /version (001.001) readonly def /FullName (Showboat) readonly def /FamilyName (Showboat) readonly def /Weight (Medium) readonly def /ItalicAngle 0 def /isFixedPitch false def /UnderlinePosition -106 def /UnderlineThickness 16 def /Notice (Showboat 1991 by David Rakowski. Alle Rechte Vorbehalten.) readonly def end readonly def /FontName /Showboat def --stdin-- .... Using this information, a possible name might be: [source,shell] .... -type1-Showboat-medium-r-normal-decorative-0-0-0-0-p-0-iso8859-1 .... The components of our name are: Foundry:: Lets just name all the new fonts `type1`. Family:: The name of the font. Weight:: Normal, bold, medium, semibold, etc. From the man:strings[1] output above, it appears that this font has a weight of __medium__. Slant:: __r__oman, __i__talic, __o__blique, etc. Since the _ItalicAngle_ is zero, _roman_ will be used. Width:: Normal, wide, condensed, extended, etc. Until it can be examined, the assumption will be __normal__. Additional style:: Usually omitted, but this will indicate that the font contains decorative capital letters. Spacing:: proportional or monospaced. _Proportional_ is used since _isFixedPitch_ is false. All of these names are arbitrary, but one should strive to be compatible with the existing conventions. A font is referenced by name with possible wild cards by an X11 program, so the name chosen should make some sense. One might begin by simply using [source,shell] .... ...-normal-r-normal-...-p-... .... as the name, and then use man:xfontsel[1] to examine it and adjust the name based on the appearance of the font. So, to complete our example: [source,shell] .... Make the font accessible to X11 % cd /usr/X11R6/lib/X11/fonts/Type1 % ln -s /usr/local/share/fonts/type1/showboat.pfb . Edit fonts.dir and fonts.scale, adding the line describing the font and incrementing the number of fonts which is found on the first line. % ex fonts.dir :1p 25 :1c 26 . :$a showboat.pfb -type1-showboat-medium-r-normal-decorative-0-0-0-0-p-0-iso8859-1 . :wq fonts.scale seems to be identical to fonts.dir... % cp fonts.dir fonts.scale Tell X11 that things have changed % xset fp rehash Examine the new font % xfontsel -pattern -type1-* .... References: man:xfontsel[1], man:xset[1], The X Windows System in a Nutshell, http://www.ora.com/[O'Reilly & Associates]. [[type1-fonts-ghostscript]] == Using Type 1 Fonts with Ghostscript Ghostscript references a font via its [.filename]#Fontmap#. This must be modified in a similar way to the X11 [.filename]#fonts.dir#. Ghostscript can use either the [.filename]#.pfa# or the [.filename]#.pfb# format fonts. Using the font from the previous example, here is how to use it with Ghostscript: [source,shell] .... Put the font in Ghostscript's font directory % cd /usr/local/share/ghostscript/fonts % ln -s /usr/local/share/fonts/type1/showboat.pfb . Edit Fontmap so Ghostscript knows about the font % cd /usr/local/share/ghostscript/4.01 % ex Fontmap :$a /Showboat (showboat.pfb) ; % From CICA /fonts/atm/showboat . :wq Use Ghostscript to examine the font % gs prfont.ps Aladdin Ghostscript 4.01 (1996-7-10) Copyright (C) 1996 Aladdin Enterprises, Menlo Park, CA. All rights reserved. This software comes with NO WARRANTY: see the file PUBLIC for details. Loading Times-Roman font from /usr/local/share/ghostscript/fonts/tir_____.pfb... /1899520 581354 1300084 13826 0 done. GS>Showboat DoFont Loading Showboat font from /usr/local/share/ghostscript/fonts/showboat.pfb... 1939688 565415 1300084 16901 0 done. >>showpage, press to continue<< >>showpage, press to continue<< >>showpage, press to continue<< GS>quit .... References: [.filename]#fonts.txt# in the Ghostscript 4.01 distribution [[type1-fonts-groff]] == Using Type 1 Fonts with Groff Now that the new font can be used by both X11 and Ghostscript, how can one use the new font with groff? First of all, since we are dealing with type 1 PostScript(R) fonts, the groff device that is applicable is the _ps_ device. A font file must be created for each font that groff can use. A groff font name is just a file in [.filename]#/usr/share/groff_font/devps#. With our example, the font file could be [.filename]#/usr/share/groff_font/devps/SHOWBOAT#. The file must be created using tools provided by groff. The first tool is `afmtodit`. This is not normally installed, so it must be retrieved from the source distribution. I found I had to change the first line of the file, so I did: [source,shell] .... % cp /usr/src/gnu/usr.bin/groff/afmtodit/afmtodit.pl /tmp % ex /tmp/afmtodit.pl :1c #!/usr/bin/perl -P- . :wq .... This tool will create the groff font file from the metrics file ([.filename]#.afm# suffix.) Continuing with our example: [source,shell] .... Many .afm files are in Mac format... ^M delimited lines We need to convert them to UNIX(R) style ^J delimited lines % cd /tmp % cat /usr/local/share/fonts/type1/showboat.afm | tr '\015' '\012' >showboat.afm Now create the groff font file % cd /usr/share/groff_font/devps % /tmp/afmtodit.pl -d DESC -e text.enc /tmp/showboat.afm generate/textmap SHOWBOAT .... The font can now be referenced with the name SHOWBOAT. If Ghostscript is used to drive the printers on the system, then nothing more needs to be done. However, if true PostScript(R) printers are used, then the font must be downloaded to the printer in order for the font to be used (unless the printer happens to have the showboat font built in or on an accessible font disk.) The final step is to create a downloadable font. The `pfbtops` tool is used to create the [.filename]#.pfa# format of the font, and [.filename]#download# is modified to reference the new font. The [.filename]#download# must reference the internal name of the font. This can easily be determined from the groff font file as illustrated: [source,shell] .... Create the .pfa font file % pfbtops /usr/local/share/fonts/type1/showboat.pfb >showboat.pfa .... Of course, if [.filename]#.pfa# is already available, just use a symbolic link to reference it. [source,shell] .... Get the internal font name % fgrep internalname SHOWBOAT internalname Showboat Tell groff that the font must be downloaded % ex download :$a Showboat showboat.pfa . :wq .... To test the font: [source,shell] .... % cd /tmp % cat >example.t <example.ps To use ghostscript/ghostview % ghostview example.ps To print it % lpr -Ppostscript example.ps .... References: [.filename]#/usr/src/gnu/usr.bin/groff/afmtodit/afmtodit.man#, man:groff_font[5], man:groff_char[7], man:pfbtops[1]. [[convert-truetype]] == Converting TrueType Fonts to a groff/PostScript Format For groff This potentially requires a bit of work, simply because it depends on some utilities that are not installed as part of the base system. They are: `ttf2pf`:: TrueType to PostScript conversion utilities. This allows conversion of a TrueType font to an ascii font metric ([.filename]#.afm#) file. + Currently available at http://sunsite.icm.edu.pl/pub/GUST/contrib/BachoTeX98/ttf2pf/[http://sunsite.icm.edu.pl/pub/GUST/contrib/BachoTeX98/ttf2pf/]. Note: These files are PostScript programs and must be downloaded to disk by holding down kbd:[Shift] when clicking on the link. Otherwise, your browser may try to launch ghostview to view them. + The files of interest are: ** [.filename]#GS_TTF.PS# ** [.filename]#PF2AFM.PS# ** [.filename]#ttf2pf.ps# + The funny upper/lower case is due to their being intended also for DOS shells. [.filename]#ttf2pf.ps# makes use of the others as upper case, so any renaming must be consistent with this. (Actually, [.filename]#GS_TTF.PS# and [.filename]#PFS2AFM.PS# are supposedly part of the Ghostscript distribution, but it is just as easy to use these as an isolated utility. FreeBSD does not seem to include the latter.) You also may want to have these installed to [.filename]#/usr/local/share/groff_font/devps#(?). `afmtodit`:: Creates font files for use with groff from ascii font metrics file. This usually resides in the directory, [.filename]#/usr/src/contrib/groff/afmtodit#, and requires some work to get going. + [NOTE] ==== If you are paranoid about working in the [.filename]#/usr/src# tree, simply copy the contents of the above directory to a work location. ==== + In the work area, you will need to make the utility. Just type: + [source,shell] .... # make -f Makefile.sub afmtodit .... + You may also need to copy [.filename]#/usr/contrib/groff/devps/generate/textmap# to [.filename]#/usr/share/groff_font/devps/generate# if it does not already exist. Once all these utilities are in place, you are ready to commence: . Create [.filename]#.afm# by typing: + [source,shell] .... % gs -dNODISPLAY -q -- ttf2pf.ps TTF_name PS_font_name AFM_name .... + Where, _TTF_name_ is your TrueType font file, _PS_font_name_ is the file name for [.filename]#.pfa#, _AFM_name_ is the name you wish for [.filename]#.afm#. If you do not specify output file names for the [.filename]#.pfa# or [.filename]#.afm# files, then default names will be generated from the TrueType font file name. + This also produces a [.filename]#.pfa#, the ascii PostScript font metrics file ([.filename]#.pfb# is for the binary form). This will not be needed, but could (I think) be useful for a fontserver. + For example, to convert the 30f9 Barcode font using the default file names, use the following command: + [source,shell] .... % gs -dNODISPLAY -- ttf2pf.ps 3of9.ttf Aladdin Ghostscript 5.10 (1997-11-23) Copyright (C) 1997 Aladdin Enterprises, Menlo Park, CA. All rights reserved. This software comes with NO WARRANTY: see the file PUBLIC for details. Converting 3of9.ttf to 3of9.pfa and 3of9.afm. .... + If you want the converted fonts to be stored in [.filename]#A.pfa# and [.filename]#B.afm#, then use this command: + [source,shell] .... % gs -dNODISPLAY -- ttf2pf.ps 3of9.ttf A B Aladdin Ghostscript 5.10 (1997-11-23) Copyright (C) 1997 Aladdin Enterprises, Menlo Park, CA. All rights reserved. This software comes with NO WARRANTY: see the file PUBLIC for details. Converting 3of9.ttf to A.pfa and B.afm. .... . Create the groff PostScript file: + Change directories to [.filename]#/usr/share/groff_font/devps# so as to make the following command easier to execute. You will probably need root privileges for this. (Or, if you are paranoid about working there, make sure you reference the files [.filename]#DESC#, [.filename]#text.enc# and [.filename]#generate/textmap# as being in this directory.) + [source,shell] .... % afmtodit -d DESC -e text.enc file.afm generate/textmap PS_font_name .... + Where, [.filename]#file.afm# is the _AFM_name_ created by `ttf2pf.ps` above, and _PS_font_name_ is the font name used from that command, as well as the name that man:groff[1] will use for references to this font. For example, assuming you used the first `tiff2pf.ps` above, then the 3of9 Barcode font can be created using the command: + [source,shell] .... % afmtodit -d DESC -e text.enc 3of9.afm generate/textmap 3of9 .... + Ensure that the resulting _PS_font_name_ file (e.g., [.filename]#3of9# in the example above) is located in the directory [.filename]#/usr/share/groff_font/devps# by copying or moving it there. + Note that if [.filename]#ttf2pf.ps# assigns a font name using the one it finds in the TrueType font file and you want to use a different name, you must edit the [.filename]#.afm# prior to running `afmtodit`. This name must also match the one used in the Fontmap file if you wish to pipe man:groff[1] into man:gs[1]. [[truetype-for-other-programs]] == Can TrueType Fonts be Used with Other Programs? The TrueType font format is used by Windows, Windows 95, and Mac's. It is quite popular and there are a great number of fonts available in this format. Unfortunately, there are few applications that I am aware of that can use this format: Ghostscript and Povray come to mind. Ghostscript's support, according to the documentation, is rudimentary and the results are likely to be inferior to type 1 fonts. Povray version 3 also has the ability to use TrueType fonts, but I rather doubt many people will be creating documents as a series of raytraced pages :-). This rather dismal situation may soon change. The http://www.freetype.org/[FreeType Project] is currently developing a useful set of FreeType tools: * The `xfsft` font server for X11 can serve TrueType fonts in addition to regular fonts. Though currently in beta, it is said to be quite usable. See http://www.dcs.ed.ac.uk/home/jec/programs/xfsft/[Juliusz Chroboczek's page] for further information. Porting instructions for FreeBSD can be found at http://math.missouri.edu/~stephen/software/[Stephen Montgomery's software page]. * xfstt is another font server for X11, available under link:ftp://sunsite.unc.edu/pub/Linux/X11/fonts/[ftp://sunsite.unc.edu/pub/Linux/X11/fonts/]. * A program called `ttf2bdf` can produce BDF files suitable for use in an X environment from TrueType files. Linux binaries are said to be available from link:ftp://crl.nmsu.edu/CLR/multiling/General/[ftp://crl.nmsu.edu/CLR/multiling/General/]. * and others ... [[obtaining-additional-fonts]] == Where Can Additional Fonts be Obtained? Many fonts are available on the Internet. They are either entirely free, or are share-ware. In addition many fonts are available in the [.filename]#x11-fonts/# category in the ports collection [[additional-questions]] == Additional Questions * What use are the [.filename]#.pfm# files? * Can one generate the [.filename]#.afm# from a [.filename]#.pfa# or [.filename]#.pfb#? * How to generate the groff character mapping files for PostScript fonts with non-standard character names? * Can xditview and devX?? devices be set up to access all the new fonts? * It would be good to have examples of using TrueType fonts with Povray and Ghostscript. diff --git a/documentation/content/en/articles/freebsd-releng/_index.adoc b/documentation/content/en/articles/freebsd-releng/_index.adoc index 5615986124..bbb3f86ff9 100644 --- a/documentation/content/en/articles/freebsd-releng/_index.adoc +++ b/documentation/content/en/articles/freebsd-releng/_index.adoc @@ -1,863 +1,863 @@ --- title: FreeBSD Release Engineering authors: - author: Glen Barber email: gjb@FreeBSD.org organizations: - organization: The FreeBSD Foundation webpage: https://www.freebsdfoundation.org/ - organization: Rubicon Communications, LLC (Netgate) webpage: https://www.netgate.com/ -description: FreeBSD Release Engineering +description: Describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization trademarks: ["freebsd", "intel", "general"] --- = FreeBSD Release Engineering :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :teamBugmeister: FreeBSD Bugmeister Team :teamDoceng: FreeBSD Documentation Engineering Team :teamPortmgr: FreeBSD Ports Management Team :teamPostmaster: FreeBSD Postmaster Team :teamRe: FreeBSD Release Engineering Team :teamSecteam: FreeBSD Security Team :branchHead: head/ :branchStable: stable/ :branchStablex: stable/12/ :branchReleng: releng/ :branchRelengx: releng/12.0/ :branchReleasex: release/12.0.0/ :branchRevision: 12.0 [.abstract-title] Abstract This article describes the release engineering process of the FreeBSD Project. ''' toc::[] [[introduction]] == Introduction to the FreeBSD Release Engineering Process Development of FreeBSD has a very specific workflow. In general, all changes to the FreeBSD base system are committed to the {branchHead} branch, which reflects the top of the source tree. After a reasonable testing period, changes can then be merged to the {branchStable} branches. The default minimum timeframe before merging to {branchStable} branches is three (3) days. Although a general rule to wait a minimum of three days before merging from {branchHead}, there are a few special circumstances where an immediate merge may be necessary, such as a critical security fix, or a bug fix that directly inhibits the release build process. After several months, and the number of changes in the {branchStable} branch have grown significantly, it is time to release the next version of FreeBSD. These releases have been historically referred to as "point" releases. In between releases from the {branchStable} branches, approximately every two (2) years, a release will be cut directly from {branchHead}. These releases have been historically referred to as "dot-zero" releases. This article will highlight the workflow and responsibilities of the {teamRe} for both "dot-zero" and "point"' releases. The following sections of this article describe: <>:: General information and preparation before starting the release cycle. <>:: Website Changes During the Release Cycle <>:: Terminology and general information, such as the "code slush" and "code freeze", used throughout this document. <>:: The Release Engineering process for a "dot-zero" release. <>:: The Release Engineering process for a "point" release. <>:: Information related to the specific procedures to build installation medium. <>:: Procedures to publish installation medium. <>:: Wrapping up the release cycle. [[releng-prep]] == General Information and Preparation Approximately two months before the start of the release cycle, the {teamRe} decides on a schedule for the release. The schedule includes the various milestone points of the release cycle, such as freeze dates, branch dates, and build dates. For example: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | Milestone | Anticipated Date |{branchHead} slush: |May 27, 2016 |{branchHead} freeze: |June 10, 2016 |{branchHead} KBI freeze: |June 24, 2016 |`doc/` tree slush [1]: |June 24, 2016 |Ports quarterly branch [2]: |July 1, 2016 |{branchStablex} branch: |July 8, 2016 |`doc/` tree tag [3]: |July 8, 2016 |BETA1 build starts: |July 8, 2016 |{branchHead} thaw: |July 9, 2016 |BETA2 build starts: |July 15, 2016 |BETA3 build starts [*]: |July 22, 2016 |{branchRelengx} branch: |July 29, 2016 |RC1 build starts: |July 29, 2016 |{branchStablex} thaw: |July 30, 2016 |RC2 build starts: |August 5, 2016 |Final Ports package builds [4]: |August 6, 2016 |Ports release tag: |August 12, 2016 |RC3 build starts [*]: |August 12, 2016 |RELEASE build starts: |August 19, 2016 |RELEASE announcement: |September 2, 2016 |=== [NOTE] ==== Items marked with "[*]" are "as needed". ==== . The `doc/` tree slush is coordinated by the {teamDoceng}. . The Ports quarterly branch used is determined by when the final `RC` build is planned. A new quarterly branch is created on the first day of the quarter, so this metric should be used when taking the release cycle milestones into account. The quarterly branch is created by the {teamPortmgr}. . The `doc/` tree is tagged by the {teamDoceng}. . The final Ports package build is done by the {teamPortmgr} after the final (or what is expected to be final) `RC` build. [NOTE] ==== If the release is being created from an existing {branchStable} branch, the KBI freeze date can be excluded, since the KBI is already considered frozen on established {branchStable} branches. ==== When writing the release cycle schedule, a number of things need to be taken into consideration, in particular milestones where the target date depends on predefined milestones upon which there is a dependency. For example, the Ports Collection release tag originates from the active quarterly branch at the time of the last `RC`. This in part defines which quarterly branch is used, when the release tag can happen, and what revision of the ports tree is used for the final `RELEASE` build. After general agreement on the schedule, the {teamRe} emails the schedule to the FreeBSD Developers. It is somewhat typical that many developers will inform the {teamRe} about various works-in-progress. In some cases, an extension for the in-progress work will be requested, and in other cases, a request for "blanket approval" to a particular subset of the tree will be made. When such requests are made, it is important to make sure timelines (even if estimated) are discussed. For blanket approvals, the length of time for the blanket approval should be made clear. For example, a FreeBSD developer may request blanket approvals from the start of the code slush until the start of the `RC` builds. [NOTE] ==== In order to keep track of blanket approvals, the {teamRe} uses an internal repository to keep a running log of such requests, which defines the area upon which a blanket approval was granted, the author(s), when the blanket approval expires, and the reason the approval was granted. One example of this is granting blanket approval to [.filename]#release/doc/# to all {teamRe} members until the final `RC` to update the release notes and other release-related documentation. ==== [NOTE] ==== The {teamRe} also uses this repository to track pending approval requests that are received just prior to starting various builds during the release cycle, which the Release Engineer specifies the cutoff period with an email to the FreeBSD developers. ==== Depending on the underlying set of code in question, and the overall impact the set of code has on FreeBSD as a whole, such requests may be approved or denied by the {teamRe}. The same applies to work-in-progress extensions. For example, in-progress work for a new device driver that is otherwise isolated from the rest of the tree may be granted an extension. A new scheduler, however, may not be feasible, especially if such dramatic changes do not exist in another branch. The schedule is also added to the Project website, in the `doc/` repository, in [.filename]#~/website/content/en/releases/{branchRevision}R/schedule.adoc#. This file is continuously updated as the release cycle progresses. [NOTE] ==== In most cases, the [.filename]#schedule.adoc# can be copied from a prior release and updated accordingly. ==== In addition to adding [.filename]#schedule.adoc# to the website, [.filename]#~/shared/releases.adoc# is also updated to add the link to the schedule to various subpages, as well as enabling the link to the schedule on the Project website index page. The schedule is also linked from [.filename]#~/website/content/en/releng/_index.adoc#. Approximately one month prior to the scheduled "code slush", the {teamRe} sends a reminder email to the FreeBSD Developers. [[releng-terms]] == Release Engineering Terminology This section describes some of the terminology used throughout the rest of this document. [[releng-terms-code-slush]] === The Code Slush Although the code slush is not a hard freeze on the tree, the {teamRe} requests that bugs in the existing code base take priority over new features. The code slush does not enforce commit approvals to the branch. [[releng-terms-code-freeze]] === The Code Freeze The code freeze marks the point in time where all commits to the branch require explicit approval from the {teamRe}. The FreeBSD Subversion repository contains several hooks to perform sanity checks before any commit is actually committed to the tree. One of these hooks will evaluate if committing to a particular branch requires specific approval. To enforce commit approvals by the {teamRe}, the Release Engineer updates [.filename]#base/svnadmin/conf/approvers#, and commits the change back to the repository. Once this is done, any change to the branch must include an "Approved by:" line in the commit message. The "Approved by:" line must match the second column in [.filename]#base/svnadmin/conf/approvers#, otherwise the commit will be rejected by the repository hooks. [NOTE] ==== During the code freeze, FreeBSD committers are urged to follow the link:https://wiki.freebsd.org/Releng/ChangeRequestGuidelines[Change Request Guidelines]. ==== [[releng-terms-kbi-freeze]] === The KBI/KPI Freeze KBI/KPI stability implies that the caller of a function across two different releases of software that implement the function results in the same end state. The caller, whether it is a process, thread, or function, expects the function to operate in a certain way, otherwise the KBI/KPI stability on the branch is broken. [[releng-website]] == Website Changes During the Release Cycle This section describes the changes to the website that should occur as the release cycle progresses. [NOTE] ==== The files specified throughout this section are relative to the `head/` branch of the `doc` repository in Subversion. ==== [[releng-website-prerelease]] === Website Changes Before the Release Cycle Begins When the release cycle schedule is available, these files need to be updated to enable various different functionalities on the FreeBSD Project website: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#~/shared/releases.adoc# |Change `beta-upcoming` from `IGNORE` to `INCLUDE` |[.filename]#~/shared/releases.adoc# |Change `beta-testing` from `IGNORE` to `INCLUDE` |=== [[releng-website-beta-rc]] === Website Changes During `BETA` or `RC` When transitioning from `PRERELEASE` to `BETA`, these files need to be updated to enable the "Help Test" block on the download page. All files are relative to [.filename]#head/# in the `doc` repository: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#share/releases.adoc# |Update `betarel-vers` to `BETA__1__` |[.filename]#~/website/data/en/news.toml# |Add an entry announcing the `BETA` |[.filename]#~/website/static/security/advisory-template.txt# |Add the new `BETA`, `RC`, or final `RELEASE` to the template |[.filename]#~/website/static/security/errata-template.txt# |Add the new `BETA`, `RC`, or final `RELEASE` to the template |=== Once the {branchRelengx} branch is created, the various release-related documents need to be generated and manually added to the `doc/` repository. Within [.filename]#release/doc#, invoke to generate [.filename]#errata.html#, [.filename]#hardware.html#, [.filename]#readme.html#, and [.filename]#relnotes.html# pages, which are then added to [.filename]#doc/head/en_US.ISO8859-1/htdocs/releases/X.YR/#, where _X.Y_ represents the major and minor version number of the release. The `fbsd:nokeywords` property must be set to `on` on the newly-added files before the pre-commit hooks will allow them to be added to the repository. [NOTE] ==== The relevant release-related documents exist in the [.filename]#doc# repository for FreeBSD 12.x and later. ==== [[releng-ports-beta-rc]] === Ports Changes During `BETA`, `RC`, and the Final `RELEASE` For each build during the release cycle, the `MANIFEST` files containing the `SHA256` of the various distribution sets, such as `base.txz`, `kernel.txz`, and so on, are added to the package:misc/freebsd-release-manifests[] port. This allows utilities other than , such as package:ports-mgmt/poudriere[], to safely use these distribution sets by providing a mechanism through which the checksums can be verified. [[releng-head]] == Release from {branchHead} This section describes the general procedures of the FreeBSD release cycle from the {branchHead} branch. [[releng-head-builds-alpha]] === FreeBSD "`ALPHA`" Builds Starting with the FreeBSD 10.0-RELEASE cycle, the notion of "`ALPHA`" builds was introduced. Unlike the `BETA` and `RC` builds, `ALPHA` builds are not included in the FreeBSD Release schedule. The idea behind `ALPHA` builds is to provide regular FreeBSD-provided builds before the creation of the {branchStable} branch. FreeBSD `ALPHA` snapshots should be built approximately once a week. For the first `ALPHA` build, the `BRANCH` value in [.filename]#sys/conf/newvers.sh# needs to be changed from `CURRENT` to `ALPHA1`. For subsequent `ALPHA` builds, increment each `ALPHA__N__` value by one. See <> for information on building the `ALPHA` images. [[releng-head-branching]] === Creating the {branchStablex} Branch When creating the {branchStable} branch, several changes are required in both the new {branchStable} branch and the {branchHead} branch. The files listed are relative to the repository root. To create the new {branchStablex} branch in Subversion: [source,shell,subs="attributes"] .... % svn cp ^/head {branchStablex} .... Once the {branchStablex} branch has been committed, make the following edits: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#stable/12/UPDATING# |Update the FreeBSD version, and remove the notice about `WITNESS` |[.filename]#stable/12/contrib/jemalloc/include/jemalloc/jemalloc_FreeBSD.h# a| [source,shell,subs="attributes"] .... #ifndef MALLOC_PRODUCTION #define MALLOC_PRODUCTION #endif .... |[.filename]#stable/12/lib/clang/llvm.build.mk# |Uncomment `-DNDEBUG` |[.filename]#stable/12/sys/\*/conf/GENERIC*# |Remove debugging support |[.filename]#stable/12/sys/*/conf/MINIMAL# |Remove debugging support |[.filename]#stable/12/release/release.conf.sample# |Update `SRCBRANCH` |[.filename]#stable/12/sys/*/conf/GENERIC-NODEBUG# |Remove these kernel configurations |[.filename]#stable/12/sys/arm/conf/std.arm*# |Remove debugging options |[.filename]#stable/12/sys/conf/newvers.sh# |Update the `BRANCH` value to reflect `BETA1` |[.filename]#stable/12/shared/mk/src.opts.mk# |Move `REPRODUCIBLE_BUILD` from `\__DEFAULT_NO_OPTIONS` to `__DEFAULT_YES_OPTIONS` |[.filename]#stable/12/shared/mk/src.opts.mk# |Move `LLVM_ASSERTIONS` from `\__DEFAULT_YES_OPTIONS` to `__DEFAULT_NO_OPTIONS` (FreeBSD 13.x and later only) |[.filename]#stable/12/libexec/rc/rc.conf# |Set `dumpdev` from `AUTO` to `NO` (it is configurable via for those that want it enabled by default) |[.filename]#stable/12/release/Makefile# |Remove the `debug.witness.trace` entries |=== Then in the {branchHead} branch, which will now become a new major version: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#head/UPDATING# |Update the FreeBSD version |[.filename]#head/sys/conf/newvers.sh# |Update the `BRANCH` value to reflect `CURRENT`, and increment `REVISION` |[.filename]#head/Makefile.inc1# |Update `TARGET_TRIPLE` and `MACHINE_TRIPLE` |[.filename]#head/sys/sys/param.h# |Update `__FreeBSD_version` |[.filename]#head/gnu/usr.bin/cc/cc_tools/freebsd-native.h# |Update `FBSD_MAJOR` and `FBSD_CC_VER` |[.filename]#head/contrib/gcc/config.gcc# |Append the `freebsdversion.h` section |[.filename]#head/lib/clang/llvm.build.mk# |Update the value of `OS_VERSION` |[.filename]#head/lib/clang/freebsd_cc_version.h# |Update `FREEBSD_CC_VERSION` |[.filename]#head/lib/clang/include/lld/Common/Version.inc# |Update `LLD_REVISION_STRING` |[.filename]#head/Makefile.libcompat# |Update `LIB32CPUFLAGS` |=== [[releng-stable]] == Release from {branchStable} This section describes the general procedures of the FreeBSD release cycle from an extablished {branchStable} branch. [[releng-stable-slush]] === FreeBSD `stable` Branch Code Slush In preparation for the code freeze on a `stable` branch, several files need to be updated to reflect the release cycle is officially in progress. These files are all relative to the top-most level of the stable branch: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#sys/conf/newvers.sh# |Update the `BRANCH` value to reflect `PRERELEASE` |[.filename]#Makefile.inc1# |Update `TARGET_TRIPLE` |[.filename]#lib/clang/llvm.build.mk# |Update `OS_VERSION` |[.filename]#Makefile.libcompat# |Update `LIB32CPUFLAGS` |[.filename]#gnu/usr.bin/groff/tmac/mdoc.local.in# |Add a new `.ds` entry for the FreeBSD version, and update `doc-default-operating-system` (FreeBSD 11.x and earlier only) |=== [[releng-stable-builds-beta]] === FreeBSD `BETA` Builds Following the code slush, the next phase of the release cycle is the code freeze. This is the point at which all commits to the stable branch require explicit approval from the {teamRe}. This is enforced by pre-commit hooks in the Subversion repository by editing [.filename]#base/svnadmin/conf/approvers# to include a regular expression matching the {branchStablex} branch for the release: [.programlisting,subs="attributes"] .... ^/{branchStablex} re ^/{branchRelengx} re .... [NOTE] ==== There are two general exceptions to requiring commit approval during the release cycle. The first is any change that needs to be committed by the Release Engineer in order to proceed with the day-to-day workflow of the release cycle, the other is security fixes that may occur during the release cycle. ==== Once the code freeze is in effect, the next build from the branch is labeled `BETA1`. This is done by updating the `BRANCH` value in [.filename]#sys/conf/newvers.sh# from `PRERELEASE` to `BETA1`. Once this is done, the first set of `BETA` builds are started. Subsequent `BETA` builds do not require updates to any files other than [.filename]#sys/conf/newvers.sh#, incrementing the `BETA` build number. [[releng-stable-branching]] === Creating the {branchRelengx} Branch When the first `RC` (Release Candidate) build is ready to begin, the {branchReleng} branch is created. This is a multi-step process that must be done in a specific order, in order to avoid anomalies such as overlaps with `__FreeBSD_version` values, for example. The paths listed below are relative to the repository root. The order of commits and what to change are: [source,shell,subs="attributes"] .... % svn cp ^/{branchStablex} {branchRelengx} .... [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#releng/12.0/sys/conf/newvers.sh# |Change `BETA__X__` to `RC1` |[.filename]#releng/12.0/sys/sys/param.h# |Update `__FreeBSD_version` |[.filename]#releng/12.0/etc/pkg/FreeBSD.conf# |Replace `latest` with `quarterly` as the default package repository location |[.filename]#releng/12.0/release/pkg_repos/release-dvd.conf# |Replace `latest` with `quarterly` as the default package repository location |[.filename]#stable/12/sys/conf/newvers.sh# |Update `BETA__X__` with `PRERELEASE` |[.filename]#stable/12/sys/sys/param.h# |Update `__FreeBSD_version` |[.filename]#svnadmin/conf/approvers# |Add a new approvers line for the releng branch as was done for the stable branch |=== [source,shell,subs="attributes"] .... % svn propdel -R svn:mergeinfo {branchRelengx} % svn commit {branchRelengx} % svn commit {branchStablex} .... Now that two new `__FreeBSD_version` values exist, also update [.filename]#~/documentation/content/en/books/porters-handbook/versions/chapter.adoc# in the Documentation Project repository. After the first `RC` build has completed and tested, the {branchStable} branch can be "thawed" by removing (or commenting) the ^/{branchStablex} entry in [.filename]#svnadmin/conf/approvers#. Following the availability of the first `RC`, {teamBugmeister} should be emailed to add the new FreeBSD `-RELEASE` to the `versions` available in the drop-down menu shown in the bug tracker. [[releng-building]] == Building FreeBSD Installation Media This section describes the general procedures producing FreeBSD development snapshots and releases. [[releng-build-scripts]] === Release Build Scripts This section describes the build scripts used by {teamRe} to produce development snapshots and releases. [[releng-build-scripts-single]] ==== The [.filename]#release.sh# Script Prior to FreeBSD 9.0-RELEASE, [.filename]#src/release/Makefile# was updated to support , and the [.filename]#src/release/generate-release.sh# script was introduced as a wrapper to automate invoking the targets. Prior to FreeBSD 9.2-RELEASE, [.filename]#src/release/release.sh# was introduced, which heavily based on [.filename]#src/release/generate-release.sh# included support to specify configuration files to override various options and environment variables. Support for configuration files provided support for cross building each architecture for a release by specifying a separate configuration file for each invocation. As a brief example of using [.filename]#src/release/release.sh# to build a single release in [.filename]#/scratch#: [source,shell,subs="attributes"] .... # /bin/sh /usr/src/release/release.sh .... As a brief example of using [.filename]#src/release/release.sh# to build a single, cross-built release using a different target directory, create a custom [.filename]#release.conf# containing: [.programlisting,subs="attributes"] .... # release.sh configuration for powerpc/powerpc64 CHROOTDIR="/scratch-powerpc64" TARGET="powerpc" TARGET_ARCH="powerpc64" KERNEL="GENERIC64" .... Then invoke [.filename]#src/release/release.sh# as: [source,shell,subs="attributes"] .... # /bin/sh /usr/src/release/release.sh -c $HOME/release.conf .... See and [.filename]#src/release/release.conf.sample# for more details and example usage. [[releng-build-scripts-multiple]] ==== The [.filename]#thermite.sh# Wrapper Script In order to make cross building the full set of architectures supported on a given branch faster, easier, and reduce human error factors, a wrapper script around [.filename]#src/release/release.sh# was written to iterate through the various combinations of architectures and invoke [.filename]#src/release/release.sh# using a configuration file specific to that architecture. The wrapper script is called [.filename]#thermite.sh#, which is available in the FreeBSD Subversion repository at `svn://svn.freebsd.org/base/user/gjb/thermite/`, in addition to configuration files used to build {branchHead} and {branchStablex} development snapshots. Using [.filename]#thermite.sh# is covered in <> and <>. Each architecture and individual kernel have their own configuration file used by [.filename]#release.sh#. Each branch has its own [.filename]#defaults-X.conf# configuration which contains entries common throughout each architecture, where overrides or special variables are set and/or overridden in the per-build files. The per-build configuration file naming scheme is in the form of [.filename]#${revision}-${TARGET_ARCH}-${KERNCONF}-${type}.conf#, where the uppercase variables are equivalent to what uses in the build system, and lowercase variables are set within the configuration files, mapping to the major version of the respective branch. Each branch also has its own [.filename]#builds-X.conf# configuration, which is used by [.filename]#thermite.sh#. The [.filename]#thermite.sh# script iterates through each ${revision}, ${TARGET_ARCH}, ${KERNCONF}, and ${type} value, creating a master list of what to build. However, a given combination from the list will only be built if the respective configuration file exists, which is where the naming scheme above is relevant. There are two paths of file sourcing: * [.filename]#builds-12.conf# - [.filename]#main.conf# + This controls [.filename]#thermite.sh# behavior * [.filename]#12-amd64-GENERIC-snap.conf# - [.filename]#defaults-12.conf# - [.filename]#main.conf# + This controls [.filename]#release/release.sh# behavior within the build [NOTE] ==== The [.filename]#builds-12.conf#, [.filename]#defaults-12.conf#, and [.filename]#main.conf# configuration files exist to reduce repetition between the various per-build files. ==== [[releng-build-snapshot]] === Building FreeBSD Development Snapshots The official release build machines have a specific filesystem layout, which using ZFS, [.filename]#thermite.sh# takes heavy advantage of with clones and snapshots, ensuring a pristine build environment. The build scripts reside in [.filename]#/releng/scripts-snapshot/scripts# or [.filename]#/releng/scripts-release/scripts# respectively, to avoid collisions between an `RC` build from a releng branch versus a `STABLE` snapshot from the respective stable branch. A separate dataset exists for the final build images, [.filename]#/snap/ftp#. This directory contains both snapshots and releases directories. They are only used if the `EVERYTHINGISFINE` variable is defined in [.filename]#main.conf#. [NOTE] ==== The `EVERYTHINGISFINE` variable name was chosen to avoid colliding with a variable that might be possibly set in the user environment, accidentally enabling the behavior that depends on it being defined. ==== As [.filename]#thermite.sh# iterates through the master list of combinations and locates the per-build configuration file, a ZFS dataset is created under [.filename]#/releng#, such as [.filename]#/releng/12-amd64-GENERIC-snap#. The `src/`, `ports/`, and `doc/` trees are checked out to separate ZFS datasets, such as [.filename]#/releng/12-src-snap#, which are then cloned and mounted into the respective build datasets. This is done to avoid checking out a given tree more than once. Assuming these filesystem paths, [.filename]#thermite.sh# would be invoked as: [source,shell,subs="attributes"] .... # cd /releng/scripts-snapshot/scripts # ./setrev.sh -b {branchStablex} # ./zfs-cleanup.sh -c ./builds-12.conf # ./thermite.sh -c ./builds-12.conf .... Once the builds have completed, additional helper scripts are available to generate development snapshot emails which are sent to the `freebsd-snapshots@freebsd.org` mailing list: [source,shell,subs="attributes"] .... # cd /releng/scripts-snapshot/scripts # ./get-checksums.sh -c ./builds-12.conf | ./generate-email.pl > snapshot-12-mail .... [NOTE] ==== The generated output should be double-checked for correctness, and the email itself should be PGP signed, in-line. ==== [NOTE] ==== These helper scripts only apply to development snapshot builds. Announcements during the release cycle (excluding the final release announcement) are created from an email template. A sample of the email template currently used can be found link:here[here]. ==== [[releng-build-release]] === Building FreeBSD Releases Similar to building FreeBSD development snapshots, [.filename]#thermite.sh# would be invoked the same way. The difference between development snapshots and release builds, `BETA` and `RC` included, is that the configuration files must be named with `release` instead of `snap` as the type, as mentioned above. In addition, the `BUILDTYPE` and `types` must be changed from `snap` to `release` in [.filename]#defaults-12.conf# and [.filename]#builds-12.conf#, respectively. When building `BETA`, `RC`, and the final `RELEASE`, also statically set `BUILDSVNREV` to the revision on the branch reflecting the name change, `BUILDDATE` to the date the builds are started in `YYYYMMDD` format. If the `doc/` and `ports/` trees have been tagged, also set `PORTBRANCH` and `DOCBRANCH` to the relevant tag path in the Subversion repository, replacing `HEAD` with the last changed revision. Also set `releasesrc` in [.filename]#builds-12.conf# to the relevant branch, such as {branchStablex} or {branchRelengx}. During the release cycle, a copy of [.filename]#CHECKSUM.SHA512# and [.filename]#CHECKSUM.SHA256# for each architecture are stored in the {teamRe} internal repository in addition to being included in the various announcement emails. Each [.filename]#MANIFEST# containing the hashes of [.filename]#base.txz#, [.filename]#kernel.txz#, etc. are added to package:misc/freebsd-release-manifests[] in the Ports Collection, as well. In preparation for the release build, several files need to be updated: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File to Edit | What to Change |[.filename]#sys/conf/newvers.sh# |Update the `BRANCH` value to `RELEASE` |[.filename]#UPDATING# |Add the anticipated announcement date |[.filename]#lib/csu/common/crtbrand.c# |Replace `__FreeBSD_version` with the value in [.filename]#sys/sys/param.h# |=== After building the final `RELEASE`, the {branchRelengx} branch is tagged as {branchReleasex} using the revision from which the `RELEASE` was built. Similar to creating the {branchStablex} and {branchRelengx} branches, this is done with `svn cp`. From the repository root: [source,shell,subs="attributes"] .... % svn cp ^/{branchRelengx}@r306420 {branchReleasex} % svn commit {branchReleasex} .... [[releng-mirrors]] == Publishing FreeBSD Installation Media to Project Mirrors This section describes the procedure to publish FreeBSD development snapshots and releases to the Project mirrors. [[releng-mirrors-staging]] === Staging FreeBSD Installation Media Images Staging FreeBSD snapshots and releases is a two part process: * Creating the directory structure to match the hierarchy on `ftp-master` + If `EVERYTHINGISFINE` is defined in the build configuration files, [.filename]#main.conf# in the case of the build scripts referenced above, this happens automatically in the after the build is complete, creating the directory structure in [.filename]#${DESTDIR}/R/ftp-stage# with a path structure matching what is expected on `ftp-master`. This is equivalent to running the following in the directly: + [source,shell,subs="attributes"] .... # make -C /usr/src/release -f Makefile.mirrors EVERYTHINGISFINE=1 ftp-stage .... + After each architecture is built, [.filename]#thermite.sh# will rsync the [.filename]#${DESTDIR}/R/ftp-stage# from the build to [.filename]#/snap/ftp/snapshots# or [.filename]#/snap/ftp/releases# on the build host, respectively. * Copying the files to a staging directory on `ftp-master` before moving the files into [.filename]#pub/# to begin propagation to the Project mirrors + Once all builds have finished, [.filename]#/snap/ftp/snapshots#, or [.filename]#/snap/ftp/releases# for a release, is polled by `ftp-master` using rsync to [.filename]#/archive/tmp/snapshots# or [.filename]#/archive/tmp/releases#, respectively. + [NOTE] ==== On `ftp-master` in the FreeBSD Project infrastructure, this step requires `root` level access, as this step must be executed as the `archive` user. ==== [[releng-mirrors-publishing]] === Publishing FreeBSD Installation Media Once the images are staged in [.filename]#/archive/tmp/#, they are ready to be made public by putting them in [.filename]#/archive/pub/FreeBSD#. In order to reduce propagation time, is used to create hard links from [.filename]#/archive/tmp# to [.filename]#/archive/pub/FreeBSD#. [NOTE] ==== In order for this to be effective, both [.filename]#/archive/tmp# and [.filename]#/archive/pub# must reside on the same logical filesystem. ==== There is a caveat, however, where rsync must be used after in order to correct the symbolic links in [.filename]#pub/FreeBSD/snapshots/ISO-IMAGES# which will replace with a hard link, increasing the propagation time. [NOTE] ==== As with the staging steps, this requires `root` level access, as this step must be executed as the `archive` user. ==== As the `archive` user: [source,shell,subs="attributes"] .... % cd /archive/tmp/snapshots % pax -r -w -l . /archive/pub/FreeBSD/snapshots % /usr/local/bin/rsync -avH /archive/tmp/snapshots/* /archive/pub/FreeBSD/snapshots/ .... Replace _snapshots_ with _releases_ as appropriate. [[releng-wrapup]] == Wrapping up the Release Cycle This section describes general post-release tasks. [[releng-wrapup-en]] === Post-Release Errata Notices As the release cycle approaches conclusion, it is common to have several EN (Errata Notice) candidates to address issues that were discovered late in the cycle. Following the release, the {teamRe} and the {teamSecteam} revisit changes that were not approved prior to the final release, and depending on the scope of the change in question, may issue an EN. [NOTE] ==== The actual process of issuing ENs is handled by the {teamSecteam}. ==== To request an Errata Notice after a release cycle has completed, a developer should fill out the https://www.freebsd.org/security/errata-template.txt[Errata Notice template], in particular the `Background`, `Problem Description`, `Impact`, and if applicable, `Workaround` sections. The completed Errata Notice template should be emailed together with either a patch against the {branchReleng} branch or a list of revisions from the {branchStable} branch. For Errata Notice requests immediately following the release, the request should be emailed to both the {teamRe} and the {teamSecteam}. Once the {branchReleng} branch has been handed over to the {teamSecteam} as described in <>, Errata Notice requests should be sent to the {teamSecteam}. [[releng-wrapup-handoff]] === Handoff to the {teamSecteam} Roughly two weeks following the release, the Release Engineer updates [.filename]#svnadmin/conf/approvers# changing the approver column from `re` to `(so|security-officer)` for the {branchRelengx} branch. [[releng-eol]] == Release End-of-Life This section describes the website-related files to update when a release reaches EoL (End-of-Life). [[releng-eol-website]] === Website Updates for End-of-Life When a release reaches End-of-Life, references to that release should be removed and/or updated on the website: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File | What to Change |[.filename]#~/website/themes/beastie/layouts/index.html# |Remove `u-relXXX-announce` and `u-relXXX-announce` references. |[.filename]#~/website/content/en/releases/_index.adoc# |Move the `u-relXXX-*` variables from the supported release list to the Legacy Releases list. |[.filename]#~/website/content/en/releng/_index.adoc# |Update the appropriate releng branch to refelect the branch is no longer supported. |[.filename]#~/website/content/en/security/_index.adoc# |Remove the branch from the supported branch list. |[.filename]#~/website/content/en/where.adoc# |Remove the URLs for the release. |[.filename]#~/website/themes/beastie/layouts/partials/sidenav.html# |Remove `u-relXXX-announce` and `u-relXXX-announce` references. |[.filename]#~/website/static/security/advisory-template.txt# |Remove references to the release and releng branch. |[.filename]#~/website/static/security/errata-template.txt# |Remove references to the release and releng branch. |=== diff --git a/documentation/content/en/articles/freebsd-update-server/_index.adoc b/documentation/content/en/articles/freebsd-update-server/_index.adoc index 6ee32d4eee..527067e752 100644 --- a/documentation/content/en/articles/freebsd-update-server/_index.adoc +++ b/documentation/content/en/articles/freebsd-update-server/_index.adoc @@ -1,612 +1,612 @@ --- title: Build Your Own FreeBSD Update Server authors: - author: Jason Helfman email: jgh@FreeBSD.org copyright: 2009-2011, 2013 Jason Helfman -description: Build Your Own FreeBSD Update Server +description: Building your own freebsd-update server allows a system administrator to perform fast updates for a number of machines from a local mirror trademarks: ["freebsd", "amd", "intel", "general"] --- = Build Your Own FreeBSD Update Server :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/authors.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This article describes building an internal FreeBSD Update Server. The https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] is written by `{cperciva}`, Security Officer Emeritus of FreeBSD. For users that think it is convenient to update their systems against an official update server, building their own FreeBSD Update Server may help to extend its functionality by supporting manually-tweaked FreeBSD releases or by providing a local mirror that will allow faster updates for a number of machines. ''' toc::[] [[acknowledgments]] == Acknowledgments This article was subsequently printed at https://people.freebsd.org/~jgh/files/fus/BSD_03_2010_EN.pdf[BSD Magazine]. [[introduction]] == Introduction Experienced users or administrators are often responsible for several machines or environments. They understand the difficult demands and challenges of maintaining such an infrastructure. Running a FreeBSD Update Server makes it easier to deploy security and software patches to selected test machines before rolling them out to production. It also means a number of systems can be updated from the local network rather than a potentially slower Internet connection. This article outlines the steps involved in creating an internal FreeBSD Update Server. [[prerequisites]] == Prerequisites To build an internal FreeBSD Update Server some requirements should be met. * A running FreeBSD system. + [NOTE] ==== At a minimum, updates require building on a FreeBSD release greater than or equal to the target release version for distribution. ==== * A user account with at least 4 GB of available space. This will allow the creation of updates for 7.1 and 7.2, but the exact space requirements may change from version to version. * An man:ssh[1] account on a remote machine to upload distributed updates. * A web server, like link:{handbook}#network-apache[Apache], with over half of the space required for the build. For instance, test builds for 7.1 and 7.2 consume a total amount of 4 GB, and the webserver space needed to distribute these updates is 2.6 GB. * Basic knowledge of shell scripting with Bourne shell, man:sh[1]. [[Configuration]] == Configuration: Installation & Setup Download the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] software by installing package:devel/subversion[] and package:security/ca_root_nss[], and execute: [source,shell] .... % svn co https://svn.freebsd.org/base/user/cperciva/freebsd-update-build freebsd-update-server .... Update [.filename]#scripts/build.conf# appropriately. It is sourced during all build operations. Here is the default [.filename]#build.conf#, which should be modified to suit your environment. [.programlisting] .... # Main configuration file for FreeBSD Update builds. The # release-specific configuration data is lower down in # the scripts tree. # Location from which to fetch releases export FTP=ftp://ftp2.freebsd.org/pub/FreeBSD/releases <.> # Host platform export HOSTPLATFORM=`uname -m` # Host name to use inside jails export BUILDHOSTNAME=${HOSTPLATFORM}-builder.daemonology.net <.> # Location of SSH key export SSHKEY=/root/.ssh/id_dsa <.> # SSH account into which files are uploaded MASTERACCT=builder@wadham.daemonology.net <.> # Directory into which files are uploaded MASTERDIR=update-master.freebsd.org <.> .... Parameters for consideration would be: <.> This is the location where ISO images are downloaded from (by the `fetchiso()` subroutine of [.filename]#scripts/build.subr#). The location configured is not limited to FTP URIs. Any URI scheme supported by standard man:fetch[1] utility should work fine. Customizations to the `fetchiso()` code can be installed by copying the default [.filename]#build.subr# script to the release and architecture-specific area at [.filename]#scripts/RELEASE/ARCHITECTURE/build.subr# and applying local changes. <.> The name of the build host. This information will be displayed on updated systems when issuing: + [source,shell] .... % uname -v .... + <.> The SSH key for uploading files to the update server. A key pair can be created by typing `ssh-keygen -t dsa`. This parameter is optional; standard password authentication will be used as a fallback authentication method when `SSHKEY` is not defined. The man:ssh-keygen[1] manual page has more detailed information about SSH and the appropriate steps for creating and using one. <.> Account for uploading files to the update server. <.> Directory on the update server where files are uploaded to. The default [.filename]#build.conf# shipped with the freebsd-update-server sources is suitable for building i386 releases of FreeBSD. As an example of building an update server for other architectures, the following steps outline the configuration changes needed for amd64: [.procedure] ==== . Create a build environment for amd64: + [source,shell] .... % mkdir -p /usr/local/freebsd-update-server/scripts/7.2-RELEASE/amd64 .... . Install a [.filename]#build.conf# in the newly created build directory. The build configuration options for FreeBSD 7.2-RELEASE on amd64 should be similar to: + [.programlisting] .... # SHA256 hash of RELEASE disc1.iso image. export RELH=1ea1f6f652d7c5f5eab7ef9f8edbed50cb664b08ed761850f95f48e86cc71ef5 <.> # Components of the world, source, and kernels export WORLDPARTS="base catpages dict doc games info manpages proflibs lib32" export SOURCEPARTS="base bin contrib crypto etc games gnu include krb5 \ lib libexec release rescue sbin secure share sys tools \ ubin usbin cddl" export KERNELPARTS="generic" # EOL date export EOL=1275289200 <.> .... + <.> The man:sha256[1] hash key for the desired release, is published within the respective link:https://www.FreeBSD.org/releases/[release announcement]. <.> To generate the "End of Life" number for [.filename]#build.conf#, refer to the "Estimated EOL" posted on the link:https://www.FreeBSD.org/security/security/[FreeBSD Security Website]. The value of `EOL` can be derived from the date listed on the web site, using the man:date[1] utility, for example: + [source,shell] .... % date -j -f '%Y%m%d-%H%M%S' '20090401-000000' '+%s' .... ==== [[build]] == Building Update Code The first step is to run [.filename]#scripts/make.sh#. This will build some binaries, create directories, and generate an RSA signing key used for approving builds. In this step, a passphrase will have to be supplied for the final creation of the signing key. [source,shell] .... # sh scripts/make.sh cc -O2 -fno-strict-aliasing -pipe findstamps.c -o findstamps findstamps.c: In function 'usage': findstamps.c:45: warning: incompatible implicit declaration of built-in function 'exit' cc -O2 -fno-strict-aliasing -pipe unstamp.c -o unstamp install findstamps ../bin install unstamp ../bin rm -f findstamps unstamp Generating RSA private key, 4096 bit long modulus ................................................................................++ ...................++ e is 65537 (0x10001) Public key fingerprint: 27ef53e48dc869eea6c3136091cc6ab8589f967559824779e855d58a2294de9e Encrypting signing key for root enter aes-256-cbc encryption password: Verifying - enter aes-256-cbc encryption password: .... [NOTE] ==== Keep a note of the generated key fingerprint. This value is required in [.filename]#/etc/freebsd-update.conf# for binary updates. ==== At this point, we are ready to stage a build. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/init.sh amd64 7.2-RELEASE .... What follows is a sample of an _initial_ build run. [source,shell] .... # sh scripts/init.sh amd64 7.2-RELEASE Mon Aug 24 16:04:36 PDT 2009 Starting fetch for FreeBSD/amd64 7.2-RELEASE /usr/local/freebsd-update-server/work/7.2-RELE100 of 588 MB 359 kBps 00m00s Mon Aug 24 16:32:38 PDT 2009 Verifying disc1 hash for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 16:32:44 PDT 2009 Extracting components for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 16:34:05 PDT 2009 Constructing world+src image for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 16:35:57 PDT 2009 Extracting world+src for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 23:36:24 UTC 2009 Building world for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:31:29 UTC 2009 Distributing world for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:32:36 UTC 2009 Building and distributing kernels for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:44:44 UTC 2009 Constructing world components for FreeBSD/amd64 7.2-RELEASE Tue Aug 25 00:44:56 UTC 2009 Distributing source for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:46:18 PDT 2009 Moving components into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:46:33 PDT 2009 Identifying extra documentation for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:47:13 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:47:18 PDT 2009 Indexing release for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 17:50:44 PDT 2009 Indexing world0 for FreeBSD/amd64 7.2-RELEASE Files built but not released: Files released but not built: Files which differ by more than contents: Files which differ between release and build: kernel|generic|/GENERIC/hptrr.ko kernel|generic|/GENERIC/kernel src|sys|/sys/conf/newvers.sh world|base|/boot/loader world|base|/boot/pxeboot world|base|/etc/mail/freebsd.cf world|base|/etc/mail/freebsd.submit.cf world|base|/etc/mail/sendmail.cf world|base|/etc/mail/submit.cf world|base|/lib/libcrypto.so.5 world|base|/usr/bin/ntpq world|base|/usr/lib/libalias.a world|base|/usr/lib/libalias_cuseeme.a world|base|/usr/lib/libalias_dummy.a world|base|/usr/lib/libalias_ftp.a ... .... Then the build of the world is performed again, with world patches. A more detailed explanation may be found in [.filename]#scripts/build.subr#. [WARNING] ==== During this second build cycle, the network time protocol daemon, man:ntpd[8], is turned off. Per `{cperciva}`, Security Officer Emeritus of FreeBSD, "the https://svnweb.freebsd.org/base/user/cperciva/freebsd-update-build/[freebsd-update-server] build code needs to identify timestamps which are stored in files so that they can be ignored when comparing builds to determine which files need to be updated. This timestamp-finding works by doing two builds 400 days apart and comparing the results." ==== [source,shell] .... Mon Aug 24 17:54:07 PDT 2009 Extracting world+src for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 00:54:34 UTC 2010 Building world for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 01:49:42 UTC 2010 Distributing world for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 01:50:50 UTC 2010 Building and distributing kernels for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 02:02:56 UTC 2010 Constructing world components for FreeBSD/amd64 7.2-RELEASE Wed Sep 29 02:03:08 UTC 2010 Distributing source for FreeBSD/amd64 7.2-RELEASE Tue Sep 28 19:04:31 PDT 2010 Moving components into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:04:46 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:04:51 PDT 2009 Indexing world1 for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:08:04 PDT 2009 Locating build stamps for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:10:19 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:10:19 PDT 2009 Preparing to copy files into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 19:10:20 PDT 2009 Copying data files into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 12:16:57 PDT 2009 Copying metadata files into staging area for FreeBSD/amd64 7.2-RELEASE Mon Aug 24 12:16:59 PDT 2009 Constructing metadata index and tag for FreeBSD/amd64 7.2-RELEASE Files found which include build stamps: kernel|generic|/GENERIC/hptrr.ko kernel|generic|/GENERIC/kernel world|base|/boot/loader world|base|/boot/pxeboot world|base|/etc/mail/freebsd.cf world|base|/etc/mail/freebsd.submit.cf world|base|/etc/mail/sendmail.cf world|base|/etc/mail/submit.cf world|base|/lib/libcrypto.so.5 world|base|/usr/bin/ntpq world|base|/usr/include/osreldate.h world|base|/usr/lib/libalias.a world|base|/usr/lib/libalias_cuseeme.a world|base|/usr/lib/libalias_dummy.a world|base|/usr/lib/libalias_ftp.a ... .... Finally, the build completes. [source,shell] .... Values of build stamps, excluding library archive headers: v1.2 (Aug 25 2009 00:40:36) v1.2 (Aug 25 2009 00:38:22) @()FreeBSD 7.2-RELEASE 0: Tue Aug 25 00:38:29 UTC 2009 FreeBSD 7.2-RELEASE 0: Tue Aug 25 00:38:29 UTC 2009 root@server.myhost.com:/usr/obj/usr/src/sys/GENERIC 7.2-RELEASE Mon Aug 24 23:55:25 UTC 2009 Mon Aug 24 23:55:25 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 built by root@server.myhost.com on Tue Aug 25 00:16:15 UTC 2009 Mon Aug 24 23:46:47 UTC 2009 ntpq 4.2.4p5-a Mon Aug 24 23:55:53 UTC 2009 (1) * Copyright (c) 1992-2009 The FreeBSD Project. Mon Aug 24 23:46:47 UTC 2009 Mon Aug 24 23:55:40 UTC 2009 Aug 25 2009 ntpd 4.2.4p5-a Mon Aug 24 23:55:52 UTC 2009 (1) ntpdate 4.2.4p5-a Mon Aug 24 23:55:53 UTC 2009 (1) ntpdc 4.2.4p5-a Mon Aug 24 23:55:53 UTC 2009 (1) Tue Aug 25 00:21:21 UTC 2009 Tue Aug 25 00:21:21 UTC 2009 Tue Aug 25 00:21:21 UTC 2009 Mon Aug 24 23:46:47 UTC 2009 FreeBSD/amd64 7.2-RELEASE initialization build complete. Please review the list of build stamps printed above to confirm that they look sensible, then run sh -e approve.sh amd64 7.2-RELEASE to sign the release. .... Approve the build if everything is correct. More information on determining this can be found in the distributed source file named [.filename]#USAGE#. Execute [.filename]#scripts/approve.sh#, as directed. This will sign the release, and move components into a staging area suitable for uploading. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/mountkey.sh .... [source,shell] .... # sh -e scripts/approve.sh amd64 7.2-RELEASE Wed Aug 26 12:50:06 PDT 2009 Signing build for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to patch source directories for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to upload staging area for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:07 PDT 2009 Updating databases for FreeBSD/amd64 7.2-RELEASE Wed Aug 26 12:50:07 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.2-RELEASE .... After the approval process is complete, the upload procedure may be started. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/upload.sh amd64 7.2-RELEASE .... [NOTE] ==== In the event update code needs to be re-uploaded, this may be done by changing to the public distributions directory for the target release and updating attributes of the _uploaded_ file. [source,shell] .... # cd /usr/local/freebsd-update-server/pub/7.2-RELEASE/amd64 # touch -t 200801010101.01 uploaded .... ==== The uploaded files will need to be in the document root of the webserver in order for updates to be distributed. The exact configuration will vary depending on the web server used. For the Apache web server, please refer to the link:{handbook}#network-apache[Configuration of Apache servers] section in the Handbook. Update client's `KeyPrint` and `ServerName` in [.filename]#/etc/freebsd-update.conf#, and perform updates as instructed in the link:{handbook}#updating-upgrading-freebsdupdate[FreeBSD Update] section of the Handbook. [IMPORTANT] ==== In order for FreeBSD Update Server to work properly, updates for both the _current_ release and the release _one wants to upgrade to_ need to be built. This is necessary for determining the differences of files between releases. For example, when upgrading a FreeBSD system from 7.1-RELEASE to 7.2-RELEASE, updates will need to be built and uploaded to your distribution server for both versions. ==== For reference, the entire run of link:../../source/articles/freebsd-update-server/init.txt[init.sh] is attached. [[patch]] == Building a Patch Every time a link:https://www.FreeBSD.org/security/advisories/[security advisory] or link:https://www.FreeBSD.org/security/notices/[security notice] is announced, a patch update can be built. For this example, 7.1-RELEASE will be used. A couple of assumptions are made for a different release build: * Setup the correct directory structure for the initial build. * Perform an initial build for 7.1-RELEASE. Create the patch directory of the respective release under [.filename]#/usr/local/freebsd-update-server/patches/#. [source,shell] .... % mkdir -p /usr/local/freebsd-update-server/patches/7.1-RELEASE/ % cd /usr/local/freebsd-update-server/patches/7.1-RELEASE .... As an example, take the patch for man:named[8]. Read the advisory, and grab the necessary file from link:https://www.FreeBSD.org/security/advisories/[FreeBSD Security Advisories]. More information on interpreting the advisory, can be found in the link:{handbook}#security-advisories[FreeBSD Handbook]. In the https://security.freebsd.org/advisories/FreeBSD-SA-09:12.bind.asc[security brief], this advisory is called `SA-09:12.bind`. After downloading the file, it is required to rename the file to an appropriate patch level. It is suggested to keep this consistent with official FreeBSD patch levels, but its name may be freely chosen. For this build, let us follow the currently established practice of FreeBSD and call this `p7`. Rename the file: [source,shell] .... % cd /usr/local/freebsd-update-server/patches/7.1-RELEASE/; mv bind.patch 7-SA-09:12.bind .... [NOTE] ==== When running a patch level build, it is assumed that previous patches are in place. When a patch build is run, it will run all patches contained in the patch directory. There can be custom patches added to any build. Use the number zero, or any other number. ==== [WARNING] ==== It is up to the administrator of the FreeBSD Update Server to take appropriate measures to verify the authenticity of every patch. ==== At this point, a _diff_ is ready to be built. The software checks first to see if a [.filename]#scripts/init.sh# has been run on the respective release prior to running the diff build. [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/diff.sh amd64 7.1-RELEASE 7 .... What follows is a sample of a _differential_ build run. [source,shell] .... # sh -e scripts/diff.sh amd64 7.1-RELEASE 7 Wed Aug 26 10:09:59 PDT 2009 Extracting world+src for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 17:10:25 UTC 2009 Building world for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:05:11 UTC 2009 Distributing world for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:06:16 UTC 2009 Building and distributing kernels for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:17:50 UTC 2009 Constructing world components for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 18:18:02 UTC 2009 Distributing source for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:19:23 PDT 2009 Moving components into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:19:37 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:19:42 PDT 2009 Indexing world0 for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 11:23:02 PDT 2009 Extracting world+src for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 18:23:29 UTC 2010 Building world for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:18:15 UTC 2010 Distributing world for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:19:18 UTC 2010 Building and distributing kernels for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:30:52 UTC 2010 Constructing world components for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 19:31:03 UTC 2010 Distributing source for FreeBSD/amd64 7.1-RELEASE-p7 Thu Sep 30 12:32:25 PDT 2010 Moving components into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:32:39 PDT 2009 Extracting extra docs for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:32:43 PDT 2009 Indexing world1 for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:35:54 PDT 2009 Locating build stamps for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:36:58 PDT 2009 Reverting changes due to build stamps for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:37:14 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:37:14 PDT 2009 Preparing to copy files into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:37:15 PDT 2009 Copying data files into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:43:23 PDT 2009 Copying metadata files into staging area for FreeBSD/amd64 7.1-RELEASE-p7 Wed Aug 26 12:43:25 PDT 2009 Constructing metadata index and tag for FreeBSD/amd64 7.1-RELEASE-p7 ... Files found which include build stamps: kernel|generic|/GENERIC/hptrr.ko kernel|generic|/GENERIC/kernel world|base|/boot/loader world|base|/boot/pxeboot world|base|/etc/mail/freebsd.cf world|base|/etc/mail/freebsd.submit.cf world|base|/etc/mail/sendmail.cf world|base|/etc/mail/submit.cf world|base|/lib/libcrypto.so.5 world|base|/usr/bin/ntpq world|base|/usr/include/osreldate.h world|base|/usr/lib/libalias.a world|base|/usr/lib/libalias_cuseeme.a world|base|/usr/lib/libalias_dummy.a world|base|/usr/lib/libalias_ftp.a ... Values of build stamps, excluding library archive headers: v1.2 (Aug 26 2009 18:13:46) v1.2 (Aug 26 2009 18:11:44) @()FreeBSD 7.1-RELEASE-p7 0: Wed Aug 26 18:11:50 UTC 2009 FreeBSD 7.1-RELEASE-p7 0: Wed Aug 26 18:11:50 UTC 2009 root@server.myhost.com:/usr/obj/usr/src/sys/GENERIC 7.1-RELEASE-p7 Wed Aug 26 17:29:15 UTC 2009 Wed Aug 26 17:29:15 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 built by root@server.myhost.com on Wed Aug 26 17:49:58 UTC 2009 Wed Aug 26 17:20:39 UTC 2009 ntpq 4.2.4p5-a Wed Aug 26 17:29:42 UTC 2009 (1) * Copyright (c) 1992-2009 The FreeBSD Project. Wed Aug 26 17:20:39 UTC 2009 Wed Aug 26 17:29:30 UTC 2009 Aug 26 2009 ntpd 4.2.4p5-a Wed Aug 26 17:29:41 UTC 2009 (1) ntpdate 4.2.4p5-a Wed Aug 26 17:29:42 UTC 2009 (1) ntpdc 4.2.4p5-a Wed Aug 26 17:29:42 UTC 2009 (1) Wed Aug 26 17:55:02 UTC 2009 Wed Aug 26 17:55:02 UTC 2009 Wed Aug 26 17:55:02 UTC 2009 Wed Aug 26 17:20:39 UTC 2009 ... .... Updates are printed, and approval is requested. [source,shell] .... New updates: kernel|generic|/GENERIC/kernel.symbols|f|0|0|0555|0|7c8dc176763f96ced0a57fc04e7c1b8d793f27e006dd13e0b499e1474ac47e10| kernel|generic|/GENERIC/kernel|f|0|0|0555|0|33197e8cf15bbbac263d17f39c153c9d489348c2c534f7ca1120a1183dec67b1| kernel|generic|/|d|0|0|0755|0|| src|base|/|d|0|0|0755|0|| src|bin|/|d|0|0|0755|0|| src|cddl|/|d|0|0|0755|0|| src|contrib|/contrib/bind9/bin/named/update.c|f|0|10000|0644|0|4d434abf0983df9bc47435670d307fa882ef4b348ed8ca90928d250f42ea0757| src|contrib|/contrib/bind9/lib/dns/openssldsa_link.c|f|0|10000|0644|0|c6805c39f3da2a06dd3f163f26c314a4692d4cd9a2d929c0acc88d736324f550| src|contrib|/contrib/bind9/lib/dns/opensslrsa_link.c|f|0|10000|0644|0|fa0f7417ee9da42cc8d0fd96ad24e7a34125e05b5ae075bd6e3238f1c022a712| ... FreeBSD/amd64 7.1-RELEASE update build complete. Please review the list of build stamps printed above and the list of updated files to confirm that they look sensible, then run sh -e approve.sh amd64 7.1-RELEASE to sign the build. .... Follow the same process as noted before for approving a build: [source,shell] .... # sh -e scripts/approve.sh amd64 7.1-RELEASE Wed Aug 26 12:50:06 PDT 2009 Signing build for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to patch source directories for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:06 PDT 2009 Copying files to upload staging area for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:07 PDT 2009 Updating databases for FreeBSD/amd64 7.1-RELEASE Wed Aug 26 12:50:07 PDT 2009 Cleaning staging area for FreeBSD/amd64 7.1-RELEASE The FreeBSD/amd64 7.1-RELEASE update build has been signed and is ready to be uploaded. Remember to run sh -e umountkey.sh to unmount the decrypted key once you have finished signing all the new builds. .... After approving the build, upload the software: [source,shell] .... # cd /usr/local/freebsd-update-server # sh scripts/upload.sh amd64 7.1-RELEASE .... For reference, the entire run of link:../../source/articles/freebsd-update-server/diff.txt[diff.sh] is attached. [[tips]] == Tips * If a custom release is built using the native `make release` link:{releng}#release-build[procedure], freebsd-update-server code will work from your release. As an example, a release without ports or documentation can be built by clearing functionality pertaining to documentation subroutines `findextradocs ()`, `addextradocs ()` and altering the download location in `fetchiso ()`, respectively, in [.filename]#scripts/build.subr#. As a last step, change the man:sha256[1] hash in [.filename]#build.conf# under your respective release and architecture and you are ready to build off your custom release. + [.programlisting] .... # Compare ${WORKDIR}/release and ${WORKDIR}/$1, identify which parts # of the world|doc subcomponent are missing from the latter, and # build a tarball out of them. findextradocs () { } # Add extra docs to ${WORKDIR}/$1 addextradocs () { } .... * Adding `-j _NUMBER_` flags to `buildworld` and `obj` targets in the [.filename]#scripts/build.subr# script may speed up processing depending on the hardware used, however it is not necessary. Using these flags in other targets is not recommended, as it may cause the build to become unreliable. + [.programlisting] .... # Build the world log "Building world" cd /usr/src && make -j 2 ${COMPATFLAGS} buildworld 2>&1 # Distribute the world log "Distributing world" cd /usr/src/release && make -j 2 obj && make ${COMPATFLAGS} release.1 release.2 2>&1 .... * Create an appropriate link:{handbook}#network-dns[DNS] SRV record for the update server, and put others behind it with variable weights. Using this facility will provide update mirrors, however this tip is not necessary unless you wish to provide a redundant service. + [.programlisting] .... _http._tcp.update.myserver.com. IN SRV 0 2 80 host1.myserver.com. IN SRV 0 1 80 host2.myserver.com. IN SRV 0 0 80 host3.myserver.com. .... diff --git a/documentation/content/en/articles/geom-class/_index.adoc b/documentation/content/en/articles/geom-class/_index.adoc index 8c93697afb..902720e465 100644 --- a/documentation/content/en/articles/geom-class/_index.adoc +++ b/documentation/content/en/articles/geom-class/_index.adoc @@ -1,402 +1,402 @@ --- title: Writing a GEOM Class authors: - author: Ivan Voras email: ivoras@FreeBSD.org -description: Writing a GEOM Class in FreeBSD +description: A guide to GEOM internals, and writing your own GEOM class trademarks: ["freebsd", "intel", "general"] --- = Writing a GEOM Class :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/en/urls.adoc[] [.abstract-title] Abstract This text documents some starting points in developing GEOM classes, and kernel modules in general. It is assumed that the reader is familiar with C userland programming. ''' toc::[] [[intro]] == Introduction [[intro-docs]] === Documentation Documentation on kernel programming is scarce - it is one of few areas where there is nearly nothing in the way of friendly tutorials, and the phrase "use the source!" really holds true. However, there are some bits and pieces (some of them seriously outdated) floating around that should be studied before beginning to code: * The link:{developers-handbook}[FreeBSD Developer's Handbook] - part of the documentation project, it does not contain anything specific to kernel programming, but rather some general useful information. * The link:{arch-handbook}[FreeBSD Architecture Handbook] - also from the documentation project, contains descriptions of several low-level facilities and procedures. The most important chapter is 13, link:{arch-handbook}#driverbasics[Writing FreeBSD device drivers]. * The Blueprints section of http://www.freebsddiary.org[FreeBSD Diary] web site - contains several interesting articles on kernel facilities. * The man pages in section 9 - for important documentation on kernel functions. * The man:geom[4] man page and http://phk.freebsd.dk/pubs/[PHK's GEOM slides] - for general introduction of the GEOM subsystem. * Man pages man:g_bio[9], man:g_event[9], man:g_data[9], man:g_geom[9], man:g_provider[9], man:g_consumer[9], man:g_access[9] & others linked from those, for documentation on specific functionalities. * The man:style[9] man page - for documentation on the coding-style conventions which must be followed for any code which is to be committed to the FreeBSD tree. [[prelim]] == Preliminaries The best way to do kernel development is to have (at least) two separate computers. One of these would contain the development environment and sources, and the other would be used to test the newly written code by network-booting and network-mounting filesystems from the first one. This way if the new code contains bugs and crashes the machine, it will not mess up the sources (and other "live" data). The second system does not even require a proper display. Instead, it could be connected with a serial cable or KVM to the first one. But, since not everybody has two or more computers handy, there are a few things that can be done to prepare an otherwise "live" system for developing kernel code. This setup is also applicable for developing in a http://www.vmware.com/[VMWare] or http://www.qemu.org/[QEmu] virtual machine (the next best thing after a dedicated development machine). [[prelim-system]] === Modifying a System for Development For any kernel programming a kernel with `INVARIANTS` enabled is a must-have. So enter these in your kernel configuration file: [.programlisting] .... options INVARIANT_SUPPORT options INVARIANTS .... For more debugging you should also include WITNESS support, which will alert you of mistakes in locking: [.programlisting] .... options WITNESS_SUPPORT options WITNESS .... For debugging crash dumps, a kernel with debug symbols is needed: [.programlisting] .... makeoptions DEBUG=-g .... With the usual way of installing the kernel (`make installkernel`) the debug kernel will not be automatically installed. It is called [.filename]#kernel.debug# and located in [.filename]#/usr/obj/usr/src/sys/KERNELNAME/#. For convenience it should be copied to [.filename]#/boot/kernel/#. Another convenience is enabling the kernel debugger so you can examine a kernel panic when it happens. For this, enter the following lines in your kernel configuration file: [.programlisting] .... options KDB options DDB options KDB_TRACE .... For this to work you might need to set a sysctl (if it is not on by default): [.programlisting] .... debug.debugger_on_panic=1 .... Kernel panics will happen, so care should be taken with the filesystem cache. In particular, having softupdates might mean the latest file version could be lost if a panic occurs before it is committed to storage. Disabling softupdates yields a great performance hit, and still does not guarantee data consistency. Mounting filesystem with the "sync" option is needed for that. For a compromise, the softupdates cache delays can be shortened. There are three sysctl's that are useful for this (best to be set in [.filename]#/etc/sysctl.conf#): [.programlisting] .... kern.filedelay=5 kern.dirdelay=4 kern.metadelay=3 .... The numbers represent seconds. For debugging kernel panics, kernel core dumps are required. Since a kernel panic might make filesystems unusable, this crash dump is first written to a raw partition. Usually, this is the swap partition. This partition must be at least as large as the physical RAM in the machine. On the next boot, the dump is copied to a regular file. This happens after filesystems are checked and mounted, and before swap is enabled. This is controlled with two [.filename]#/etc/rc.conf# variables: [.programlisting] .... dumpdev="/dev/ad0s4b" dumpdir="/usr/core .... The `dumpdev` variable specifies the swap partition and `dumpdir` tells the system where in the filesystem to relocate the core dump on reboot. Writing kernel core dumps is slow and takes a long time so if you have lots of memory (>256M) and lots of panics it could be frustrating to sit and wait while it is done (twice - first to write it to swap, then to relocate it to filesystem). It is convenient then to limit the amount of RAM the system will use via a [.filename]#/boot/loader.conf# tunable: [.programlisting] .... hw.physmem="256M" .... If the panics are frequent and filesystems large (or you simply do not trust softupdates+background fsck) it is advisable to turn background fsck off via [.filename]#/etc/rc.conf# variable: [.programlisting] .... background_fsck="NO" .... This way, the filesystems will always get checked when needed. Note that with background fsck, a new panic could happen while it is checking the disks. Again, the safest way is not to have many local filesystems by using another computer as an NFS server. [[prelim-starting]] === Starting the Project For the purpose of creating a new GEOM class, an empty subdirectory has to be created under an arbitrary user-accessible directory. You do not have to create the module directory under [.filename]#/usr/src#. [[prelim-makefile]] === The Makefile It is good practice to create [.filename]#Makefiles# for every nontrivial coding project, which of course includes kernel modules. Creating the [.filename]#Makefile# is simple thanks to an extensive set of helper routines provided by the system. In short, here is how a minimal [.filename]#Makefile# looks for a kernel module: [.programlisting] .... SRCS=g_journal.c KMOD=geom_journal .include .... This [.filename]#Makefile# (with changed filenames) will do for any kernel module, and a GEOM class can reside in just one kernel module. If more than one file is required, list it in the `SRCS` variable, separated with whitespace from other filenames. [[kernelprog]] == On FreeBSD Kernel Programming [[kernelprog-memalloc]] === Memory Allocation See man:malloc[9]. Basic memory allocation is only slightly different than its userland equivalent. Most notably, `malloc`() and `free`() accept additional parameters as is described in the man page. A "malloc type" must be declared in the declaration section of a source file, like this: [.programlisting] .... static MALLOC_DEFINE(M_GJOURNAL, "gjournal data", "GEOM_JOURNAL Data"); .... To use this macro, [.filename]#sys/param.h#, [.filename]#sys/kernel.h# and [.filename]#sys/malloc.h# headers must be included. There is another mechanism for allocating memory, the UMA (Universal Memory Allocator). See man:uma[9] for details, but it is a special type of allocator mainly used for speedy allocation of lists comprised of same-sized items (for example, dynamic arrays of structs). [[kernelprog-lists]] === Lists and Queues See man:queue[3]. There are a LOT of cases when a list of things needs to be maintained. Fortunately, this data structure is implemented (in several ways) by C macros included in the system. The most used list type is TAILQ because it is the most flexible. It is also the one with largest memory requirements (its elements are doubly-linked) and also the slowest (although the speed variation is on the order of several CPU instructions more, so it should not be taken seriously). If data retrieval speed is very important, see man:tree[3] and man:hashinit[9]. [[kernelprog-bios]] === BIOs Structure `bio` is used for any and all Input/Output operations concerning GEOM. It basically contains information about what device ('provider') should satisfy the request, request type, offset, length, pointer to a buffer, and a bunch of "user-specific" flags and fields that can help implement various hacks. The important thing here is that ``bio``s are handled asynchronously. That means that, in most parts of the code, there is no analogue to userland's man:read[2] and man:write[2] calls that do not return until a request is done. Rather, a developer-supplied function is called as a notification when the request gets completed (or results in error). The asynchronous programming model (also called "event-driven") is somewhat harder than the much more used imperative one used in userland (at least it takes a while to get used to it). In some cases the helper routines `g_write_data`() and `g_read_data`() can be used, but __not always__. In particular, they cannot be used when a mutex is held; for example, the GEOM topology mutex or the internal mutex held during the `.start`() and `.stop`() functions. [[geom]] == On GEOM Programming [[geom-ggate]] === Ggate If maximum performance is not needed, a much simpler way of making a data transformation is to implement it in userland via the ggate (GEOM gate) facility. Unfortunately, there is no easy way to convert between, or even share code between the two approaches. [[geom-class]] === GEOM Class GEOM classes are transformations on the data. These transformations can be combined in a tree-like fashion. Instances of GEOM classes are called __geoms__. Each GEOM class has several "class methods" that get called when there is no geom instance available (or they are simply not bound to a single instance): * `.init` is called when GEOM becomes aware of a GEOM class (when the kernel module gets loaded.) * `.fini` gets called when GEOM abandons the class (when the module gets unloaded) * `.taste` is called next, once for each provider the system has available. If applicable, this function will usually create and start a geom instance. * `.destroy_geom` is called when the geom should be disbanded * `.ctlconf` is called when user requests reconfiguration of existing geom Also defined are the GEOM event functions, which will get copied to the geom instance. Field `.geom` in the `g_class` structure is a LIST of geoms instantiated from the class. These functions are called from the g_event kernel thread. [[geom-softc]] === Softc The name "softc" is a legacy term for "driver private data". The name most probably comes from the archaic term "software control block". In GEOM, it is a structure (more precise: pointer to a structure) that can be attached to a geom instance to hold whatever data is private to the geom instance. Most GEOM classes have the following members: * `struct g_provider *provider` : The "provider" this geom instantiates * `uint16_t n_disks` : Number of consumer this geom consumes * `struct g_consumer \**disks` : Array of `struct g_consumer*`. (It is not possible to use just single indirection because struct g_consumer* are created on our behalf by GEOM). The `softc` structure contains all the state of geom instance. Every geom instance has its own softc. [[geom-metadata]] === Metadata Format of metadata is more-or-less class-dependent, but MUST start with: * 16 byte buffer for null-terminated signature (usually the class name) * uint32 version ID It is assumed that geom classes know how to handle metadata with version ID's lower than theirs. Metadata is located in the last sector of the provider (and thus must fit in it). (All this is implementation-dependent but all existing code works like that, and it is supported by libraries.) [[geom-creating]] === Labeling/creating a GEOM The sequence of events is: * user calls man:geom[8] utility (or one of its hardlinked friends) * the utility figures out which geom class it is supposed to handle and searches for [.filename]#geom_CLASSNAME.so# library (usually in [.filename]#/lib/geom#). * it man:dlopen[3]-s the library, extracts the definitions of command-line parameters and helper functions. In the case of creating/labeling a new geom, this is what happens: * man:geom[8] looks in the command-line argument for the command (usually `label`), and calls a helper function. * The helper function checks parameters and gathers metadata, which it proceeds to write to all concerned providers. * This "spoils" existing geoms (if any) and initializes a new round of "tasting" of the providers. The intended geom class recognizes the metadata and brings the geom up. (The above sequence of events is implementation-dependent but all existing code works like that, and it is supported by libraries.) [[geom-command]] === GEOM Command Structure The helper [.filename]#geom_CLASSNAME.so# library exports `class_commands` structure, which is an array of `struct g_command` elements. Commands are of uniform format and look like: [.programlisting] .... verb [-options] geomname [other] .... Common verbs are: * label - to write metadata to devices so they can be recognized at tasting and brought up in geoms * destroy - to destroy metadata, so the geoms get destroyed Common options are: * `-v` : be verbose * `-f` : force Many actions, such as labeling and destroying metadata can be performed in userland. For this, `struct g_command` provides field `gc_func` that can be set to a function (in the same [.filename]#.so#) that will be called to process a verb. If `gc_func` is NULL, the command will be passed to kernel module, to `.ctlreq` function of the geom class. [[geom-geoms]] === Geoms Geoms are instances of GEOM classes. They have internal data (a softc structure) and some functions with which they respond to external events. The event functions are: * `.access` : calculates permissions (read/write/exclusive) * `.dumpconf` : returns XML-formatted information about the geom * `.orphan` : called when some underlying provider gets disconnected * `.spoiled` : called when some underlying provider gets written to * `.start` : handles I/O These functions are called from the `g_down` kernel thread and there can be no sleeping in this context, (see definition of sleeping elsewhere) which limits what can be done quite a bit, but forces the handling to be fast. Of these, the most important function for doing actual useful work is the `.start`() function, which is called when a BIO request arrives for a provider managed by a instance of geom class. [[geom-threads]] === GEOM Threads There are three kernel threads created and run by the GEOM framework: * `g_down` : Handles requests coming from high-level entities (such as a userland request) on the way to physical devices * `g_up` : Handles responses from device drivers to requests made by higher-level entities * `g_event` : Handles all other cases: creation of geom instances, access counting, "spoil" events, etc. When a user process issues "read data X at offset Y of a file" request, this is what happens: * The filesystem converts the request into a struct bio instance and passes it to the GEOM subsystem. It knows what geom instance should handle it because filesystems are hosted directly on a geom instance. * The request ends up as a call to the `.start`() function made on the g_down thread and reaches the top-level geom instance. * This top-level geom instance (for example the partition slicer) determines that the request should be routed to a lower-level instance (for example the disk driver). It makes a copy of the bio request (bio requests _ALWAYS_ need to be copied between instances, with `g_clone_bio`()!), modifies the data offset and target provider fields and executes the copy with `g_io_request`() * The disk driver gets the bio request also as a call to `.start`() on the `g_down` thread. It talks to hardware, gets the data back, and calls `g_io_deliver`() on the bio. * Now, the notification of bio completion "bubbles up" in the `g_up` thread. First the partition slicer gets `.done`() called in the `g_up` thread, it uses information stored in the bio to free the cloned `bio` structure (with `g_destroy_bio`()) and calls `g_io_deliver`() on the original request. * The filesystem gets the data and transfers it to userland. See man:g_bio[9] man page for information how the data is passed back and forth in the `bio` structure (note in particular the `bio_parent` and `bio_children` fields and how they are handled). One important feature is: __THERE CAN BE NO SLEEPING IN G_UP AND G_DOWN THREADS__. This means that none of the following things can be done in those threads (the list is of course not complete, but only informative): * Calls to `msleep`() and `tsleep`(), obviously. * Calls to `g_write_data`() and `g_read_data`(), because these sleep between passing the data to consumers and returning. * Waiting for I/O. * Calls to man:malloc[9] and `uma_zalloc`() with `M_WAITOK` flag set * sx and other sleepable locks This restriction is here to stop GEOM code clogging the I/O request path, since sleeping is usually not time-bound and there can be no guarantees on how long will it take (there are some other, more technical reasons also). It also means that there is not much that can be done in those threads; for example, almost any complex thing requires memory allocation. Fortunately, there is a way out: creating additional kernel threads. [[geom-kernelthreads]] === Kernel Threads for Use in GEOM Code Kernel threads are created with man:kthread_create[9] function, and they are sort of similar to userland threads in behavior, only they cannot return to caller to signify termination, but must call man:kthread_exit[9]. In GEOM code, the usual use of threads is to offload processing of requests from `g_down` thread (the `.start`() function). These threads look like "event handlers": they have a linked list of event associated with them (on which events can be posted by various functions in various threads so it must be protected by a mutex), take the events from the list one by one and process them in a big `switch`() statement. The main benefit of using a thread to handle I/O requests is that it can sleep when needed. Now, this sounds good, but should be carefully thought out. Sleeping is well and very convenient but can very effectively destroy performance of the geom transformation. Extremely performance-sensitive classes probably should do all the work in `.start`() function call, taking great care to handle out-of-memory and similar errors. The other benefit of having a event-handler thread like that is to serialize all the requests and responses coming from different geom threads into one thread. This is also very convenient but can be slow. In most cases, handling of `.done`() requests can be left to the `g_up` thread. Mutexes in FreeBSD kernel (see man:mutex[9]) have one distinction from their more common userland cousins - the code cannot sleep while holding a mutex). If the code needs to sleep a lot, man:sx[9] locks may be more appropriate. On the other hand, if you do almost everything in a single thread, you may get away with no mutexes at all. diff --git a/documentation/content/en/articles/hubs/_index.adoc b/documentation/content/en/articles/hubs/_index.adoc index 3819b76d71..55f3b50f0e 100644 --- a/documentation/content/en/articles/hubs/_index.adoc +++ b/documentation/content/en/articles/hubs/_index.adoc @@ -1,379 +1,379 @@ --- title: Mirroring FreeBSD authors: - author: Jun Kuriyama email: kuriyama@FreeBSD.org - author: Valentino Vaschetto email: logo@FreeBSD.org - author: Daniel Lang email: dl@leo.org - author: Ken Smith email: kensmith@FreeBSD.org -description: How to mirror FreeBSD +description: The all in one guide for mirroring the FreeBSD website, FTP servers, and more trademarks: ["freebsd", "general"] --- = Mirroring FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] include::shared/releases.adoc[] [.abstract-title] Abstract An in-progress article on how to mirror FreeBSD, aimed at hub administrators. ''' toc::[] [NOTE] ==== We are not accepting new mirrors at this time. ==== [[mirror-contact]] == Contact Information The Mirror System Coordinators can be reached through email at mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org]. There is also a {freebsd-hubs}. [[mirror-requirements]] == Requirements for FreeBSD Mirrors [[mirror-diskspace]] === Disk Space Disk space is one of the most important requirements. Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind that _official_ mirrors are probably required to be complete. The web pages should always be mirrored completely. Also note that the numbers stated here are reflecting the current state (at {rel120-current}-RELEASE/{rel113-current}-RELEASE). Further development and releases will only increase the required amount. Also make sure to keep some (ca. 10-20%) extra space around just to be sure. Here are some approximate figures: * Full FTP Distribution: 1.4 TB * CTM deltas: 10 GB * Web pages: 1GB The current disk usage of FTP Distribution can be found at link:ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes[ftp://ftp.FreeBSD.org/pub/FreeBSD/dir.sizes]. [[mirror-bandwidth]] === Network Connection/Bandwidth Of course, you need to be connected to the Internet. The required bandwidth depends on your intended use of the mirror. If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller than if you want to make the files publicly available. If you intend to become an official mirror, the bandwidth required will be even higher. We can only give rough estimates here: * Local site, no public access: basically no minimum, but < 2 Mbps could make syncing too slow. * Unofficial public site: 34 Mbps is probably a good start. * Official site: > 100 Mbps is recommended, and your host should be connected as close as possible to your border router. [[mirror-system]] === System Requirements, CPU, RAM One thing this depends on the expected number of clients, which is determined by the server's policy. It is also affected by the types of services you want to offer. Plain FTP or HTTP services may not require a huge amount of resources. Watch out if you provide rsync. This can have a huge impact on CPU and memory requirements as it is considered a memory hog. The following are just examples to give you a very rough hint. For a moderately visited site that offers rsync, you might consider a current CPU with around 800MHz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an _official_ site. For a frequently used site you definitely need more RAM (consider 2GB as a good start) and possibly more CPU, which could also mean that you need to go for a SMP system. You also want to consider a fast disk subsystem. Operations on the SVN repository require a fast disk subsystem (RAID is highly advised). A SCSI controller that has a cache of its own can also speed up things since most of these services incur a large number of small modifications to the disk. [[mirror-services]] === Services to Offer Every mirror site is required to have a set of core services available. In addition to these required services, there are a number of optional services that server administrators may choose to offer. This section explains which services you can provide and how to go about implementing them. [[mirror-serv-ftp]] ==== FTP (required for FTP Fileset) This is one of the most basic services, and it is required for each mirror offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and _must_ never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path [.filename]#/pub/FreeBSD#. There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order). * `/usr/libexec/ftpd`: FreeBSD's own ftpd can be used. Be sure to read man:ftpd[8]. * package:ftp/ncftpd[]: A commercial package, free for educational use. * package:ftp/oftpd[]: An ftpd designed with security as a main focus. * package:ftp/proftpd[]: A modular and very flexible ftpd. * package:ftp/pure-ftpd[]: Another ftpd developed with security in mind. * package:ftp/twoftpd[]: As above. * package:ftp/vsftpd[]: The "very secure" ftpd. FreeBSD's `ftpd`, `proftpd` and maybe `ncftpd` are among the most commonly used FTPds. The others do not have a large userbase among mirror sites. One thing to consider is that you may need flexibility in limiting how many simultaneous connections are allowed, thus limiting how much network bandwidth and system resources are consumed. [[mirror-serv-rsync]] ==== Rsync (optional for FTP Fileset) Rsync is often offered for access to the contents of the FTP area of FreeBSD, so other mirror sites can use your system as their source. The protocol is different from FTP in many ways. It is much more bandwidth friendly, as only differences between files are transferred instead of whole files when they change. Rsync does require a significant amount of memory for each instance. The size depends on the size of the synced module in terms of the number of directories and files. Rsync can use `rsh` and `ssh` (now default) as a transport, or use its own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available: * package:net/rsync[] [[mirror-serv-http]] ==== HTTP (required for Web Pages, Optional for FTP Fileset) If you want to offer the FreeBSD web pages, you will need to install a web server. You may optionally offer the FTP fileset via HTTP. The choice of web server software is left up to the mirror administrator. Some of the most popular choices are: * package:www/apache24[]: Apache is still one of the most widely deployed web servers on the Internet. It is used extensively by the FreeBSD Project. * package:www/boa[]: Boa is a single-tasking HTTP server. Unlike traditional web servers, it does not fork for each incoming connection, nor does it fork many copies of itself to handle multiple connections. Although, it should provide considerably great performance for purely static content. * package:www/cherokee[]: Cherokee is a very fast, flexible and easy to configure web server. It supports the widespread technologies nowadays: FastCGI, SCGI, PHP, CGI, SSL/TLS encrypted connections, vhosts, users authentication, on the fly encoding and load balancing. It also generates Apache compatible log files. * package:www/lighttpd[]: lighttpd is a secure, fast, compliant and very flexible web server which has been optimized for high-performance environments. It has a very low memory footprint compared to other web servers and takes care of cpu-load. * package:www/nginx[]: nginx is a high performance edge web server with a low memory footprint and key features to build a modern and efficient web infrastructure. Features include a HTTP server, HTTP and mail reverse proxy, caching, load balancing, compression, request throttling, connection multiplexing and reuse, SSL offload and HTTP media streaming. * package:www/thttpd[]: If you are going to be serving a large amount of static content you may find that using an application such as thttpd is more efficient than others. It is also optimized for excellent performance on FreeBSD. [[mirror-howto]] == How to Mirror FreeBSD Ok, now you know the requirements and how to offer the services, but not how to get it. :-) This section explains how to actually mirror the various parts of FreeBSD, what tools to use, and where to mirror from. [[mirror-ftp-rsync]] === Mirroring the FTP Site The FTP area is the largest amount of data that needs to be mirrored. It includes the _distribution sets_ required for network installation, the _branches_ which are actually snapshots of checked-out source trees, the _ISO Images_ to write CD-ROMs with the installation distribution, a live file system, and a snapshot of the ports tree. All of course for various FreeBSD versions, and various architectures. The best way to mirror the FTP area is rsync. You can install the port package:net/rsync[] and then use rsync to sync with your upstream host. rsync is already mentioned in <>. Since rsync access is not required, your preferred upstream site may not allow it. You may need to hunt around a little bit to find a site that allows rsync access. [NOTE] ==== Since the number of rsync clients will have a significant impact on the server machine, most admins impose limitations on their server. For a mirror, you should ask the site maintainer you are syncing from about their policy, and maybe an exception for your host (since you are a mirror). ==== A command line to mirror FreeBSD might look like: [source,shell] .... % rsync -vaHz --delete rsync://ftp4.de.FreeBSD.org/FreeBSD/ /pub/FreeBSD/ .... Consult the documentation for rsync, which is also available at http://rsync.samba.org/[http://rsync.samba.org/], about the various options to be used with rsync. If you sync the whole module (unlike subdirectories), be aware that the module-directory (here "FreeBSD") will not be created, so you cannot omit the target directory. Also you might want to set up a script framework that calls such a command via man:cron[8]. [[mirror-www]] === Mirroring the WWW Pages The FreeBSD website should only be mirrored via rsync. A command line to mirror the FreeBSD web site might look like: [source,shell] .... % rsync -vaHz --delete rsync://bit0.us-west.freebsd.org/FreeBSD-www-data/ /usr/local/www/ .... [[mirror-pkgs]] === Mirroring Packages Due to very high requirements of bandwidth, storage and adminstration the FreeBSD Project has decided not to allow public mirrors of packages. For sites with lots of machines, it might be advantagous to run a caching HTTP proxy for the man:pkg[8] process. Alternatively specific packages and their dependencies can be fetched by running something like the following: [source,shell] .... % pkg fetch -d -o /usr/local/mirror vim .... Once those packages have been fetched, the repository metadata must be generated by running: [source,shell] .... % pkg repo /usr/local/mirror .... Once the packages have been fetched and the metadata for the repository has been generated, serve the packages up to the client machines via HTTP. For additional information see the man pages for man:pkg[8], specifically the man:pkg-repo[8] page. [[mirror-how-often]] === How Often Should I Mirror? Every mirror should be updated at a minimum of once per day. Certainly a script with locking to prevent multiple runs happening at the same time will be needed to run from man:cron[8]. Since nearly every admin does this in their own way, specific instructions cannot be provided. It could work something like this: [.procedure] ==== . Put the command to run your mirroring application in a script. Use of a plain `/bin/sh` script is recommended. . Add some output redirections so diagnostic messages are logged to a file. . Test if your script works. Check the logs. . Use man:crontab[1] to add the script to the appropriate user's man:crontab[5]. This should be a different user than what your FTP daemon runs as so that if file permissions inside your FTP area are not world-readable those files cannot be accessed by anonymous FTP. This is used to "stage" releases - making sure all of the official mirror sites have all of the necessary release files on release day. ==== Here are some recommended schedules: * FTP fileset: daily * WWW pages: daily [[mirror-where]] == Where to Mirror From This is an important issue. So this section will spend some effort to explain the backgrounds. We will say this several times: under no circumstances should you mirror from `ftp.FreeBSD.org`. [[mirror-where-organization]] === A few Words About the Organization Mirrors are organized by country. All official mirrors have a DNS entry of the form `ftpN.CC.FreeBSD.org`. _CC_ (i.e., country code) is the _top level domain_ (TLD) of the country where this mirror is located. _N_ is a number, telling that the host would be the _Nth_ mirror in that country. (Same applies to `wwwN.CC.FreeBSD.org`, etc.) There are mirrors with no _CC_ part. These are the mirror sites that are very well connected and allow a large number of concurrent users. `ftp.FreeBSD.org` is actually two machines, one currently located in Denmark and the other in the United States. It is _NOT_ a master site and should never be used to mirror from. Lots of online documentation leads "interactive"users to `ftp.FreeBSD.org` so automated mirroring systems should find a different machine to mirror from. Additionally there exists a hierarchy of mirrors, which is described in terms of __tiers__. The master sites are not referred to but can be described as __Tier-0__. Mirrors that mirror from these sites can be considered __Tier-1__, mirrors of __Tier-1__-mirrors, are __Tier-2__, etc. Official sites are encouraged to be of a low __tier__, but the lower the tier the higher the requirements in terms as described in <>. Also access to low-tier-mirrors may be restricted, and access to master sites is definitely restricted. The __tier__-hierarchy is not reflected by DNS and generally not documented anywhere except for the master sites. However, official mirrors with low numbers like 1-4, are usually _Tier-1_ (this is just a rough hint, and there is no rule). [[mirror-where-where]] === Ok, but Where Should I get the Stuff Now? Under no circumstances should you mirror from `ftp.FreeBSD.org`. The short answer is: from the site that is closest to you in Internet terms, or gives you the fastest access. [[mirror-where-simple]] ==== I Just Want to Mirror from Somewhere! If you have no special intentions or requirements, the statement in <> applies. This means: [.procedure] ==== . Check for those which provide fastest access (number of hops, round-trip-times) and offer the services you intend to use (like rsync). . Contact the administrators of your chosen site stating your request, and asking about their terms and policies. . Set up your mirror as described above. ==== [[mirror-where-official]] ==== I am an Official Mirror, What is the Right Rite for Me? In general the description in <> still applies. Of course you may want to put some weight on the fact that your upstream should be of a low tier. There are some other considerations about _official_ mirrors that are described in <>. [[mirror-where-master]] ==== I Want to Access the Master Sites! If you have good reasons and good prerequisites, you may want and get access to one of the master sites. Access to these sites is generally restricted, and there are special policies for access. If you are already an _official_ mirror, this certainly helps you getting access. In any other case make sure your country really needs another mirror. If it already has three or more, ask the "zone administrator" (mailto:hostmaster@CC.FreeBSD.org[hostmaster@CC.FreeBSD.org]) or {freebsd-hubs} first. Whoever helped you become, an _official_ should have helped you gain access to an appropriate upstream host, either one of the master sites or a suitable Tier-1 site. If not, you can send email to mailto:mirror-admin@FreeBSD.org[mirror-admin@FreeBSD.org] to request help with that. There is one master site for the FTP fileset. [[mirror-where-master-ftp]] ===== ftp-master.FreeBSD.org This is the master site for the FTP fileset. `ftp-master.FreeBSD.org` provides rsync access, in addition to FTP. Refer to <>. Mirrors are also encouraged to allow rsync access for the FTP contents, since they are __Tier-1__-mirrors. [[mirror-official]] == Official Mirrors Official mirrors are mirrors that * a) have a `FreeBSD.org` DNS entry (usually a CNAME). * b) are listed as an official mirror in the FreeBSD documentation (like handbook). So far to distinguish official mirrors. Official mirrors are not necessarily __Tier-1__-mirrors. However you probably will not find a __Tier-1__-mirror, that is not also official. [[mirror-official-requirements]] === Special Requirements for Official (tier-1) Mirrors It is not so easy to state requirements for all official mirrors, since the project is sort of tolerant here. It is more easy to say, what _official tier-1 mirrors_ are required to. All other official mirrors can consider this a big __should__. Tier-1 mirrors are required to: * carry the complete fileset * allow access to other mirror sites * provide FTP and rsync access Furthermore, admins should be subscribed to the {freebsd-hubs}. See link:{handbook}#eresources-mail[this link] for details, how to subscribe. [IMPORTANT] ==== It is _very_ important for a hub administrator, especially Tier-1 hub admins, to check the https://www.FreeBSD.org/releng/[release schedule] for the next FreeBSD release. This is important because it will tell you when the next release is scheduled to come out, and thus giving you time to prepare for the big spike of traffic which follows it. It is also important that hub administrators try to keep their mirrors as up-to-date as possible (again, even more crucial for Tier-1 mirrors). If Mirror1 does not update for a while, lower tier mirrors will begin to mirror old data from Mirror1 and thus begins a downward spiral... Keep your mirrors up to date! ==== [[mirror-official-become]] === How to Become Official Then? We are not accepting any new mirrors at this time. [[mirror-statpages]] == Some Statistics from Mirror Sites Here are links to the stat pages of your favorite mirrors (aka the only ones who feel like providing stats). [[mirror-statpagesftp]] === FTP Site Statistics * ftp.is.FreeBSD.org - mailto:hostmaster@is.FreeBSD.org[hostmaster@is.FreeBSD.org] - http://www.rhnet.is/status/draupnir/draupnir.html[ (Bandwidth)] http://www.rhnet.is/status/ftp/ftp-notendur.html[(FTP processes)] http://www.rhnet.is/status/ftp/http-notendur.html[(HTTP processes)] * ftp2.ru.FreeBSD.org - mailto:mirror@macomnet.ru[mirror@macomnet.ru] - http://mirror.macomnet.net/mrtg/mirror.macomnet.net_195.128.64.25.html[(Bandwidth)] http://mirror.macomnet.net/mrtg/mirror.macomnet.net_proc.html[(HTTP and FTP users)] diff --git a/documentation/content/en/articles/leap-seconds/_index.adoc b/documentation/content/en/articles/leap-seconds/_index.adoc index 8ad3c6154e..9dc11b3483 100644 --- a/documentation/content/en/articles/leap-seconds/_index.adoc +++ b/documentation/content/en/articles/leap-seconds/_index.adoc @@ -1,84 +1,84 @@ --- title: FreeBSD Support for Leap Seconds -description: FreeBSD Support for Leap Seconds +description: A short description of how leap seconds are handled on FreeBSD --- = FreeBSD Support for Leap Seconds :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/en/urls.adoc[] ''' toc::[] [[leapseconds-definition]] == Introduction A _leap second_ is an one second adjustment made at specific times of year to UTC to synchronize atomic time scales with variations in the rotation of the Earth. This article describes how FreeBSD interacts with leap seconds. As of this writing, the next leap second will occur at 2015-Jun-30 23:59:60 UTC. This leap second will occur during a business day for North and South America and the Asia/Pacific region. Leap seconds are announced by http://datacenter.iers.org/[IERS] on http://datacenter.iers.org/web/guest/bulletins/-/somos/5Rgv/product/16[Bulletin C]. Standard leap second behavior is described in https://tools.ietf.org/html/rfc7164#section-3[RFC 7164]. Also see man:time2posix[3]. [[leapseconds-posix]] == Default Leap Second Handling on FreeBSD The easiest way to handle leap seconds is with the POSIX time rules FreeBSD uses by default, combined with link:{handbook}#network-ntp[NTP]. When man:ntpd[8] is running and the time is synchronized with upstream NTP servers that handle leap seconds correctly, the leap second will cause the system time to automatically repeat the last second of the day. No other adjustments are necessary. If the upstream NTP servers do not handle leap seconds correctly, man:ntpd[8] will step the time by one second after the errant upstream server has noticed and stepped itself. If NTP is not being used, manual adjustment of the system clock will be required after the leap second has passed. [[leapseconds-cautions]] == Cautions Leap seconds are inserted at the same instant all over the world: UTC midnight. In Japan that is mid-morning, in the Pacific mid-day, in the Americas late afternoon, and in Europe at night. We believe and expect that FreeBSD, if provided correct and stable NTP service, will work as designed during this leap second, as it did during the previous ones. However, we caution that practically no applications have ever asked the kernel about leap seconds. Our experience is that, as designed, leap seconds are essentially a replay of the second before the leap second, and this is a surprise to most application programmers. Other operating systems and other computers may or may not handle the leap-second the same way as FreeBSD, and systems without correct and stable NTP service will not know anything about leap seconds at all. It is not unheard of for computers to crash because of leap seconds, and experience has shown that a large fraction of all public NTP servers might handle and announce the leap second incorrectly. Please try to make sure nothing horrible happens because of the leap second. [[leapseconds-testing]] == Testing It is possible to test whether a leap second will be used. Due to the nature of NTP, the test might work up to 24 hours before the leap second. Some major reference clock sources only announce leap seconds one hour ahead of the event. Query the NTP daemon: [source,shell] .... % ntpq -c 'rv 0 leap' .... Output that includes `leap_add_sec` indicates proper support of the leap second. Before the 24 hours leading up to the leap second, or after the leap second has passed, `leap_none` will be shown. [[leapseconds-conclusion]] == Conclusion In practice, leap seconds are usually not a problem on FreeBSD. We hope that this overview helps clarify what to expect and how to make the leap second event proceed more smoothly. diff --git a/documentation/content/en/articles/linux-emulation/_index.adoc b/documentation/content/en/articles/linux-emulation/_index.adoc index 701bf524cc..d5c6d985ed 100644 --- a/documentation/content/en/articles/linux-emulation/_index.adoc +++ b/documentation/content/en/articles/linux-emulation/_index.adoc @@ -1,1416 +1,1416 @@ --- title: Linux® emulation in FreeBSD authors: - author: Roman Divacky email: rdivacky@FreeBSD.org -description: Linux® emulation in FreeBSD +description: A technical description about the internals of the Linux emulation layer in FreeBSD trademarks: ["freebsd", "ibm", "adobe", "netbsd", "realnetworks", "oracle", "linux", "sun", "general"] --- = Linux(R) emulation in FreeBSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/authors.adoc[] [.abstract-title] Abstract This masters thesis deals with updating the Linux(R) emulation layer (the so called _Linuxulator_). The task was to update the layer to match the functionality of Linux(R) 2.6. As a reference implementation, the Linux(R) 2.6.16 kernel was chosen. The concept is loosely based on the NetBSD implementation. Most of the work was done in the summer of 2006 as a part of the Google Summer of Code students program. The focus was on bringing the _NPTL_ (new POSIX(R) thread library) support into the emulation layer, including _TLS_ (thread local storage), _futexes_ (fast user space mutexes), _PID mangling_, and some other minor things. Many small problems were identified and fixed in the process. My work was integrated into the main FreeBSD source repository and will be shipped in the upcoming 7.0R release. We, the emulation development team, are working on making the Linux(R) 2.6 emulation the default emulation layer in FreeBSD. ''' toc::[] [[intro]] == Introduction In the last few years the open source UNIX(R) based operating systems started to be widely deployed on server and client machines. Among these operating systems I would like to point out two: FreeBSD, for its BSD heritage, time proven code base and many interesting features and Linux(R) for its wide user base, enthusiastic open developer community and support from large companies. FreeBSD tends to be used on server class machines serving heavy duty networking tasks with less usage on desktop class machines for ordinary users. While Linux(R) has the same usage on servers, but it is used much more by home based users. This leads to a situation where there are many binary only programs available for Linux(R) that lack support for FreeBSD. Naturally, a need for the ability to run Linux(R) binaries on a FreeBSD system arises and this is what this thesis deals with: the emulation of the Linux(R) kernel in the FreeBSD operating system. During the Summer of 2006 Google Inc. sponsored a project which focused on extending the Linux(R) emulation layer (the so called Linuxulator) in FreeBSD to include Linux(R) 2.6 facilities. This thesis is written as a part of this project. [[inside]] == A look inside... In this section we are going to describe every operating system in question. How they deal with syscalls, trapframes etc., all the low-level stuff. We also describe the way they understand common UNIX(R) primitives like what a PID is, what a thread is, etc. In the third subsection we talk about how UNIX(R) on UNIX(R) emulation could be done in general. [[what-is-unix]] === What is UNIX(R) UNIX(R) is an operating system with a long history that has influenced almost every other operating system currently in use. Starting in the 1960s, its development continues to this day (although in different projects). UNIX(R) development soon forked into two main ways: the BSDs and System III/V families. They mutually influenced themselves by growing a common UNIX(R) standard. Among the contributions originated in BSD we can name virtual memory, TCP/IP networking, FFS, and many others. The System V branch contributed to SysV interprocess communication primitives, copy-on-write, etc. UNIX(R) itself does not exist any more but its ideas have been used by many other operating systems world wide thus forming the so called UNIX(R)-like operating systems. These days the most influential ones are Linux(R), Solaris, and possibly (to some extent) FreeBSD. There are in-company UNIX(R) derivatives (AIX, HP-UX etc.), but these have been more and more migrated to the aforementioned systems. Let us summarize typical UNIX(R) characteristics. [[tech-details]] === Technical details Every running program constitutes a process that represents a state of the computation. Running process is divided between kernel-space and user-space. Some operations can be done only from kernel space (dealing with hardware etc.), but the process should spend most of its lifetime in the user space. The kernel is where the management of the processes, hardware, and low-level details take place. The kernel provides a standard unified UNIX(R) API to the user space. The most important ones are covered below. [[kern-proc-comm]] ==== Communication between kernel and user space process Common UNIX(R) API defines a syscall as a way to issue commands from a user space process to the kernel. The most common implementation is either by using an interrupt or specialized instruction (think of `SYSENTER`/`SYSCALL` instructions for ia32). Syscalls are defined by a number. For example in FreeBSD, the syscall number 85 is the man:swapon[2] syscall and the syscall number 132 is man:mkfifo[2]. Some syscalls need parameters, which are passed from the user-space to the kernel-space in various ways (implementation dependant). Syscalls are synchronous. Another possible way to communicate is by using a _trap_. Traps occur asynchronously after some event occurs (division by zero, page fault etc.). A trap can be transparent for a process (page fault) or can result in a reaction like sending a _signal_ (division by zero). [[proc-proc-comm]] ==== Communication between processes There are other APIs (System V IPC, shared memory etc.) but the single most important API is signal. Signals are sent by processes or by the kernel and received by processes. Some signals can be ignored or handled by a user supplied routine, some result in a predefined action that cannot be altered or ignored. [[proc-mgmt]] ==== Process management Kernel instances are processed first in the system (so called init). Every running process can create its identical copy using the man:fork[2] syscall. Some slightly modified versions of this syscall were introduced but the basic semantic is the same. Every running process can morph into some other process using the man:exec[3] syscall. Some modifications of this syscall were introduced but all serve the same basic purpose. Processes end their lives by calling the man:exit[2] syscall. Every process is identified by a unique number called PID. Every process has a defined parent (identified by its PID). [[thread-mgmt]] ==== Thread management Traditional UNIX(R) does not define any API nor implementation for threading, while POSIX(R) defines its threading API but the implementation is undefined. Traditionally there were two ways of implementing threads. Handling them as separate processes (1:1 threading) or envelope the whole thread group in one process and managing the threading in userspace (1:N threading). Comparing main features of each approach: 1:1 threading * - heavyweight threads * - the scheduling cannot be altered by the user (slightly mitigated by the POSIX(R) API) * + no syscall wrapping necessary * + can utilize multiple CPUs 1:N threading * + lightweight threads * + scheduling can be easily altered by the user * - syscalls must be wrapped * - cannot utilize more than one CPU [[what-is-freebsd]] === What is FreeBSD? The FreeBSD project is one of the oldest open source operating systems currently available for daily use. It is a direct descendant of the genuine UNIX(R) so it could be claimed that it is a true UNIX(R) although licensing issues do not permit that. The start of the project dates back to the early 1990's when a crew of fellow BSD users patched the 386BSD operating system. Based on this patchkit a new operating system arose named FreeBSD for its liberal license. Another group created the NetBSD operating system with different goals in mind. We will focus on FreeBSD. FreeBSD is a modern UNIX(R)-based operating system with all the features of UNIX(R). Preemptive multitasking, multiuser facilities, TCP/IP networking, memory protection, symmetric multiprocessing support, virtual memory with merged VM and buffer cache, they are all there. One of the interesting and extremely useful features is the ability to emulate other UNIX(R)-like operating systems. As of December 2006 and 7-CURRENT development, the following emulation functionalities are supported: * FreeBSD/i386 emulation on FreeBSD/amd64 * FreeBSD/i386 emulation on FreeBSD/ia64 * Linux(R)-emulation of Linux(R) operating system on FreeBSD * NDIS-emulation of Windows networking drivers interface * NetBSD-emulation of NetBSD operating system * PECoff-support for PECoff FreeBSD executables * SVR4-emulation of System V revision 4 UNIX(R) Actively developed emulations are the Linux(R) layer and various FreeBSD-on-FreeBSD layers. Others are not supposed to work properly nor be usable these days. [[freebsd-tech-details]] ==== Technical details FreeBSD is traditional flavor of UNIX(R) in the sense of dividing the run of processes into two halves: kernel space and user space run. There are two types of process entry to the kernel: a syscall and a trap. There is only one way to return. In the subsequent sections we will describe the three gates to/from the kernel. The whole description applies to the i386 architecture as the Linuxulator only exists there but the concept is similar on other architectures. The information was taken from [1] and the source code. [[freebsd-sys-entries]] ===== System entries FreeBSD has an abstraction called an execution class loader, which is a wedge into the man:execve[2] syscall. This employs a structure `sysentvec`, which describes an executable ABI. It contains things like errno translation table, signal translation table, various functions to serve syscall needs (stack fixup, coredumping, etc.). Every ABI the FreeBSD kernel wants to support must define this structure, as it is used later in the syscall processing code and at some other places. System entries are handled by trap handlers, where we can access both the kernel-space and the user-space at once. [[freebsd-syscalls]] ===== Syscalls Syscalls on FreeBSD are issued by executing interrupt `0x80` with register `%eax` set to a desired syscall number with arguments passed on the stack. When a process issues an interrupt `0x80`, the `int0x80` syscall trap handler is issued (defined in [.filename]#sys/i386/i386/exception.s#), which prepares arguments (i.e. copies them on to the stack) for a call to a C function man:syscall[2] (defined in [.filename]#sys/i386/i386/trap.c#), which processes the passed in trapframe. The processing consists of preparing the syscall (depending on the `sysvec` entry), determining if the syscall is 32-bit or 64-bit one (changes size of the parameters), then the parameters are copied, including the syscall. Next, the actual syscall function is executed with processing of the return code (special cases for `ERESTART` and `EJUSTRETURN` errors). Finally an `userret()` is scheduled, switching the process back to the users-pace. The parameters to the actual syscall handler are passed in the form of `struct thread *td`, `struct syscall args *` arguments where the second parameter is a pointer to the copied in structure of parameters. [[freebsd-traps]] ===== Traps Handling of traps in FreeBSD is similar to the handling of syscalls. Whenever a trap occurs, an assembler handler is called. It is chosen between alltraps, alltraps with regs pushed or calltrap depending on the type of the trap. This handler prepares arguments for a call to a C function `trap()` (defined in [.filename]#sys/i386/i386/trap.c#), which then processes the occurred trap. After the processing it might send a signal to the process and/or exit to userland using `userret()`. [[freebsd-exits]] ===== Exits Exits from kernel to userspace happen using the assembler routine `doreti` regardless of whether the kernel was entered via a trap or via a syscall. This restores the program status from the stack and returns to the userspace. [[freebsd-unix-primitives]] ===== UNIX(R) primitives FreeBSD operating system adheres to the traditional UNIX(R) scheme, where every process has a unique identification number, the so called _PID_ (Process ID). PID numbers are allocated either linearly or randomly ranging from `0` to `PID_MAX`. The allocation of PID numbers is done using linear searching of PID space. Every thread in a process receives the same PID number as result of the man:getpid[2] call. There are currently two ways to implement threading in FreeBSD. The first way is M:N threading followed by the 1:1 threading model. The default library used is M:N threading (`libpthread`) and you can switch at runtime to 1:1 threading (`libthr`). The plan is to switch to 1:1 library by default soon. Although those two libraries use the same kernel primitives, they are accessed through different API(es). The M:N library uses the `kse_*` family of syscalls while the 1:1 library uses the `thr_*` family of syscalls. Due to this, there is no general concept of thread ID shared between kernel and userspace. Of course, both threading libraries implement the pthread thread ID API. Every kernel thread (as described by `struct thread`) has td tid identifier but this is not directly accessible from userland and solely serves the kernel's needs. It is also used for 1:1 threading library as pthread's thread ID but handling of this is internal to the library and cannot be relied on. As stated previously there are two implementations of threading in FreeBSD. The M:N library divides the work between kernel space and userspace. Thread is an entity that gets scheduled in the kernel but it can represent various number of userspace threads. M userspace threads get mapped to N kernel threads thus saving resources while keeping the ability to exploit multiprocessor parallelism. Further information about the implementation can be obtained from the man page or [1]. The 1:1 library directly maps a userland thread to a kernel thread thus greatly simplifying the scheme. None of these designs implement a fairness mechanism (such a mechanism was implemented but it was removed recently because it caused serious slowdown and made the code more difficult to deal with). [[what-is-linux]] === What is Linux(R) Linux(R) is a UNIX(R)-like kernel originally developed by Linus Torvalds, and now being contributed to by a massive crowd of programmers all around the world. From its mere beginnings to today, with wide support from companies such as IBM or Google, Linux(R) is being associated with its fast development pace, full hardware support and benevolent dictator model of organization. Linux(R) development started in 1991 as a hobbyist project at University of Helsinki in Finland. Since then it has obtained all the features of a modern UNIX(R)-like OS: multiprocessing, multiuser support, virtual memory, networking, basically everything is there. There are also highly advanced features like virtualization etc. As of 2006 Linux(R) seems to be the most widely used open source operating system with support from independent software vendors like Oracle, RealNetworks, Adobe, etc. Most of the commercial software distributed for Linux(R) can only be obtained in a binary form so recompilation for other operating systems is impossible. Most of the Linux(R) development happens in a Git version control system. Git is a distributed system so there is no central source of the Linux(R) code, but some branches are considered prominent and official. The version number scheme implemented by Linux(R) consists of four numbers A.B.C.D. Currently development happens in 2.6.C.D, where C represents major version, where new features are added or changed while D is a minor version for bugfixes only. More information can be obtained from [3]. [[linux-tech-details]] ==== Technical details Linux(R) follows the traditional UNIX(R) scheme of dividing the run of a process in two halves: the kernel and user space. The kernel can be entered in two ways: via a trap or via a syscall. The return is handled only in one way. The further description applies to Linux(R) 2.6 on the i386(TM) architecture. This information was taken from [2]. [[linux-syscalls]] ===== Syscalls Syscalls in Linux(R) are performed (in userspace) using `syscallX` macros where X substitutes a number representing the number of parameters of the given syscall. This macro translates to a code that loads `%eax` register with a number of the syscall and executes interrupt `0x80`. After this syscall return is called, which translates negative return values to positive `errno` values and sets `res` to `-1` in case of an error. Whenever the interrupt `0x80` is called the process enters the kernel in system call trap handler. This routine saves all registers on the stack and calls the selected syscall entry. Note that the Linux(R) calling convention expects parameters to the syscall to be passed via registers as shown here: . parameter -> `%ebx` . parameter -> `%ecx` . parameter -> `%edx` . parameter -> `%esi` . parameter -> `%edi` . parameter -> `%ebp` There are some exceptions to this, where Linux(R) uses different calling convention (most notably the `clone` syscall). [[linux-traps]] ===== Traps The trap handlers are introduced in [.filename]#arch/i386/kernel/traps.c# and most of these handlers live in [.filename]#arch/i386/kernel/entry.S#, where handling of the traps happens. [[linux-exits]] ===== Exits Return from the syscall is managed by syscall man:exit[3], which checks for the process having unfinished work, then checks whether we used user-supplied selectors. If this happens stack fixing is applied and finally the registers are restored from the stack and the process returns to the userspace. [[linux-unix-primitives]] ===== UNIX(R) primitives In the 2.6 version, the Linux(R) operating system redefined some of the traditional UNIX(R) primitives, notably PID, TID and thread. PID is defined not to be unique for every process, so for some processes (threads) man:getppid[2] returns the same value. Unique identification of process is provided by TID. This is because _NPTL_ (New POSIX(R) Thread Library) defines threads to be normal processes (so called 1:1 threading). Spawning a new process in Linux(R) 2.6 happens using the `clone` syscall (fork variants are reimplemented using it). This clone syscall defines a set of flags that affect behavior of the cloning process regarding thread implementation. The semantic is a bit fuzzy as there is no single flag telling the syscall to create a thread. Implemented clone flags are: * `CLONE_VM` - processes share their memory space * `CLONE_FS` - share umask, cwd and namespace * `CLONE_FILES` - share open files * `CLONE_SIGHAND` - share signal handlers and blocked signals * `CLONE_PARENT` - share parent * `CLONE_THREAD` - be thread (further explanation below) * `CLONE_NEWNS` - new namespace * `CLONE_SYSVSEM` - share SysV undo structures * `CLONE_SETTLS` - setup TLS at supplied address * `CLONE_PARENT_SETTID` - set TID in the parent * `CLONE_CHILD_CLEARTID` - clear TID in the child * `CLONE_CHILD_SETTID` - set TID in the child `CLONE_PARENT` sets the real parent to the parent of the caller. This is useful for threads because if thread A creates thread B we want thread B to be parented to the parent of the whole thread group. `CLONE_THREAD` does exactly the same thing as `CLONE_PARENT`, `CLONE_VM` and `CLONE_SIGHAND`, rewrites PID to be the same as PID of the caller, sets exit signal to be none and enters the thread group. `CLONE_SETTLS` sets up GDT entries for TLS handling. The `CLONE_*_*TID` set of flags sets/clears user supplied address to TID or 0. As you can see the `CLONE_THREAD` does most of the work and does not seem to fit the scheme very well. The original intention is unclear (even for authors, according to comments in the code) but I think originally there was one threading flag, which was then parcelled among many other flags but this separation was never fully finished. It is also unclear what this partition is good for as glibc does not use that so only hand-written use of the clone permits a programmer to access this features. For non-threaded programs the PID and TID are the same. For threaded programs the first thread PID and TID are the same and every created thread shares the same PID and gets assigned a unique TID (because `CLONE_THREAD` is passed in) also parent is shared for all processes forming this threaded program. The code that implements man:pthread_create[3] in NPTL defines the clone flags like this: [.programlisting] .... int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGNAL | CLONE_SETTLS | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID | CLONE_SYSVSEM #if __ASSUME_NO_CLONE_DETACHED == 0 | CLONE_DETACHED #endif | 0); .... The `CLONE_SIGNAL` is defined like [.programlisting] .... #define CLONE_SIGNAL (CLONE_SIGHAND | CLONE_THREAD) .... the last 0 means no signal is sent when any of the threads exits. [[what-is-emu]] === What is emulation According to a dictionary definition, emulation is the ability of a program or device to imitate another program or device. This is achieved by providing the same reaction to a given stimulus as the emulated object. In practice, the software world mostly sees three types of emulation - a program used to emulate a machine (QEMU, various game console emulators etc.), software emulation of a hardware facility (OpenGL emulators, floating point units emulation etc.) and operating system emulation (either in kernel of the operating system or as a userspace program). Emulation is usually used in a place, where using the original component is not feasible nor possible at all. For example someone might want to use a program developed for a different operating system than they use. Then emulation comes in handy. Sometimes there is no other way but to use emulation - e.g. when the hardware device you try to use does not exist (yet/anymore) then there is no other way but emulation. This happens often when porting an operating system to a new (non-existent) platform. Sometimes it is just cheaper to emulate. Looking from an implementation point of view, there are two main approaches to the implementation of emulation. You can either emulate the whole thing - accepting possible inputs of the original object, maintaining inner state and emitting correct output based on the state and/or input. This kind of emulation does not require any special conditions and basically can be implemented anywhere for any device/program. The drawback is that implementing such emulation is quite difficult, time-consuming and error-prone. In some cases we can use a simpler approach. Imagine you want to emulate a printer that prints from left to right on a printer that prints from right to left. It is obvious that there is no need for a complex emulation layer but simply reversing of the printed text is sufficient. Sometimes the emulating environment is very similar to the emulated one so just a thin layer of some translation is necessary to provide fully working emulation! As you can see this is much less demanding to implement, so less time-consuming and error-prone than the previous approach. But the necessary condition is that the two environments must be similar enough. The third approach combines the two previous. Most of the time the objects do not provide the same capabilities so in a case of emulating the more powerful one on the less powerful we have to emulate the missing features with full emulation described above. This master thesis deals with emulation of UNIX(R) on UNIX(R), which is exactly the case, where only a thin layer of translation is sufficient to provide full emulation. The UNIX(R) API consists of a set of syscalls, which are usually self contained and do not affect some global kernel state. There are a few syscalls that affect inner state but this can be dealt with by providing some structures that maintain the extra state. No emulation is perfect and emulations tend to lack some parts but this usually does not cause any serious drawbacks. Imagine a game console emulator that emulates everything but music output. No doubt that the games are playable and one can use the emulator. It might not be that comfortable as the original game console but its an acceptable compromise between price and comfort. The same goes with the UNIX(R) API. Most programs can live with a very limited set of syscalls working. Those syscalls tend to be the oldest ones (man:read[2]/man:write[2], man:fork[2] family, man:signal[3] handling, man:exit[3], man:socket[2] API) hence it is easy to emulate because their semantics is shared among all UNIX(R)es, which exist todays. [[freebsd-emulation]] == Emulation === How emulation works in FreeBSD As stated earlier, FreeBSD supports running binaries from several other UNIX(R)es. This works because FreeBSD has an abstraction called the execution class loader. This wedges into the man:execve[2] syscall, so when man:execve[2] is about to execute a binary it examines its type. There are basically two types of binaries in FreeBSD. Shell-like text scripts which are identified by `#!` as their first two characters and normal (typically _ELF_) binaries, which are a representation of a compiled executable object. The vast majority (one could say all of them) of binaries in FreeBSD are from type ELF. ELF files contain a header, which specifies the OS ABI for this ELF file. By reading this information, the operating system can accurately determine what type of binary the given file is. Every OS ABI must be registered in the FreeBSD kernel. This applies to the FreeBSD native OS ABI, as well. So when man:execve[2] executes a binary it iterates through the list of registered APIs and when it finds the right one it starts to use the information contained in the OS ABI description (its syscall table, `errno` translation table, etc.). So every time the process calls a syscall, it uses its own set of syscalls instead of some global one. This effectively provides a very elegant and easy way of supporting execution of various binary formats. The nature of emulation of different OSes (and also some other subsystems) led developers to invite a handler event mechanism. There are various places in the kernel, where a list of event handlers are called. Every subsystem can register an event handler and they are called accordingly. For example, when a process exits there is a handler called that possibly cleans up whatever the subsystem needs to be cleaned. Those simple facilities provide basically everything that is needed for the emulation infrastructure and in fact these are basically the only things necessary to implement the Linux(R) emulation layer. [[freebsd-common-primitives]] === Common primitives in the FreeBSD kernel Emulation layers need some support from the operating system. I am going to describe some of the supported primitives in the FreeBSD operating system. [[freebsd-locking-primitives]] ==== Locking primitives Contributed by: `{attilio}` The FreeBSD synchronization primitive set is based on the idea to supply a rather huge number of different primitives in a way that the better one can be used for every particular, appropriate situation. To a high level point of view you can consider three kinds of synchronization primitives in the FreeBSD kernel: * atomic operations and memory barriers * locks * scheduling barriers Below there are descriptions for the 3 families. For every lock, you should really check the linked manpage (where possible) for more detailed explanations. [[freebsd-atomic-op]] ===== Atomic operations and memory barriers Atomic operations are implemented through a set of functions performing simple arithmetics on memory operands in an atomic way with respect to external events (interrupts, preemption, etc.). Atomic operations can guarantee atomicity just on small data types (in the magnitude order of the `.long.` architecture C data type), so should be rarely used directly in the end-level code, if not only for very simple operations (like flag setting in a bitmap, for example). In fact, it is rather simple and common to write down a wrong semantic based on just atomic operations (usually referred as lock-less). The FreeBSD kernel offers a way to perform atomic operations in conjunction with a memory barrier. The memory barriers will guarantee that an atomic operation will happen following some specified ordering with respect to other memory accesses. For example, if we need that an atomic operation happen just after all other pending writes (in terms of instructions reordering buffers activities) are completed, we need to explicitly use a memory barrier in conjunction to this atomic operation. So it is simple to understand why memory barriers play a key role for higher-level locks building (just as refcounts, mutexes, etc.). For a detailed explanatory on atomic operations, please refer to man:atomic[9]. It is far, however, noting that atomic operations (and memory barriers as well) should ideally only be used for building front-ending locks (as mutexes). [[freebsd-refcounts]] ===== Refcounts Refcounts are interfaces for handling reference counters. They are implemented through atomic operations and are intended to be used just for cases, where the reference counter is the only one thing to be protected, so even something like a spin-mutex is deprecated. Using the refcount interface for structures, where a mutex is already used is often wrong since we should probably close the reference counter in some already protected paths. A manpage discussing refcount does not exist currently, just check [.filename]#sys/refcount.h# for an overview of the existing API. [[freebsd-locks]] ===== Locks FreeBSD kernel has huge classes of locks. Every lock is defined by some peculiar properties, but probably the most important is the event linked to contesting holders (or in other terms, the behavior of threads unable to acquire the lock). FreeBSD's locking scheme presents three different behaviors for contenders: . spinning . blocking . sleeping [NOTE] ==== numbers are not casual ==== [[freebsd-spinlocks]] ===== Spinning locks Spin locks let waiters to spin until they cannot acquire the lock. An important matter do deal with is when a thread contests on a spin lock if it is not descheduled. Since the FreeBSD kernel is preemptive, this exposes spin lock at the risk of deadlocks that can be solved just disabling interrupts while they are acquired. For this and other reasons (like lack of priority propagation support, poorness in load balancing schemes between CPUs, etc.), spin locks are intended to protect very small paths of code, or ideally not to be used at all if not explicitly requested (explained later). [[freebsd-blocking]] ===== Blocking Block locks let waiters to be descheduled and blocked until the lock owner does not drop it and wakes up one or more contenders. In order to avoid starvation issues, blocking locks do priority propagation from the waiters to the owner. Block locks must be implemented through the turnstile interface and are intended to be the most used kind of locks in the kernel, if no particular conditions are met. [[freebsd-sleeping]] ===== Sleeping Sleep locks let waiters to be descheduled and fall asleep until the lock holder does not drop it and wakes up one or more waiters. Since sleep locks are intended to protect large paths of code and to cater asynchronous events, they do not do any form of priority propagation. They must be implemented through the man:sleepqueue[9] interface. The order used to acquire locks is very important, not only for the possibility to deadlock due at lock order reversals, but even because lock acquisition should follow specific rules linked to locks natures. If you give a look at the table above, the practical rule is that if a thread holds a lock of level n (where the level is the number listed close to the kind of lock) it is not allowed to acquire a lock of superior levels, since this would break the specified semantic for a path. For example, if a thread holds a block lock (level 2), it is allowed to acquire a spin lock (level 1) but not a sleep lock (level 3), since block locks are intended to protect smaller paths than sleep lock (these rules are not about atomic operations or scheduling barriers, however). This is a list of lock with their respective behaviors: * spin mutex - spinning - man:mutex[9] * sleep mutex - blocking - man:mutex[9] * pool mutex - blocking - man:mtx[pool] * sleep family - sleeping - man:sleep[9] pause tsleep msleep msleep spin msleep rw msleep sx * condvar - sleeping - man:condvar[9] * rwlock - blocking - man:rwlock[9] * sxlock - sleeping - man:sx[9] * lockmgr - sleeping - man:lockmgr[9] * semaphores - sleeping - man:sema[9] Among these locks only mutexes, sxlocks, rwlocks and lockmgrs are intended to handle recursion, but currently recursion is only supported by mutexes and lockmgrs. [[freebsd-scheduling]] ===== Scheduling barriers Scheduling barriers are intended to be used in order to drive scheduling of threading. They consist mainly of three different stubs: * critical sections (and preemption) * sched_bind * sched_pin Generally, these should be used only in a particular context and even if they can often replace locks, they should be avoided because they do not let the diagnose of simple eventual problems with locking debugging tools (as man:witness[4]). [[freebsd-critical]] ===== Critical sections The FreeBSD kernel has been made preemptive basically to deal with interrupt threads. In fact, in order to avoid high interrupt latency, time-sharing priority threads can be preempted by interrupt threads (in this way, they do not need to wait to be scheduled as the normal path previews). Preemption, however, introduces new racing points that need to be handled, as well. Often, in order to deal with preemption, the simplest thing to do is to completely disable it. A critical section defines a piece of code (borderlined by the pair of functions man:critical_enter[9] and man:critical_exit[9], where preemption is guaranteed to not happen (until the protected code is fully executed). This can often replace a lock effectively but should be used carefully in order to not lose the whole advantage that preemption brings. [[freebsd-schedpin]] ===== sched_pin/sched_unpin Another way to deal with preemption is the `sched_pin()` interface. If a piece of code is closed in the `sched_pin()` and `sched_unpin()` pair of functions it is guaranteed that the respective thread, even if it can be preempted, it will always be executed on the same CPU. Pinning is very effective in the particular case when we have to access at per-cpu datas and we assume other threads will not change those data. The latter condition will determine a critical section as a too strong condition for our code. [[freebsd-schedbind]] ===== sched_bind/sched_unbind `sched_bind` is an API used in order to bind a thread to a particular CPU for all the time it executes the code, until a `sched_unbind` function call does not unbind it. This feature has a key role in situations where you cannot trust the current state of CPUs (for example, at very early stages of boot), as you want to avoid your thread to migrate on inactive CPUs. Since `sched_bind` and `sched_unbind` manipulate internal scheduler structures, they need to be enclosed in `sched_lock` acquisition/releasing when used. [[freebsd-proc]] ==== Proc structure Various emulation layers sometimes require some additional per-process data. It can manage separate structures (a list, a tree etc.) containing these data for every process but this tends to be slow and memory consuming. To solve this problem the FreeBSD `proc` structure contains `p_emuldata`, which is a void pointer to some emulation layer specific data. This `proc` entry is protected by the proc mutex. The FreeBSD `proc` structure contains a `p_sysent` entry that identifies, which ABI this process is running. In fact, it is a pointer to the `sysentvec` described above. So by comparing this pointer to the address where the `sysentvec` structure for the given ABI is stored we can effectively determine whether the process belongs to our emulation layer. The code typically looks like: [.programlisting] .... if (__predict_true(p->p_sysent != &elf_Linux(R)_sysvec)) return; .... As you can see, we effectively use the `__predict_true` modifier to collapse the most common case (FreeBSD process) to a simple return operation thus preserving high performance. This code should be turned into a macro because currently it is not very flexible, i.e. we do not support Linux(R)64 emulation nor A.OUT Linux(R) processes on i386. [[freebsd-vfs]] ==== VFS The FreeBSD VFS subsystem is very complex but the Linux(R) emulation layer uses just a small subset via a well defined API. It can either operate on vnodes or file handlers. Vnode represents a virtual vnode, i.e. representation of a node in VFS. Another representation is a file handler, which represents an opened file from the perspective of a process. A file handler can represent a socket or an ordinary file. A file handler contains a pointer to its vnode. More then one file handler can point to the same vnode. [[freebsd-namei]] ===== namei The man:namei[9] routine is a central entry point to pathname lookup and translation. It traverses the path point by point from the starting point to the end point using lookup function, which is internal to VFS. The man:namei[9] syscall can cope with symlinks, absolute and relative paths. When a path is looked up using man:namei[9] it is inputed to the name cache. This behavior can be suppressed. This routine is used all over the kernel and its performance is very critical. [[freebsd-vn]] ===== vn_fullpath The man:vn_fullpath[9] function takes the best effort to traverse VFS name cache and returns a path for a given (locked) vnode. This process is unreliable but works just fine for the most common cases. The unreliability is because it relies on VFS cache (it does not traverse the on medium structures), it does not work with hardlinks, etc. This routine is used in several places in the Linuxulator. [[freebsd-vnode]] ===== Vnode operations * `fgetvp` - given a thread and a file descriptor number it returns the associated vnode * man:vn_lock[9] - locks a vnode * `vn_unlock` - unlocks a vnode * man:VOP_READDIR[9] - reads a directory referenced by a vnode * man:VOP_GETATTR[9] - gets attributes of a file or a directory referenced by a vnode * man:VOP_LOOKUP[9] - looks up a path to a given directory * man:VOP_OPEN[9] - opens a file referenced by a vnode * man:VOP_CLOSE[9] - closes a file referenced by a vnode * man:vput[9] - decrements the use count for a vnode and unlocks it * man:vrele[9] - decrements the use count for a vnode * man:vref[9] - increments the use count for a vnode [[freebsd-file-handler]] ===== File handler operations * `fget` - given a thread and a file descriptor number it returns associated file handler and references it * `fdrop` - drops a reference to a file handler * `fhold` - references a file handler [[md]] == Linux(R) emulation layer -MD part This section deals with implementation of Linux(R) emulation layer in FreeBSD operating system. It first describes the machine dependent part talking about how and where interaction between userland and kernel is implemented. It talks about syscalls, signals, ptrace, traps, stack fixup. This part discusses i386 but it is written generally so other architectures should not differ very much. The next part is the machine independent part of the Linuxulator. This section only covers i386 and ELF handling. A.OUT is obsolete and untested. [[syscall-handling]] === Syscall handling Syscall handling is mostly written in [.filename]#linux_sysvec.c#, which covers most of the routines pointed out in the `sysentvec` structure. When a Linux(R) process running on FreeBSD issues a syscall, the general syscall routine calls linux prepsyscall routine for the Linux(R) ABI. [[linux-prepsyscall]] ==== Linux(R) prepsyscall Linux(R) passes arguments to syscalls via registers (that is why it is limited to 6 parameters on i386) while FreeBSD uses the stack. The Linux(R) prepsyscall routine must copy parameters from registers to the stack. The order of the registers is: `%ebx`, `%ecx`, `%edx`, `%esi`, `%edi`, `%ebp`. The catch is that this is true for only _most_ of the syscalls. Some (most notably `clone`) uses a different order but it is luckily easy to fix by inserting a dummy parameter in the `linux_clone` prototype. [[syscall-writing]] ==== Syscall writing Every syscall implemented in the Linuxulator must have its prototype with various flags in [.filename]#syscalls.master#. The form of the file is: [.programlisting] .... ... AUE_FORK STD { int linux_fork(void); } ... AUE_CLOSE NOPROTO { int close(int fd); } ... .... The first column represents the syscall number. The second column is for auditing support. The third column represents the syscall type. It is either `STD`, `OBSOL`, `NOPROTO` and `UNIMPL`. `STD` is a standard syscall with full prototype and implementation. `OBSOL` is obsolete and defines just the prototype. `NOPROTO` means that the syscall is implemented elsewhere so do not prepend ABI prefix, etc. `UNIMPL` means that the syscall will be substituted with the `nosys` syscall (a syscall just printing out a message about the syscall not being implemented and returning `ENOSYS`). From [.filename]#syscalls.master# a script generates three files: [.filename]#linux_syscall.h#, [.filename]#linux_proto.h# and [.filename]#linux_sysent.c#. The [.filename]#linux_syscall.h# contains definitions of syscall names and their numerical value, e.g.: [.programlisting] .... ... #define LINUX_SYS_linux_fork 2 ... #define LINUX_SYS_close 6 ... .... The [.filename]#linux_proto.h# contains structure definitions of arguments to every syscall, e.g.: [.programlisting] .... struct linux_fork_args { register_t dummy; }; .... And finally, [.filename]#linux_sysent.c# contains structure describing the system entry table, used to actually dispatch a syscall, e.g.: [.programlisting] .... { 0, (sy_call_t *)linux_fork, AUE_FORK, NULL, 0, 0 }, /* 2 = linux_fork */ { AS(close_args), (sy_call_t *)close, AUE_CLOSE, NULL, 0, 0 }, /* 6 = close */ .... As you can see `linux_fork` is implemented in Linuxulator itself so the definition is of `STD` type and has no argument, which is exhibited by the dummy argument structure. On the other hand `close` is just an alias for real FreeBSD man:close[2] so it has no linux arguments structure associated and in the system entry table it is not prefixed with linux as it calls the real man:close[2] in the kernel. [[dummy-syscalls]] ==== Dummy syscalls The Linux(R) emulation layer is not complete, as some syscalls are not implemented properly and some are not implemented at all. The emulation layer employs a facility to mark unimplemented syscalls with the `DUMMY` macro. These dummy definitions reside in [.filename]#linux_dummy.c# in a form of `DUMMY(syscall);`, which is then translated to various syscall auxiliary files and the implementation consists of printing a message saying that this syscall is not implemented. The `UNIMPL` prototype is not used because we want to be able to identify the name of the syscall that was called in order to know what syscalls are more important to implement. [[signal-handling]] === Signal handling Signal handling is done generally in the FreeBSD kernel for all binary compatibilities with a call to a compat-dependent layer. Linux(R) compatibility layer defines `linux_sendsig` routine for this purpose. [[linux-sendsig]] ==== Linux(R) sendsig This routine first checks whether the signal has been installed with a `SA_SIGINFO` in which case it calls `linux_rt_sendsig` routine instead. Furthermore, it allocates (or reuses an already existing) signal handle context, then it builds a list of arguments for the signal handler. It translates the signal number based on the signal translation table, assigns a handler, translates sigset. Then it saves context for the `sigreturn` routine (various registers, translated trap number and signal mask). Finally, it copies out the signal context to the userspace and prepares context for the actual signal handler to run. [[linux-rt-sendsig]] ==== linux_rt_sendsig This routine is similar to `linux_sendsig` just the signal context preparation is different. It adds `siginfo`, `ucontext`, and some POSIX(R) parts. It might be worth considering whether those two functions could not be merged with a benefit of less code duplication and possibly even faster execution. [[linux-sigreturn]] ==== linux_sigreturn This syscall is used for return from the signal handler. It does some security checks and restores the original process context. It also unmasks the signal in process signal mask. [[ptrace]] === Ptrace Many UNIX(R) derivates implement the man:ptrace[2] syscall in order to allow various tracking and debugging features. This facility enables the tracing process to obtain various information about the traced process, like register dumps, any memory from the process address space, etc. and also to trace the process like in stepping an instruction or between system entries (syscalls and traps). man:ptrace[2] also lets you set various information in the traced process (registers etc.). man:ptrace[2] is a UNIX(R)-wide standard implemented in most UNIX(R)es around the world. Linux(R) emulation in FreeBSD implements the man:ptrace[2] facility in [.filename]#linux_ptrace.c#. The routines for converting registers between Linux(R) and FreeBSD and the actual man:ptrace[2] syscall emulation syscall. The syscall is a long switch block that implements its counterpart in FreeBSD for every man:ptrace[2] command. The man:ptrace[2] commands are mostly equal between Linux(R) and FreeBSD so usually just a small modification is needed. For example, `PT_GETREGS` in Linux(R) operates on direct data while FreeBSD uses a pointer to the data so after performing a (native) man:ptrace[2] syscall, a copyout must be done to preserve Linux(R) semantics. The man:ptrace[2] implementation in Linuxulator has some known weaknesses. There have been panics seen when using `strace` (which is a man:ptrace[2] consumer) in the Linuxulator environment. Also `PT_SYSCALL` is not implemented. [[traps]] === Traps Whenever a Linux(R) process running in the emulation layer traps the trap itself is handled transparently with the only exception of the trap translation. Linux(R) and FreeBSD differs in opinion on what a trap is so this is dealt with here. The code is actually very short: [.programlisting] .... static int translate_traps(int signal, int trap_code) { if (signal != SIGBUS) return signal; switch (trap_code) { case T_PROTFLT: case T_TSSFLT: case T_DOUBLEFLT: case T_PAGEFLT: return SIGSEGV; default: return signal; } } .... [[stack-fixup]] === Stack fixup The RTLD run-time link-editor expects so called AUX tags on stack during an `execve` so a fixup must be done to ensure this. Of course, every RTLD system is different so the emulation layer must provide its own stack fixup routine to do this. So does Linuxulator. The `elf_linux_fixup` simply copies out AUX tags to the stack and adjusts the stack of the user space process to point right after those tags. So RTLD works in a smart way. [[aout-support]] === A.OUT support The Linux(R) emulation layer on i386 also supports Linux(R) A.OUT binaries. Pretty much everything described in the previous sections must be implemented for A.OUT support (beside traps translation and signals sending). The support for A.OUT binaries is no longer maintained, especially the 2.6 emulation does not work with it but this does not cause any problem, as the linux-base in ports probably do not support A.OUT binaries at all. This support will probably be removed in future. Most of the stuff necessary for loading Linux(R) A.OUT binaries is in [.filename]#imgact_linux.c# file. [[mi]] == Linux(R) emulation layer -MI part This section talks about machine independent part of the Linuxulator. It covers the emulation infrastructure needed for Linux(R) 2.6 emulation, the thread local storage (TLS) implementation (on i386) and futexes. Then we talk briefly about some syscalls. [[nptl-desc]] === Description of NPTL One of the major areas of progress in development of Linux(R) 2.6 was threading. Prior to 2.6, the Linux(R) threading support was implemented in the linuxthreads library. The library was a partial implementation of POSIX(R) threading. The threading was implemented using separate processes for each thread using the `clone` syscall to let them share the address space (and other things). The main weaknesses of this approach was that every thread had a different PID, signal handling was broken (from the pthreads perspective), etc. Also the performance was not very good (use of `SIGUSR` signals for threads synchronization, kernel resource consumption, etc.) so to overcome these problems a new threading system was developed and named NPTL. The NPTL library focused on two things but a third thing came along so it is usually considered a part of NPTL. Those two things were embedding of threads into a process structure and futexes. The additional third thing was TLS, which is not directly required by NPTL but the whole NPTL userland library depends on it. Those improvements yielded in much improved performance and standards conformance. NPTL is a standard threading library in Linux(R) systems these days. The FreeBSD Linuxulator implementation approaches the NPTL in three main areas. The TLS, futexes and PID mangling, which is meant to simulate the Linux(R) threads. Further sections describe each of these areas. [[linux26-emu]] === Linux(R) 2.6 emulation infrastructure These sections deal with the way Linux(R) threads are managed and how we simulate that in FreeBSD. [[linux26-runtime]] ==== Runtime determining of 2.6 emulation The Linux(R) emulation layer in FreeBSD supports runtime setting of the emulated version. This is done via man:sysctl[8], namely `compat.linux.osrelease`. Setting this man:sysctl[8] affects runtime behavior of the emulation layer. When set to 2.6.x it sets the value of `linux_use_linux26` while setting to something else keeps it unset. This variable (plus per-prison variables of the very same kind) determines whether 2.6 infrastructure (mainly PID mangling) is used in the code or not. The version setting is done system-wide and this affects all Linux(R) processes. The man:sysctl[8] should not be changed when running any Linux(R) binary as it might harm things. [[linux-proc-thread]] ==== Linux(R) processes and thread identifiers The semantics of Linux(R) threading are a little confusing and uses entirely different nomenclature to FreeBSD. A process in Linux(R) consists of a `struct task` embedding two identifier fields - PID and TGID. PID is _not_ a process ID but it is a thread ID. The TGID identifies a thread group in other words a process. For single-threaded process the PID equals the TGID. The thread in NPTL is just an ordinary process that happens to have TGID not equal to PID and have a group leader not equal to itself (and shared VM etc. of course). Everything else happens in the same way as to an ordinary process. There is no separation of a shared status to some external structure like in FreeBSD. This creates some duplication of information and possible data inconsistency. The Linux(R) kernel seems to use task -> group information in some places and task information elsewhere and it is really not very consistent and looks error-prone. Every NPTL thread is created by a call to the `clone` syscall with a specific set of flags (more in the next subsection). The NPTL implements strict 1:1 threading. In FreeBSD we emulate NPTL threads with ordinary FreeBSD processes that share VM space, etc. and the PID gymnastic is just mimicked in the emulation specific structure attached to the process. The structure attached to the process looks like: [.programlisting] .... struct linux_emuldata { pid_t pid; int *child_set_tid; /* in clone(): Child.s TID to set on clone */ int *child_clear_tid;/* in clone(): Child.s TID to clear on exit */ struct linux_emuldata_shared *shared; int pdeath_signal; /* parent death signal */ LIST_ENTRY(linux_emuldata) threads; /* list of linux threads */ }; .... The PID is used to identify the FreeBSD process that attaches this structure. The `child_se_tid` and `child_clear_tid` are used for TID address copyout when a process exits and is created. The `shared` pointer points to a structure shared among threads. The `pdeath_signal` variable identifies the parent death signal and the `threads` pointer is used to link this structure to the list of threads. The `linux_emuldata_shared` structure looks like: [.programlisting] .... struct linux_emuldata_shared { int refs; pid_t group_pid; LIST_HEAD(, linux_emuldata) threads; /* head of list of linux threads */ }; .... The `refs` is a reference counter being used to determine when we can free the structure to avoid memory leaks. The `group_pid` is to identify PID ( = TGID) of the whole process ( = thread group). The `threads` pointer is the head of the list of threads in the process. The `linux_emuldata` structure can be obtained from the process using `em_find`. The prototype of the function is: [.programlisting] .... struct linux_emuldata *em_find(struct proc *, int locked); .... Here, `proc` is the process we want the emuldata structure from and the locked parameter determines whether we want to lock or not. The accepted values are `EMUL_DOLOCK` and `EMUL_DOUNLOCK`. More about locking later. [[pid-mangling]] ==== PID mangling As there is a difference in view as what to the idea of a process ID and thread ID is between FreeBSD and Linux(R) we have to translate the view somehow. We do it by PID mangling. This means that we fake what a PID (=TGID) and TID (=PID) is between kernel and userland. The rule of thumb is that in kernel (in Linuxulator) PID = PID and TGID = shared -> group pid and to userland we present `PID = shared -> group_pid` and `TID = proc -> p_pid`. The PID member of `linux_emuldata structure` is a FreeBSD PID. The above affects mainly getpid, getppid, gettid syscalls. Where we use PID/TGID respectively. In copyout of TIDs in `child_clear_tid` and `child_set_tid` we copy out FreeBSD PID. [[clone-syscall]] ==== Clone syscall The `clone` syscall is the way threads are created in Linux(R). The syscall prototype looks like this: [.programlisting] .... int linux_clone(l_int flags, void *stack, void *parent_tidptr, int dummy, void * child_tidptr); .... The `flags` parameter tells the syscall how exactly the processes should be cloned. As described above, Linux(R) can create processes sharing various things independently, for example two processes can share file descriptors but not VM, etc. Last byte of the `flags` parameter is the exit signal of the newly created process. The `stack` parameter if non-`NULL` tells, where the thread stack is and if it is `NULL` we are supposed to copy-on-write the calling process stack (i.e. do what normal man:fork[2] routine does). The `parent_tidptr` parameter is used as an address for copying out process PID (i.e. thread id) once the process is sufficiently instantiated but is not runnable yet. The `dummy` parameter is here because of the very strange calling convention of this syscall on i386. It uses the registers directly and does not let the compiler do it what results in the need of a dummy syscall. The `child_tidptr` parameter is used as an address for copying out PID once the process has finished forking and when the process exits. The syscall itself proceeds by setting corresponding flags depending on the flags passed in. For example, `CLONE_VM` maps to RFMEM (sharing of VM), etc. The only nit here is `CLONE_FS` and `CLONE_FILES` because FreeBSD does not allow setting this separately so we fake it by not setting RFFDG (copying of fd table and other fs information) if either of these is defined. This does not cause any problems, because those flags are always set together. After setting the flags the process is forked using the internal `fork1` routine, the process is instrumented not to be put on a run queue, i.e. not to be set runnable. After the forking is done we possibly reparent the newly created process to emulate `CLONE_PARENT` semantics. Next part is creating the emulation data. Threads in Linux(R) does not signal their parents so we set exit signal to be 0 to disable this. After that setting of `child_set_tid` and `child_clear_tid` is performed enabling the functionality later in the code. At this point we copy out the PID to the address specified by `parent_tidptr`. The setting of process stack is done by simply rewriting thread frame `%esp` register (`%rsp` on amd64). Next part is setting up TLS for the newly created process. After this man:vfork[2] semantics might be emulated and finally the newly created process is put on a run queue and copying out its PID to the parent process via `clone` return value is done. The `clone` syscall is able and in fact is used for emulating classic man:fork[2] and man:vfork[2] syscalls. Newer glibc in a case of 2.6 kernel uses `clone` to implement man:fork[2] and man:vfork[2] syscalls. [[locking]] ==== Locking The locking is implemented to be per-subsystem because we do not expect a lot of contention on these. There are two locks: `emul_lock` used to protect manipulating of `linux_emuldata` and `emul_shared_lock` used to manipulate `linux_emuldata_shared`. The `emul_lock` is a nonsleepable blocking mutex while `emul_shared_lock` is a sleepable blocking `sx_lock`. Due to of the per-subsystem locking we can coalesce some locks and that is why the em find offers the non-locking access. [[tls]] === TLS This section deals with TLS also known as thread local storage. [[trheading-intro]] ==== Introduction to threading Threads in computer science are entities within a process that can be scheduled independently from each other. The threads in the process share process wide data (file descriptors, etc.) but also have their own stack for their own data. Sometimes there is a need for process-wide data specific to a given thread. Imagine a name of the thread in execution or something like that. The traditional UNIX(R) threading API, pthreads provides a way to do it via man:pthread_key_create[3], man:pthread_setspecific[3] and man:pthread_getspecific[3] where a thread can create a key to the thread local data and using man:pthread_getspecific[3] or man:pthread_getspecific[3] to manipulate those data. You can easily see that this is not the most comfortable way this could be accomplished. So various producers of C/C++ compilers introduced a better way. They defined a new modifier keyword thread that specifies that a variable is thread specific. A new method of accessing such variables was developed as well (at least on i386). The pthreads method tends to be implemented in userspace as a trivial lookup table. The performance of such a solution is not very good. So the new method uses (on i386) segment registers to address a segment, where TLS area is stored so the actual accessing of a thread variable is just appending the segment register to the address thus addressing via it. The segment registers are usually `%gs` and `%fs` acting like segment selectors. Every thread has its own area where the thread local data are stored and the segment must be loaded on every context switch. This method is very fast and used almost exclusively in the whole i386 UNIX(R) world. Both FreeBSD and Linux(R) implement this approach and it yields very good results. The only drawback is the need to reload the segment on every context switch which can slowdown context switches. FreeBSD tries to avoid this overhead by using only 1 segment descriptor for this while Linux(R) uses 3. Interesting thing is that almost nothing uses more than 1 descriptor (only Wine seems to use 2) so Linux(R) pays this unnecessary price for context switches. [[i386-segs]] ==== Segments on i386 The i386 architecture implements the so called segments. A segment is a description of an area of memory. The base address (bottom) of the memory area, the end of it (ceiling), type, protection, etc. The memory described by a segment can be accessed using segment selector registers (`%cs`, `%ds`, `%ss`, `%es`, `%fs`, `%gs`). For example let us suppose we have a segment which base address is 0x1234 and length and this code: [.programlisting] .... mov %edx,%gs:0x10 .... This will load the content of the `%edx` register into memory location 0x1244. Some segment registers have a special use, for example `%cs` is used for code segment and `%ss` is used for stack segment but `%fs` and `%gs` are generally unused. Segments are either stored in a global GDT table or in a local LDT table. LDT is accessed via an entry in the GDT. The LDT can store more types of segments. LDT can be per process. Both tables define up to 8191 entries. [[linux-i386]] ==== Implementation on Linux(R) i386 There are two main ways of setting up TLS in Linux(R). It can be set when cloning a process using the `clone` syscall or it can call `set_thread_area`. When a process passes `CLONE_SETTLS` flag to `clone`, the kernel expects the memory pointed to by the `%esi` register a Linux(R) user space representation of a segment, which gets translated to the machine representation of a segment and loaded into a GDT slot. The GDT slot can be specified with a number or -1 can be used meaning that the system itself should choose the first free slot. In practice, the vast majority of programs use only one TLS entry and does not care about the number of the entry. We exploit this in the emulation and in fact depend on it. [[tls-emu]] ==== Emulation of Linux(R) TLS [[tls-i386]] ===== i386 Loading of TLS for the current thread happens by calling `set_thread_area` while loading TLS for a second process in `clone` is done in the separate block in `clone`. Those two functions are very similar. The only difference being the actual loading of the GDT segment, which happens on the next context switch for the newly created process while `set_thread_area` must load this directly. The code basically does this. It copies the Linux(R) form segment descriptor from the userland. The code checks for the number of the descriptor but because this differs between FreeBSD and Linux(R) we fake it a little. We only support indexes of 6, 3 and -1. The 6 is genuine Linux(R) number, 3 is genuine FreeBSD one and -1 means autoselection. Then we set the descriptor number to constant 3 and copy out this to the userspace. We rely on the userspace process using the number from the descriptor but this works most of the time (have never seen a case where this did not work) as the userspace process typically passes in 1. Then we convert the descriptor from the Linux(R) form to a machine dependant form (i.e. operating system independent form) and copy this to the FreeBSD defined segment descriptor. Finally we can load it. We assign the descriptor to threads PCB (process control block) and load the `%gs` segment using `load_gs`. This loading must be done in a critical section so that nothing can interrupt us. The `CLONE_SETTLS` case works exactly like this just the loading using `load_gs` is not performed. The segment used for this (segment number 3) is shared for this use between FreeBSD processes and Linux(R) processes so the Linux(R) emulation layer does not add any overhead over plain FreeBSD. [[tls-amd64]] ===== amd64 The amd64 implementation is similar to the i386 one but there was initially no 32bit segment descriptor used for this purpose (hence not even native 32bit TLS users worked) so we had to add such a segment and implement its loading on every context switch (when a flag signaling use of 32bit is set). Apart from this the TLS loading is exactly the same just the segment numbers are different and the descriptor format and the loading differs slightly. [[futexes]] === Futexes [[sync-intro]] ==== Introduction to synchronization Threads need some kind of synchronization and POSIX(R) provides some of them: mutexes for mutual exclusion, read-write locks for mutual exclusion with biased ratio of reads and writes and condition variables for signaling a status change. It is interesting to note that POSIX(R) threading API lacks support for semaphores. Those synchronization routines implementations are heavily dependant on the type threading support we have. In pure 1:M (userspace) model the implementation can be solely done in userspace and thus be very fast (the condition variables will probably end up being implemented using signals, i.e. not fast) and simple. In 1:1 model, the situation is also quite clear - the threads must be synchronized using kernel facilities (which is very slow because a syscall must be performed). The mixed M:N scenario just combines the first and second approach or rely solely on kernel. Threads synchronization is a vital part of thread-enabled programming and its performance can affect resulting program a lot. Recent benchmarks on FreeBSD operating system showed that an improved sx_lock implementation yielded 40% speedup in _ZFS_ (a heavy sx user), this is in-kernel stuff but it shows clearly how important the performance of synchronization primitives is. Threaded programs should be written with as little contention on locks as possible. Otherwise, instead of doing useful work the thread just waits on a lock. As a result of this, the most well written threaded programs show little locks contention. [[futex-intro]] ==== Futexes introduction Linux(R) implements 1:1 threading, i.e. it has to use in-kernel synchronization primitives. As stated earlier, well written threaded programs have little lock contention. So a typical sequence could be performed as two atomic increase/decrease mutex reference counter, which is very fast, as presented by the following example: [.programlisting] .... pthread_mutex_lock(&mutex); ... pthread_mutex_unlock(&mutex); .... 1:1 threading forces us to perform two syscalls for those mutex calls, which is very slow. The solution Linux(R) 2.6 implements is called futexes. Futexes implement the check for contention in userspace and call kernel primitives only in a case of contention. Thus the typical case takes place without any kernel intervention. This yields reasonably fast and flexible synchronization primitives implementation. [[futex-api]] ==== Futex API The futex syscall looks like this: [.programlisting] .... int futex(void *uaddr, int op, int val, struct timespec *timeout, void *uaddr2, int val3); .... In this example `uaddr` is an address of the mutex in userspace, `op` is an operation we are about to perform and the other parameters have per-operation meaning. Futexes implement the following operations: * `FUTEX_WAIT` * `FUTEX_WAKE` * `FUTEX_FD` * `FUTEX_REQUEUE` * `FUTEX_CMP_REQUEUE` * `FUTEX_WAKE_OP` [[futex-wait]] ===== FUTEX_WAIT This operation verifies that on address `uaddr` the value `val` is written. If not, `EWOULDBLOCK` is returned, otherwise the thread is queued on the futex and gets suspended. If the argument `timeout` is non-zero it specifies the maximum time for the sleeping, otherwise the sleeping is infinite. [[futex-wake]] ===== FUTEX_WAKE This operation takes a futex at `uaddr` and wakes up `val` first futexes queued on this futex. [[futex-fd]] ===== FUTEX_FD This operations associates a file descriptor with a given futex. [[futex-requeue]] ===== FUTEX_REQUEUE This operation takes `val` threads queued on futex at `uaddr`, wakes them up, and takes `val2` next threads and requeues them on futex at `uaddr2`. [[futex-cmp-requeue]] ===== FUTEX_CMP_REQUEUE This operation does the same as `FUTEX_REQUEUE` but it checks that `val3` equals to `val` first. [[futex-wake-op]] ===== FUTEX_WAKE_OP This operation performs an atomic operation on `val3` (which contains coded some other value) and `uaddr`. Then it wakes up `val` threads on futex at `uaddr` and if the atomic operation returned a positive number it wakes up `val2` threads on futex at `uaddr2`. The operations implemented in `FUTEX_WAKE_OP`: * `FUTEX_OP_SET` * `FUTEX_OP_ADD` * `FUTEX_OP_OR` * `FUTEX_OP_AND` * `FUTEX_OP_XOR` [NOTE] ==== There is no `val2` parameter in the futex prototype. The `val2` is taken from the `struct timespec *timeout` parameter for operations `FUTEX_REQUEUE`, `FUTEX_CMP_REQUEUE` and `FUTEX_WAKE_OP`. ==== [[futex-emu]] ==== Futex emulation in FreeBSD The futex emulation in FreeBSD is taken from NetBSD and further extended by us. It is placed in `linux_futex.c` and [.filename]#linux_futex.h# files. The `futex` structure looks like: [.programlisting] .... struct futex { void *f_uaddr; int f_refcount; LIST_ENTRY(futex) f_list; TAILQ_HEAD(lf_waiting_paroc, waiting_proc) f_waiting_proc; }; .... And the structure `waiting_proc` is: [.programlisting] .... struct waiting_proc { struct thread *wp_t; struct futex *wp_new_futex; TAILQ_ENTRY(waiting_proc) wp_list; }; .... [[futex-get]] ===== futex_get / futex_put A futex is obtained using the `futex_get` function, which searches a linear list of futexes and returns the found one or creates a new futex. When releasing a futex from the use we call the `futex_put` function, which decreases a reference counter of the futex and if the refcount reaches zero it is released. [[futex-sleep]] ===== futex_sleep When a futex queues a thread for sleeping it creates a `working_proc` structure and puts this structure to the list inside the futex structure then it just performs a man:tsleep[9] to suspend the thread. The sleep can be timed out. After man:tsleep[9] returns (the thread was woken up or it timed out) the `working_proc` structure is removed from the list and is destroyed. All this is done in the `futex_sleep` function. If we got woken up from `futex_wake` we have `wp_new_futex` set so we sleep on it. This way the actual requeueing is done in this function. [[futex-wake-2]] ===== futex_wake Waking up a thread sleeping on a futex is performed in the `futex_wake` function. First in this function we mimic the strange Linux(R) behavior, where it wakes up N threads for all operations, the only exception is that the REQUEUE operations are performed on N+1 threads. But this usually does not make any difference as we are waking up all threads. Next in the function in the loop we wake up n threads, after this we check if there is a new futex for requeueing. If so, we requeue up to n2 threads on the new futex. This cooperates with `futex_sleep`. [[futex-wake-op-2]] ===== futex_wake_op The `FUTEX_WAKE_OP` operation is quite complicated. First we obtain two futexes at addresses `uaddr` and `uaddr2` then we perform the atomic operation using `val3` and `uaddr2`. Then `val` waiters on the first futex is woken up and if the atomic operation condition holds we wake up `val2` (i.e. `timeout`) waiter on the second futex. [[futex-atomic-op]] ===== futex atomic operation The atomic operation takes two parameters `encoded_op` and `uaddr`. The encoded operation encodes the operation itself, comparing value, operation argument, and comparing argument. The pseudocode for the operation is like this one: [.programlisting] .... oldval = *uaddr2 *uaddr2 = oldval OP oparg .... And this is done atomically. First a copying in of the number at `uaddr` is performed and the operation is done. The code handles page faults and if no page fault occurs `oldval` is compared to `cmparg` argument with cmp comparator. [[futex-locking]] ===== Futex locking Futex implementation uses two lock lists protecting `sx_lock` and global locks (either Giant or another `sx_lock`). Every operation is performed locked from the start to the very end. [[syscall-impl]] === Various syscalls implementation In this section I am going to describe some smaller syscalls that are worth mentioning because their implementation is not obvious or those syscalls are interesting from other point of view. [[syscall-at]] ==== *at family of syscalls During development of Linux(R) 2.6.16 kernel, the *at syscalls were added. Those syscalls (`openat` for example) work exactly like their at-less counterparts with the slight exception of the `dirfd` parameter. This parameter changes where the given file, on which the syscall is to be performed, is. When the `filename` parameter is absolute `dirfd` is ignored but when the path to the file is relative, it comes to the play. The `dirfd` parameter is a directory relative to which the relative pathname is checked. The `dirfd` parameter is a file descriptor of some directory or `AT_FDCWD`. So for example the `openat` syscall can be like this: [.programlisting] .... file descriptor 123 = /tmp/foo/, current working directory = /tmp/ openat(123, /tmp/bah\, flags, mode) /* opens /tmp/bah */ openat(123, bah\, flags, mode) /* opens /tmp/foo/bah */ openat(AT_FDWCWD, bah\, flags, mode) /* opens /tmp/bah */ openat(stdio, bah\, flags, mode) /* returns error because stdio is not a directory */ .... This infrastructure is necessary to avoid races when opening files outside the working directory. Imagine that a process consists of two threads, thread A and thread B. Thread A issues `open(./tmp/foo/bah., flags, mode)` and before returning it gets preempted and thread B runs. Thread B does not care about the needs of thread A and renames or removes [.filename]#/tmp/foo/#. We got a race. To avoid this we can open [.filename]#/tmp/foo# and use it as `dirfd` for `openat` syscall. This also enables user to implement per-thread working directories. Linux(R) family of *at syscalls contains: `linux_openat`, `linux_mkdirat`, `linux_mknodat`, `linux_fchownat`, `linux_futimesat`, `linux_fstatat64`, `linux_unlinkat`, `linux_renameat`, `linux_linkat`, `linux_symlinkat`, `linux_readlinkat`, `linux_fchmodat` and `linux_faccessat`. All these are implemented using the modified man:namei[9] routine and simple wrapping layer. [[implementation]] ===== Implementation The implementation is done by altering the man:namei[9] routine (described above) to take additional parameter `dirfd` in its `nameidata` structure, which specifies the starting point of the pathname lookup instead of using the current working directory every time. The resolution of `dirfd` from file descriptor number to a vnode is done in native *at syscalls. When `dirfd` is `AT_FDCWD` the `dvp` entry in `nameidata` structure is `NULL` but when `dirfd` is a different number we obtain a file for this file descriptor, check whether this file is valid and if there is vnode attached to it then we get a vnode. Then we check this vnode for being a directory. In the actual man:namei[9] routine we simply substitute the `dvp` vnode for `dp` variable in the man:namei[9] function, which determines the starting point. The man:namei[9] is not used directly but via a trace of different functions on various levels. For example the `openat` goes like this: [.programlisting] .... openat() --> kern_openat() --> vn_open() -> namei() .... For this reason `kern_open` and `vn_open` must be altered to incorporate the additional `dirfd` parameter. No compat layer is created for those because there are not many users of this and the users can be easily converted. This general implementation enables FreeBSD to implement their own *at syscalls. This is being discussed right now. [[ioctl]] ==== Ioctl The ioctl interface is quite fragile due to its generality. We have to bear in mind that devices differ between Linux(R) and FreeBSD so some care must be applied to do ioctl emulation work right. The ioctl handling is implemented in [.filename]#linux_ioctl.c#, where `linux_ioctl` function is defined. This function simply iterates over sets of ioctl handlers to find a handler that implements a given command. The ioctl syscall has three parameters, the file descriptor, command and an argument. The command is a 16-bit number, which in theory is divided into high 8 bits determining class of the ioctl command and low 8 bits, which are the actual command within the given set. The emulation takes advantage of this division. We implement handlers for each set, like `sound_handler` or `disk_handler`. Each handler has a maximum command and a minimum command defined, which is used for determining what handler is used. There are slight problems with this approach because Linux(R) does not use the set division consistently so sometimes ioctls for a different set are inside a set they should not belong to (SCSI generic ioctls inside cdrom set, etc.). FreeBSD currently does not implement many Linux(R) ioctls (compared to NetBSD, for example) but the plan is to port those from NetBSD. The trend is to use Linux(R) ioctls even in the native FreeBSD drivers because of the easy porting of applications. [[debugging]] ==== Debugging Every syscall should be debuggable. For this purpose we introduce a small infrastructure. We have the ldebug facility, which tells whether a given syscall should be debugged (settable via a sysctl). For printing we have LMSG and ARGS macros. Those are used for altering a printable string for uniform debugging messages. [[conclusion]] == Conclusion [[results]] === Results As of April 2007 the Linux(R) emulation layer is capable of emulating the Linux(R) 2.6.16 kernel quite well. The remaining problems concern futexes, unfinished *at family of syscalls, problematic signals delivery, missing `epoll` and `inotify` and probably some bugs we have not discovered yet. Despite this we are capable of running basically all the Linux(R) programs included in FreeBSD Ports Collection with Fedora Core 4 at 2.6.16 and there are some rudimentary reports of success with Fedora Core 6 at 2.6.16. The Fedora Core 6 linux_base was recently committed enabling some further testing of the emulation layer and giving us some more hints where we should put our effort in implementing missing stuff. We are able to run the most used applications like package:www/linux-firefox[], package:net-im/skype[] and some games from the Ports Collection. Some of the programs exhibit bad behavior under 2.6 emulation but this is currently under investigation and hopefully will be fixed soon. The only big application that is known not to work is the Linux(R) Java(TM) Development Kit and this is because of the requirement of `epoll` facility which is not directly related to the Linux(R) kernel 2.6. We hope to enable 2.6.16 emulation by default some time after FreeBSD 7.0 is released at least to expose the 2.6 emulation parts for some wider testing. Once this is done we can switch to Fedora Core 6 linux_base, which is the ultimate plan. [[future-work]] === Future work Future work should focus on fixing the remaining issues with futexes, implement the rest of the *at family of syscalls, fix the signal delivery and possibly implement the `epoll` and `inotify` facilities. We hope to be able to run the most important programs flawlessly soon, so we will be able to switch to the 2.6 emulation by default and make the Fedora Core 6 the default linux_base because our currently used Fedora Core 4 is not supported any more. The other possible goal is to share our code with NetBSD and DragonflyBSD. NetBSD has some support for 2.6 emulation but its far from finished and not really tested. DragonflyBSD has expressed some interest in porting the 2.6 improvements. Generally, as Linux(R) develops we would like to keep up with their development, implementing newly added syscalls. Splice comes to mind first. Some already implemented syscalls are also heavily crippled, for example `mremap` and others. Some performance improvements can also be made, finer grained locking and others. [[team]] === Team I cooperated on this project with (in alphabetical order): * `{jhb}` * `{kib}` * Emmanuel Dreyfus * Scot Hetzel * `{jkim}` * `{netchild}` * `{ssouhlal}` * Li Xiao * `{davidxu}` I would like to thank all those people for their advice, code reviews and general support. [[literatures]] == Literatures . Marshall Kirk McKusick - George V. Nevile-Neil. Design and Implementation of the FreeBSD operating system. Addison-Wesley, 2005. . https://tldp.org[https://tldp.org] . https://www.kernel.org[https://www.kernel.org] diff --git a/documentation/content/en/articles/mailing-list-faq/_index.adoc b/documentation/content/en/articles/mailing-list-faq/_index.adoc index 81bb9f2b07..f5b4eb064d 100644 --- a/documentation/content/en/articles/mailing-list-faq/_index.adoc +++ b/documentation/content/en/articles/mailing-list-faq/_index.adoc @@ -1,191 +1,191 @@ --- title: Frequently Asked Questions About The FreeBSD Mailing Lists authors: - author: The FreeBSD Documentation Project copyright: 2004-2021 The FreeBSD Documentation Project -description: Frequently Asked Questions About The FreeBSD Mailing Lists +description: How to best use the mailing lists, such as how to help avoid frequently-repeated discussions --- = Frequently Asked Questions About The FreeBSD Mailing Lists :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/authors.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This is the FAQ for the FreeBSD mailing lists. If you are interested in helping with this project, send email to the {freebsd-doc}. The latest version of this document is always available from the link:.[FreeBSD World Wide Web server]. It may also be downloaded as one large link:.[HTML] file with HTTP or as plain text, PostScript, PDF, etc. from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server]. You may also want to link:https://www.FreeBSD.org/search/[Search the FAQ]. ''' toc::[] [[introduction]] == Introduction As is usual with FAQs, this document aims to cover the most frequently asked questions concerning the FreeBSD mailing lists (and of course answer them!). Although originally intended to reduce bandwidth and avoid the same old questions being asked over and over again, FAQs have become recognized as valuable information resources. This document attempts to represent a community consensus, and as such it can never really be __authoritative__. However, if you find technical errors within this document, or have suggestions about items that should be added, please either submit a PR, or email the {freebsd-doc}. Thanks. === What is the purpose of the FreeBSD mailing lists? The FreeBSD mailing lists serve as the primary communication channels for the FreeBSD community, covering many different topic areas and communities of interest. === Who is the audience for the FreeBSD mailing lists? This depends on charter of each individual list. Some lists are more oriented to developers; some are more oriented towards the FreeBSD community as a whole. Please see http://lists.FreeBSD.org/mailman/listinfo[this list] for the current summary. === Are the FreeBSD mailing lists open for anyone to participate? Again, this depends on charter of each individual list. Please read the charter of a mailing list before you post to it, and respect it when you post. This will help everyone to have a better experience with the lists. If after reading the above lists, you still do not know which mailing list to post a question to, you will probably want to post to freebsd-questions (but see below, first). Also note that the mailing lists have traditionally been open to postings from non-subscribers. This has been a deliberate choice, to help make joining the FreeBSD community an easier process, and to encourage open sharing of ideas. However, due to past abuse by some individuals, certain lists now have a policy where postings from non-subscribers must be manually screened to ensure that they are appropriate. === How can I subscribe? You can use http://lists.FreeBSD.org/mailman/listinfo[the Mailman web interface] to subscribe to any of the public lists. === How can I unsubscribe? You can use the same interface as above; or, you can follow the instructions that are at the bottom of every mailing list message that is sent. Please do not send unsubscribe messages directly to the public lists themselves. First, this will not accomplish your goal, and second, it will irritate the existing subscribers, and you will probably get flamed. This is a classical mistake when using mailing lists; please try to avoid it. === Are archives available? Yes. Threaded archives are available http://docs.FreeBSD.org/mail/[here]. === Are mailing lists available in a digest format? Yes. See http://lists.FreeBSD.org/mailman/listinfo[the Mailman web interface]. [[etiquette]] == Mailing List Etiquette Participation in the mailing lists, like participation in any community, requires a common basis for communication. Please make only appropriate postings, and follow common rules of etiquette. === What should I do before I post? You have already taken the most important step by reading this document. However, if you are new to FreeBSD, you may first need to familiarize yourself with the software, and all the social history around it, by reading the numerous link:https://www.FreeBSD.org/docs/books/[books and articles] that are available. Items of particular interest include the link:{faq}[FreeBSD Frequently Asked Questions (FAQ)] document, the link:{handbook}[FreeBSD Handbook], and the articles link:{freebsd-questions-article}[How to get best results from the FreeBSD-questions mailing list], link:{explaining-bsd}[Explaining BSD], and link:{new-users}[FreeBSD First Steps]. It is always considered bad form to ask a question that is already answered in the above documents. This is not because the volunteers who work on this project are particularly mean people, but after a certain number of times answering the same questions over and over again, frustration begins to set in. This is particularly true if there is an existing answer to the question that is already available. Always keep in mind that almost all of the work done on FreeBSD is done by volunteers, and that we are only human. === What constitutes an inappropriate posting? * Postings must be in accordance with the charter of the mailing list. * Personal attacks are discouraged. As good net-citizens, we should try to hold ourselves to high standards of behavior. * Spam is not allowed, ever. The mailing lists are actively processed to ban offenders to this rule. === What is considered proper etiquette when posting to the mailing lists? * Please wrap lines at 75 characters, since not everyone uses fancy GUI mail reading programs. * Please respect the fact that bandwidth is not infinite. Not everyone reads email through high-speed connections, so if your posting involves something like the content of [.filename]#config.log# or an extensive stack trace, please consider putting that information up on a website somewhere and just provide a URL to it. Remember, too, that these postings will be archived indefinitely, so huge postings will simply inflate the size of the archives long after their purpose has expired. * Format your message so that it is legible, and PLEASE DO NOT SHOUT!!!!!. Do not underestimate the effect that a poorly formatted mail message has, and not just on the FreeBSD mailing lists. Your mail message is all that people see of you, and if it is poorly formatted, badly spelled, full of errors, and/or has lots of exclamation points, it will give people a poor impression of you. * Please use an appropriate human language for a particular mailing list. Many non-English mailing lists are link:https://www.FreeBSD.org/community/mailinglists/[available]. + For the ones that are not, we do appreciate that many people do not speak English as their first language, and we try to make allowances for that. It is considered particularly poor form to criticize non-native speakers for spelling or grammatical errors. FreeBSD has an excellent track record in this regard; please, help us to uphold that tradition. * Please use a standards-compliant Mail User Agent (MUA). A lot of badly formatted messages come from http://www.lemis.com/grog/email/email.php[bad mailers or badly configured mailers]. The following mailers are known to send out badly formatted messages without you finding out about them: ** exmh ** Microsoft(R) Exchange ** Microsoft(R) Outlook(R) + Try not to use MIME: a lot of people use mailers which do not get on very well with MIME. * Make sure your time and time zone are set correctly. This may seem a little silly, since your message still gets there, but many of the people on these mailing lists get several hundred messages a day. They frequently sort the incoming messages by subject and by date, and if your message does not come before the first answer, they may assume that they missed it and not bother to look. * A lot of the information you need to supply is the output of programs, such as man:dmesg[8], or console messages, which usually appear in [.filename]#/var/log/messages#. Do not try to copy this information by typing it in again; not only it is a real pain, but you are bound to make a mistake. To send log file contents, either make a copy of the file and use an editor to trim the information to what is relevant, or cut and paste into your message. For the output of programs like `dmesg`, redirect the output to a file and include that. For example, + [source,shell] .... % dmesg > /tmp/dmesg.out .... + This redirects the information to the file [.filename]#/tmp/dmesg.out#. * When using cut-and-paste, please be aware that some such operations badly mangle their messages. This is of particular concern when posting contents of [.filename]#Makefiles#, where `tab` is a significant character. This is a very common, and very annoying, problem with submissions to the link:https://www.FreeBSD.org/support/[Problem Reports database]. [.filename]#Makefiles# with tabs changed to either spaces, or the annoying `=3B` escape sequence, create a great deal of aggravation for committers. === What are the special etiquette consideration when replying to an existing posting on the mailing lists? * Please include relevant text from the original message. Trim it to the minimum, but do not overdo it. It should still be possible for somebody who did not read the original message to understand what you are talking about. + This is especially important for postings of the type "yes, I see this too", where the initial posting was dozens or hundreds of lines. * Use some technique to identify which text came from the original message, and which text you add. A common convention is to prepend "`>`" to the original message. Leaving white space after the "`>`" and leaving empty lines between your text and the original text both make the result more readable. * Please ensure that the attributions of the text you are quoting is correct. People can become offended if you attribute words to them that they themselves did not write. * Please do not `top post`. By this, we mean that if you are replying to a message, please put your replies after the text that you copy in your reply. + ** A: Because it reverses the logical flow of conversation. ** Q: Why is top posting frowned upon? + (Thanks to Randy Bush for the joke.) [[recurring]] == Recurring Topics On The Mailing Lists Participation in the mailing lists, like participation in any community, requires a common basis for communication. Many of the mailing lists presuppose a knowledge of the Project's history. In particular, there are certain topics that seem to regularly occur to newcomers to the community. It is the responsibility of each poster to ensure that their postings do not fall into one of these categories. By doing so, you will help the mailing lists to stay on-topic, and probably save yourself being flamed in the process. The best method to avoid this is to familiarize yourself with the http://docs.FreeBSD.org/mail/[mailing list archives], to help yourself understand the background of what has gone before. In this, the https://www.FreeBSD.org/search/#mailinglists[mailing list search interface] is invaluable. (If that method does not yield useful results, please supplement it with a search with your favorite major search engine). By familiarizing yourself with the archives, not only will you learn what topics have been discussed before, but also how discussion tends to proceed on that list, who the participants are, and who the target audience is. These are always good things to know before you post to any mailing list, not just a FreeBSD mailing list. There is no doubt that the archives are quite extensive, and some questions recur more often than others, sometimes as followups where the subject line no longer accurately reflects the new content. Nevertheless, the burden is on you, the poster, to do your homework to help avoid these recurring topics. [[bikeshed]] == What Is A "Bikeshed"? Literally, a `bikeshed` is a small outdoor shelter into which one may store one's two-wheeled form of transportation. However, in FreeBSD parlance, the term refers to topics that are simple enough that (nearly) anyone can offer an opinion about, and often (nearly) everyone does. The genesis of this term is explained in more detail link:{faq}#bikeshed-painting[in this document]. You simply must have a working knowledge of this concept before posting to any FreeBSD mailing list. More generally, a bikeshed is a topic that will tend to generate immediate meta-discussions and flames if you have not read up on their past history. Please help us to keep the mailing lists as useful for as many people as possible by avoiding bikesheds whenever you can. Thanks. [[acknowledgments]] == Acknowledgments `{grog}`:: Original author of most of the material on mailing list etiquette, taken from the article on link:{freebsd-questions-article}[How to get best results from the FreeBSD-questions mailing list]. `{linimon}`:: Creation of the rough draft of this FAQ. diff --git a/documentation/content/en/articles/pam/_index.adoc b/documentation/content/en/articles/pam/_index.adoc index 6f3d5a22d6..03672f1ba2 100644 --- a/documentation/content/en/articles/pam/_index.adoc +++ b/documentation/content/en/articles/pam/_index.adoc @@ -1,641 +1,641 @@ --- title: Pluggable Authentication Modules authors: - author: Dag-Erling Smørgrav copyright: 2001-2003 Networks Associates Technology, Inc. -description: Pluggable Authentication Modules (PAM) in FreeBSD +description: A guide to the PAM system and modules under FreeBSD trademarks: ["pam", "freebsd", "linux", "opengroup", "sun", "general"] --- = Pluggable Authentication Modules :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: [.abstract-title] Abstract This article describes the underlying principles and mechanisms of the Pluggable Authentication Modules (PAM) library, and explains how to configure PAM, how to integrate PAM into applications, and how to write PAM modules. ''' toc::[] [[pam-intro]] == Introduction The Pluggable Authentication Modules (PAM) library is a generalized API for authentication-related services which allows a system administrator to add new authentication methods simply by installing new PAM modules, and to modify authentication policies by editing configuration files. PAM was defined and developed in 1995 by Vipin Samar and Charlie Lai of Sun Microsystems, and has not changed much since. In 1997, the Open Group published the X/Open Single Sign-on (XSSO) preliminary specification, which standardized the PAM API and added extensions for single (or rather integrated) sign-on. At the time of this writing, this specification has not yet been adopted as a standard. Although this article focuses primarily on FreeBSD 5.x, which uses OpenPAM, it should be equally applicable to FreeBSD 4.x, which uses Linux-PAM, and other operating systems such as Linux and Solaris(TM). [[pam-terms]] == Terms and Conventions [[pam-definitions]] === Definitions The terminology surrounding PAM is rather confused. Neither Samar and Lai's original paper nor the XSSO specification made any attempt at formally defining terms for the various actors and entities involved in PAM, and the terms that they do use (but do not define) are sometimes misleading and ambiguous. The first attempt at establishing a consistent and unambiguous terminology was a whitepaper written by Andrew G. Morgan (author of Linux-PAM) in 1999. While Morgan's choice of terminology was a huge leap forward, it is in this author's opinion by no means perfect. What follows is an attempt, heavily inspired by Morgan, to define precise and unambiguous terms for all actors and entities involved in PAM. account:: The set of credentials the applicant is requesting from the arbitrator. applicant:: The user or entity requesting authentication. arbitrator:: The user or entity who has the privileges necessary to verify the applicant's credentials and the authority to grant or deny the request. chain:: A sequence of modules that will be invoked in response to a PAM request. The chain includes information about the order in which to invoke the modules, what arguments to pass to them, and how to interpret the results. client:: The application responsible for initiating an authentication request on behalf of the applicant and for obtaining the necessary authentication information from him. facility:: One of the four basic groups of functionality provided by PAM: authentication, account management, session management and authentication token update. module:: A collection of one or more related functions implementing a particular authentication facility, gathered into a single (normally dynamically loadable) binary file and identified by a single name. policy:: The complete set of configuration statements describing how to handle PAM requests for a particular service. A policy normally consists of four chains, one for each facility, though some services do not use all four facilities. server:: The application acting on behalf of the arbitrator to converse with the client, retrieve authentication information, verify the applicant's credentials and grant or deny requests. service:: A class of servers providing similar or related functionality and requiring similar authentication. PAM policies are defined on a per-service basis, so all servers that claim the same service name will be subject to the same policy. session:: The context within which service is rendered to the applicant by the server. One of PAM's four facilities, session management, is concerned exclusively with setting up and tearing down this context. token:: A chunk of information associated with the account, such as a password or passphrase, which the applicant must provide to prove his identity. transaction:: A sequence of requests from the same applicant to the same instance of the same server, beginning with authentication and session set-up and ending with session tear-down. [[pam-usage-examples]] === Usage Examples This section aims to illustrate the meanings of some of the terms defined above by way of a handful of simple examples. ==== Client and Server Are One This simple example shows `alice` man:su[1]'ing to `root`. [source,shell] .... % whoami alice % ls -l `which su` -r-sr-xr-x 1 root wheel 10744 Dec 6 19:06 /usr/bin/su % su - Password: xi3kiune # whoami root .... * The applicant is `alice`. * The account is `root`. * The man:su[1] process is both client and server. * The authentication token is `xi3kiune`. * The arbitrator is `root`, which is why man:su[1] is setuid `root`. ==== Client and Server Are Separate The example below shows `eve` try to initiate an man:ssh[1] connection to `login.example.com`, ask to log in as `bob`, and succeed. Bob should have chosen a better password! [source,shell] .... % whoami eve % ssh bob@login.example.com bob@login.example.com's password: % god Last login: Thu Oct 11 09:52:57 2001 from 192.168.0.1 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 4.4-STABLE (LOGIN) 4: Tue Nov 27 18:10:34 PST 2001 Welcome to FreeBSD! % .... * The applicant is `eve`. * The client is Eve's man:ssh[1] process. * The server is the man:sshd[8] process on `login.example.com` * The account is `bob`. * The authentication token is `god`. * Although this is not shown in this example, the arbitrator is `root`. ==== Sample Policy The following is FreeBSD's default policy for `sshd`: [.programlisting] .... sshd auth required pam_nologin.so no_warn sshd auth required pam_unix.so no_warn try_first_pass sshd account required pam_login_access.so sshd account required pam_unix.so sshd session required pam_lastlog.so no_fail sshd password required pam_permit.so .... * This policy applies to the `sshd` service (which is not necessarily restricted to the man:sshd[8] server.) * `auth`, `account`, `session` and `password` are facilities. * [.filename]#pam_nologin.so#, [.filename]#pam_unix.so#, [.filename]#pam_login_access.so#, [.filename]#pam_lastlog.so# and [.filename]#pam_permit.so# are modules. It is clear from this example that [.filename]#pam_unix.so# provides at least two facilities (authentication and account management.) [[pam-essentials]] == PAM Essentials [[pam-facilities-primitives]] === Facilities and Primitives The PAM API offers six different authentication primitives grouped in four facilities, which are described below. `auth`:: _Authentication._ This facility concerns itself with authenticating the applicant and establishing the account credentials. It provides two primitives: ** man:pam_authenticate[3] authenticates the applicant, usually by requesting an authentication token and comparing it with a value stored in a database or obtained from an authentication server. ** man:pam_setcred[3] establishes account credentials such as user ID, group membership and resource limits. `account`:: _Account management._ This facility handles non-authentication-related issues of account availability, such as access restrictions based on the time of day or the server's work load. It provides a single primitive: ** man:pam_acct_mgmt[3] verifies that the requested account is available. `session`:: _Session management._ This facility handles tasks associated with session set-up and tear-down, such as login accounting. It provides two primitives: ** man:pam_open_session[3] performs tasks associated with session set-up: add an entry in the [.filename]#utmp# and [.filename]#wtmp# databases, start an SSH agent, etc. ** man:pam_close_session[3] performs tasks associated with session tear-down: add an entry in the [.filename]#utmp# and [.filename]#wtmp# databases, stop the SSH agent, etc. `password`:: _Password management._ This facility is used to change the authentication token associated with an account, either because it has expired or because the user wishes to change it. It provides a single primitive: ** man:pam_chauthtok[3] changes the authentication token, optionally verifying that it is sufficiently hard to guess, has not been used previously, etc. [[pam-modules]] === Modules Modules are a very central concept in PAM; after all, they are the "M" in "PAM". A PAM module is a self-contained piece of program code that implements the primitives in one or more facilities for one particular mechanism; possible mechanisms for the authentication facility, for instance, include the UNIX(R) password database, NIS, LDAP and Radius. [[pam-module-naming]] ==== Module Naming FreeBSD implements each mechanism in a single module, named `pam_mechanism.so` (for instance, `pam_unix.so` for the UNIX(R) mechanism.) Other implementations sometimes have separate modules for separate facilities, and include the facility name as well as the mechanism name in the module name. To name one example, Solaris(TM) has a `pam_dial_auth.so.1` module which is commonly used to authenticate dialup users. [[pam-module-versioning]] ==== Module Versioning FreeBSD's original PAM implementation, based on Linux-PAM, did not use version numbers for PAM modules. This would commonly cause problems with legacy applications, which might be linked against older versions of the system libraries, as there was no way to load a matching version of the required modules. OpenPAM, on the other hand, looks for modules that have the same version number as the PAM library (currently 2), and only falls back to an unversioned module if no versioned module could be loaded. Thus legacy modules can be provided for legacy applications, while allowing new (or newly built) applications to take advantage of the most recent modules. Although Solaris(TM) PAM modules commonly have a version number, they are not truly versioned, because the number is a part of the module name and must be included in the configuration. [[pam-chains-policies]] === Chains and Policies When a server initiates a PAM transaction, the PAM library tries to load a policy for the service specified in the man:pam_start[3] call. The policy specifies how authentication requests should be processed, and is defined in a configuration file. This is the other central concept in PAM: the possibility for the admin to tune the system security policy (in the wider sense of the word) simply by editing a text file. A policy consists of four chains, one for each of the four PAM facilities. Each chain is a sequence of configuration statements, each specifying a module to invoke, some (optional) parameters to pass to the module, and a control flag that describes how to interpret the return code from the module. Understanding the control flags is essential to understanding PAM configuration files. There are four different control flags: `binding`:: If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the rest of the chain is executed, but the request is ultimately denied. + This control flag was introduced by Sun in Solaris(TM) 9 (SunOS(TM) 5.9), and is also supported by OpenPAM. `required`:: If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the rest of the chain is also executed, but the request is ultimately denied. `requisite`:: If the module succeeds, the rest of the chain is executed, and the request is granted unless some other module fails. If the module fails, the chain is immediately terminated and the request is denied. `sufficient`:: If the module succeeds and no earlier module in the chain has failed, the chain is immediately terminated and the request is granted. If the module fails, the module is ignored and the rest of the chain is executed. + As the semantics of this flag may be somewhat confusing, especially when it is used for the last module in a chain, it is recommended that the `binding` control flag be used instead if the implementation supports it. `optional`:: The module is executed, but its result is ignored. If all modules in a chain are marked `optional`, all requests will always be granted. When a server invokes one of the six PAM primitives, PAM retrieves the chain for the facility the primitive belongs to, and invokes each of the modules listed in the chain, in the order they are listed, until it reaches the end, or determines that no further processing is necessary (either because a `binding` or `sufficient` module succeeded, or because a `requisite` module failed.) The request is granted if and only if at least one module was invoked, and all non-optional modules succeeded. Note that it is possible, though not very common, to have the same module listed several times in the same chain. For instance, a module that looks up user names and passwords in a directory server could be invoked multiple times with different parameters specifying different directory servers to contact. PAM treat different occurrences of the same module in the same chain as different, unrelated modules. [[pam-transactions]] === Transactions The lifecycle of a typical PAM transaction is described below. Note that if any of these steps fails, the server should report a suitable error message to the client and abort the transaction. . If necessary, the server obtains arbitrator credentials through a mechanism independent of PAM-most commonly by virtue of having been started by `root`, or of being setuid `root`. . The server calls man:pam_start[3] to initialize the PAM library and specify its service name and the target account, and register a suitable conversation function. . The server obtains various information relating to the transaction (such as the applicant's user name and the name of the host the client runs on) and submits it to PAM using man:pam_set_item[3]. . The server calls man:pam_authenticate[3] to authenticate the applicant. . The server calls man:pam_acct_mgmt[3] to verify that the requested account is available and valid. If the password is correct but has expired, man:pam_acct_mgmt[3] will return `PAM_NEW_AUTHTOK_REQD` instead of `PAM_SUCCESS`. . If the previous step returned `PAM_NEW_AUTHTOK_REQD`, the server now calls man:pam_chauthtok[3] to force the client to change the authentication token for the requested account. . Now that the applicant has been properly authenticated, the server calls man:pam_setcred[3] to establish the credentials of the requested account. It is able to do this because it acts on behalf of the arbitrator, and holds the arbitrator's credentials. . Once the correct credentials have been established, the server calls man:pam_open_session[3] to set up the session. . The server now performs whatever service the client requested-for instance, provide the applicant with a shell. . Once the server is done serving the client, it calls man:pam_close_session[3] to tear down the session. . Finally, the server calls man:pam_end[3] to notify the PAM library that it is done and that it can release whatever resources it has allocated in the course of the transaction. [[pam-config]] == PAM Configuration [[pam-config-file]] === PAM Policy Files [[pam-config-pam.conf]] ==== The [.filename]#/etc/pam.conf# The traditional PAM policy file is [.filename]#/etc/pam.conf#. This file contains all the PAM policies for your system. Each line of the file describes one step in a chain, as shown below: [.programlisting] .... login auth required pam_nologin.so no_warn .... The fields are, in order: service name, facility name, control flag, module name, and module arguments. Any additional fields are interpreted as additional module arguments. A separate chain is constructed for each service / facility pair, so while the order in which lines for the same service and facility appear is significant, the order in which the individual services and facilities are listed is not. The examples in the original PAM paper grouped configuration lines by facility, and the Solaris(TM) stock [.filename]#pam.conf# still does that, but FreeBSD's stock configuration groups configuration lines by service. Either way is fine; either way makes equal sense. [[pam-config-pam.d]] ==== The [.filename]#/etc/pam.d# OpenPAM and Linux-PAM support an alternate configuration mechanism, which is the preferred mechanism in FreeBSD. In this scheme, each policy is contained in a separate file bearing the name of the service it applies to. These files are stored in [.filename]#/etc/pam.d/#. These per-service policy files have only four fields instead of [.filename]#pam.conf#'s five: the service name field is omitted. Thus, instead of the sample [.filename]#pam.conf# line from the previous section, one would have the following line in [.filename]#/etc/pam.d/login#: [.programlisting] .... auth required pam_nologin.so no_warn .... As a consequence of this simplified syntax, it is possible to use the same policy for multiple services by linking each service name to a same policy file. For instance, to use the same policy for the `su` and `sudo` services, one could do as follows: [source,shell] .... # cd /etc/pam.d # ln -s su sudo .... This works because the service name is determined from the file name rather than specified in the policy file, so the same file can be used for multiple differently-named services. Since each service's policy is stored in a separate file, the [.filename]#pam.d# mechanism also makes it very easy to install additional policies for third-party software packages. [[pam-config-file-order]] ==== The Policy Search Order As we have seen above, PAM policies can be found in a number of places. What happens if policies for the same service exist in multiple places? It is essential to understand that PAM's configuration system is centered on chains. [[pam-config-breakdown]] === Breakdown of a Configuration Line As explained in <>, each line in [.filename]#/etc/pam.conf# consists of four or more fields: the service name, the facility name, the control flag, the module name, and zero or more module arguments. The service name is generally (though not always) the name of the application the statement applies to. If you are unsure, refer to the individual application's documentation to determine what service name it uses. Note that if you use [.filename]#/etc/pam.d/# instead of [.filename]#/etc/pam.conf#, the service name is specified by the name of the policy file, and omitted from the actual configuration lines, which then start with the facility name. The facility is one of the four facility keywords described in <>. Likewise, the control flag is one of the four keywords described in <>, describing how to interpret the return code from the module. Linux-PAM supports an alternate syntax that lets you specify the action to associate with each possible return code, but this should be avoided as it is non-standard and closely tied in with the way Linux-PAM dispatches service calls (which differs greatly from the way Solaris(TM) and OpenPAM do it.) Unsurprisingly, OpenPAM does not support this syntax. [[pam-policies]] === Policies To configure PAM correctly, it is essential to understand how policies are interpreted. When an application calls man:pam_start[3], the PAM library loads the policy for the specified service and constructs four module chains (one for each facility.) If one or more of these chains are empty, the corresponding chains from the policy for the `other` service are substituted. When the application later calls one of the six PAM primitives, the PAM library retrieves the chain for the corresponding facility and calls the appropriate service function in each module listed in the chain, in the order in which they were listed in the configuration. After each call to a service function, the module type and the error code returned by the service function are used to determine what happens next. With a few exceptions, which we discuss below, the following table applies: .PAM Chain Execution Summary [cols="1,1,1,1", options="header"] |=== | | PAM_SUCCESS | PAM_IGNORE | other |binding |if (!fail) break; |- |fail = true; |required |- |- |fail = true; |requisite |- |- |fail = true; break; |sufficient |if (!fail) break; |- |- |optional |- |- |- |=== If `fail` is true at the end of a chain, or when a "break" is reached, the dispatcher returns the error code returned by the first module that failed. Otherwise, it returns `PAM_SUCCESS`. The first exception of note is that the error code `PAM_NEW_AUTHTOK_REQD` is treated like a success, except that if no module failed, and at least one module returned `PAM_NEW_AUTHTOK_REQD`, the dispatcher will return `PAM_NEW_AUTHTOK_REQD`. The second exception is that man:pam_setcred[3] treats `binding` and `sufficient` modules as if they were `required`. The third and final exception is that man:pam_chauthtok[3] runs the entire chain twice (once for preliminary checks and once to actually set the password), and in the preliminary phase it treats `binding` and `sufficient` modules as if they were `required`. [[pam-freebsd-modules]] == FreeBSD PAM Modules [[pam-modules-deny]] === man:pam_deny[8] The man:pam_deny[8] module is one of the simplest modules available; it responds to any request with `PAM_AUTH_ERR`. It is useful for quickly disabling a service (add it to the top of every chain), or for terminating chains of `sufficient` modules. [[pam-modules-echo]] === man:pam_echo[8] The man:pam_echo[8] module simply passes its arguments to the conversation function as a `PAM_TEXT_INFO` message. It is mostly useful for debugging, but can also serve to display messages such as "Unauthorized access will be prosecuted" before starting the authentication procedure. [[pam-modules-exec]] === man:pam_exec[8] The man:pam_exec[8] module takes its first argument to be the name of a program to execute, and the remaining arguments are passed to that program as command-line arguments. One possible application is to use it to run a program at login time which mounts the user's home directory. [[pam-modules-ftpusers]] === man:pam_ftpusers[8] The man:pam_ftpusers[8] module [[pam-modules-group]] === man:pam_group[8] The man:pam_group[8] module accepts or rejects applicants on the basis of their membership in a particular file group (normally `wheel` for man:su[1]). It is primarily intended for maintaining the traditional behavior of BSD man:su[1], but has many other uses, such as excluding certain groups of users from a particular service. [[pam-modules-guest]] === man:pam_guest[8] The man:pam_guest[8] module allows guest logins using fixed login names. Various requirements can be placed on the password, but the default behavior is to allow any password as long as the login name is that of a guest account. The man:pam_guest[8] module can easily be used to implement anonymous FTP logins. [[pam-modules-krb5]] === man:pam_krb5[8] The man:pam_krb5[8] module [[pam-modules-ksu]] === man:pam_ksu[8] The man:pam_ksu[8] module [[pam-modules-lastlog]] === man:pam_lastlog[8] The man:pam_lastlog[8] module [[pam-modules-login-access]] === man:pam_login_access[8] The man:pam_login_access[8] module provides an implementation of the account management primitive which enforces the login restrictions specified in the man:login.access[5] table. [[pam-modules-nologin]] === man:pam_nologin[8] The man:pam_nologin[8] module refuses non-root logins when [.filename]#/var/run/nologin# exists. This file is normally created by man:shutdown[8] when less than five minutes remain until the scheduled shutdown time. [[pam-modules-opie]] === man:pam_opie[8] The man:pam_opie[8] module implements the man:opie[4] authentication method. The man:opie[4] system is a challenge-response mechanism where the response to each challenge is a direct function of the challenge and a passphrase, so the response can be easily computed "just in time" by anyone possessing the passphrase, eliminating the need for password lists. Moreover, since man:opie[4] never reuses a challenge that has been correctly answered, it is not vulnerable to replay attacks. [[pam-modules-opieaccess]] === man:pam_opieaccess[8] The man:pam_opieaccess[8] module is a companion module to man:pam_opie[8]. Its purpose is to enforce the restrictions codified in man:opieaccess[5], which regulate the conditions under which a user who would normally authenticate herself using man:opie[4] is allowed to use alternate methods. This is most often used to prohibit the use of password authentication from untrusted hosts. In order to be effective, the man:pam_opieaccess[8] module must be listed as `requisite` immediately after a `sufficient` entry for man:pam_opie[8], and before any other modules, in the `auth` chain. [[pam-modules-passwdqc]] === man:pam_passwdqc[8] The man:pam_passwdqc[8] module [[pam-modules-permit]] === man:pam_permit[8] The man:pam_permit[8] module is one of the simplest modules available; it responds to any request with `PAM_SUCCESS`. It is useful as a placeholder for services where one or more chains would otherwise be empty. [[pam-modules-radius]] === man:pam_radius[8] The man:pam_radius[8] module [[pam-modules-rhosts]] === man:pam_rhosts[8] The man:pam_rhosts[8] module [[pam-modules-rootok]] === man:pam_rootok[8] The man:pam_rootok[8] module reports success if and only if the real user id of the process calling it (which is assumed to be run by the applicant) is 0. This is useful for non-networked services such as man:su[1] or man:passwd[1], to which the `root` should have automatic access. [[pam-modules-securetty]] === man:pam_securetty[8] The man:pam_securetty[8] module [[pam-modules-self]] === man:pam_self[8] The man:pam_self[8] module reports success if and only if the names of the applicant matches that of the target account. It is most useful for non-networked services such as man:su[1], where the identity of the applicant can be easily verified. [[pam-modules-ssh]] === man:pam_ssh[8] The man:pam_ssh[8] module provides both authentication and session services. The authentication service allows users who have passphrase-protected SSH secret keys in their [.filename]#~/.ssh# directory to authenticate themselves by typing their passphrase. The session service starts man:ssh-agent[1] and preloads it with the keys that were decrypted in the authentication phase. This feature is particularly useful for local logins, whether in X (using man:xdm[1] or another PAM-aware X login manager) or at the console. [[pam-modules-tacplus]] === man:pam_tacplus[8] The man:pam_tacplus[8] module [[pam-modules-unix]] === man:pam_unix[8] The man:pam_unix[8] module implements traditional UNIX(R) password authentication, using man:getpwnam[3] to obtain the target account's password and compare it with the one provided by the applicant. It also provides account management services (enforcing account and password expiration times) and password-changing services. This is probably the single most useful module, as the great majority of admins will want to maintain historical behavior for at least some services. [[pam-appl-prog]] == PAM Application Programming This section has not yet been written. [[pam-module-prog]] == PAM Module Programming This section has not yet been written. :sectnums!: [appendix] [[pam-sample-appl]] == Sample PAM Application The following is a minimal implementation of man:su[1] using PAM. Note that it uses the OpenPAM-specific man:openpam_ttyconv[3] conversation function, which is prototyped in [.filename]#security/openpam.h#. If you wish build this application on a system with a different PAM library, you will have to provide your own conversation function. A robust conversation function is surprisingly difficult to implement; the one presented in <> is a good starting point, but should not be used in real-world applications. [.programlisting] .... include::static/source/articles/pam/su.c[] .... :sectnums!: [appendix] [[pam-sample-module]] == Sample PAM Module The following is a minimal implementation of man:pam_unix[8], offering only authentication services. It should build and run with most PAM implementations, but takes advantage of OpenPAM extensions if available: note the use of man:pam_get_authtok[3], which enormously simplifies prompting the user for a password. [.programlisting] .... include::static/source/articles/pam/pam_unix.c[] .... :sectnums!: [appendix] [[pam-sample-conv]] == Sample PAM Conversation Function The conversation function presented below is a greatly simplified version of OpenPAM's man:openpam_ttyconv[3]. It is fully functional, and should give the reader a good idea of how a conversation function should behave, but it is far too simple for real-world use. Even if you are not using OpenPAM, feel free to download the source code and adapt man:openpam_ttyconv[3] to your uses; we believe it to be as robust as a tty-oriented conversation function can reasonably get. [.programlisting] .... include::static/source/articles/pam/converse.c[] .... :sectnums!: [[pam-further]] == Further Reading === Papers Making Login Services Independent of Authentication Technologies Vipin Samar. Charlie Lai. Sun Microsystems. _link:https://pubs.opengroup.org/onlinepubs/8329799/toc.htm[X/Open Single Sign-on Preliminary Specification]_. The Open Group. 1-85912-144-6. June 1997. _link:https://mirrors.kernel.org/pub/linux/libs/pam/pre/doc/draft-morgan-pam-07.txt[Pluggable Authentication Modules]_. Andrew G. Morgan. 1999-10-06. === User Manuals _link:https://docs.oracle.com/cd/E26505_01/html/E27224/pam-1.html[PAM Administration]_. Sun Microsystems. === Related Web Pages _link:https://www.openpam.org/[OpenPAM homepage]_ Dag-Erling Smørgrav. ThinkSec AS. _link:http://www.kernel.org/pub/linux/libs/pam/[Linux-PAM homepage]_ Andrew Morgan. _Solaris PAM homepage_. Sun Microsystems. diff --git a/documentation/content/en/articles/problem-reports/_index.adoc b/documentation/content/en/articles/problem-reports/_index.adoc index d61055bdfc..ac334b78b4 100644 --- a/documentation/content/en/articles/problem-reports/_index.adoc +++ b/documentation/content/en/articles/problem-reports/_index.adoc @@ -1,271 +1,271 @@ --- title: Writing FreeBSD Problem Reports authors: - author: Dag-Erling Smørgrav - author: Mark Linimon -description: Writing FreeBSD Problem Reports +description: How to best formulate and submit a problem report to the FreeBSD Project trademarks: ["freebsd", "ibm", "intel", "sun", "general"] --- = Writing FreeBSD Problem Reports :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This article describes how to best formulate and submit a problem report to the FreeBSD Project. ''' toc::[] [[pr-intro]] == Introduction One of the most frustrating experiences one can have as a software user is to submit a problem report only to have it summarily closed with a terse and unhelpful explanation like "not a bug" or "bogus PR". Similarly, one of the most frustrating experiences as a software developer is to be flooded with problem reports that are not really problem reports but requests for support, or that contain little or no information about what the problem is and how to reproduce it. This document attempts to describe how to write good problem reports. What, one asks, is a good problem report? Well, to go straight to the bottom line, a good problem report is one that can be analyzed and dealt with swiftly, to the mutual satisfaction of both user and developer. Although the primary focus of this article is on FreeBSD problem reports, most of it should apply quite well to other software projects. Note that this article is organized thematically, not chronologically. Read the entire document before submitting a problem report, rather than treating it as a step-by-step tutorial. [[pr-when]] == When to Submit a Problem Report There are many types of problems, and not all of them should engender a problem report. Of course, nobody is perfect, and there will be times when what seems to be a bug in a program is, in fact, a misunderstanding of the syntax for a command or a typographical error in a configuration file (though that in itself may sometimes be indicative of poor documentation or poor error handling in the application). There are still many cases where submitting a problem report is clearly _not_ the right course of action, and will only serve to frustrate both the submitter and the developers. Conversely, there are cases where it might be appropriate to submit a problem report about something else than a bug-an enhancement or a new feature, for instance. So how does one determine what is a bug and what is not? As a simple rule of thumb, the problem is _not_ a bug if it can be expressed as a question (usually of the form "How do I do X?" or "Where can I find Y?"). It is not always quite so black and white, but the question rule covers a large majority of cases. When looking for an answer, consider posing the question to the {freebsd-questions}. Consider these factors when submitting PRs about ports or other software that is not part of FreeBSD itself: * Please do not submit problem reports that simply state that a newer version of an application is available. Ports maintainers are automatically notified by portscout when a new version of an application becomes available. Actual patches to update a port to the latest version are welcome. * For unmaintained ports (`MAINTAINER` is `ports@FreeBSD.org`), a PR without an included patch is unlikely to get picked up by a committer. To become the maintainer of an unmaintained port, submit a PR with the request (patch preferred but not required). * In either case, following the process described in link:{porters-handbook}#port-upgrading[Porter's Handbook] will yield the best results. (You might also wish to read link:{contributing}#ports-contributing[Contributing to the FreeBSD Ports Collection].) A bug that cannot be reproduced can rarely be fixed. If the bug only occurred once and you cannot reproduce it, and it does not seem to happen to anybody else, chances are none of the developers will be able to reproduce it or figure out what is wrong. That does not mean it did not happen, but it does mean that the chances of your problem report ever leading to a bug fix are very slim. To make matters worse, often these kinds of bugs are actually caused by failing hard drives or overheating processors - you should always try to rule out these causes, whenever possible, before submitting a PR. Next, to decide to whom you should file your problem report, you need to understand that the software that makes up FreeBSD is composed of several different elements: * Code in the base system that is written and maintained by FreeBSD contributors, such as the kernel, the C library, and the device drivers (categorized as `kern`); the binary utilities (`bin`); the manual pages and documentation (`docs`); and the web pages (`www`). All bugs in these areas should be reported to the FreeBSD developers. * Code in the base system that is written and maintained by others, and imported into FreeBSD and adapted. Examples include man:clang[1], and man:sendmail[8]. Most bugs in these areas should be reported to the FreeBSD developers; but in some cases they may need to be reported to the original authors instead if the problems are not FreeBSD-specific. * Individual applications that are not in the base system but are instead part of the FreeBSD Ports Collection (category `ports`). Most of these applications are not written by FreeBSD developers; what FreeBSD provides is merely a framework for installing the application. Therefore, only report a problem to the FreeBSD developers when the problem is believed to be FreeBSD-specific; otherwise, report it to the authors of the software. Then, ascertain whether the problem is timely. There are few things that will annoy a developer more than receiving a problem report about a bug she has already fixed. If the problem is in the base system, first read the FAQ section on link:{faq}#LATEST-VERSION[FreeBSD versions], if you are not already familiar with the topic. It is not possible for FreeBSD to fix problems in anything other than certain recent branches of the base system, so filing a bug report about an older version will probably only result in a developer advising you to upgrade to a supported version to see if the problem still recurs. The Security Officer team maintains the link:https://www.FreeBSD.org/security/[list of supported versions]. If the problem is in a port, consider filing a bug with the upstream. The FreeBSD Project can not fix all bugs in all software. [[pr-prep]] == Preparations A good rule to follow is to always do a background search before submitting a problem report. Maybe the problem has already been reported; maybe it is being discussed on the mailing lists, or recently was; it may even already be fixed in a newer version than what you are running. You should therefore check all the obvious places before submitting your problem report. For FreeBSD, this means: * The FreeBSD link:{faq}[Frequently Asked Questions] (FAQ) list. The FAQ attempts to provide answers for a wide range of questions, such as those concerning link:{faq}#hardware[hardware compatibility], link:{faq}#applications[user applications], and link:{faq}#kernelconfig[kernel configuration]. * The link:{handbook}#eresources-mail[mailing lists]-if you are not subscribed, use https://www.FreeBSD.org/search/#mailinglists[the searchable archives] on the FreeBSD web site. If the problem has not been discussed on the lists, you might try posting a message about it and waiting a few days to see if someone can spot something that has been overlooked. * Optionally, the entire web-use your favorite search engine to locate any references to the problem. You may even get hits from archived mailing lists or newsgroups you did not know of or had not thought to search through. * Next, the searchable https://bugs.freebsd.org/bugzilla/query.cgi[FreeBSD PR database] (Bugzilla). Unless the problem is recent or obscure, there is a fair chance it has already been reported. * Most importantly, attempt to see if existing documentation in the source base addresses your problem. + For the base FreeBSD code, you should carefully study the contents of [.filename]#/usr/src/UPDATING# on your system or the latest version at https://cgit.freebsd.org/src/tree/UPDATING[https://cgit.freebsd.org/src/tree/UPDATING]. (This is vital information if you are upgrading from one version to another-especially if you are upgrading to the FreeBSD-CURRENT branch). + However, if the problem is in something that was installed as a part of the FreeBSD Ports Collection, you should refer to [.filename]#/usr/ports/UPDATING# (for individual ports) or [.filename]#/usr/ports/CHANGES# (for changes that affect the entire Ports Collection). https://cgit.freebsd.org/ports/tree/UPDATING[https://cgit.freebsd.org/ports/tree/UPDATING] and https://cgit.freebsd.org/ports/tree/CHANGES[https://cgit.freebsd.org/ports/tree/CHANGES] are also available via cgit. [[pr-writing]] == Writing the Problem Report Now that you have decided that your issue merits a problem report, and that it is a FreeBSD problem, it is time to write the actual problem report. Before we get into the mechanics of the program used to generate and submit PRs, here are some tips and tricks to help make sure that your PR will be most effective. [[pr-writing-tips]] == Tips and Tricks for Writing a Good Problem Report * _Do not leave the "Summary" line empty._ The PRs go both onto a mailing list that goes all over the world (where the "Summary" is used for the `Subject:` line), but also into a database. Anyone who comes along later and browses the database by synopsis, and finds a PR with a blank subject line, tends just to skip over it. Remember that PRs stay in this database until they are closed by someone; an anonymous one will usually just disappear in the noise. * _Avoid using a weak "Summary" line._ You should not assume that anyone reading your PR has any context for your submission, so the more you provide, the better. For instance, what part of the system does the problem apply to? Do you only see the problem while installing, or while running? To illustrate, instead of `Summary: portupgrade is broken`, see how much more informative this seems: `Summary: port ports-mgmt/portupgrade coredumps on -current`. (In the case of ports, it is especially helpful to have both the category and portname in the "Summary" line.) * _If you have a patch, say so._ A PR with a patch included is much more likely to be looked at than one without. Please set the `patch` Keyword in Bugzilla. * _If you are a maintainer, say so._ If you are maintaining a part of the source code (for instance, an existing port), you definitely should set the "Class" of your PR to `maintainer-update`. This way any committer that handles your PR will not have to check. * _Be specific._ The more information you supply about what problem you are having, the better your chance of getting a response. ** Include the version of FreeBSD you are running (there is a place to put that, see below) and on which architecture. You should include whether you are running from a release (e.g., from a CD-ROM or download), or from a system maintained by Git (and, if so, what hash and branch you are at). If you are tracking the FreeBSD-CURRENT branch, that is the very first thing someone will ask, because fixes (especially for high-profile problems) tend to get committed very quickly, and FreeBSD-CURRENT users are expected to keep up. ** Include which global options you have specified in your [.filename]#make.conf#, [.filename]#src.conf#, and [.filename]#src-env.conf#. Given the infinite number of options, not every combination may be fully supported. ** If the problem can be reproduced easily, include information that will help a developer to reproduce it themselves. If a problem can be demonstrated with specific input then include an example of that input if possible, and include both the actual and the expected output. If this data is large or cannot be made public, then do try to create a minimal file that exhibits the same issue and that can be included within the PR. ** If this is a kernel problem, then be prepared to supply the following information. (You do not have to include these by default, which only tends to fill up the database, but you should include excerpts that you think might be relevant): *** your kernel configuration (including which hardware devices you have installed) *** whether or not you have debugging options enabled (such as `WITNESS`), and if so, whether the problem persists when you change the sense of that option *** the full text of any backtrace, panic or other console output, or entries in [.filename]#/var/log/messages#, if any were generated *** the output of `pciconf -l` and relevant parts of your `dmesg` output if your problem relates to a specific piece of hardware *** the fact that you have read [.filename]#src/UPDATING# and that your problem is not listed there (someone is guaranteed to ask) *** whether or not you can run any other kernel as a fallback (this is to rule out hardware-related issues such as failing disks and overheating CPUs, which can masquerade as kernel problems) ** If this is a ports problem, then be prepared to supply the following information. (You do not have to include these by default, which only tends to fill up the database, but you should include excerpts that you think might be relevant): *** which ports you have installed *** any environment variables that override the defaults in [.filename]#bsd.port.mk#, such as `PORTSDIR` *** the fact that you have read [.filename]#ports/UPDATING# and that your problem is not listed there (someone is guaranteed to ask) * _Avoid vague requests for features._ PRs of the form "someone should really implement something that does so-and-so" are less likely to get results than very specific requests. Remember, the source is available to everyone, so if you want a feature, the best way to ensure it being included is to get to work! Also consider the fact that many things like this would make a better topic for discussion on `freebsd-questions` than an entry in the PR database, as discussed above. * _Make sure no one else has already submitted a similar PR._ Although this has already been mentioned above, it bears repeating here. It only take a minute or two to use the web-based search engine at https://bugs.freebsd.org/bugzilla/query.cgi[https://bugs.freebsd.org/bugzilla/query.cgi]. (Of course, everyone is guilty of forgetting to do this now and then.) * _Report only one issue per Problem Report._ Avoid including two or more problems within the same report unless they are related. When submitting patches, avoid adding multiple features or fixing multiple bugs in the same PR unless they are closely related-such PRs often take longer to resolve. * _Avoid controversial requests._ If your PR addresses an area that has been controversial in the past, you should probably be prepared to not only offer patches, but also justification for why the patches are "The Right Thing To Do". As noted above, a careful search of the mailing lists using the archives at https://www.FreeBSD.org/search/#mailinglists[https://www.FreeBSD.org/search/#mailinglists] is always good preparation. * _Be polite._ Almost anyone who would potentially work on your PR is a volunteer. No one likes to be told that they have to do something when they are already doing it for some motivation other than monetary gain. This is a good thing to keep in mind at all times on Open Source projects. [[pr-writing-before-beginning]] == Before Beginning Similar considerations apply to use of the https://bugs.freebsd.org/bugzilla/enter_bug.cgi[web-based PR submission form]. Be careful of cut-and-paste operations that might change whitespace or other text formatting. Finally, if the submission is lengthy, prepare the work offline so that nothing will be lost if there is a problem submitting it. [[pr-writing-attaching-patches]] == Attaching Patches or Files When attaching a patch, be sure to use either `git diff` or man:diff[1] with the `-u` option to create a unified diff and make sure to specify the Git hash and branch of the repository against which you modified files, so the developers who read your report will be able to apply them easily. For problems with the kernel or the base utilities, a patch against FreeBSD-CURRENT (the main Git branch) is preferred since all new code should be applied and tested there first. After appropriate or substantial testing has been done, the code will be merged/migrated to the FreeBSD-STABLE branch. If you attach a patch inline, instead of as an attachment, note that the most common problem by far is the tendency of some email programs to render tabs as spaces, which will completely ruin anything intended to be part of a Makefile. Do not send patches as attachments using `Content-Transfer-Encoding: quoted-printable`. These will perform character escaping and the entire patch will be useless. Also note that while including small patches in a PR is generally all right-particularly when they fix the problem described in the PR-large patches and especially new code which may require substantial review before committing should be placed on a web or ftp server, and the URL should be included in the PR instead of the patch. Patches in email tend to get mangled, and the larger the patch, the harder it will be for interested parties to unmangle it. Also, posting a patch on the web allows you to modify it without having to resubmit the entire patch in a followup to the original PR. Finally, large patches simply increase the size of the database, since closed PRs are not actually deleted but instead kept and simply marked as complete. You should also take note that unless you explicitly specify otherwise in your PR or in the patch itself, any patches you submit will be assumed to be licensed under the same terms as the original file you modified. [[pr-writing-filling-template]] == Filling out the Form [NOTE] ==== The email address you use will become public information and may become available to spammers. You should either have spam handling procedures in place, or use a temporary email account. However, please note that if you do not use a valid email account at all, we will not be able to ask you questions about your PR. ==== When you file a bug, you will find the following fields: * _Summary:_ Fill this out with a short and accurate description of the problem. The synopsis is used as the subject of the problem report email, and is used in problem report listings and summaries; problem reports with obscure synopses tend to get ignored. * _Severity:_ One of `Affects only me`, `Affects some people` or `Affects many people`. Do not overreact; refrain from labeling your problem `Affects many people` unless it really does. FreeBSD developers will not necessarily work on your problem faster if you inflate its importance since there are so many other people who have done exactly that. * _Category:_ Choose an appropriate category. + The first thing you need to do is to decide what part of the system your problem lies in. Remember, FreeBSD is a complete operating system, which installs both a kernel, the standard libraries, many peripheral drivers, and a large number of utilities (the "base system"). However, there are thousands of additional applications in the Ports Collection. You'll first need to decide if the problem is in the base system or something installed via the Ports Collection. + Here is a description of the major categories: ** If a problem is with the kernel, the libraries (such as standard C library `libc`), or a peripheral driver in the base system, in general you will use the `kern` category. (There are a few exceptions; see below). In general these are things that are described in section 2, 3, or 4 of the manual pages. ** If a problem is with a binary program such as man:sh[1] or man:mount[8], you will first need to determine whether these programs are in the base system or were added via the Ports Collection. If you are unsure, you can do `whereis _programname_`. FreeBSD's convention for the Ports Collection is to install everything underneath [.filename]#/usr/local#, although this can be overridden by a system administrator. For these, you will use the `ports` category (yes, even if the port's category is `www`; see below). If the location is [.filename]#/bin#, [.filename]#/usr/bin#, [.filename]#/sbin#, or [.filename]#/usr/sbin#, it is part of the base system, and you should use the `bin` category. These are all things that are described in section 1 or 8 of the manual pages. ** If you believe that the error is in the startup `(rc)` scripts, or in some kind of other non-executable configuration file, then the right category is `conf` (configuration). These are things that are described in section 5 of the manual pages. ** If you have found a problem in the documentation set (articles, books, man pages) or website the correct choice is `docs`. + [NOTE] ==== if you are having a problem with something from a port named `www/_someportname_`, this nevertheless goes in the `ports` category. ==== + There are a few more specialized categories. ** If the problem would otherwise be filed in `kern` but has to do with the USB subsystem, the correct choice is `usb`. ** If the problem would otherwise be filed in `kern` but has to do with the threading libraries, the correct choice is `threads`. ** If the problem would otherwise be in the base system, but has to do with our adherence to standards such as POSIX(R), the correct choice is `standards`. ** If you are convinced that the problem will only occur under the processor architecture you are using, select one of the architecture-specific categories: commonly `i386` for Intel-compatible machines in 32-bit mode; `amd64` for AMD machines running in 64-bit mode (this also includes Intel-compatible machines running in EMT64 mode); and less commonly `arm` or `powerpc`. + [NOTE] ==== These categories are quite often misused for "I do not know" problems. Rather than guessing, please just use `misc`. ==== + .Correct Use of Arch-Specific Category [example] ==== You have a common PC-based machine, and think you have encountered a problem specific to a particular chipset or a particular motherboard: `i386` is the right category. ==== + .Incorrect Use of Arch-Specific Category [example] ==== You are having a problem with an add-in peripheral card on a commonly seen bus, or a problem with a particular type of hard disk drive: in this case, it probably applies to more than one architecture, and `kern` is the right category. ==== ** If you really do not know where the problem lies (or the explanation does not seem to fit into the ones above), use the `misc` category. Before you do so, you may wish to ask for help on the {freebsd-questions} first. You may be advised that one of the existing categories really is a better choice. * _Environment:_ This should describe, as accurately as possible, the environment in which the problem has been observed. This includes the operating system version, the version of the specific program or file that contains the problem, and any other relevant items such as system configuration, other installed software that influences the problem, etc.-quite simply everything a developer needs to know to reconstruct the environment in which the problem occurs. * __Description:__A complete and accurate description of the problem you are experiencing. Try to avoid speculating about the causes of the problem unless you are certain that you are on the right track, as it may mislead a developer into making incorrect assumptions about the problem. It should include the actions you need to take to reproduce the problem. If you know any workaround, include it. It not only helps other people with the same problem work around it, but may also help a developer understand the cause for the problem. [[pr-followup]] == Follow-up Once the problem report has been filed, you will receive a confirmation by email which will include the tracking number that was assigned to your problem report and a URL you can use to check its status. With a little luck, someone will take an interest in your problem and try to address it, or, as the case may be, explain why it is not a problem. You will be automatically notified of any change of status, and you will receive copies of any comments or patches someone may attach to your problem report's audit trail. If someone requests additional information from you, or you remember or discover something you did not mention in the initial report, please submit a follow up. The number one reason for a bug not getting fixed is lack of communication with the originator. The easiest way is to use the comment option on the individual PR's web page, which you can reach from the https://bugs.freebsd.org/bugzilla/query.cgi[PR search page]. If the problem report remains open after the problem has gone away, just add a comment saying that the problem report can be closed, and, if possible, explaining how or when the problem was fixed. Sometimes there is a delay of a week or two where the problem report remains untouched, not assigned or commented on by anyone. This can happen when there is an increased problem report backlog or during a holiday season. When a problem report has not received attention after several weeks, it is worth finding a committer particularly interested in working on it. There are a few ways to do so, ideally in the following order, with a few days between attempting each communication channel: * Find the relevant FreeBSD mailing list for the problem report from the link:{handbook}#eresources-mail[list in the Handbook] and send a message to that list asking about assistance or comments on the problem report. * Join the relevant IRC channels. A partial listing is here: https://wiki.freebsd.org/IrcChannels[]. Inform the people in that channel about the problem report and ask for assistance. Be patient and stay in the channel after posting, so that the people from different time zones around the world have a chance to catch up. * Find committers interested in the problem that was reported. If the problem was in a particular tool, binary, port, document, or source file, check the https://cgit.FreeBSD.org[Git Repository]. Locate the last few committers who made substantive changes to the file, and try to reach them via IRC or email. A list of committers and their emails can be found in the link:{contributors}[Contributors to FreeBSD] article. Remember that these people are volunteers, just like maintainers and users, so they might not be immediately available to assist with the problem report. Patience and consistency in the follow-ups is highly advised and appreciated. With enough care and effort dedicated to that follow-up process, finding a committer to take care of the problem report is just a matter of time. [[pr-problems]] == If There Are Problems If you found an issue with the bug system, file a bug! There is a category for exactly this purpose. If you are unable to do so, contact the bug wranglers at mailto:bugmeister@FreeBSD.org[bugmeister@FreeBSD.org]. [[pr-further]] == Further Reading This is a list of resources relevant to the proper writing and processing of problem reports. It is by no means complete. * https://github.com/smileytechguy/reporting-bugs-effectively/blob/master/ENGLISH.md[How to Report Bugs Effectively]-an excellent essay by Simon G. Tatham on composing useful (non-FreeBSD-specific) problem reports. * link:{pr-guidelines}[Problem Report Handling Guidelines]-valuable insight into how problem reports are handled by the FreeBSD developers. diff --git a/documentation/content/en/articles/rc-scripting/_index.adoc b/documentation/content/en/articles/rc-scripting/_index.adoc index 72e71bffbd..50507c8412 100644 --- a/documentation/content/en/articles/rc-scripting/_index.adoc +++ b/documentation/content/en/articles/rc-scripting/_index.adoc @@ -1,803 +1,803 @@ --- title: Practical rc.d scripting in BSD authors: - author: Yar Tikhiy email: yar@FreeBSD.org copyright: 2005-2006, 2012 The FreeBSD Project -description: Practical rc.d scripting in BSD +description: A guide to writing new rc.d scripts and understanding those already written trademarks: ["freebsd", "netbsd", "general"] --- = Practical rc.d scripting in BSD :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/en/urls.adoc[] [.abstract-title] Abstract Beginners may find it difficult to relate the facts from the formal documentation on the BSD [.filename]#rc.d# framework with the practical tasks of [.filename]#rc.d# scripting. In this article, we consider a few typical cases of increasing complexity, show [.filename]#rc.d# features suited for each case, and discuss how they work. Such an examination should provide reference points for further study of the design and efficient application of [.filename]#rc.d#. ''' toc::[] [[rcng-intro]] == Introduction The historical BSD had a monolithic startup script, [.filename]#/etc/rc#. It was invoked by man:init[8] at system boot time and performed all userland tasks required for multi-user operation: checking and mounting file systems, setting up the network, starting daemons, and so on. The precise list of tasks was not the same in every system; admins needed to customize it. With few exceptions, [.filename]#/etc/rc# had to be modified, and true hackers liked it. The real problem with the monolithic approach was that it provided no control over the individual components started from [.filename]#/etc/rc#. For instance, [.filename]#/etc/rc# could not restart a single daemon. The system admin had to find the daemon process by hand, kill it, wait until it actually exited, then browse through [.filename]#/etc/rc# for the flags, and finally type the full command line to start the daemon again. The task would become even more difficult and prone to errors if the service to restart consisted of more than one daemon or demanded additional actions. In a few words, the single script failed to fulfil what scripts are for: to make the system admin's life easier. Later there was an attempt to split out some parts of [.filename]#/etc/rc# for the sake of starting the most important subsystems separately. The notorious example was [.filename]#/etc/netstart# to bring up networking. It did allow for accessing the network from single-user mode, but it did not integrate well into the automatic startup process because parts of its code needed to interleave with actions essentially unrelated to networking. That was why [.filename]#/etc/netstart# mutated into [.filename]#/etc/rc.network#. The latter was no longer an ordinary script; it comprised of large, tangled man:sh[1] functions called from [.filename]#/etc/rc# at different stages of system startup. However, as the startup tasks grew diverse and sophisticated, the "quasi-modular" approach became even more of a drag than the monolithic [.filename]#/etc/rc# had been. Without a clean and well-designed framework, the startup scripts had to bend over backwards to satisfy the needs of rapidly developing BSD-based operating systems. It became obvious at last that more steps are necessary on the way to a fine-grained and extensible [.filename]#rc# system. Thus BSD [.filename]#rc.d# was born. Its acknowledged fathers were Luke Mewburn and the NetBSD community. Later it was imported into FreeBSD. Its name refers to the location of system scripts for individual services, which is in [.filename]#/etc/rc.d#. Soon we will learn about more components of the [.filename]#rc.d# system and see how the individual scripts are invoked. The basic ideas behind BSD [.filename]#rc.d# are _fine modularity_ and __code reuse__. _Fine modularity_ means that each basic "service" such as a system daemon or primitive startup task gets its own man:sh[1] script able to start the service, stop it, reload it, check its status. A particular action is chosen by the command-line argument to the script. The [.filename]#/etc/rc# script still drives system startup, but now it merely invokes the smaller scripts one by one with the `start` argument. It is easy to perform shutdown tasks as well by running the same set of scripts with the `stop` argument, which is done by [.filename]#/etc/rc.shutdown#. Note how closely this follows the Unix way of having a set of small specialized tools, each fulfilling its task as well as possible. _Code reuse_ means that common operations are implemented as man:sh[1] functions and collected in [.filename]#/etc/rc.subr#. Now a typical script can be just a few lines' worth of man:sh[1] code. Finally, an important part of the [.filename]#rc.d# framework is man:rcorder[8], which helps [.filename]#/etc/rc# to run the small scripts orderly with respect to dependencies between them. It can help [.filename]#/etc/rc.shutdown#, too, because the proper order for the shutdown sequence is opposite to that of startup. The BSD [.filename]#rc.d# design is described in <>, and the [.filename]#rc.d# components are documented in great detail in <>. However, it might not appear obvious to an [.filename]#rc.d# newbie how to tie the numerous bits and pieces together in order to create a well-styled script for a particular task. Therefore this article will try a different approach to describe [.filename]#rc.d#. It will show which features should be used in a number of typical cases, and why. Note that this is not a how-to document because our aim is not at giving ready-made recipes, but at showing a few easy entrances into the [.filename]#rc.d# realm. Neither is this article a replacement for the relevant manual pages. Do not hesitate to refer to them for more formal and complete documentation while reading this article. There are prerequisites to understanding this article. First of all, you should be familiar with the man:sh[1] scripting language in order to master [.filename]#rc.d#. In addition, you should know how the system performs userland startup and shutdown tasks, which is described in man:rc[8]. This article focuses on the FreeBSD branch of [.filename]#rc.d#. Nevertheless, it may be useful to NetBSD developers, too, because the two branches of BSD [.filename]#rc.d# not only share the same design but also stay similar in their aspects visible to script authors. [[rcng-task]] == Outlining the task A little consideration before starting `$EDITOR` will not hurt. In order to write a well-tempered [.filename]#rc.d# script for a system service, we should be able to answer the following questions first: * Is the service mandatory or optional? * Will the script serve a single program, e.g., a daemon, or perform more complex actions? * Which other services will our service depend on, and vice versa? From the examples that follow we will see why it is important to know the answers to these questions. [[rcng-dummy]] == A dummy script The following script just emits a message each time the system boots up: [.programlisting] .... #!/bin/sh <.> . /etc/rc.subr <.> name="dummy" <.> start_cmd="${name}_start" <.> stop_cmd=":" <.> dummy_start() <.> { echo "Nothing started." } load_rc_config $name <.> run_rc_command "$1" <.> .... Things to note are: ➊ An interpreted script should begin with the magic "shebang" line. That line specifies the interpreter program for the script. Due to the shebang line, the script can be invoked exactly like a binary program provided that it has the execute bit set. (See man:chmod[1].) For example, a system admin can run our script manually, from the command line: [source,shell] .... # /etc/rc.d/dummy start .... [NOTE] ==== In order to be properly managed by the [.filename]#rc.d# framework, its scripts need to be written in the man:sh[1] language. If you have a service or port that uses a binary control utility or a startup routine written in another language, install that element in [.filename]#/usr/sbin# (for the system) or [.filename]#/usr/local/sbin# (for ports) and call it from a man:sh[1] script in the appropriate [.filename]#rc.d# directory. ==== [TIP] ==== If you would like to learn the details of why [.filename]#rc.d# scripts must be written in the man:sh[1] language, see how [.filename]#/etc/rc# invokes them by means of `run_rc_script`, then study the implementation of `run_rc_script` in [.filename]#/etc/rc.subr#. ==== ➋ In [.filename]#/etc/rc.subr#, a number of man:sh[1] functions are defined for an [.filename]#rc.d# script to use. The functions are documented in man:rc.subr[8]. While it is theoretically possible to write an [.filename]#rc.d# script without ever using man:rc.subr[8], its functions prove extremely handy and make the job an order of magnitude easier. So it is no surprise that everybody resorts to man:rc.subr[8] in [.filename]#rc.d# scripts. We are not going to be an exception. An [.filename]#rc.d# script must "source"[.filename]#/etc/rc.subr# (include it using "`.`") _before_ it calls man:rc.subr[8] functions so that man:sh[1] has an opportunity to learn the functions. The preferred style is to source [.filename]#/etc/rc.subr# first of all. [NOTE] ==== Some useful functions related to networking are provided by another include file, [.filename]#/etc/network.subr#. ==== ➌ [[name-var]]The mandatory variable `name` specifies the name of our script. It is required by man:rc.subr[8]. That is, each [.filename]#rc.d# script _must_ set `name` before it calls man:rc.subr[8] functions. Now it is the right time to choose a unique name for our script once and for all. We will use it in a number of places while developing the script. For a start, let us give the same name to the script file, too. [NOTE] ==== The current style of [.filename]#rc.d# scripting is to enclose values assigned to variables in double quotes. Keep in mind that it is just a style issue that may not always be applicable. You can safely omit quotes from around simple words without man:sh[1] metacharacters in them, while in certain cases you will need single quotes to prevent any interpretation of the value by man:sh[1]. A programmer should be able to tell the language syntax from style conventions and use both of them wisely. ==== ➍ The main idea behind man:rc.subr[8] is that an [.filename]#rc.d# script provides handlers, or methods, for man:rc.subr[8] to invoke. In particular, `start`, `stop`, and other arguments to an [.filename]#rc.d# script are handled this way. A method is a man:sh[1] expression stored in a variable named `argument_cmd`, where _argument_ corresponds to what can be specified on the script's command line. We will see later how man:rc.subr[8] provides default methods for the standard arguments. [NOTE] ==== To make the code in [.filename]#rc.d# more uniform, it is common to use `${name}` wherever appropriate. Thus a number of lines can be just copied from one script to another. ==== ➎ We should keep in mind that man:rc.subr[8] provides default methods for the standard arguments. Consequently, we must override a standard method with a no-op man:sh[1] expression if we want it to do nothing. ➏ The body of a sophisticated method can be implemented as a function. It is a good idea to make the function name meaningful. [IMPORTANT] ==== It is strongly recommended to add the prefix `${name}` to the names of all functions defined in our script so they never clash with the functions from man:rc.subr[8] or another common include file. ==== ➐ This call to man:rc.subr[8] loads man:rc.conf[5] variables. Our script makes no use of them yet, but it still is recommended to load man:rc.conf[5] because there can be man:rc.conf[5] variables controlling man:rc.subr[8] itself. ➑ Usually this is the last command in an [.filename]#rc.d# script. It invokes the man:rc.subr[8] machinery to perform the requested action using the variables and methods our script has provided. [[rcng-confdummy]] == A configurable dummy script Now let us add some controls to our dummy script. As you may know, [.filename]#rc.d# scripts are controlled with man:rc.conf[5]. Fortunately, man:rc.subr[8] hides all the complications from us. The following script uses man:rc.conf[5] via man:rc.subr[8] to see whether it is enabled in the first place, and to fetch a message to show at boot time. These two tasks in fact are independent. On the one hand, an [.filename]#rc.d# script can just support enabling and disabling its service. On the other hand, a mandatory [.filename]#rc.d# script can have configuration variables. We will do both things in the same script though: [.programlisting] .... #!/bin/sh . /etc/rc.subr name=dummy rcvar=dummy_enable <.> start_cmd="${name}_start" stop_cmd=":" load_rc_config $name <.> : ${dummy_enable:=no} <.> : ${dummy_msg="Nothing started."} <.> dummy_start() { echo "$dummy_msg" <.> } run_rc_command "$1" .... What changed in this example? ➊ The variable `rcvar` specifies the name of the ON/OFF knob variable. ➋ Now `load_rc_config` is invoked earlier in the script, before any man:rc.conf[5] variables are accessed. [NOTE] ==== While examining [.filename]#rc.d# scripts, keep in mind that man:sh[1] defers the evaluation of expressions in a function until the latter is called. Therefore it is not an error to invoke `load_rc_config` as late as just before `run_rc_command` and still access man:rc.conf[5] variables from the method functions exported to `run_rc_command`. This is because the method functions are to be called by `run_rc_command`, which is invoked _after_ `load_rc_config`. ==== ➌ A warning will be emitted by `run_rc_command` if `rcvar` itself is set, but the indicated knob variable is unset. If your [.filename]#rc.d# script is for the base system, you should add a default setting for the knob to [.filename]#/etc/defaults/rc.conf# and document it in man:rc.conf[5]. Otherwise it is your script that should provide a default setting for the knob. The canonical approach to the latter case is shown in the example. [NOTE] ==== You can make man:rc.subr[8] act as though the knob is set to `ON`, irrespective of its current setting, by prefixing the argument to the script with `one` or `force`, as in `onestart` or `forcestop`. Keep in mind though that `force` has other dangerous effects we will touch upon below, while `one` just overrides the ON/OFF knob. E.g., assume that `dummy_enable` is `OFF`. The following command will run the `start` method in spite of the setting: [source,shell] .... # /etc/rc.d/dummy onestart .... ==== ➍ Now the message to be shown at boot time is no longer hard-coded in the script. It is specified by an man:rc.conf[5] variable named `dummy_msg`. This is a trivial example of how man:rc.conf[5] variables can control an [.filename]#rc.d# script. [IMPORTANT] ==== The names of all man:rc.conf[5] variables used exclusively by our script _must_ have the same prefix: `${name}_`. For example: `dummy_mode`, `dummy_state_file`, and so on. ==== [NOTE] ==== While it is possible to use a shorter name internally, e.g., just `msg`, adding the unique prefix `${name}_` to all global names introduced by our script will save us from possible collisions with the man:rc.subr[8] namespace. As a rule, [.filename]#rc.d# scripts of the base system need not provide defaults for their man:rc.conf[5] variables because the defaults should be set in [.filename]#/etc/defaults/rc.conf# instead. On the other hand, [.filename]#rc.d# scripts for ports should provide the defaults as shown in the example. ==== ➎ Here we use `dummy_msg` to actually control our script, i.e., to emit a variable message. Use of a shell function is overkill here, since it only runs a single command; an equally valid alternative is: [.programlisting] .... start_cmd="echo \"$dummy_msg\"" .... [[rcng-daemon]] == Startup and shutdown of a simple daemon We said earlier that man:rc.subr[8] could provide default methods. Obviously, such defaults cannot be too general. They are suited for the common case of starting and shutting down a simple daemon program. Let us assume now that we need to write an [.filename]#rc.d# script for such a daemon called `mumbled`. Here it is: [.programlisting] .... #!/bin/sh . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" <.> load_rc_config $name run_rc_command "$1" .... Pleasingly simple, isn't it? Let us examine our little script. The only new thing to note is as follows: ➊ The `command` variable is meaningful to man:rc.subr[8]. If it is set, man:rc.subr[8] will act according to the scenario of serving a conventional daemon. In particular, the default methods will be provided for such arguments: `start`, `stop`, `restart`, `poll`, and `status`. The daemon will be started by running `$command` with command-line flags specified by `$mumbled_flags`. Thus all the input data for the default `start` method are available in the variables set by our script. Unlike `start`, other methods may require additional information about the process started. For instance, `stop` must know the PID of the process to terminate it. In the present case, man:rc.subr[8] will scan through the list of all processes, looking for a process with its name equal to `procname`. The latter is another variable of meaning to man:rc.subr[8], and its value defaults to that of `command`. In other words, when we set `command`, `procname` is effectively set to the same value. This enables our script to kill the daemon and to check if it is running in the first place. [NOTE] ==== Some programs are in fact executable scripts. The system runs such a script by starting its interpreter and passing the name of the script to it as a command-line argument. This is reflected in the list of processes, which can confuse man:rc.subr[8]. You should additionally set `command_interpreter` to let man:rc.subr[8] know the actual name of the process if `$command` is a script. For each [.filename]#rc.d# script, there is an optional man:rc.conf[5] variable that takes precedence over `command`. Its name is constructed as follows: `${name}_program`, where `name` is the mandatory variable we discussed <>. E.g., in this case it will be `mumbled_program`. It is man:rc.subr[8] that arranges `${name}_program` to override `command`. Of course, man:sh[1] will permit you to set `${name}_program` from man:rc.conf[5] or the script itself even if `command` is unset. In that case, the special properties of `${name}_program` are lost, and it becomes an ordinary variable your script can use for its own purposes. However, the sole use of `${name}_program` is discouraged because using it together with `command` became an idiom of [.filename]#rc.d# scripting. ==== For more detailed information on default methods, refer to man:rc.subr[8]. [[rcng-daemon-adv]] == Startup and shutdown of an advanced daemon Let us add some meat onto the bones of the previous script and make it more complex and featureful. The default methods can do a good job for us, but we may need some of their aspects tweaked. Now we will learn how to tune the default methods to our needs. [.programlisting] .... #!/bin/sh . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" command_args="mock arguments > /dev/null 2>&1" <.> pidfile="/var/run/${name}.pid" <.> required_files="/etc/${name}.conf /usr/share/misc/${name}.rules" <.> sig_reload="USR1" <.> start_precmd="${name}_prestart" <.> stop_postcmd="echo Bye-bye" <.> extra_commands="reload plugh xyzzy" <.> plugh_cmd="mumbled_plugh" <.> xyzzy_cmd="echo 'Nothing happens.'" mumbled_prestart() { if checkyesno mumbled_smart; then <.> rc_flags="-o smart ${rc_flags}" <.> fi case "$mumbled_mode" in foo) rc_flags="-frotz ${rc_flags}" ;; bar) rc_flags="-baz ${rc_flags}" ;; *) warn "Invalid value for mumbled_mode" <.> return 1 <.> ;; esac run_rc_command xyzzy <.> return 0 } mumbled_plugh() <.> { echo 'A hollow voice says "plugh".' } load_rc_config $name run_rc_command "$1" .... ➊ Additional arguments to `$command` can be passed in `command_args`. They will be added to the command line after `$mumbled_flags`. Since the final command line is passed to `eval` for its actual execution, input and output redirections can be specified in `command_args`. [NOTE] ==== _Never_ include dashed options, like `-X` or `--foo`, in `command_args`. The contents of `command_args` will appear at the end of the final command line, hence they are likely to follow arguments present in `${name}_flags`; but most commands will not recognize dashed options after ordinary arguments. A better way of passing additional options to `$command` is to add them to the beginning of `${name}_flags`. Another way is to modify `rc_flags` <>. ==== ➋ A good-mannered daemon should create a _pidfile_ so that its process can be found more easily and reliably. The variable `pidfile`, if set, tells man:rc.subr[8] where it can find the pidfile for its default methods to use. [NOTE] ==== In fact, man:rc.subr[8] will also use the pidfile to see if the daemon is already running before starting it. This check can be skipped by using the `faststart` argument. ==== ➌ If the daemon cannot run unless certain files exist, just list them in `required_files`, and man:rc.subr[8] will check that those files do exist before starting the daemon. There also are `required_dirs` and `required_vars` for directories and environment variables, respectively. They all are described in detail in man:rc.subr[8]. [NOTE] ==== The default method from man:rc.subr[8] can be forced to skip the prerequisite checks by using `forcestart` as the argument to the script. ==== ➍ We can customize signals to send to the daemon in case they differ from the well-known ones. In particular, `sig_reload` specifies the signal that makes the daemon reload its configuration; it is SIGHUP by default. Another signal is sent to stop the daemon process; the default is SIGTERM, but this can be changed by setting `sig_stop` appropriately. [NOTE] ==== The signal names should be specified to man:rc.subr[8] without the `SIG` prefix, as it is shown in the example. The FreeBSD version of man:kill[1] can recognize the `SIG` prefix, but the versions from other OS types may not. ==== ➎➏ Performing additional tasks before or after the default methods is easy. For each command-argument supported by our script, we can define `argument_precmd` and `argument_postcmd`. These man:sh[1] commands are invoked before and after the respective method, as it is evident from their names. [NOTE] ==== Overriding a default method with a custom `argument_cmd` still does not prevent us from making use of `argument_precmd` or `argument_postcmd` if we need to. In particular, the former is good for checking custom, sophisticated conditions that should be met before performing the command itself. Using `argument_precmd` along with `argument_cmd` lets us logically separate the checks from the action. Do not forget that you can cram any valid man:sh[1] expressions into the methods, pre-, and post-commands you define. Just invoking a function that makes the real job is a good style in most cases, but never let style limit your understanding of what is going on behind the curtain. ==== ➐ If we would like to implement custom arguments, which can also be thought of as _commands_ to our script, we need to list them in `extra_commands` and provide methods to handle them. [NOTE] ==== The `reload` command is special. On the one hand, it has a preset method in man:rc.subr[8]. On the other hand, `reload` is not offered by default. The reason is that not all daemons use the same reload mechanism and some have nothing to reload at all. So we need to ask explicitly that the builtin functionality be provided. We can do so via `extra_commands`. What do we get from the default method for `reload`? Quite often daemons reload their configuration upon reception of a signal - typically, SIGHUP. Therefore man:rc.subr[8] attempts to reload the daemon by sending a signal to it. The signal is preset to SIGHUP but can be customized via `sig_reload` if necessary. ==== ➑⓮ Our script supports two non-standard commands, `plugh` and `xyzzy`. We saw them listed in `extra_commands`, and now it is time to provide methods for them. The method for `xyzzy` is just inlined while that for `plugh` is implemented as the `mumbled_plugh` function. Non-standard commands are not invoked during startup or shutdown. Usually they are for the system admin's convenience. They can also be used from other subsystems, e.g., man:devd[8] if specified in man:devd.conf[5]. The full list of available commands can be found in the usage line printed by man:rc.subr[8] when the script is invoked without arguments. For example, here is the usage line from the script under study: [source,shell] .... # /etc/rc.d/mumbled Usage: /etc/rc.d/mumbled [fast|force|one](start|stop|restart|rcvar|reload|plugh|xyzzy|status|poll) .... ⓭ A script can invoke its own standard or non-standard commands if needed. This may look similar to calling functions, but we know that commands and shell functions are not always the same thing. For instance, `xyzzy` is not implemented as a function here. In addition, there can be a pre-command and post-command, which should be invoked orderly. So the proper way for a script to run its own command is by means of man:rc.subr[8], as shown in the example. ➒ A handy function named `checkyesno` is provided by man:rc.subr[8]. It takes a variable name as its argument and returns a zero exit code if and only if the variable is set to `YES`, or `TRUE`, or `ON`, or `1`, case insensitive; a non-zero exit code is returned otherwise. In the latter case, the function tests the variable for being set to `NO`, `FALSE`, `OFF`, or `0`, case insensitive; it prints a warning message if the variable contains anything else, i.e., junk. Keep in mind that for man:sh[1] a zero exit code means true and a non-zero exit code means false. [IMPORTANT] ==== The `checkyesno` function takes a __variable name__. Do not pass the expanded _value_ of a variable to it; it will not work as expected. The following is the correct usage of `checkyesno`: [.programlisting] .... if checkyesno mumbled_enable; then foo fi .... On the contrary, calling `checkyesno` as shown below will not work - at least not as expected: [.programlisting] .... if checkyesno "${mumbled_enable}"; then foo fi .... ==== ➓ [[rc-flags]]We can affect the flags to be passed to `$command` by modifying `rc_flags` in `$start_precmd`. ⓫ In certain cases we may need to emit an important message that should go to `syslog` as well. This can be done easily with the following man:rc.subr[8] functions: `debug`, `info`, `warn`, and `err`. The latter function then exits the script with the code specified. ⓬ The exit codes from methods and their pre-commands are not just ignored by default. If `argument_precmd` returns a non-zero exit code, the main method will not be performed. In turn, `argument_postcmd` will not be invoked unless the main method returns a zero exit code. [NOTE] ==== However, man:rc.subr[8] can be instructed from the command line to ignore those exit codes and invoke all commands anyway by prefixing an argument with `force`, as in `forcestart`. ==== [[rcng-hookup]] == Connecting a script to the rc.d framework After a script has been written, it needs to be integrated into [.filename]#rc.d#. The crucial step is to install the script in [.filename]#/etc/rc.d# (for the base system) or [.filename]#/usr/local/etc/rc.d# (for ports). Both [.filename]#bsd.prog.mk# and [.filename]#bsd.port.mk# provide convenient hooks for that, and usually you do not have to worry about the proper ownership and mode. System scripts should be installed from [.filename]#src/etc/rc.d# through the [.filename]#Makefile# found there. Port scripts can be installed using `USE_RC_SUBR` as described link:{porters-handbook}#rc-scripts[in the Porter's Handbook]. However, we should consider beforehand the place of our script in the system startup sequence. The service handled by our script is likely to depend on other services. For instance, a network daemon cannot function without the network interfaces and routing up and running. Even if a service seems to demand nothing, it can hardly start before the basic filesystems have been checked and mounted. We mentioned man:rcorder[8] already. Now it is time to have a close look at it. In a nutshell, man:rcorder[8] takes a set of files, examines their contents, and prints a dependency-ordered list of files from the set to `stdout`. The point is to keep dependency information _inside_ the files so that each file can speak for itself only. A file can specify the following information: * the names of the "conditions" (which means services to us) it __provides__; * the names of the "conditions" it __requires__; * the names of the "conditions" this file should run __before__; * additional _keywords_ that can be used to select a subset from the whole set of files (man:rcorder[8] can be instructed via options to include or omit the files having particular keywords listed.) It is no surprise that man:rcorder[8] can handle only text files with a syntax close to that of man:sh[1]. That is, special lines understood by man:rcorder[8] look like man:sh[1] comments. The syntax of such special lines is rather rigid to simplify their processing. See man:rcorder[8] for details. Besides using man:rcorder[8] special lines, a script can insist on its dependency upon another service by just starting it forcibly. This can be needed when the other service is optional and will not start by itself because the system admin has disabled it mistakenly in man:rc.conf[5]. With this general knowledge in mind, let us consider the simple daemon script enhanced with dependency stuff: [.programlisting] .... #!/bin/sh # PROVIDE: mumbled oldmumble <.> # REQUIRE: DAEMON cleanvar frotz <.> # BEFORE: LOGIN <.> # KEYWORD: nojail shutdown <.> . /etc/rc.subr name=mumbled rcvar=mumbled_enable command="/usr/sbin/${name}" start_precmd="${name}_prestart" mumbled_prestart() { if ! checkyesno frotz_enable && \ ! /etc/rc.d/frotz forcestatus 1>/dev/null 2>&1; then force_depend frotz || return 1 <.> fi return 0 } load_rc_config $name run_rc_command "$1" .... As before, detailed analysis follows: ➊ That line declares the names of "conditions" our script provides. Now other scripts can record a dependency on our script by those names. [NOTE] ==== Usually a script specifies a single condition provided. However, nothing prevents us from listing several conditions there, e.g., for compatibility reasons. In any case, the name of the main, or the only, `PROVIDE:` condition should be the same as `${name}`. ==== ➋➌ So our script indicates which "conditions" provided by other scripts it depends on. According to the lines, our script asks man:rcorder[8] to put it after the script(s) providing [.filename]#DAEMON# and [.filename]#cleanvar#, but before that providing [.filename]#LOGIN#. [NOTE] ==== The `BEFORE:` line should not be abused to work around an incomplete dependency list in the other script. The appropriate case for using `BEFORE:` is when the other script does not care about ours, but our script can do its task better if run before the other one. A typical real-life example is the network interfaces vs. the firewall: While the interfaces do not depend on the firewall in doing their job, the system security will benefit from the firewall being ready before there is any network traffic. Besides conditions corresponding to a single service each, there are meta-conditions and their "placeholder" scripts used to ensure that certain groups of operations are performed before others. These are denoted by [.filename]#UPPERCASE# names. Their list and purposes can be found in man:rc[8]. Keep in mind that putting a service name in the `REQUIRE:` line does not guarantee that the service will actually be running by the time our script starts. The required service may fail to start or just be disabled in man:rc.conf[5]. Obviously, man:rcorder[8] cannot track such details, and man:rc[8] will not do that either. Consequently, the application started by our script should be able to cope with any required services being unavailable. In certain cases, we can help it as discussed <> ==== [[keywords]]➍ As we remember from the above text, man:rcorder[8] keywords can be used to select or leave out some scripts. Namely any man:rcorder[8] consumer can specify through `-k` and `-s` options which keywords are on the "keep list" and "skip list", respectively. From all the files to be dependency sorted, man:rcorder[8] will pick only those having a keyword from the keep list (unless empty) and not having a keyword from the skip list. In FreeBSD, man:rcorder[8] is used by [.filename]#/etc/rc# and [.filename]#/etc/rc.shutdown#. These two scripts define the standard list of FreeBSD [.filename]#rc.d# keywords and their meanings as follows: [[forcedep]]➎ To begin with, `force_depend` should be used with much care. It is generally better to revise the hierarchy of configuration variables for your [.filename]#rc.d# scripts if they are interdependent. If you still cannot do without `force_depend`, the example offers an idiom of how to invoke it conditionally. In the example, our `mumbled` daemon requires that another one, `frotz`, be started in advance. However, `frotz` is optional, too; and man:rcorder[8] knows nothing about such details. Fortunately, our script has access to all man:rc.conf[5] variables. If `frotz_enable` is true, we hope for the best and rely on [.filename]#rc.d# to have started `frotz`. Otherwise we forcibly check the status of `frotz`. Finally, we enforce our dependency on `frotz` if it is found to be not running. A warning message will be emitted by `force_depend` because it should be invoked only if a misconfiguration has been detected. [[rcng-args]] == Giving more flexibility to an rc.d script When invoked during startup or shutdown, an [.filename]#rc.d# script is supposed to act on the entire subsystem it is responsible for. E.g., [.filename]#/etc/rc.d/netif# should start or stop all network interfaces described by man:rc.conf[5]. Either task can be uniquely indicated by a single command argument such as `start` or `stop`. Between startup and shutdown, [.filename]#rc.d# scripts help the admin to control the running system, and it is when the need for more flexibility and precision arises. For instance, the admin may want to add the settings of a new network interface to man:rc.conf[5] and then to start it without interfering with the operation of the existing interfaces. Next time the admin may need to shut down a single network interface. In the spirit of the command line, the respective [.filename]#rc.d# script calls for an extra argument, the interface name. Fortunately, man:rc.subr[8] allows for passing any number of arguments to script's methods (within the system limits). Due to that, the changes in the script itself can be minimal. How can man:rc.subr[8] gain access to the extra command-line arguments. Should it just grab them directly? Not by any means. Firstly, an man:sh[1] function has no access to the positional parameters of its caller, but man:rc.subr[8] is just a sack of such functions. Secondly, the good manner of [.filename]#rc.d# dictates that it is for the main script to decide which arguments are to be passed to its methods. So the approach adopted by man:rc.subr[8] is as follows: `run_rc_command` passes on all its arguments but the first one to the respective method verbatim. The first, omitted, argument is the name of the method itself: `start`, `stop`, etc. It will be shifted out by `run_rc_command`, so what is `$2` in the original command line will be presented as `$1` to the method, and so on. To illustrate this opportunity, let us modify the primitive dummy script so that its messages depend on the additional arguments supplied. Here we go: [.programlisting] .... #!/bin/sh . /etc/rc.subr name="dummy" start_cmd="${name}_start" stop_cmd=":" kiss_cmd="${name}_kiss" extra_commands="kiss" dummy_start() { if [ $# -gt 0 ]; then <.> echo "Greeting message: $*" else echo "Nothing started." fi } dummy_kiss() { echo -n "A ghost gives you a kiss" if [ $# -gt 0 ]; then <.> echo -n " and whispers: $*" fi case "$*" in *[.!?]) echo ;; *) echo . ;; esac } load_rc_config $name run_rc_command "$@" <.> .... What essential changes can we notice in the script? ➊ All arguments you type after `start` can end up as positional parameters to the respective method. We can use them in any way according to our task, skills, and fancy. In the current example, we just pass all of them to man:echo[1] as one string in the next line - note `$*` within the double quotes. Here is how the script can be invoked now: [source,shell] .... # /etc/rc.d/dummy start Nothing started. # /etc/rc.d/dummy start Hello world! Greeting message: Hello world! .... ➋ The same applies to any method our script provides, not only to a standard one. We have added a custom method named `kiss`, and it can take advantage of the extra arguments not less than `start` does. E.g.: [source,shell] .... # /etc/rc.d/dummy kiss A ghost gives you a kiss. # /etc/rc.d/dummy kiss Once I was Etaoin Shrdlu... A ghost gives you a kiss and whispers: Once I was Etaoin Shrdlu... .... ➌ If we want just to pass all extra arguments to any method, we can merely substitute `"$@"` for `"$1"` in the last line of our script, where we invoke `run_rc_command`. [IMPORTANT] ==== An man:sh[1] programmer ought to understand the subtle difference between `$*` and `$@` as the ways to designate all positional parameters. For its in-depth discussion, refer to a good handbook on man:sh[1] scripting. _Do not_ use the expressions until you fully understand them because their misuse will result in buggy and insecure scripts. ==== [NOTE] ==== Currently `run_rc_command` may have a bug that prevents it from keeping the original boundaries between arguments. That is, arguments with embedded whitespace may not be processed correctly. The bug stems from `$*` misuse. ==== [[rcng-furthur]] == Further reading [[lukem]]http://www.mewburn.net/luke/papers/rc.d.pdf[The original article by Luke Mewburn] offers a general overview of [.filename]#rc.d# and detailed rationale for its design decisions. It provides insight on the whole [.filename]#rc.d# framework and its place in a modern BSD operating system. [[manpages]]The manual pages man:rc[8], man:rc.subr[8], and man:rcorder[8] document the [.filename]#rc.d# components in great detail. You cannot fully use the [.filename]#rc.d# power without studying the manual pages and referring to them while writing your own scripts. The major source of working, real-life examples is [.filename]#/etc/rc.d# in a live system. Its contents are easy and pleasant to read because most rough corners are hidden deep in man:rc.subr[8]. Keep in mind though that the [.filename]#/etc/rc.d# scripts were not written by angels, so they might suffer from bugs and suboptimal design decisions. Now you can improve them! diff --git a/documentation/content/en/articles/releng/_index.adoc b/documentation/content/en/articles/releng/_index.adoc index 665c3cea45..ef1c6c5035 100644 --- a/documentation/content/en/articles/releng/_index.adoc +++ b/documentation/content/en/articles/releng/_index.adoc @@ -1,444 +1,444 @@ --- -title: FreeBSD Release Engineering +title: Legacy FreeBSD Release Engineering authors: - author: Murray Stokely email: murray@FreeBSD.org webpage: https://people.FreeBSD.org/~murray/ -description: This paper describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System +description: This paper describes the approach used by past the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System trademarks: ["freebsd", "intel", "general"] --- = FreeBSD Release Engineering :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: :xrefstyle: full include::shared/releases.adoc[] include::shared/authors.adoc[] include::shared/en/teams.adoc[lines=16..-1] include::shared/en/mailing-lists.adoc[] include::shared/en/urls.adoc[] ifeval::["{backend}" == "html5"] :imagesdir: ../../../images/articles/releng/ endif::[] ifeval::["{backend}" == "pdf"] :imagesdir: ../../../../static/images/articles/releng/ endif::[] ifeval::["{backend}" == "epub3"] :imagesdir: ../../../../static/images/articles/releng/ endif::[] [.abstract-title] Abstract [NOTE] ==== This document is outdated and does not accurately describe the current release procedures of the FreeBSD Release Engineering team. It is retained for historical purposes. The current procedures used by the FreeBSD Release Engineering team are available in the link:{freebsd-releng}[FreeBSD Release Engineering] article. ==== This paper describes the approach used by the FreeBSD release engineering team to make production quality releases of the FreeBSD Operating System. It details the methodology used for the official FreeBSD releases and describes the tools available for those interested in producing customized FreeBSD releases for corporate rollouts or commercial productization. ''' toc::[] [[introduction]] == Introduction The development of FreeBSD is a very open process. FreeBSD is comprised of contributions from thousands of people around the world. The FreeBSD Project provides Subversion footnote:[Subversion, http://subversion.apache.org] access to the general public so that others can have access to log messages, diffs (patches) between development branches, and other productivity enhancements that formal source code management provides. This has been a huge help in attracting more talented developers to FreeBSD. However, I think everyone would agree that chaos would soon manifest if write access to the main repository was opened up to everyone on the Internet. Therefore only a "select" group of nearly 300 people are given write access to the Subversion repository. These link:{contributors}#staff-committers[FreeBSD committers]footnote:[link:{contributors}#staff-committers[FreeBSD committers]] are usually the people who do the bulk of FreeBSD development. An elected link:https://www.FreeBSD.org/administration/#t-core[Core Team]footnote:[link:https://www.FreeBSD.org/administration/#t-core[FreeBSD Core Team]] of developers provide some level of direction over the project. The rapid pace of `FreeBSD` development makes the main development branch unsuitable for the everyday use by the general public. In particular, stabilizing efforts are required for polishing the development system into a production quality release. To solve this conflict, development continues on several parallel tracks. The main development branch is the _HEAD_ or _trunk_ of our Subversion tree, known as "FreeBSD-CURRENT" or "-CURRENT" for short. A set of more stable branches are maintained, known as "FreeBSD-STABLE" or "-STABLE" for short. All branches live in a master Subversion repository maintained by the FreeBSD Project. FreeBSD-CURRENT is the "bleeding-edge" of FreeBSD development where all new changes first enter the system. FreeBSD-STABLE is the development branch from which major releases are made. Changes go into this branch at a different pace, and with the general assumption that they have first gone into FreeBSD-CURRENT and have been thoroughly tested by our user community. The term _stable_ in the name of the branch refers to the presumed Application Binary Interface stability, which is promised by the project. This means that a user application compiled on an older version of the system from the same branch works on a newer system from the same branch. The ABI stability has improved greatly from the compared to previous releases. In most cases, binaries from the older _STABLE_ systems run unmodified on newer systems, including __HEAD__, assuming that the system management interfaces are not used. In the interim period between releases, weekly snapshots are built automatically by the FreeBSD Project build machines and made available for download from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/`. The widespread availability of binary release snapshots, and the tendency of our user community to keep up with -STABLE development with Subversion and "`make buildworld`" footnote:[link:{handbook}#makeworld[Rebuilding world]] helps to keep FreeBSD-STABLE in a very reliable condition even before the quality assurance activities ramp up pending a major release. In addition to installation ISO snapshots, weekly virtual machine images are also provided for use with VirtualBox, qemu, or other popular emulation software. The virtual machine images can be downloaded from `ftp://ftp.FreeBSD.org/pub/FreeBSD/snapshots/VM-IMAGES/`. The virtual machine images are approximately 150MB man:xz[1] compressed, and contain a 10GB sparse filesystem when attached to a virtual machine. Bug reports and feature requests are continuously submitted by users throughout the release cycle. Problems reports are entered into our Bugzilla database through the web interface provided at https://www.freebsd.org/support/bugreports/[https://www.freebsd.org/support/bugreports/]. To service our most conservative users, individual release branches were introduced with FreeBSD 4.3. These release branches are created shortly before a final release is made. After the release goes out, only the most critical security fixes and additions are merged onto the release branch. In addition to source updates via Subversion, binary patchkits are available to keep systems on the _releng/X.Y_ branches updated. === What This Article Describes The following sections of this article describe: <>:: The different phases of the release engineering process leading up to the actual system build. <>:: The actual build process. <>:: How the base release may be extended by third parties. <>:: Some of the lessons learned through the release of FreeBSD 4.4. <>:: Future directions of development. [[release-proc]] == Release Process New releases of FreeBSD are released from the -STABLE branch at approximately four month intervals. The FreeBSD release process begins to ramp up 70-80 days before the anticipated release date when the release engineer sends an email to the development mailing lists to remind developers that they only have 15 days to integrate new changes before the code freeze. During this time, many developers perform what have become known as "MFC sweeps". MFC stands for "Merge From CURRENT" and it describes the process of merging a tested change from our -CURRENT development branch to our -STABLE branch. Project policy requires any change to be first applied to trunk, and merged to the -STABLE branches after sufficient external testing was done by -CURRENT users (developers are expected to extensively test the change before committing to -CURRENT, but it is impossible for a person to exercise all usages of the general-purpose operating system). Minimal MFC period is 3 days, which is typically used only for trivial or critical bugfixes. === Code Review Sixty days before the anticipated release, the source repository enters a "code freeze". During this time, all commits to the -STABLE branch must be approved by `{re}`. The approval process is technically enforced by a pre-commit hook. The kinds of changes that are allowed during this period include: * Bug fixes. * Documentation updates. * Security-related fixes of any kind. * Minor changes to device drivers, such as adding new Device IDs. * Driver updates from the vendors. * Any additional change that the release engineering team feels is justified, given the potential risk. Shortly after the code freeze is started, a _BETA1_ image is built and released for widespread testing. During the code freeze, at least one beta image or release candidate is released every two weeks until the final release is ready. During the days preceding the final release, the release engineering team is in constant communication with the security-officer team, the documentation maintainers, and the port maintainers to ensure that all of the different components required for a successful release are available. After the quality of the BETA images is satisfying enough, and no large and potentially risky changes are planned, the release branch is created and _Release Candidate_ (RC) images are built from the release branch, instead of the BETA images from the STABLE branch. Also, the freeze on the STABLE branch is lifted and release branch enters a "hard code freeze" where it becomes much harder to justify new changes to the system unless a serious bug-fix or security issue is involved. === Final Release Checklist When several BETA images have been made available for widespread testing and all major issues have been resolved, the final release "polishing" can begin. [[rel-branch]] ==== Creating the Release Branch [NOTE] ==== In all examples below, `$FSVN` refers to the location of the FreeBSD Subversion repository, `svn+ssh://svn.FreeBSD.org/base/`. ==== The layout of FreeBSD branches in Subversion is described in the link:{committers-guide}#subversion-primer-base-layout[Committer's Guide]. The first step in creating a branch is to identify the revision of the `stable/_X_` sources that you want to branch _from_. [source,shell] .... # svn log -v $FSVN/stable/9 .... The next step is to create the _release branch_ [source,shell] .... # svn cp $FSVN/stable/9@REVISION $FSVN/releng/9.2 .... This branch can be checked out: [source,shell] .... # svn co $FSVN/releng/9.2 src .... [NOTE] ==== Creating the `releng` branch and `release` tags is done by the link:https://www.FreeBSD.org/administration/#t-re[Release Engineering Team]. ==== image::branches-head.png[FreeBSD Development Branch] image::branches-releng3.png[FreeBSD 3.x STABLE Branch] image::branches-releng4.png[FreeBSD 4.x STABLE Branch] image::branches-releng5.png[FreeBSD 5.x STABLE Branch] image::branches-releng6.png[FreeBSD 6.x STABLE Branch] image::branches-releng7.png[FreeBSD 7.x STABLE Branch] image::branches-releng8.png[FreeBSD 8.x STABLE Branch] image::branches-releng9.png[FreeBSD 9.x STABLE Branch] [[versionbump]] ==== Bumping up the Version Number Before the final release can be tagged, built, and released, the following files need to be modified to reflect the correct version of FreeBSD: * [.filename]#doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.xml# * [.filename]#doc/en_US.ISO8859-1/books/porters-handbook/book.xml# * [.filename]#doc/en_US.ISO8859-1/htdocs/cgi/ports.cgi# * [.filename]#ports/Tools/scripts/release/config# * [.filename]#doc/shared/xml/freebsd.ent# * [.filename]#src/Makefile.inc1# * [.filename]#src/UPDATING# * [.filename]#src/gnu/usr.bin/groff/tmac/mdoc.local# * [.filename]#src/release/Makefile# * [.filename]#src/release/doc/en_US.ISO8859-1/shared/xml/release.dsl# * [.filename]#src/release/doc/shared/examples/Makefile.relnotesng# * [.filename]#src/release/doc/shared/xml/release.ent# * [.filename]#src/sys/conf/newvers.sh# * [.filename]#src/sys/sys/param.h# * [.filename]#src/usr.sbin/pkg_install/add/main.c# * [.filename]#doc/en_US.ISO8859-1/htdocs/search/opensearch/man.xml# The release notes and errata files also need to be adjusted for the new release (on the release branch) and truncated appropriately (on the stable/current branch): * [.filename]#src/release/doc/en_US.ISO8859-1/relnotes/common/new.xml# * [.filename]#src/release/doc/en_US.ISO8859-1/errata/article.xml# Sysinstall should be updated to note the number of available ports and the amount of disk space required for the Ports Collection. footnote:[FreeBSD Ports Collection https://www.FreeBSD.org/ports] This information is currently kept in [.filename]#src/usr.sbin/bsdinstall/dist.c#. After the release has been built, a number of files should be updated to announce the release to the world. These files are relative to `head/` within the `doc/` subversion tree. * [.filename]#share/images/articles/releng/branches-relengX.pic# * [.filename]#head/shared/xml/release.ent# * [.filename]#en_US.ISO8859-1/htdocs/releases/*# * [.filename]#en_US.ISO8859-1/htdocs/releng/index.xml# * [.filename]#share/xml/news.xml# Additionally, update the "BSD Family Tree" file: * [.filename]#src/shared/misc/bsd-family-tree# ==== Creating the Release Tag When the final release is ready, the following command will create the `release/9.2.0` tag. [source,shell] .... # svn cp $FSVN/releng/9.2 $FSVN/release/9.2.0 .... The Documentation and Ports managers are responsible for tagging their respective trees with the `tags/RELEASE_9_2_0` tag. When the Subversion `svn cp` command is used to create a __release tag__, this identifies the source at a specific point in time. By creating tags, we ensure that future release builders will always be able to use the exact same source we used to create the official FreeBSD Project releases. [[release-build]] == Release Building FreeBSD "releases" can be built by anyone with a fast machine and access to a source repository. (That should be everyone, since we offer Subversion access! See the link:{handbook}#svn[Subversion section in the Handbook] for details.) The _only_ special requirement is that the man:md[4] device must be available. If the device is not loaded into your kernel, then the kernel module should be automatically loaded when man:mdconfig[8] is executed during the boot media creation phase. All of the tools necessary to build a release are available from the Subversion repository in [.filename]#src/release#. These tools aim to provide a consistent way to build FreeBSD releases. A complete release can actually be built with only a single command, including the creation of ISO images suitable for burning to CDROM or DVD, and an FTP install directory. man:release[7] fully documents the `src/release/generate-release.sh` script which is used to build a release. `generate-release.sh` is a wrapper around the Makefile target: `make release`. === Building a Release man:release[7] documents the exact commands required to build a FreeBSD release. The following sequences of commands can build an 9.2.0 release: [source,shell] .... # cd /usr/src/release # sh generate-release.sh release/9.2.0 /local3/release .... After running these commands, all prepared release files are available in [.filename]#/local3/release/R# directory. The release [.filename]#Makefile# can be broken down into several distinct steps. * Creation of a sanitized system environment in a separate directory hierarchy with "`make installworld`". * Checkout from Subversion of a clean version of the system source, documentation, and ports into the release build hierarchy. * Population of [.filename]#/etc# and [.filename]#/dev# in the chrooted environment. * chroot into the release build hierarchy, to make it harder for the outside environment to taint this build. * `make world` in the chrooted environment. * Build of Kerberos-related binaries. * Build [.filename]#GENERIC# kernel. * Creation of a staging directory tree where the binary distributions will be built and packaged. * Build and installation of the documentation toolchain needed to convert the documentation source (SGML) into HTML and text documents that will accompany the release. * Build and installation of the actual documentation (user manuals, tutorials, release notes, hardware compatibility lists, and so on.) * Package up distribution tarballs of the binaries and sources. * Create FTP installation hierarchy. * _(optionally)_ Create ISO images for CDROM/DVD media. For more information about the release build infrastructure, please see man:release[7]. [NOTE] ==== It is important to remove any site-specific settings from [.filename]#/etc/make.conf#. For example, it would be unwise to distribute binaries that were built on a system with `CPUTYPE` set to a specific processor. ==== === Contributed Software ("ports") The https://www.FreeBSD.org/ports[FreeBSD Ports collection] is a collection of over {numports} third-party software packages available for FreeBSD. The `{portmgr}` is responsible for maintaining a consistent ports tree that can be used to create the binary packages that accompany official FreeBSD releases. === Release ISOs Starting with FreeBSD 4.4, the FreeBSD Project decided to release all four ISO images that were previously sold on the _BSDi/Wind River Systems/FreeBSD Mall_ "official" CDROM distributions. Each of the four discs must contain a [.filename]#README.TXT# file that explains the contents of the disc, a [.filename]#CDROM.INF# file that provides meta-data for the disc so that man:bsdinstall[8] can validate and use the contents, and a [.filename]#filename.txt# file that provides a manifest for the disc. This _manifest_ can be created with a simple command: [source,shell] .... /stage/cdrom# find . -type f | sed -e 's/^\.\///' | sort > filename.txt .... The specific requirements of each CD are outlined below. ==== Disc 1 The first disc is almost completely created by `make release`. The only changes that should be made to the [.filename]#disc1# directory are the addition of a [.filename]#tools# directory, and as many popular third party software packages as will fit on the disc. The [.filename]#tools# directory contains software that allow users to create installation floppies from other operating systems. This disc should be made bootable so that users of modern PCs do not need to create installation floppy disks. If a custom kernel of FreeBSD is to be included, then man:bsdinstall[8] and man:release[7] must be updated to include installation instructions. The relevant code is contained in [.filename]#src/release# and [.filename]#src/usr.sbin/bsdinstall#. Specifically, the file [.filename]#src/release/Makefile#, and [.filename]#dist.c#, [.filename]#dist.h#, [.filename]#menus.c#, [.filename]#install.c#, and [.filename]#Makefile# will need to be updated under [.filename]#src/usr.sbin/bsdinstall#. Optionally, you may choose to update [.filename]#bsdinstall.8#. ==== Disc 2 The second disc is also largely created by `make release`. This disc contains a "live filesystem" that can be used from man:bsdinstall[8] to troubleshoot a FreeBSD installation. This disc should be bootable and should also contain a compressed copy of the CVS repository in the [.filename]#CVSROOT# directory and commercial software demos in the [.filename]#commerce# directory. ==== Multi-volume Support Sysinstall supports multiple volume package installations. This requires that each disc have an [.filename]#INDEX# file containing all of the packages on all volumes of a set, along with an extra field that indicates which volume that particular package is on. Each volume in the set must also have the `CD_VOLUME` variable set in the [.filename]#cdrom.inf# file so that bsdinstall can tell which volume is which. When a user attempts to install a package that is not on the current disc, bsdinstall will prompt the user to insert the appropriate one. [[distribution]] == Distribution [[dist-ftp]] === FTP Sites When the release has been thoroughly tested and packaged for distribution, the master FTP site must be updated. The official FreeBSD public FTP sites are all mirrors of a master server that is open only to other FTP sites. This site is known as `ftp-master`. When the release is ready, the following files must be modified on `ftp-master`: [.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/#:: The installable FTP directory as output from `make release`. [.filename]#/pub/FreeBSD/ports/arch/packages-X.Y-release/#:: The complete package build for this release. [.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/tools#:: A symlink to [.filename]#../../../tools#. [.filename]#/pub/FreeBSD/releases/arch/X.Y-RELEASE/packages#:: A symlink to [.filename]#../../../ports/arch/packages-X.Y-release#. [.filename]#/pub/FreeBSD/releases/arch/ISO-IMAGES/X.Y/X.Y-RELEASE-arch-*.iso#:: The ISO images. The "*" is [.filename]#disc1#, [.filename]#disc2#, etc. Only if there is a [.filename]#disc1# and there is an alternative first installation CD (for example a stripped-down install with no windowing system) there may be a [.filename]#mini# as well. For more information about the distribution mirror architecture of the FreeBSD FTP sites, please see the link:{hubs}[Mirroring FreeBSD] article. It may take many hours to two days after updating `ftp-master` before a majority of the Tier-1 FTP sites have the new software depending on whether or not a package set got loaded at the same time. It is imperative that the release engineers coordinate with the {mirror-announce} before announcing the general availability of new software on the FTP sites. Ideally the release package set should be loaded at least four days prior to release day. The release bits should be loaded between 24 and 48 hours before the planned release time with "other" file permissions turned off. This will allow the mirror sites to download it but the general public will not be able to download it from the mirror sites. Mail should be sent to {mirror-announce} at the time the release bits get posted saying the release has been staged and giving the time that the mirror sites should begin allowing access. Be sure to include a time zone with the time, for example make it relative to GMT. [[dist-cdrom]] === CD-ROM Replication Coming soon: Tips for sending FreeBSD ISOs to a replicator and quality assurance measures to be taken. [[extensibility]] == Extensibility Although FreeBSD forms a complete operating system, there is nothing that forces you to use the system exactly as we have packaged it up for distribution. We have tried to design the system to be as extensible as possible so that it can serve as a platform that other commercial products can be built on top of. The only "rule" we have about this is that if you are going to distribute FreeBSD with non-trivial changes, we encourage you to document your enhancements! The FreeBSD community can only help support users of the software we provide. We certainly encourage innovation in the form of advanced installation and administration tools, for example, but we cannot be expected to answer questions about it. === Scripting `bsdinstall` The FreeBSD system installation and configuration tool, man:bsdinstall[8], can be scripted to provide automated installs for large sites. This functionality can be used in conjunction with Intel(R) PXE footnote:[link:{handbook}#network-diskless[Diskless Operation with PXE]] to bootstrap systems from the network. [[lessons-learned]] == Lessons Learned from FreeBSD 4.4 The release engineering process for 4.4 formally began on August 1st, 2001. After that date all commits to the `RELENG_4` branch of FreeBSD had to be explicitly approved by the `{re}`. The first release candidate for the x86 architecture was released on August 16, followed by 4 more release candidates leading up to the final release on September 18th. The security officer was very involved in the last week of the process as several security issues were found in the earlier release candidates. A total of over _500_ emails were sent to the `{re}` in little over a month. Our user community has made it very clear that the security and stability of a FreeBSD release should not be sacrificed for any self-imposed deadlines or target release dates. The FreeBSD Project has grown tremendously over its lifetime and the need for standardized release engineering procedures has never been more apparent. This will become even more important as FreeBSD is ported to new platforms. [[future]] == Future Directions It is imperative for our release engineering activities to scale with our growing userbase. Along these lines we are working very hard to document the procedures involved in producing FreeBSD releases. * _Parallelism_ - Certain portions of the release build are actually "embarrassingly parallel". Most of the tasks are very I/O intensive, so having multiple high-speed disk drives is actually more important than using multiple processors in speeding up the `make release` process. If multiple disks are used for different hierarchies in the man:chroot[2] environment, then the CVS checkout of the [.filename]#ports# and [.filename]#doc# trees can be happening simultaneously as the `make world` on another disk. Using a RAID solution (hardware or software) can significantly decrease the overall build time. * _Cross-building releases_ - Building IA-64 or Alpha release on x86 hardware? `make TARGET=ia64 release`. * _Regression Testing_ - We need better automated correctness testing for FreeBSD. * _Installation Tools_ - Our installation program has long since outlived its intended life span. Several projects are under development to provide a more advanced installation mechanism. The libh project was one such project that aimed to provide an intelligent new package framework and GUI installation program. [[ackno]] == Acknowledgements I would like to thank Jordan Hubbard for giving me the opportunity to take on some of the release engineering responsibilities for FreeBSD 4.4 and also for all of his work throughout the years making FreeBSD what it is today. Of course the release would not have been possible without all of the release-related work done by `{asami}`, `{steve}`, `{bmah}`, `{nik}`, `{obrien}`, `{kris}`, `{jhb}` and the rest of the FreeBSD development community. I would also like to thank `{rgrimes}`, `{phk}`, and others who worked on the release engineering tools in the very early days of FreeBSD. This article was influenced by release engineering documents from the CSRG footnote:[Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic: link:http://docs.FreeBSD.org/44doc/papers/releng.html[The Release Engineering of 4.3BSD]] , the NetBSD Project, footnote:[NetBSD Developer Documentation: Release Engineering http://www.NetBSD.org/developers/releng/index.html] , and John Baldwin's proposed release engineering process notes. footnote:[John Baldwin's FreeBSD Release Engineering Proposal https://people.FreeBSD.org/~jhb/docs/releng.txt] diff --git a/documentation/content/en/articles/remote-install/_index.adoc b/documentation/content/en/articles/remote-install/_index.adoc index e5343b75a9..03d62cad6a 100644 --- a/documentation/content/en/articles/remote-install/_index.adoc +++ b/documentation/content/en/articles/remote-install/_index.adoc @@ -1,376 +1,376 @@ --- title: Remote Installation of the FreeBSD Operating System Without a Remote Console authors: - author: Daniel Gerzo email: danger@FreeBSD.org copyright: 2008-2021 The FreeBSD Documentation Project -description: Remote Installation of the FreeBSD Operating System Without a Remote Console +description: Describes the remote installation of the FreeBSD operating system when the console of the remote system is unavailable trademarks: ["freebsd", "general"] --- = Remote Installation of the FreeBSD Operating System Without a Remote Console :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/authors.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This article documents the remote installation of the FreeBSD operating system when the console of the remote system is unavailable. The main idea behind this article is the result of a collaboration with `{mm}` with valuable input provided by `{pjd}`. ''' toc::[] [[background]] == Background There are many server hosting providers in the world, but very few of them are officially supporting FreeBSD. They usually provide support for a Linux(R) distribution to be installed on the servers they offer. In some cases, these companies will install your preferred Linux(R) distribution if you request it. Using this option, we will attempt to install FreeBSD. In other cases, they may offer a rescue system which would be used in an emergency. It is possible to use this for our purposes as well. This article covers the basic installation and configuration steps required to bootstrap a remote installation of FreeBSD with RAID-1 and ZFS capabilities. [[intro]] == Introduction This section will summarize the purpose of this article and better explain what is covered herein. The instructions included in this article will benefit those using services provided by colocation facilities not supporting FreeBSD. [.procedure] ==== . As we have mentioned in the <> section, many of the reputable server hosting companies provide some kind of rescue system, which is booted from their LAN and accessible over SSH. They usually provide this support in order to help their customers fix broken operating systems. As this article will explain, it is possible to install FreeBSD with the help of these rescue systems. + . The next section of this article will describe how to configure, and build minimalistic FreeBSD on the local machine. That version will eventually be running on the remote machine from a ramdisk, which will allow us to install a complete FreeBSD operating system from an FTP mirror using the sysinstall utility. . The rest of this article will describe the installation procedure itself, as well as the configuration of the ZFS file system. ==== [[requirements]] === Requirements To continue successfully, you must: * Have a network accessible operating system with SSH access * Understand the FreeBSD installation process * Be familiar with the man:sysinstall[8] utility * Have the FreeBSD installation SO image or CD handy [[preparation]] == Preparation - mfsBSD Before FreeBSD may be installed on the target system, it is necessary to build the minimal FreeBSD operating system image which will boot from the hard drive. This way the new system can be accessed from the network, and the rest of the installation can be done without remote access to the system console. The mfsBSD tool-set can be used to build a tiny FreeBSD image. As the name of mfsBSD suggests ("mfs" means "memory file system"), the resulting image runs entirely from a ramdisk. Thanks to this feature, the manipulation of hard drives will not be limited, therefore it will be possible to install a complete FreeBSD operating system. The mfsBSD http://mfsbsd.vx.sk/[home page] includes pointers to the latest release of the toolset. Please note that the internals of mfsBSD and how it all fits together is beyond the scope of this article. The interested reader should consult the original documentation of mfsBSD for more details. Download and extract the latest mfsBSD release and change your working directory to the directory where the mfsBSD scripts will reside: [source,shell] .... # fetch http://mfsbsd.vx.sk/release/mfsbsd-2.1.tar.gz # tar xvzf mfsbsd-2.1.tar.gz # cd mfsbsd-2.1/ .... [[mfsbsd-config]] === Configuration of mfsBSD Before booting mfsBSD, a few important configuration options have to be set. The most important that we have to get right is, naturally, the network setup. The most suitable method to configure networking options depends on whether we know beforehand the type of the network interface we will use, and the network interface driver to be loaded for our hardware. We will see how mfsBSD can be configured in either case. Another important thing to set is the `root` password. This can be done by editing [.filename]#conf/loader.conf#. Please see the included comments. ==== The [.filename]#conf/interfaces.conf# method When the installed network interface card is unknown, it is possible to use the auto-detection features of mfsBSD. The startup scripts of mfsBSD can detect the correct driver to use, based on the MAC address of the interface, if we set the following options in [.filename]#conf/interfaces.conf#: [.programlisting] .... mac_interfaces="ext1" ifconfig_ext1_mac="00:00:00:00:00:00" ifconfig_ext1="inet 192.168.0.2/24" .... Do not forget to add the `defaultrouter` information to [.filename]#conf/rc.conf#: [.programlisting] .... defaultrouter="192.168.0.1" .... ==== The [.filename]#conf/rc.conf# Method When the network interface driver is known, it is more convenient to use [.filename]#conf/rc.conf# for networking options. The syntax of this file is the same as the one used in the standard man:rc.conf[5] file of FreeBSD. For example, if you know that a man:re[4] network interface is going to be available, you can set the following options in [.filename]#conf/rc.conf#: [.programlisting] .... defaultrouter="192.168.0.1" ifconfig_re0="inet 192.168.0.2/24" .... [[mfsbsd-build]] === Building an mfsBSD Image The process of building an mfsBSD image is pretty straightforward. The first step is to mount the FreeBSD installation CD, or the installation ISO image to [.filename]#/cdrom#. For the sake of example, in this article we will assume that you have downloaded the FreeBSD 10.1-RELEASE ISO. Mounting this ISO image to the [.filename]#/cdrom# directory is easy with the man:mdconfig[8] utility: [source,shell] .... # mdconfig -a -t vnode -u 10 -f FreeBSD-10.1-RELEASE-amd64-disc1.iso # mount_cd9660 /dev/md10 /cdrom .... Since the recent FreeBSD releases do not contain regular distribution sets, it is required to extract the FreeBSD distribution files from the distribution archives located on the ISO image: [source,shell] .... # mkdir DIST # tar -xvf /cdrom/usr/freebsd-dist/base.txz -C DIST # tar -xvf /cdrom/usr/freebsd-dist/kernel.txz -C DIST .... Next, build the bootable mfsBSD image: [source,shell] .... # make BASE=DIST .... [NOTE] ==== The above `make` has to be run from the top level of the mfsBSD directory tree, for example [.filename]#~/mfsbsd-2.1/#. ==== === Booting mfsBSD Now that the mfsBSD image is ready, it must be uploaded to the remote system running a live rescue system or pre-installed Linux(R) distribution. The most suitable tool for this task is scp: [source,shell] .... # scp disk.img root@192.168.0.2:. .... To boot mfsBSD image properly, it must be placed on the first (bootable) device of the given machine. This may be accomplished using this example providing that [.filename]#sda# is the first bootable disk device: [source,shell] .... # dd if=/root/disk.img of=/dev/sda bs=1m .... If all went well, the image should now be in the MBR of the first device and the machine can be rebooted. Watch for the machine to boot up properly with the man:ping[8] tool. Once it has came back on-line, it should be possible to access it over man:ssh[1] as user `root` with the configured password. [[installation]] == Installation of the FreeBSD Operating System The mfsBSD has been successfully booted and it should be possible to log in through man:ssh[1]. This section will describe how to create and label slices, set up `gmirror` for RAID-1, and how to use `sysinstall` to install a minimal distribution of the FreeBSD operating system. === Preparation of Hard Drives The first task is to allocate disk space for FreeBSD, i.e.: to create slices and partitions. Obviously, the currently running system is fully loaded in system memory and therefore there will be no problems with manipulating hard drives. To complete this task, it is possible to use either `sysinstall` or man:fdisk[8] in conjunction to man:bsdlabel[8]. At the start, mark all system disks as empty. Repeat the following command for each hard drive: [source,shell] .... # dd if=/dev/zero of=/dev/ad0 count=2 .... Next, create slices and label them with your preferred tool. While it is considered easier to use `sysinstall`, a powerful and also probably less buggy method will be to use standard text-based UNIX(R) tools, such as man:fdisk[8] and man:bsdlabel[8], which will also be covered in this section. The former option is well documented in the link:{handbook}#install-steps[Installing FreeBSD] chapter of the FreeBSD Handbook. As it was mentioned in the introduction, this article will present how to set up a system with RAID-1 and ZFS capabilities. Our set up will consist of a small man:gmirror[8] mirrored [.filename]#/# (root), [.filename]#/usr# and [.filename]#/var# dataset, and the rest of the disk space will be allocated for a man:zpool[8] mirrored ZFS file system. Please note, that the ZFS file system will be configured after the FreeBSD operating system is successfully installed and booted. The following example will describe how to create slices and labels, initialize man:gmirror[8] on each partition and how to create a UFS2 file system in each mirrored partition: [source,shell] .... # fdisk -BI /dev/ad0 <.> # fdisk -BI /dev/ad1 # bsdlabel -wB /dev/ad0s1 <.> # bsdlabel -wB /dev/ad1s1 # bsdlabel -e /dev/ad0s1 <.> # bsdlabel /dev/ad0s1 > /tmp/bsdlabel.txt && bsdlabel -R /dev/ad1s1 /tmp/bsdlabel.txt <.> # gmirror label root /dev/ad[01]s1a <.> # gmirror label var /dev/ad[01]s1d # gmirror label usr /dev/ad[01]s1e # gmirror label -F swap /dev/ad[01]s1b <.> # newfs /dev/mirror/root <.> # newfs /dev/mirror/var # newfs /dev/mirror/usr .... <.> Create a slice covering the entire disk and initialize the boot code contained in sector 0 of the given disk. Repeat this command for all hard drives in the system. <.> Write a standard label for each disk including the bootstrap code. <.> Now, manually edit the label of the given disk. Refer to the man:bsdlabel[8] manual page in order to find out how to create partitions. Create partitions `a` for [.filename]#/# (root) file system, `b` for swap, `d` for [.filename]#/var#, `e` for [.filename]#/usr# and finally `f` which will later be used for ZFS. <.> Import the recently created label for the second hard drive, so both hard drives will be labeled in the same way. <.> Initialize man:gmirror[8] on each partition. <.> Note that `-F` is used for the swap partition. This instructs man:gmirror[8] to assume that the device is in the consistent state after the power/system failure. <.> Create a UFS2 file system on each mirrored partition. === System Installation This is the most important part. This section will describe how to actually install the minimal distribution of FreeBSD on the hard drives that we have prepared in the previous section. To accomplish this goal, all file systems need to be mounted so `sysinstall` may write the contents of FreeBSD to the hard drives: [source,shell] .... # mount /dev/mirror/root /mnt # mkdir /mnt/var /mnt/usr # mount /dev/mirror/var /mnt/var # mount /dev/mirror/usr /mnt/usr .... When you are done, start man:sysinstall[8]. Select the [.guimenuitem]#Custom# installation from the main menu. Select [.guimenuitem]#Options# and press kbd:[Enter]. With the help of arrow keys, move the cursor on the `Install Root` item, press kbd:[Space] and change it to [.filename]#/mnt#. Press kbd:[Enter] to submit your changes and exit the [.guimenuitem]#Options# menu by pressing kbd:[q]. [WARNING] ==== Note that this step is very important and if skipped, `sysinstall` will be unable to install FreeBSD. ==== Go to the [.guimenuitem]#Distributions# menu, move the cursor with the arrow keys to `Minimal`, and check it by pressing kbd:[Space]. This article uses the Minimal distribution in order to save network traffic, because the system itself will be installed over ftp. Exit this menu by choosing `Exit`. [NOTE] ==== The [.guimenuitem]#Partition# and [.guimenuitem]#Label# menus will be skipped, as these are useless now. ==== In the [.guimenuitem]#Media# menu, select `FTP`. Select the nearest mirror and let `sysinstall` assume that the network is already configured. You will be returned back to the [.guimenuitem]#Custom# menu. Finally, perform the system installation by selecting the last option, [.guimenuitem]#Commit#. Exit `sysinstall` when it finishes the installation. === Post Installation Steps The FreeBSD operating system should be installed now; however, the process is not finished yet. It is necessary to perform some post installation steps in order to allow FreeBSD to boot in the future and to be able to log in to the system. You must now man:chroot[8] into the freshly installed system in order to finish the installation. Use the following command: [source,shell] .... # chroot /mnt .... To complete our goal, perform these steps: * Copy the `GENERIC` kernel to the [.filename]#/boot/kernel# directory: + [source,shell] .... # cp -Rp /boot/GENERIC/* /boot/kernel .... * Create the [.filename]#/etc/rc.conf#, [.filename]#/etc/resolv.conf# and [.filename]#/etc/fstab# files. Do not forget to properly set the network information and to enable sshd in [.filename]#/etc/rc.conf#. The contents of [.filename]#/etc/fstab# will be similar to the following: + [.programlisting] .... # Device Mountpoint FStype Options Dump Pass# /dev/mirror/swap none swap sw 0 0 /dev/mirror/root / ufs rw 1 1 /dev/mirror/usr /usr ufs rw 2 2 /dev/mirror/var /var ufs rw 2 2 /dev/cd0 /cdrom cd9660 ro,noauto 0 0 .... * Create [.filename]#/boot/loader.conf# with the following contents: + [.programlisting] .... geom_mirror_load="YES" zfs_load="YES" .... * Perform the following command, which will make ZFS available on the next boot: + [source,shell] .... # sysrc zfs_enable="YES" .... * Add additional users to the system using the man:adduser[8] tool. Do not forget to add a user to the `wheel` group so you may obtain root access after the reboot. * Double-check all your settings. The system should now be ready for the next boot. Use the man:reboot[8] command to reboot your system. [[zfs]] == ZFS If your system survived the reboot, it should now be possible to log in. Welcome to the fresh FreeBSD installation, performed remotely without the use of a remote console! The only remaining step is to configure man:zpool[8] and create some man:zfs[8] file systems. Creating and administering ZFS is very straightforward. First, create a mirrored pool: [source,shell] .... # zpool create tank mirror /dev/ad[01]s1f .... Next, create some file systems: [source,shell] .... # zfs create tank/ports # zfs create tank/src # zfs set compression=gzip tank/ports # zfs set compression=on tank/src # zfs set mountpoint=/usr/ports tank/ports # zfs set mountpoint=/usr/src tank/src .... That is all. If you are interested in more details about ZFS on FreeBSD, please refer to the https://wiki.freebsd.org/ZFS[ZFS] section of the FreeBSD Wiki. diff --git a/documentation/content/en/articles/serial-uart/_index.adoc b/documentation/content/en/articles/serial-uart/_index.adoc index 9bd53781e0..4a9d5b783f 100644 --- a/documentation/content/en/articles/serial-uart/_index.adoc +++ b/documentation/content/en/articles/serial-uart/_index.adoc @@ -1,1163 +1,1163 @@ --- title: Serial and UART Tutorial authors: - author: Frank Durda email: uhclem@FreeBSD.org -description: How to use serial hardware and UART with FreeBSD +description: Detailed information about the use of serial ports and UART with FreeBSD trademarks: ["freebsd", "microsoft", "general"] --- = Serial and UART Tutorial :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/authors.adoc[] include::shared/en/urls.adoc[] [.abstract-title] Abstract This article talks about using serial hardware with FreeBSD. ''' toc::[] [[uart]] == The UART: What it is and how it works _Copyright (R) 1996 `{uhclem}`, All Rights Reserved. 13 January 1996._ The Universal Asynchronous Receiver/Transmitter (UART) controller is the key component of the serial communications subsystem of a computer. The UART takes bytes of data and transmits the individual bits in a sequential fashion. At the destination, a second UART re-assembles the bits into complete bytes. Serial transmission is commonly used with modems and for non-networked communication between computers, terminals and other devices. There are two primary forms of serial transmission: Synchronous and Asynchronous. Depending on the modes that are supported by the hardware, the name of the communication sub-system will usually include a `A` if it supports Asynchronous communications, and a `S` if it supports Synchronous communications. Both forms are described below. Some common acronyms are: [.blockquote] UART Universal Asynchronous Receiver/Transmitter [.blockquote] USART Universal Synchronous-Asynchronous Receiver/Transmitter === Synchronous Serial Transmission Synchronous serial transmission requires that the sender and receiver share a clock with one another, or that the sender provide a strobe or other timing signal so that the receiver knows when to "read" the next bit of the data. In most forms of serial Synchronous communication, if there is no data available at a given instant to transmit, a fill character must be sent instead so that data is always being transmitted. Synchronous communication is usually more efficient because only data bits are transmitted between sender and receiver, and synchronous communication can be more costly if extra wiring and circuits are required to share a clock signal between the sender and receiver. A form of Synchronous transmission is used with printers and fixed disk devices in that the data is sent on one set of wires while a clock or strobe is sent on a different wire. Printers and fixed disk devices are not normally serial devices because most fixed disk interface standards send an entire word of data for each clock or strobe signal by using a separate wire for each bit of the word. In the PC industry, these are known as Parallel devices. The standard serial communications hardware in the PC does not support Synchronous operations. This mode is described here for comparison purposes only. === Asynchronous Serial Transmission Asynchronous transmission allows data to be transmitted without the sender having to send a clock signal to the receiver. Instead, the sender and receiver must agree on timing parameters in advance and special bits are added to each word which are used to synchronize the sending and receiving units. When a word is given to the UART for Asynchronous transmissions, a bit called the "Start Bit" is added to the beginning of each word that is to be transmitted. The Start Bit is used to alert the receiver that a word of data is about to be sent, and to force the clock in the receiver into synchronization with the clock in the transmitter. These two clocks must be accurate enough to not have the frequency drift by more than 10% during the transmission of the remaining bits in the word. (This requirement was set in the days of mechanical teleprinters and is easily met by modern electronic equipment.) After the Start Bit, the individual bits of the word of data are sent, with the Least Significant Bit (LSB) being sent first. Each bit in the transmission is transmitted for exactly the same amount of time as all of the other bits, and the receiver "looks" at the wire at approximately halfway through the period assigned to each bit to determine if the bit is a `1` or a `0`. For example, if it takes two seconds to send each bit, the receiver will examine the signal to determine if it is a `1` or a `0` after one second has passed, then it will wait two seconds and then examine the value of the next bit, and so on. The sender does not know when the receiver has "looked" at the value of the bit. The sender only knows when the clock says to begin transmitting the next bit of the word. When the entire data word has been sent, the transmitter may add a Parity Bit that the transmitter generates. The Parity Bit may be used by the receiver to perform simple error checking. Then at least one Stop Bit is sent by the transmitter. When the receiver has received all of the bits in the data word, it may check for the Parity Bits (both sender and receiver must agree on whether a Parity Bit is to be used), and then the receiver looks for a Stop Bit. If the Stop Bit does not appear when it is supposed to, the UART considers the entire word to be garbled and will report a Framing Error to the host processor when the data word is read. The usual cause of a Framing Error is that the sender and receiver clocks were not running at the same speed, or that the signal was interrupted. Regardless of whether the data was received correctly or not, the UART automatically discards the Start, Parity and Stop bits. If the sender and receiver are configured identically, these bits are not passed to the host. If another word is ready for transmission, the Start Bit for the new word can be sent as soon as the Stop Bit for the previous word has been sent. As asynchronous data is "self synchronizing", if there is no data to transmit, the transmission line can be idle. === Other UART Functions In addition to the basic job of converting data from parallel to serial for transmission and from serial to parallel on reception, a UART will usually provide additional circuits for signals that can be used to indicate the state of the transmission media, and to regulate the flow of data in the event that the remote device is not prepared to accept more data. For example, when the device connected to the UART is a modem, the modem may report the presence of a carrier on the phone line while the computer may be able to instruct the modem to reset itself or to not take calls by raising or lowering one more of these extra signals. The function of each of these additional signals is defined in the EIA RS232-C standard. === The RS232-C and V.24 Standards In most computer systems, the UART is connected to circuitry that generates signals that comply with the EIA RS232-C specification. There is also a CCITT standard named V.24 that mirrors the specifications included in RS232-C. ==== RS232-C Bit Assignments (Marks and Spaces) In RS232-C, a value of `1` is called a `Mark` and a value of `0` is called a `Space`. When a communication line is idle, the line is said to be "Marking", or transmitting continuous `1` values. The Start bit always has a value of `0` (a Space). The Stop Bit always has a value of `1` (a Mark). This means that there will always be a Mark (1) to Space (0) transition on the line at the start of every word, even when multiple word are transmitted back to back. This guarantees that sender and receiver can resynchronize their clocks regardless of the content of the data bits that are being transmitted. The idle time between Stop and Start bits does not have to be an exact multiple (including zero) of the bit rate of the communication link, but most UARTs are designed this way for simplicity. In RS232-C, the "Marking" signal (a `1`) is represented by a voltage between -2 VDC and -12 VDC, and a "Spacing" signal (a `0`) is represented by a voltage between 0 and +12 VDC. The transmitter is supposed to send +12 VDC or -12 VDC, and the receiver is supposed to allow for some voltage loss in long cables. Some transmitters in low power devices (like portable computers) sometimes use only +5 VDC and -5 VDC, but these values are still acceptable to a RS232-C receiver, provided that the cable lengths are short. ==== RS232-C Break Signal RS232-C also specifies a signal called a `Break`, which is caused by sending continuous Spacing values (no Start or Stop bits). When there is no electricity present on the data circuit, the line is considered to be sending `Break`. The `Break` signal must be of a duration longer than the time it takes to send a complete byte plus Start, Stop and Parity bits. Most UARTs can distinguish between a Framing Error and a Break, but if the UART cannot do this, the Framing Error detection can be used to identify Breaks. In the days of teleprinters, when numerous printers around the country were wired in series (such as news services), any unit could cause a `Break` by temporarily opening the entire circuit so that no current flowed. This was used to allow a location with urgent news to interrupt some other location that was currently sending information. In modern systems there are two types of Break signals. If the Break is longer than 1.6 seconds, it is considered a "Modem Break", and some modems can be programmed to terminate the conversation and go on-hook or enter the modems' command mode when the modem detects this signal. If the Break is smaller than 1.6 seconds, it signifies a Data Break and it is up to the remote computer to respond to this signal. Sometimes this form of Break is used as an Attention or Interrupt signal and sometimes is accepted as a substitute for the ASCII CONTROL-C character. Marks and Spaces are also equivalent to "Holes" and "No Holes" in paper tape systems. [NOTE] ==== Breaks cannot be generated from paper tape or from any other byte value, since bytes are always sent with Start and Stop bit. The UART is usually capable of generating the continuous Spacing signal in response to a special command from the host processor. ==== ==== RS232-C DTE and DCE Devices The RS232-C specification defines two types of equipment: the Data Terminal Equipment (DTE) and the Data Carrier Equipment (DCE). Usually, the DTE device is the terminal (or computer), and the DCE is a modem. Across the phone line at the other end of a conversation, the receiving modem is also a DCE device and the computer that is connected to that modem is a DTE device. The DCE device receives signals on the pins that the DTE device transmits on, and vice versa. When two devices that are both DTE or both DCE must be connected together without a modem or a similar media translator between them, a NULL modem must be used. The NULL modem electrically re-arranges the cabling so that the transmitter output is connected to the receiver input on the other device, and vice versa. Similar translations are performed on all of the control signals so that each device will see what it thinks are DCE (or DTE) signals from the other device. The number of signals generated by the DTE and DCE devices are not symmetrical. The DTE device generates fewer signals for the DCE device than the DTE device receives from the DCE. ==== RS232-C Pin Assignments The EIA RS232-C specification (and the ITU equivalent, V.24) calls for a twenty-five pin connector (usually a DB25) and defines the purpose of most of the pins in that connector. In the IBM Personal Computer and similar systems, a subset of RS232-C signals are provided via nine pin connectors (DB9). The signals that are not included on the PC connector deal mainly with synchronous operation, and this transmission mode is not supported by the UART that IBM selected for use in the IBM PC. Depending on the computer manufacturer, a DB25, a DB9, or both types of connector may be used for RS232-C communications. (The IBM PC also uses a DB25 connector for the parallel printer interface which causes some confusion.) Below is a table of the RS232-C signal assignments in the DB25 and DB9 connectors. [.informaltable] [cols="1,1,1,1,1,1,1", frame="none", options="header"] |=== | DB25 RS232-C Pin | DB9 IBM PC Pin | EIA Circuit Symbol | CCITT Circuit Symbol | Common Name | Signal Source | Description |1 |- |AA |101 |PG/FG |- |Frame/Protective Ground |2 |3 |BA |103 |TD |DTE |Transmit Data |3 |2 |BB |104 |RD |DCE |Receive Data |4 |7 |CA |105 |RTS |DTE |Request to Send |5 |8 |CB |106 |CTS |DCE |Clear to Send |6 |6 |CC |107 |DSR |DCE |Data Set Ready |7 |5 |AV |102 |SG/GND |- |Signal Ground |8 |1 |CF |109 |DCD/CD |DCE |Data Carrier Detect |9 |- |- |- |- |- |Reserved for Test |10 |- |- |- |- |- |Reserved for Test |11 |- |- |- |- |- |Reserved for Test |12 |- |CI |122 |SRLSD |DCE |Sec. Recv. Line Signal Detector |13 |- |SCB |121 |SCTS |DCE |Secondary Clear to Send |14 |- |SBA |118 |STD |DTE |Secondary Transmit Data |15 |- |DB |114 |TSET |DCE |Trans. Sig. Element Timing |16 |- |SBB |119 |SRD |DCE |Secondary Received Data |17 |- |DD |115 |RSET |DCE |Receiver Signal Element Timing |18 |- |- |141 |LOOP |DTE |Local Loopback |19 |- |SCA |120 |SRS |DTE |Secondary Request to Send |20 |4 |CD |108.2 |DTR |DTE |Data Terminal Ready |21 |- |- |- |RDL |DTE |Remote Digital Loopback |22 |9 |CE |125 |RI |DCE |Ring Indicator |23 |- |CH |111 |DSRS |DTE |Data Signal Rate Selector |24 |- |DA |113 |TSET |DTE |Trans. Sig. Element Timing |25 |- |- |142 |- |DCE |Test Mode |=== === Bits, Baud and Symbols Baud is a measurement of transmission speed in asynchronous communication. Due to advances in modem communication technology, this term is frequently misused when describing the data rates in newer devices. Traditionally, a Baud Rate represents the number of bits that are actually being sent over the media, not the amount of data that is actually moved from one DTE device to the other. The Baud count includes the overhead bits Start, Stop and Parity that are generated by the sending UART and removed by the receiving UART. This means that seven-bit words of data actually take 10 bits to be completely transmitted. Therefore, a modem capable of moving 300 bits per second from one place to another can normally only move 30 7-bit words if Parity is used and one Start and Stop bit are present. If 8-bit data words are used and Parity bits are also used, the data rate falls to 27.27 words per second, because it now takes 11 bits to send the eight-bit words, and the modem still only sends 300 bits per second. The formula for converting bytes per second into a baud rate and vice versa was simple until error-correcting modems came along. These modems receive the serial stream of bits from the UART in the host computer (even when internal modems are used the data is still frequently serialized) and converts the bits back into bytes. These bytes are then combined into packets and sent over the phone line using a Synchronous transmission method. This means that the Stop, Start, and Parity bits added by the UART in the DTE (the computer) were removed by the modem before transmission by the sending modem. When these bytes are received by the remote modem, the remote modem adds Start, Stop and Parity bits to the words, converts them to a serial format and then sends them to the receiving UART in the remote computer, who then strips the Start, Stop and Parity bits. The reason all these extra conversions are done is so that the two modems can perform error correction, which means that the receiving modem is able to ask the sending modem to resend a block of data that was not received with the correct checksum. This checking is handled by the modems, and the DTE devices are usually unaware that the process is occurring. By striping the Start, Stop and Parity bits, the additional bits of data that the two modems must share between themselves to perform error-correction are mostly concealed from the effective transmission rate seen by the sending and receiving DTE equipment. For example, if a modem sends ten 7-bit words to another modem without including the Start, Stop and Parity bits, the sending modem will be able to add 30 bits of its own information that the receiving modem can use to do error-correction without impacting the transmission speed of the real data. The use of the term Baud is further confused by modems that perform compression. A single 8-bit word passed over the telephone line might represent a dozen words that were transmitted to the sending modem. The receiving modem will expand the data back to its original content and pass that data to the receiving DTE. Modern modems also include buffers that allow the rate that bits move across the phone line (DCE to DCE) to be a different speed than the speed that the bits move between the DTE and DCE on both ends of the conversation. Normally the speed between the DTE and DCE is higher than the DCE to DCE speed because of the use of compression by the modems. As the number of bits needed to describe a byte varied during the trip between the two machines plus the differing bits-per-seconds speeds that are used present on the DTE-DCE and DCE-DCE links, the usage of the term Baud to describe the overall communication speed causes problems and can misrepresent the true transmission speed. So Bits Per Second (bps) is the correct term to use to describe the transmission rate seen at the DCE to DCE interface and Baud or Bits Per Second are acceptable terms to use when a connection is made between two systems with a wired connection, or if a modem is in use that is not performing error-correction or compression. Modern high speed modems (2400, 9600, 14,400, and 19,200bps) in reality still operate at or below 2400 baud, or more accurately, 2400 Symbols per second. High speed modem are able to encode more bits of data into each Symbol using a technique called Constellation Stuffing, which is why the effective bits per second rate of the modem is higher, but the modem continues to operate within the limited audio bandwidth that the telephone system provides. Modems operating at 28,800 and higher speeds have variable Symbol rates, but the technique is the same. === The IBM Personal Computer UART Starting with the original IBM Personal Computer, IBM selected the National Semiconductor INS8250 UART for use in the IBM PC Parallel/Serial Adapter. Subsequent generations of compatible computers from IBM and other vendors continued to use the INS8250 or improved versions of the National Semiconductor UART family. ==== National Semiconductor UART Family Tree There have been several versions and subsequent generations of the INS8250 UART. Each major version is described below. [.programlisting] .... INS8250 -> INS8250B \ \ \-> INS8250A -> INS82C50A \ \ \-> NS16450 -> NS16C450 \ \ \-> NS16550 -> NS16550A -> PC16550D .... INS8250:: This part was used in the original IBM PC and IBM PC/XT. The original name for this part was the INS8250 ACE (Asynchronous Communications Element) and it is made from NMOS technology. + The 8250 uses eight I/O ports and has a one-byte send and a one-byte receive buffer. This original UART has several race conditions and other flaws. The original IBM BIOS includes code to work around these flaws, but this made the BIOS dependent on the flaws being present, so subsequent parts like the 8250A, 16450 or 16550 could not be used in the original IBM PC or IBM PC/XT. INS8250-B:: This is the slower speed of the INS8250 made from NMOS technology. It contains the same problems as the original INS8250. INS8250A:: An improved version of the INS8250 using XMOS technology with various functional flaws corrected. The INS8250A was used initially in PC clone computers by vendors who used "clean" BIOS designs. Due to the corrections in the chip, this part could not be used with a BIOS compatible with the INS8250 or INS8250B. INS82C50A:: This is a CMOS version (low power consumption) of the INS8250A and has similar functional characteristics. NS16450:: Same as NS8250A with improvements so it can be used with faster CPU bus designs. IBM used this part in the IBM AT and updated the IBM BIOS to no longer rely on the bugs in the INS8250. NS16C450:: This is a CMOS version (low power consumption) of the NS16450. NS16550:: Same as NS16450 with a 16-byte send and receive buffer but the buffer design was flawed and could not be reliably be used. NS16550A:: Same as NS16550 with the buffer flaws corrected. The 16550A and its successors have become the most popular UART design in the PC industry, mainly due to its ability to reliably handle higher data rates on operating systems with sluggish interrupt response times. NS16C552:: This component consists of two NS16C550A CMOS UARTs in a single package. PC16550D:: Same as NS16550A with subtle flaws corrected. This is revision D of the 16550 family and is the latest design available from National Semiconductor. ==== The NS16550AF and the PC16550D are the same thing National reorganized their part numbering system a few years ago, and the NS16550AFN no longer exists by that name. (If you have a NS16550AFN, look at the date code on the part, which is a four digit number that usually starts with a nine. The first two digits of the number are the year, and the last two digits are the week in that year when the part was packaged. If you have a NS16550AFN, it is probably a few years old.) The new numbers are like PC16550DV, with minor differences in the suffix letters depending on the package material and its shape. (A description of the numbering system can be found below.) It is important to understand that in some stores, you may pay $15(US) for a NS16550AFN made in 1990 and in the next bin are the new PC16550DN parts with minor fixes that National has made since the AFN part was in production, the PC16550DN was probably made in the past six months and it costs half (as low as $5(US) in volume) as much as the NS16550AFN because they are readily available. As the supply of NS16550AFN chips continues to shrink, the price will probably continue to increase until more people discover and accept that the PC16550DN really has the same function as the old part number. ==== National Semiconductor Part Numbering System The older NS``__nnnnnrqp__`` part numbers are now of the format PC``__nnnnnrgp__``. The `_r_` is the revision field. The current revision of the 16550 from National Semiconductor is `D`. The `_p_` is the package-type field. The types are: [.informaltable] [cols="1,1,1", frame="none"] |=== |"F" |QFP |(quad flat pack) L lead type |"N" |DIP |(dual inline package) through hole straight lead type |"V" |LPCC |(lead plastic chip carrier) J lead type |=== The _g_ is the product grade field. If an `I` precedes the package-type letter, it indicates an "industrial" grade part, which has higher specs than a standard part but not as high as Military Specification (Milspec) component. This is an optional field. So what we used to call a NS16550AFN (DIP Package) is now called a PC16550DN or PC16550DIN. === Other Vendors and Similar UARTs Over the years, the 8250, 8250A, 16450 and 16550 have been licensed or copied by other chip vendors. In the case of the 8250, 8250A and 16450, the exact circuit (the "megacell") was licensed to many vendors, including Western Digital and Intel. Other vendors reverse-engineered the part or produced emulations that had similar behavior. In internal modems, the modem designer will frequently emulate the 8250A/16450 with the modem microprocessor, and the emulated UART will frequently have a hidden buffer consisting of several hundred bytes. Due to the size of the buffer, these emulations can be as reliable as a 16550A in their ability to handle high speed data. However, most operating systems will still report that the UART is only a 8250A or 16450, and may not make effective use of the extra buffering present in the emulated UART unless special drivers are used. Some modem makers are driven by market forces to abandon a design that has hundreds of bytes of buffer and instead use a 16550A UART so that the product will compare favorably in market comparisons even though the effective performance may be lowered by this action. A common misconception is that all parts with "16550A" written on them are identical in performance. There are differences, and in some cases, outright flaws in most of these 16550A clones. When the NS16550 was developed, the National Semiconductor obtained several patents on the design and they also limited licensing, making it harder for other vendors to provide a chip with similar features. As a result of the patents, reverse-engineered designs and emulations had to avoid infringing the claims covered by the patents. Subsequently, these copies almost never perform exactly the same as the NS16550A or PC16550D, which are the parts most computer and modem makers want to buy but are sometimes unwilling to pay the price required to get the genuine part. Some of the differences in the clone 16550A parts are unimportant, while others can prevent the device from being used at all with a given operating system or driver. These differences may show up when using other drivers, or when particular combinations of events occur that were not well tested or considered in the Windows(R) driver. This is because most modem vendors and 16550-clone makers use the Microsoft drivers from Windows(R) for Workgroups 3.11 and the Microsoft(R) MS-DOS(R) utility as the primary tests for compatibility with the NS16550A. This over-simplistic criteria means that if a different operating system is used, problems could appear due to subtle differences between the clones and genuine components. National Semiconductor has made available a program named COMTEST that performs compatibility tests independent of any OS drivers. It should be remembered that the purpose of this type of program is to demonstrate the flaws in the products of the competition, so the program will report major as well as extremely subtle differences in behavior in the part being tested. In a series of tests performed by the author of this document in 1994, components made by National Semiconductor, TI, StarTech, and CMD as well as megacells and emulations embedded in internal modems were tested with COMTEST. A difference count for some of these components is listed below. Since these tests were performed in 1994, they may not reflect the current performance of the given product from a vendor. It should be noted that COMTEST normally aborts when an excessive number or certain types of problems have been detected. As part of this testing, COMTEST was modified so that it would not abort no matter how many differences were encountered. [.informaltable] [cols="1,1,1", frame="none", options="header"] |=== | Vendor | Part Number | Errors (aka "differences" reported) |National |(PC16550DV) |0 |National |(NS16550AFN) |0 |National |(NS16C552V) |0 |TI |(TL16550AFN) |3 |CMD |(16C550PE) |19 |StarTech |(ST16C550J) |23 |Rockwell |Reference modem with internal 16550 or an emulation (RC144DPi/C3000-25) |117 |Sierra |Modem with an internal 16550 (SC11951/SC11351) |91 |=== [NOTE] ==== To date, the author of this document has not found any non-National parts that report zero differences using the COMTEST program. It should also be noted that National has had five versions of the 16550 over the years and the newest parts behave a bit differently than the classic NS16550AFN that is considered the benchmark for functionality. COMTEST appears to turn a blind eye to the differences within the National product line and reports no errors on the National parts (except for the original 16550) even when there are official erratas that describe bugs in the A, B and C revisions of the parts, so this bias in COMTEST must be taken into account. ==== It is important to understand that a simple count of differences from COMTEST does not reveal a lot about what differences are important and which are not. For example, about half of the differences reported in the two modems listed above that have internal UARTs were caused by the clone UARTs not supporting five- and six-bit character modes. The real 16550, 16450, and 8250 UARTs all support these modes and COMTEST checks the functionality of these modes so over fifty differences are reported. However, almost no modern modem supports five- or six-bit characters, particularly those with error-correction and compression capabilities. This means that the differences related to five- and six-bit character modes can be discounted. Many of the differences COMTEST reports have to do with timing. In many of the clone designs, when the host reads from one port, the status bits in some other port may not update in the same amount of time (some faster, some slower) as a _real_ NS16550AFN and COMTEST looks for these differences. This means that the number of differences can be misleading in that one device may only have one or two differences but they are extremely serious, and some other device that updates the status registers faster or slower than the reference part (that would probably never affect the operation of a properly written driver) could have dozens of differences reported. COMTEST can be used as a screening tool to alert the administrator to the presence of potentially incompatible components that might cause problems or have to be handled as a special case. If you run COMTEST on a 16550 that is in a modem or a modem is attached to the serial port, you need to first issue a ATE0&W command to the modem so that the modem will not echo any of the test characters. If you forget to do this, COMTEST will report at least this one difference: [source,shell] .... Error (6)...Timeout interrupt failed: IIR = c1 LSR = 61 .... === 8250/16450/16550 Registers The 8250/16450/16550 UART occupies eight contiguous I/O port addresses. In the IBM PC, there are two defined locations for these eight ports and they are known collectively as [.filename]#COM1# and [.filename]#COM2#. The makers of PC-clones and add-on cards have created two additional areas known as [.filename]#COM3# and [.filename]#COM4#, but these extra COM ports conflict with other hardware on some systems. The most common conflict is with video adapters that provide IBM 8514 emulation. [.filename]#COM1# is located from 0x3f8 to 0x3ff and normally uses IRQ 4. [.filename]#COM2# is located from 0x2f8 to 0x2ff and normally uses IRQ 3. [.filename]#COM3# is located from 0x3e8 to 0x3ef and has no standardized IRQ. [.filename]#COM4# is located from 0x2e8 to 0x2ef and has no standardized IRQ. A description of the I/O ports of the 8250/16450/16550 UART is provided below. [.informaltable] [cols="10%,10%,80%", frame="none", options="header"] |=== | I/O Port | Access Allowed | Description |+0x00 |write (DLAB==0) | Transmit Holding Register (THR). Information written to this port are treated as data words and will be transmitted by the UART. |+0x00 |read (DLAB==0) | Receive Buffer Register (RBR). Any data words received by the UART form the serial link are accessed by the host by reading this port. |+0x00 |write/read (DLAB==1) | Divisor Latch LSB (DLL) This value will be divided from the master input clock (in the IBM PC, the master clock is 1.8432MHz) and the resulting clock will determine the baud rate of the UART. This register holds bits 0 thru 7 of the divisor. |+0x01 |write/read (DLAB==1) | Divisor Latch MSB (DLH) This value will be divided from the master input clock (in the IBM PC, the master clock is 1.8432MHz) and the resulting clock will determine the baud rate of the UART. This register holds bits 8 thru 15 of the divisor. |+0x01 |write/read (DLAB==0) |Interrupt Enable Register (IER) + The 8250/16450/16550 UART classifies events into one of four categories. Each category can be configured to generate an interrupt when any of the events occurs. The 8250/16450/16550 UART generates a single external interrupt signal regardless of how many events in the enabled categories have occurred. It is up to the host processor to respond to the interrupt and then poll the enabled interrupt categories (usually all categories have interrupts enabled) to determine the true cause(s) of the interrupt. + Bit 7 -> Reserved, always 0. + Bit 6 -> Reserved, always 0. + Bit 5 -> Reserved, always 0. + Bit 4 -> Reserved, always 0. + Bit 3 -> Enable Modem Status Interrupt (EDSSI). Setting this bit to "1" allows the UART to generate an interrupt when a change occurs on one or more of the status lines. + Bit 2 -> Enable Receiver Line Status Interrupt (ELSI) Setting this bit to "1" causes the UART to generate an interrupt when the an error (or a BREAK signal) has been detected in the incoming data. + Bit 1 -> Enable Transmitter Holding Register Empty Interrupt (ETBEI) Setting this bit to "1" causes the UART to generate an interrupt when the UART has room for one or more additional characters that are to be transmitted. + Bit 0 -> Enable Received Data Available Interrupt (ERBFI) Setting this bit to "1" causes the UART to generate an interrupt when the UART has received enough characters to exceed the trigger level of the FIFO, or the FIFO timer has expired (stale data), or a single character has been received when the FIFO is disabled. |+0x02 |write |FIFO Control Register (FCR) (This port does not exist on the 8250 and 16450 UART.) + Bit 7 -> Receiver Trigger Bit #1 + Bit 6 -> Receiver Trigger Bit #0 + These two bits control at what point the receiver is to generate an interrupt when the FIFO is active. + 7 6 How many words are received before an interrupt is generated + 0 0 1 + 0 1 4 + 1 0 8 + 1 1 14 + Bit 5 -> Reserved, always 0. + Bit 4 -> Reserved, always 0. + Bit 3 -> DMA Mode Select. If Bit 0 is set to "1" (FIFOs enabled), setting this bit changes the operation of the -RXRDY and -TXRDY signals from Mode 0 to Mode 1. + Bit 2 -> Transmit FIFO Reset. When a "1" is written to this bit, the contents of the FIFO are discarded. Any word currently being transmitted will be sent intact. This function is useful in aborting transfers. + Bit 1 -> Receiver FIFO Reset. When a "1" is written to this bit, the contents of the FIFO are discarded. Any word currently being assembled in the shift register will be received intact. + Bit 0 -> 16550 FIFO Enable. When set, both the transmit and receive FIFOs are enabled. Any contents in the holding register, shift registers or FIFOs are lost when FIFOs are enabled or disabled. + |+0x02 |read |Interrupt Identification Register + Bit 7 -> FIFOs enabled. On the 8250/16450 UART, this bit is zero. + Bit 6 -> FIFOs enabled. On the 8250/16450 UART, this bit is zero. + Bit 5 -> Reserved, always 0. + Bit 4 -> Reserved, always 0. + Bit 3 -> Interrupt ID Bit #2. On the 8250/16450 UART, this bit is zero. + Bit 2 -> Interrupt ID Bit #1 + Bit 1 -> Interrupt ID Bit #0.These three bits combine to report the category of event that caused the interrupt that is in progress. These categories have priorities, so if multiple categories of events occur at the same time, the UART will report the more important events first and the host must resolve the events in the order they are reported. All events that caused the current interrupt must be resolved before any new interrupts will be generated. (This is a limitation of the PC architecture.) + 2 1 0 Priority Description + 0 1 1 First Received Error (OE, PE, BI, or FE) + 0 1 0 Second Received Data Available + 1 1 0 Second Trigger level identification (Stale data in receive buffer) + 0 0 1 Third Transmitter has room for more words (THRE) + 0 0 0 Fourth Modem Status Change (-CTS, -DSR, -RI, or -DCD) + Bit 0 -> Interrupt Pending Bit. If this bit is set to "0", then at least one interrupt is pending. |+0x03 |write/read |Line Control Register (LCR) + Bit 7 -> Divisor Latch Access Bit (DLAB). When set, access to the data transmit/receive register (THR/RBR) and the Interrupt Enable Register (IER) is disabled. Any access to these ports is now redirected to the Divisor Latch Registers. Setting this bit, loading the Divisor Registers, and clearing DLAB should be done with interrupts disabled. + Bit 6 -> Set Break. When set to "1", the transmitter begins to transmit continuous Spacing until this bit is set to "0". This overrides any bits of characters that are being transmitted. + Bit 5 -> Stick Parity. When parity is enabled, setting this bit causes parity to always be "1" or "0", based on the value of Bit 4. Bit 4 -> Even Parity Select (EPS). When parity is enabled and Bit 5 is "0", setting this bit causes even parity to be transmitted and expected. Otherwise, odd parity is used. + Bit 3 -> Parity Enable (PEN). When set to "1", a parity bit is inserted between the last bit of the data and the Stop Bit. The UART will also expect parity to be present in the received data. + Bit 2 -> Number of Stop Bits (STB). If set to "1" and using 5-bit data words, 1.5 Stop Bits are transmitted and expected in each data word. For 6, 7 and 8-bit data words, 2 Stop Bits are transmitted and expected. When this bit is set to "0", one Stop Bit is used on each data word. + Bit 1 -> Word Length Select Bit #1 (WLSB1) + Bit 0 -> Word Length Select Bit #0 (WLSB0) + Together these bits specify the number of bits in each data word. + 1 0 Word Length + 0 0 5 Data Bits + 0 1 6 Data Bits + 1 0 7 Data Bits + 1 1 8 Data Bits + |+0x04 |write/read |Modem Control Register (MCR) + Bit 7 -> Reserved, always 0. + Bit 6 -> Reserved, always 0. + Bit 5 -> Reserved, always 0. + Bit 4 -> Loop-Back Enable. When set to "1", the UART transmitter and receiver are internally connected together to allow diagnostic operations. In addition, the UART modem control outputs are connected to the UART modem control inputs. CTS is connected to RTS, DTR is connected to DSR, OUT1 is connected to RI, and OUT 2 is connected to DCD. + Bit 3 -> OUT 2. An auxiliary output that the host processor may set high or low. In the IBM PC serial adapter (and most clones), OUT 2 is used to tri-state (disable) the interrupt signal from the 8250/16450/16550 UART. + Bit 2 -> OUT 1. An auxiliary output that the host processor may set high or low. This output is not used on the IBM PC serial adapter. + Bit 1 -> Request to Send (RTS). When set to "1", the output of the UART -RTS line is Low (Active). + Bit 0 -> Data Terminal Ready (DTR). When set to "1", the output of the UART -DTR line is Low (Active). + |+0x05 |write/read |Line Status Register (LSR) + Bit 7 -> Error in Receiver FIFO. On the 8250/16450 UART, this bit is zero. This bit is set to "1" when any of the bytes in the FIFO have one or more of the following error conditions: PE, FE, or BI. + Bit 6 -> Transmitter Empty (TEMT). When set to "1", there are no words remaining in the transmit FIFO or the transmit shift register. The transmitter is completely idle. + Bit 5 -> Transmitter Holding Register Empty (THRE). When set to "1", the FIFO (or holding register) now has room for at least one additional word to transmit. The transmitter may still be transmitting when this bit is set to "1". + Bit 4 -> Break Interrupt (BI). The receiver has detected a Break signal. + Bit 3 -> Framing Error (FE). A Start Bit was detected but the Stop Bit did not appear at the expected time. The received word is probably garbled. + Bit 2 -> Parity Error (PE). The parity bit was incorrect for the word received. + Bit 1 -> Overrun Error (OE). A new word was received and there was no room in the receive buffer. The newly-arrived word in the shift register is discarded. On 8250/16450 UARTs, the word in the holding register is discarded and the newly- arrived word is put in the holding register. + Bit 0 -> Data Ready (DR) One or more words are in the receive FIFO that the host may read. A word must be completely received and moved from the shift register into the FIFO (or holding register for 8250/16450 designs) before this bit is set. |+0x06 |write/read |Modem Status Register (MSR) + Bit 7 -> Data Carrier Detect (DCD). Reflects the state of the DCD line on the UART. + Bit 6 -> Ring Indicator (RI). Reflects the state of the RI line on the UART. + Bit 5 -> Data Set Ready (DSR). Reflects the state of the DSR line on the UART. + Bit 4 -> Clear To Send (CTS). Reflects the state of the CTS line on the UART. + Bit 3 -> Delta Data Carrier Detect (DDCD). Set to "1" if the -DCD line has changed state one more time since the last time the MSR was read by the host. + Bit 2 -> Trailing Edge Ring Indicator (TERI). Set to "1" if the -RI line has had a low to high transition since the last time the MSR was read by the host. + Bit 1 -> Delta Data Set Ready (DDSR). Set to "1" if the -DSR line has changed state one more time since the last time the MSR was read by the host. + Bit 0 -> Delta Clear To Send (DCTS). Set to "1" if the -CTS line has changed state one more time since the last time the MSR was read by the host. + |+0x07 |write/read |Scratch Register (SCR). This register performs no function in the UART. Any value can be written by the host to this location and read by the host later on. |=== === Beyond the 16550A UART Although National Semiconductor has not offered any components compatible with the 16550 that provide additional features, various other vendors have. Some of these components are described below. It should be understood that to effectively utilize these improvements, drivers may have to be provided by the chip vendor since most of the popular operating systems do not support features beyond those provided by the 16550. ST16650:: By default this part is similar to the NS16550A, but an extended 32-byte send and receive buffer can be optionally enabled. Made by StarTech. TIL16660:: By default this part behaves similar to the NS16550A, but an extended 64-byte send and receive buffer can be optionally enabled. Made by Texas Instruments. Hayes ESP:: This proprietary plug-in card contains a 2048-byte send and receive buffer, and supports data rates to 230.4Kbit/sec. Made by Hayes. In addition to these "dumb" UARTs, many vendors produce intelligent serial communication boards. This type of design usually provides a microprocessor that interfaces with several UARTs, processes and buffers the data, and then alerts the main PC processor when necessary. As the UARTs are not directly accessed by the PC processor in this type of communication system, it is not necessary for the vendor to use UARTs that are compatible with the 8250, 16450, or the 16550 UART. This leaves the designer free to components that may have better performance characteristics. [[sio]] == Configuring the [.filename]#sio# driver The [.filename]#sio# driver provides support for NS8250-, NS16450-, NS16550 and NS16550A-based EIA RS-232C (CCITT V.24) communications interfaces. Several multiport cards are supported as well. See the man:sio[4] manual page for detailed technical documentation. === Digi International (DigiBoard) PC/8 _Contributed by `{awebster}`. 26 August 1995._ Here is a config snippet from a machine with a Digi International PC/8 with 16550. It has 8 modems connected to these 8 lines, and they work just great. Do not forget to add `options COM_MULTIPORT` or it will not work very well! [.programlisting] .... device sio4 at isa? port 0x100 flags 0xb05 device sio5 at isa? port 0x108 flags 0xb05 device sio6 at isa? port 0x110 flags 0xb05 device sio7 at isa? port 0x118 flags 0xb05 device sio8 at isa? port 0x120 flags 0xb05 device sio9 at isa? port 0x128 flags 0xb05 device sio10 at isa? port 0x130 flags 0xb05 device sio11 at isa? port 0x138 flags 0xb05 irq 9 .... The trick in setting this up is that the MSB of the flags represent the last SIO port, in this case 11 so flags are 0xb05. === Boca 16 _Contributed by `{whiteside}`. 26 August 1995._ The procedures to make a Boca 16 port board with FreeBSD are pretty straightforward, but you will need a couple things to make it work: . You either need the kernel sources installed so you can recompile the necessary options or you will need someone else to compile it for you. The 2.0.5 default kernel does _not_ come with multiport support enabled and you will need to add a device entry for each port anyways. . Two, you will need to know the interrupt and IO setting for your Boca Board so you can set these options properly in the kernel. One important note - the actual UART chips for the Boca 16 are in the connector box, not on the internal board itself. So if you have it unplugged, probes of those ports will fail. I have never tested booting with the box unplugged and plugging it back in, and I suggest you do not either. If you do not already have a custom kernel configuration file set up, refer to link:{handbook}#kernelconfig[Kernel Configuration] chapter of the FreeBSD Handbook for general procedures. The following are the specifics for the Boca 16 board and assume you are using the kernel name MYKERNEL and editing with vi. [.procedure] ==== . Add the line + [.programlisting] .... options COM_MULTIPORT .... to the config file. . Where the current `device sio__n__` lines are, you will need to add 16 more devices. The following example is for a Boca Board with an interrupt of 3, and a base IO address 100h. The IO address for Each port is +8 hexadecimal from the previous port, thus the 100h, 108h, 110h... addresses. + [.programlisting] .... device sio1 at isa? port 0x100 flags 0x1005 device sio2 at isa? port 0x108 flags 0x1005 device sio3 at isa? port 0x110 flags 0x1005 device sio4 at isa? port 0x118 flags 0x1005 ... device sio15 at isa? port 0x170 flags 0x1005 device sio16 at isa? port 0x178 flags 0x1005 irq 3 .... + The flags entry _must_ be changed from this example unless you are using the exact same sio assignments. Flags are set according to 0x``__MYY__`` where _M_ indicates the minor number of the master port (the last port on a Boca 16) and _YY_ indicates if FIFO is enabled or disabled(enabled), IRQ sharing is used(yes) and if there is an AST/4 compatible IRQ control register(no). In this example, + [.programlisting] .... flags 0x1005 .... indicates that the master port is sio16. If I added another board and assigned sio17 through sio28, the flags for all 16 ports on _that_ board would be 0x1C05, where 1C indicates the minor number of the master port. Do not change the 05 setting. . Save and complete the kernel configuration, recompile, install and reboot. Presuming you have successfully installed the recompiled kernel and have it set to the correct address and IRQ, your boot message should indicate the successful probe of the Boca ports as follows: (obviously the sio numbers, IO and IRQ could be different) + [source,shell] .... sio1 at 0x100-0x107 flags 0x1005 on isa sio1: type 16550A (multiport) sio2 at 0x108-0x10f flags 0x1005 on isa sio2: type 16550A (multiport) sio3 at 0x110-0x117 flags 0x1005 on isa sio3: type 16550A (multiport) sio4 at 0x118-0x11f flags 0x1005 on isa sio4: type 16550A (multiport) sio5 at 0x120-0x127 flags 0x1005 on isa sio5: type 16550A (multiport) sio6 at 0x128-0x12f flags 0x1005 on isa sio6: type 16550A (multiport) sio7 at 0x130-0x137 flags 0x1005 on isa sio7: type 16550A (multiport) sio8 at 0x138-0x13f flags 0x1005 on isa sio8: type 16550A (multiport) sio9 at 0x140-0x147 flags 0x1005 on isa sio9: type 16550A (multiport) sio10 at 0x148-0x14f flags 0x1005 on isa sio10: type 16550A (multiport) sio11 at 0x150-0x157 flags 0x1005 on isa sio11: type 16550A (multiport) sio12 at 0x158-0x15f flags 0x1005 on isa sio12: type 16550A (multiport) sio13 at 0x160-0x167 flags 0x1005 on isa sio13: type 16550A (multiport) sio14 at 0x168-0x16f flags 0x1005 on isa sio14: type 16550A (multiport) sio15 at 0x170-0x177 flags 0x1005 on isa sio15: type 16550A (multiport) sio16 at 0x178-0x17f irq 3 flags 0x1005 on isa sio16: type 16550A (multiport master) .... + If the messages go by too fast to see, + [source,shell] .... # dmesg | more .... will show you the boot messages. . Next, appropriate entries in [.filename]#/dev# for the devices must be made using the [.filename]#/dev/MAKEDEV# script. This step can be omitted if you are running FreeBSD 5.X with a kernel that has man:devfs[5] support compiled in. + If you do need to create the [.filename]#/dev# entries, run the following as `root`: + [source,shell] .... # cd /dev # ./MAKEDEV tty1 # ./MAKEDEV cua1 (everything in between) # ./MAKEDEV ttyg # ./MAKEDEV cuag .... + If you do not want or need call-out devices for some reason, you can dispense with making the [.filename]#cua*# devices. . If you want a quick and sloppy way to make sure the devices are working, you can simply plug a modem into each port and (as root) + [source,shell] .... # echo at > ttyd* .... for each device you have made. You _should_ see the RX lights flash for each working port. ==== === Support for Cheap Multi-UART Cards _Contributed by Helge Oldach_ mailto:hmo@sep.hamburg.com[hmo@sep.hamburg.com], September 1999 Ever wondered about FreeBSD support for your 20$ multi-I/O card with two (or more) COM ports, sharing IRQs? Here is how: Usually the only option to support these kind of boards is to use a distinct IRQ for each port. For example, if your CPU board has an on-board [.filename]#COM1# port (aka [.filename]#sio0#-I/O address 0x3F8 and IRQ 4) and you have an extension board with two UARTs, you will commonly need to configure them as [.filename]#COM2# (aka [.filename]#sio1#-I/O address 0x2F8 and IRQ 3), and the third port (aka [.filename]#sio2#) as I/O 0x3E8 and IRQ 5. Obviously this is a waste of IRQ resources, as it should be basically possible to run both extension board ports using a single IRQ with the `COM_MULTIPORT` configuration described in the previous sections. Such cheap I/O boards commonly have a 4 by 3 jumper matrix for the COM ports, similar to the following: [.programlisting] .... o o o * Port A | o * o * Port B | o * o o IRQ 2 3 4 5 .... Shown here is port A wired for IRQ 5 and port B wired for IRQ 3. The IRQ columns on your specific board may vary-other boards may supply jumpers for IRQs 3, 4, 5, and 7 instead. One could conclude that wiring both ports for IRQ 3 using a handcrafted wire-made jumper covering all three connection points in the IRQ 3 column would solve the issue, but no. You cannot duplicate IRQ 3 because the output drivers of each UART are wired in a "totem pole" fashion, so if one of the UARTs drives IRQ 3, the output signal will not be what you would expect. Depending on the implementation of the extension board or your motherboard, the IRQ 3 line will continuously stay up, or always stay low. You need to decouple the IRQ drivers for the two UARTs, so that the IRQ line of the board only goes up if (and only if) one of the UARTs asserts a IRQ, and stays low otherwise. The solution was proposed by Joerg Wunsch mailto:j@ida.interface-business.de[j@ida.interface-business.de]: To solder up a wired-or consisting of two diodes (Germanium or Schottky-types strongly preferred) and a 1 kOhm resistor. Here is the schematic, starting from the 4 by 3 jumper field above: [.programlisting] .... Diode +---------->|-------+ / | o * o o | 1 kOhm Port A +----|######|-------+ o * o o | | Port B `-------------------+ ==+== o * o o | Ground \ | +--------->|-------+ IRQ 2 3 4 5 Diode .... The cathodes of the diodes are connected to a common point, together with a 1 kOhm pull-down resistor. It is essential to connect the resistor to ground to avoid floating of the IRQ line on the bus. Now we are ready to configure a kernel. Staying with this example, we would configure: [.programlisting] .... # standard on-board COM1 port device sio0 at isa? port "IO_COM1" flags 0x10 # patched-up multi-I/O extension board options COM_MULTIPORT device sio1 at isa? port "IO_COM2" flags 0x205 device sio2 at isa? port "IO_COM3" flags 0x205 irq 3 .... Note that the `flags` setting for [.filename]#sio1# and [.filename]#sio2# is truly essential; refer to man:sio[4] for details. (Generally, the `2` in the "flags" attribute refers to [.filename]#sio#`2` which holds the IRQ, and you surely want a `5` low nibble.) With kernel verbose mode turned on this should yield something similar to this: [source,shell] .... sio0: irq maps: 0x1 0x11 0x1 0x1 sio0 at 0x3f8-0x3ff irq 4 flags 0x10 on isa sio0: type 16550A sio1: irq maps: 0x1 0x9 0x1 0x1 sio1 at 0x2f8-0x2ff flags 0x205 on isa sio1: type 16550A (multiport) sio2: irq maps: 0x1 0x9 0x1 0x1 sio2 at 0x3e8-0x3ef irq 3 flags 0x205 on isa sio2: type 16550A (multiport master) .... Though [.filename]#/sys/i386/isa/sio.c# is somewhat cryptic with its use of the "irq maps" array above, the basic idea is that you observe `0x1` in the first, third, and fourth place. This means that the corresponding IRQ was set upon output and cleared after, which is just what we would expect. If your kernel does not display this behavior, most likely there is something wrong with your wiring. [[cy]] == Configuring the [.filename]#cy# driver _Contributed by Alex Nash. 6 June 1996._ The Cyclades multiport cards are based on the [.filename]#cy# driver instead of the usual [.filename]#sio# driver used by other multiport cards. Configuration is a simple matter of: [.procedure] ==== . Add the [.filename]#cy# device to your kernel configuration (note that your irq and iomem settings may differ). + [.programlisting] .... device cy0 at isa? irq 10 iomem 0xd4000 iosiz 0x2000 .... . Rebuild and install the new kernel. . Make the device nodes by typing (the following example assumes an 8-port board): + [source,shell] .... # cd /dev # for i in 0 1 2 3 4 5 6 7;do ./MAKEDEV cuac$i ttyc$i;done .... . If appropriate, add dialup entries to [.filename]#/etc/ttys# by duplicating serial device (`ttyd`) entries and using `ttyc` in place of `ttyd`. For example: + [.programlisting] .... ttyc0 "/usr/libexec/getty std.38400" unknown on insecure ttyc1 "/usr/libexec/getty std.38400" unknown on insecure ttyc2 "/usr/libexec/getty std.38400" unknown on insecure ... ttyc7 "/usr/libexec/getty std.38400" unknown on insecure .... . Reboot with the new kernel. ==== == Configuring the [.filename]#si# driver _Contributed by `{nsayer}`. 25 March 1998._ The Specialix SI/XIO and SX multiport cards use the [.filename]#si# driver. A single machine can have up to 4 host cards. The following host cards are supported: * ISA SI/XIO host card (2 versions) * EISA SI/XIO host card * PCI SI/XIO host card * ISA SX host card * PCI SX host card Although the SX and SI/XIO host cards look markedly different, their functionality are basically the same. The host cards do not use I/O locations, but instead require a 32K chunk of memory. The factory configuration for ISA cards places this at `0xd0000-0xd7fff`. They also require an IRQ. PCI cards will, of course, auto-configure themselves. You can attach up to 4 external modules to each host card. The external modules contain either 4 or 8 serial ports. They come in the following varieties: * SI 4 or 8 port modules. Up to 57600 bps on each port supported. * XIO 8 port modules. Up to 115200 bps on each port supported. One type of XIO module has 7 serial and 1 parallel port. * SXDC 8 port modules. Up to 921600 bps on each port supported. Like XIO, a module is available with one parallel port as well. To configure an ISA host card, add the following line to your kernel configuration file, changing the numbers as appropriate: [.programlisting] .... device si0 at isa? iomem 0xd0000 irq 11 .... Valid IRQ numbers are 9, 10, 11, 12 and 15 for SX ISA host cards and 11, 12 and 15 for SI/XIO ISA host cards. To configure an EISA or PCI host card, use this line: [.programlisting] .... device si0 .... After adding the configuration entry, rebuild and install your new kernel. [NOTE] ==== The following step, is not necessary if you are using man:devfs[5] in FreeBSD 5._X_. ==== After rebooting with the new kernel, you need to make the device nodes in [.filename]#/dev#. The [.filename]#MAKEDEV# script will take care of this for you. Count how many total ports you have and type: [source,shell] .... # cd /dev # ./MAKEDEV ttyAnn cuaAnn .... (where _nn_ is the number of ports) If you want login prompts to appear on these ports, you will need to add lines like this to [.filename]#/etc/ttys#: [.programlisting] .... ttyA01 "/usr/libexec/getty std.9600" vt100 on insecure .... Change the terminal type as appropriate. For modems, `dialup` or `unknown` is fine. diff --git a/documentation/content/en/articles/solid-state/_index.adoc b/documentation/content/en/articles/solid-state/_index.adoc index 4a1b022fdb..7238aa8dad 100644 --- a/documentation/content/en/articles/solid-state/_index.adoc +++ b/documentation/content/en/articles/solid-state/_index.adoc @@ -1,304 +1,304 @@ --- title: FreeBSD and Solid State Devices authors: - author: John Kozubik email: john@kozubik.com copyright: 2001 - 2021 The FreeBSD Documentation Project -description: FreeBSD and Solid State Devices +description: The use of solid state disk devices in FreeBSD trademarks: ["freebsd", "general"] --- = FreeBSD and Solid State Devices :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: [.abstract-title] Abstract This article covers the use of solid state disk devices in FreeBSD to create embedded systems. Embedded systems have the advantage of increased stability due to the lack of integral moving parts (hard drives). Account must be taken, however, for the generally low disk space available in the system and the durability of the storage medium. Specific topics to be covered include the types and attributes of solid state media suitable for disk use in FreeBSD, kernel options that are of interest in such an environment, the [.filename]#rc.initdiskless# mechanisms that automate the initialization of such systems and the need for read-only filesystems, and building filesystems from scratch. The article will conclude with some general strategies for small and read-only FreeBSD environments. ''' toc::[] [[intro]] == Solid State Disk Devices The scope of this article will be limited to solid state disk devices made from flash memory. Flash memory is a solid state memory (no moving parts) that is non-volatile (the memory maintains data even after all power sources have been disconnected). Flash memory can withstand tremendous physical shock and is reasonably fast (the flash memory solutions covered in this article are slightly slower than a EIDE hard disk for write operations, and much faster for read operations). One very important aspect of flash memory, the ramifications of which will be discussed later in this article, is that each sector has a limited rewrite capacity. You can only write, erase, and write again to a sector of flash memory a certain number of times before the sector becomes permanently unusable. Although many flash memory products automatically map bad blocks, and although some even distribute write operations evenly throughout the unit, the fact remains that there exists a limit to the amount of writing that can be done to the device. Competitive units have between 1,000,000 and 10,000,000 writes per sector in their specification. This figure varies due to the temperature of the environment. Specifically, we will be discussing ATA compatible compact-flash units, which are quite popular as storage media for digital cameras. Of particular interest is the fact that they pin out directly to the IDE bus and are compatible with the ATA command set. Therefore, with a very simple and low-cost adaptor, these devices can be attached directly to an IDE bus in a computer. Once implemented in this manner, operating systems such as FreeBSD see the device as a normal hard disk (albeit small). Other solid state disk solutions do exist, but their expense, obscurity, and relative unease of use places them beyond the scope of this article. [[kernel]] == Kernel Options A few kernel options are of specific interest to those creating an embedded FreeBSD system. All embedded FreeBSD systems that use flash memory as system disk will be interested in memory disks and memory filesystems. As a result of the limited number of writes that can be done to flash memory, the disk and the filesystems on the disk will most likely be mounted read-only. In this environment, filesystems such as [.filename]#/tmp# and [.filename]#/var# are mounted as memory filesystems to allow the system to create logs and update counters and temporary files. Memory filesystems are a critical component to a successful solid state FreeBSD implementation. You should make sure the following lines exist in your kernel configuration file: [.programlisting] .... options MFS # Memory Filesystem options MD_ROOT # md device usable as a potential root device pseudo-device md # memory disk .... [[ro-fs]] == The `rc` Subsystem and Read-Only Filesystems The post-boot initialization of an embedded FreeBSD system is controlled by [.filename]#/etc/rc.initdiskless#. [.filename]#/etc/rc.d/var# mounts [.filename]#/var# as a memory filesystem, makes a configurable list of directories in [.filename]#/var# with the man:mkdir[1] command, and changes modes on some of those directories. In the execution of [.filename]#/etc/rc.d/var#, one other [.filename]#rc.conf# variable comes into play - `varsize`. A [.filename]#/var# partition is created by [.filename]#/etc/rc.d/var# based on the value of this variable in [.filename]#rc.conf#: [.programlisting] .... varsize=8192 .... Remember that this value is in sectors by default. The fact that [.filename]#/var# is a read-write filesystem is an important distinction, as the [.filename]#/# partition (and any other partitions you may have on your flash media) should be mounted read-only. Remember that in <> we detailed the limitations of flash memory - specifically the limited write capability. The importance of not mounting filesystems on flash media read-write, and the importance of not using a swap file, cannot be overstated. A swap file on a busy system can burn through a piece of flash media in less than one year. Heavy logging or temporary file creation and destruction can do the same. Therefore, in addition to removing the `swap` entry from your [.filename]#/etc/fstab#, you should also change the Options field for each filesystem to `ro` as follows: [.programlisting] .... # Device Mountpoint FStype Options Dump Pass# /dev/ad0s1a / ufs ro 1 1 .... A few applications in the average system will immediately begin to fail as a result of this change. For instance, cron will not run properly as a result of missing cron tabs in the [.filename]#/var# created by [.filename]#/etc/rc.d/var#, and syslog and dhcp will encounter problems as well as a result of the read-only filesystem and missing items in the [.filename]#/var# that [.filename]#/etc/rc.d/var# has created. These are only temporary problems though, and are addressed, along with solutions to the execution of other common software packages in <>. An important thing to remember is that a filesystem that was mounted read-only with [.filename]#/etc/fstab# can be made read-write at any time by issuing the command: [source,shell] .... # /sbin/mount -uw partition .... and can be toggled back to read-only with the command: [source,shell] .... # /sbin/mount -ur partition .... == Building a File System from Scratch Since ATA compatible compact-flash cards are seen by FreeBSD as normal IDE hard drives, you could theoretically install FreeBSD from the network using the kern and mfsroot floppies or from a CD. However, even a small installation of FreeBSD using normal installation procedures can produce a system in size of greater than 200 megabytes. Most people will be using smaller flash memory devices (128 megabytes is considered fairly large - 32 or even 16 megabytes is common), so an installation using normal mechanisms is not possible-there is simply not enough disk space for even the smallest of conventional installations. The easiest way to overcome this space limitation is to install FreeBSD using conventional means to a normal hard disk. After the installation is complete, pare down the operating system to a size that will fit onto your flash media, then tar the entire filesystem. The following steps will guide you through the process of preparing a piece of flash memory for your tarred filesystem. Remember, because a normal installation is not being performed, operations such as partitioning, labeling, file-system creation, etc. need to be performed by hand. In addition to the kern and mfsroot floppy disks, you will also need to use the fixit floppy. [.procedure] ==== . Partitioning Your Flash Media Device + After booting with the kern and mfsroot floppies, choose `custom` from the installation menu. In the custom installation menu, choose `partition`. In the partition menu, you should delete all existing partitions using kbd:[d]. After deleting all existing partitions, create a partition using kbd:[c] and accept the default value for the size of the partition. When asked for the type of the partition, make sure the value is set to `165`. Now write this partition table to the disk by pressing kbd:[w] (this is a hidden option on this screen). If you are using an ATA compatible compact flash card, you should choose the FreeBSD Boot Manager. Now press kbd:[q] to quit the partition menu. You will be shown the boot manager menu once more - repeat the choice you made earlier. . Creating Filesystems on Your Flash Memory Device + Exit the custom installation menu, and from the main installation menu choose the `fixit` option. After entering the fixit environment, enter the following command: + [source,shell] .... # disklabel -e /dev/ad0c .... + At this point you will have entered the vi editor under the auspices of the disklabel command. Next, you need to add an `a:` line at the end of the file. This `a:` line should look like: + [.programlisting] .... a: 123456 0 4.2BSD 0 0 .... + Where _123456_ is a number that is exactly the same as the number in the existing `c:` entry for size. Basically you are duplicating the existing `c:` line as an `a:` line, making sure that fstype is `4.2BSD`. Save the file and exit. + [source,shell] .... # disklabel -B -r /dev/ad0c # newfs /dev/ad0a .... . Placing Your Filesystem on the Flash Media + Mount the newly prepared flash media: + [source,shell] .... # mount /dev/ad0a /flash .... + Bring this machine up on the network so we may transfer our tar file and explode it onto our flash media filesystem. One example of how to do this is: + [source,shell] .... # ifconfig xl0 192.168.0.10 netmask 255.255.255.0 # route add default 192.168.0.1 .... + Now that the machine is on the network, transfer your tar file. You may be faced with a bit of a dilemma at this point - if your flash memory part is 128 megabytes, for instance, and your tar file is larger than 64 megabytes, you cannot have your tar file on the flash media at the same time as you explode it - you will run out of space. One solution to this problem, if you are using FTP, is to untar the file while it is transferred over FTP. If you perform your transfer in this manner, you will never have the tar file and the tar contents on your disk at the same time: + [source,shell] .... ftp> get tarfile.tar "| tar xvf -" .... + If your tarfile is gzipped, you can accomplish this as well: + [source,shell] .... ftp> get tarfile.tar "| zcat | tar xvf -" .... + After the contents of your tarred filesystem are on your flash memory filesystem, you can unmount the flash memory and reboot: + [source,shell] .... # cd / # umount /flash # exit .... + Assuming that you configured your filesystem correctly when it was built on the normal hard disk (with your filesystems mounted read-only, and with the necessary options compiled into the kernel) you should now be successfully booting your FreeBSD embedded system. ==== [[strategies]] == System Strategies for Small and Read Only Environments In <>, it was pointed out that the [.filename]#/var# filesystem constructed by [.filename]#/etc/rc.d/var# and the presence of a read-only root filesystem causes problems with many common software packages used with FreeBSD. In this article, suggestions for successfully running cron, syslog, ports installations, and the Apache web server will be provided. === Cron Upon boot, [.filename]#/var# gets populated by [.filename]#/etc/rc.d/var# using the list from [.filename]#/etc/mtree/BSD.var.dist#, so the [.filename]#cron#, [.filename]#cron/tabs#, [.filename]#at#, and a few other standard directories get created. However, this does not solve the problem of maintaining cron tabs across reboots. When the system reboots, the [.filename]#/var# filesystem that is in memory will disappear and any cron tabs you may have had in it will also disappear. Therefore, one solution would be to create cron tabs for the users that need them, mount your [.filename]#/# filesystem as read-write and copy those cron tabs to somewhere safe, like [.filename]#/etc/tabs#, then add a line to the end of [.filename]#/etc/rc.initdiskless# that copies those crontabs into [.filename]#/var/cron/tabs# after that directory has been created during system initialization. You may also need to add a line that changes modes and permissions on the directories you create and the files you copy with [.filename]#/etc/rc.initdiskless#. === Syslog [.filename]#syslog.conf# specifies the locations of certain log files that exist in [.filename]#/var/log#. These files are not created by [.filename]#/etc/rc.d/var# upon system initialization. Therefore, somewhere in [.filename]#/etc/rc.d/var#, after the section that creates the directories in [.filename]#/var#, you will need to add something like this: [source,shell] .... # touch /var/log/security /var/log/maillog /var/log/cron /var/log/messages # chmod 0644 /var/log/* .... === Ports Installation Before discussing the changes necessary to successfully use the ports tree, a reminder is necessary regarding the read-only nature of your filesystems on the flash media. Since they are read-only, you will need to temporarily mount them read-write using the mount syntax shown in <>. You should always remount those filesystems read-only when you are done with any maintenance - unnecessary writes to the flash media could considerably shorten its lifespan. To make it possible to enter a ports directory and successfully run `make install`, we must create a packages directory on a non-memory filesystem that will keep track of our packages across reboots. As it is necessary to mount your filesystems as read-write for the installation of a package anyway, it is sensible to assume that an area on the flash media can also be used for package information to be written to. First, create a package database directory. This is normally in [.filename]#/var/db/pkg#, but we cannot place it there as it will disappear every time the system is booted. [source,shell] .... # mkdir /etc/pkg .... Now, add a line to [.filename]#/etc/rc.d/var# that links the [.filename]#/etc/pkg# directory to [.filename]#/var/db/pkg#. An example: [source,shell] .... # ln -s /etc/pkg /var/db/pkg .... Now, any time that you mount your filesystems as read-write and install a package, the `make install` will work, and package information will be written successfully to [.filename]#/etc/pkg# (because the filesystem will, at that time, be mounted read-write) which will always be available to the operating system as [.filename]#/var/db/pkg#. === Apache Web Server [NOTE] ==== The steps in this section are only necessary if Apache is set up to write its pid or log information outside of [.filename]#/var#. By default, Apache keeps its pid file in [.filename]#/var/run/httpd.pid# and its log files in [.filename]#/var/log#. ==== It is now assumed that Apache keeps its log files in a directory [.filename]#apache_log_dir# outside of [.filename]#/var#. When this directory lives on a read-only filesystem, Apache will not be able to save any log files, and may have problems working. If so, it is necessary to add a new directory to the list of directories in [.filename]#/etc/rc.d/var# to create in [.filename]#/var#, and to link [.filename]#apache_log_dir# to [.filename]#/var/log/apache#. It is also necessary to set permissions and ownership on this new directory. First, add the directory `log/apache` to the list of directories to be created in [.filename]#/etc/rc.d/var#. Second, add these commands to [.filename]#/etc/rc.d/var# after the directory creation section: [source,shell] .... # chmod 0774 /var/log/apache # chown nobody:nobody /var/log/apache .... Finally, remove the existing [.filename]#apache_log_dir# directory, and replace it with a link: [source,shell] .... # rm -rf apache_log_dir # ln -s /var/log/apache apache_log_dir .... diff --git a/documentation/content/en/articles/vm-design/_index.adoc b/documentation/content/en/articles/vm-design/_index.adoc index 12a60f6962..884e88bd76 100644 --- a/documentation/content/en/articles/vm-design/_index.adoc +++ b/documentation/content/en/articles/vm-design/_index.adoc @@ -1,411 +1,411 @@ --- title: Design elements of the FreeBSD VM system authors: - author: Matthew Dillon email: dillon@apollo.backplane.com -description: Design elements of the FreeBSD VM system +description: An easy to follow description of the design of the FreeBSD virtual memory system trademarks: ["freebsd", "linux", "microsoft", "opengroup", "daemon-news", "general"] --- = Design elements of the FreeBSD VM system :doctype: article :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: ifeval::["{backend}" == "html5"] :imagesdir: ../../../images/articles/vm-design/ endif::[] ifeval::["{backend}" == "pdf"] :imagesdir: ../../../../static/images/articles/vm-design/ endif::[] ifeval::["{backend}" == "epub3"] :imagesdir: ../../../../static/images/articles/vm-design/ endif::[] [.abstract-title] Abstract The title is really just a fancy way of saying that I am going to attempt to describe the whole VM enchilada, hopefully in a way that everyone can follow. For the last year I have concentrated on a number of major kernel subsystems within FreeBSD, with the VM and Swap subsystems being the most interesting and NFS being "a necessary chore". I rewrote only small portions of the code. In the VM arena the only major rewrite I have done is to the swap subsystem. Most of my work was cleanup and maintenance, with only moderate code rewriting and no major algorithmic adjustments within the VM subsystem. The bulk of the VM subsystem's theoretical base remains unchanged and a lot of the credit for the modernization effort in the last few years belongs to John Dyson and David Greenman. Not being a historian like Kirk I will not attempt to tag all the various features with peoples names, since I will invariably get it wrong. ''' toc::[] [[introduction]] == Introduction Before moving along to the actual design let's spend a little time on the necessity of maintaining and modernizing any long-living codebase. In the programming world, algorithms tend to be more important than code and it is precisely due to BSD's academic roots that a great deal of attention was paid to algorithm design from the beginning. More attention paid to the design generally leads to a clean and flexible codebase that can be fairly easily modified, extended, or replaced over time. While BSD is considered an "old" operating system by some people, those of us who work on it tend to view it more as a "mature" codebase which has various components modified, extended, or replaced with modern code. It has evolved, and FreeBSD is at the bleeding edge no matter how old some of the code might be. This is an important distinction to make and one that is unfortunately lost to many people. The biggest error a programmer can make is to not learn from history, and this is precisely the error that many other modern operating systems have made. Windows NT(R) is the best example of this, and the consequences have been dire. Linux also makes this mistake to some degree-enough that we BSD folk can make small jokes about it every once in a while, anyway. Linux's problem is simply one of a lack of experience and history to compare ideas against, a problem that is easily and rapidly being addressed by the Linux community in the same way it has been addressed in the BSD community-by continuous code development. The Windows NT(R) folk, on the other hand, repeatedly make the same mistakes solved by UNIX(R) decades ago and then spend years fixing them. Over and over again. They have a severe case of "not designed here" and "we are always right because our marketing department says so". I have little tolerance for anyone who cannot learn from history. Much of the apparent complexity of the FreeBSD design, especially in the VM/Swap subsystem, is a direct result of having to solve serious performance issues that occur under various conditions. These issues are not due to bad algorithmic design but instead rise from environmental factors. In any direct comparison between platforms, these issues become most apparent when system resources begin to get stressed. As I describe FreeBSD's VM/Swap subsystem the reader should always keep two points in mind: . The most important aspect of performance design is what is known as "Optimizing the Critical Path". It is often the case that performance optimizations add a little bloat to the code in order to make the critical path perform better. . A solid, generalized design outperforms a heavily-optimized design over the long run. While a generalized design may end up being slower than an heavily-optimized design when they are first implemented, the generalized design tends to be easier to adapt to changing conditions and the heavily-optimized design winds up having to be thrown away. Any codebase that will survive and be maintainable for years must therefore be designed properly from the beginning even if it costs some performance. Twenty years ago people were still arguing that programming in assembly was better than programming in a high-level language because it produced code that was ten times as fast. Today, the fallibility of that argument is obvious - as are the parallels to algorithmic design and code generalization. [[vm-objects]] == VM Objects The best way to begin describing the FreeBSD VM system is to look at it from the perspective of a user-level process. Each user process sees a single, private, contiguous VM address space containing several types of memory objects. These objects have various characteristics. Program code and program data are effectively a single memory-mapped file (the binary file being run), but program code is read-only while program data is copy-on-write. Program BSS is just memory allocated and filled with zeros on demand, called demand zero page fill. Arbitrary files can be memory-mapped into the address space as well, which is how the shared library mechanism works. Such mappings can require modifications to remain private to the process making them. The fork system call adds an entirely new dimension to the VM management problem on top of the complexity already given. A program binary data page (which is a basic copy-on-write page) illustrates the complexity. A program binary contains a preinitialized data section which is initially mapped directly from the program file. When a program is loaded into a process's VM space, this area is initially memory-mapped and backed by the program binary itself, allowing the VM system to free/reuse the page and later load it back in from the binary. The moment a process modifies this data, however, the VM system must make a private copy of the page for that process. Since the private copy has been modified, the VM system may no longer free it, because there is no longer any way to restore it later on. You will notice immediately that what was originally a simple file mapping has become much more complex. Data may be modified on a page-by-page basis whereas the file mapping encompasses many pages at once. The complexity further increases when a process forks. When a process forks, the result is two processes-each with their own private address spaces, including any modifications made by the original process prior to the call to `fork()`. It would be silly for the VM system to make a complete copy of the data at the time of the `fork()` because it is quite possible that at least one of the two processes will only need to read from that page from then on, allowing the original page to continue to be used. What was a private page is made copy-on-write again, since each process (parent and child) expects their own personal post-fork modifications to remain private to themselves and not effect the other. FreeBSD manages all of this with a layered VM Object model. The original binary program file winds up being the lowest VM Object layer. A copy-on-write layer is pushed on top of that to hold those pages which had to be copied from the original file. If the program modifies a data page belonging to the original file the VM system takes a fault and makes a copy of the page in the higher layer. When a process forks, additional VM Object layers are pushed on. This might make a little more sense with a fairly basic example. A `fork()` is a common operation for any *BSD system, so this example will consider a program that starts up, and forks. When the process starts, the VM system creates an object layer, let's call this A: image::fig1.png[A picture] A represents the file-pages may be paged in and out of the file's physical media as necessary. Paging in from the disk is reasonable for a program, but we really do not want to page back out and overwrite the executable. The VM system therefore creates a second layer, B, that will be physically backed by swap space: image::fig2.png[] On the first write to a page after this, a new page is created in B, and its contents are initialized from A. All pages in B can be paged in or out to a swap device. When the program forks, the VM system creates two new object layers-C1 for the parent, and C2 for the child-that rest on top of B: image::fig3.png[] In this case, let's say a page in B is modified by the original parent process. The process will take a copy-on-write fault and duplicate the page in C1, leaving the original page in B untouched. Now, let's say the same page in B is modified by the child process. The process will take a copy-on-write fault and duplicate the page in C2. The original page in B is now completely hidden since both C1 and C2 have a copy and B could theoretically be destroyed if it does not represent a "real" file; however, this sort of optimization is not trivial to make because it is so fine-grained. FreeBSD does not make this optimization. Now, suppose (as is often the case) that the child process does an `exec()`. Its current address space is usually replaced by a new address space representing a new file. In this case, the C2 layer is destroyed: image::fig4.png[] In this case, the number of children of B drops to one, and all accesses to B now go through C1. This means that B and C1 can be collapsed together. Any pages in B that also exist in C1 are deleted from B during the collapse. Thus, even though the optimization in the previous step could not be made, we can recover the dead pages when either of the processes exit or `exec()`. This model creates a number of potential problems. The first is that you can wind up with a relatively deep stack of layered VM Objects which can cost scanning time and memory when you take a fault. Deep layering can occur when processes fork and then fork again (either parent or child). The second problem is that you can wind up with dead, inaccessible pages deep in the stack of VM Objects. In our last example if both the parent and child processes modify the same page, they both get their own private copies of the page and the original page in B is no longer accessible by anyone. That page in B can be freed. FreeBSD solves the deep layering problem with a special optimization called the "All Shadowed Case". This case occurs if either C1 or C2 take sufficient COW faults to completely shadow all pages in B. Lets say that C1 achieves this. C1 can now bypass B entirely, so rather then have C1->B->A and C2->B->A we now have C1->A and C2->B->A. But look what also happened-now B has only one reference (C2), so we can collapse B and C2 together. The end result is that B is deleted entirely and we have C1->A and C2->A. It is often the case that B will contain a large number of pages and neither C1 nor C2 will be able to completely overshadow it. If we fork again and create a set of D layers, however, it is much more likely that one of the D layers will eventually be able to completely overshadow the much smaller dataset represented by C1 or C2. The same optimization will work at any point in the graph and the grand result of this is that even on a heavily forked machine VM Object stacks tend to not get much deeper then 4. This is true of both the parent and the children and true whether the parent is doing the forking or whether the children cascade forks. The dead page problem still exists in the case where C1 or C2 do not completely overshadow B. Due to our other optimizations this case does not represent much of a problem and we simply allow the pages to be dead. If the system runs low on memory it will swap them out, eating a little swap, but that is it. The advantage to the VM Object model is that `fork()` is extremely fast, since no real data copying need take place. The disadvantage is that you can build a relatively complex VM Object layering that slows page fault handling down a little, and you spend memory managing the VM Object structures. The optimizations FreeBSD makes proves to reduce the problems enough that they can be ignored, leaving no real disadvantage. [[swap-layers]] == SWAP Layers Private data pages are initially either copy-on-write or zero-fill pages. When a change, and therefore a copy, is made, the original backing object (usually a file) can no longer be used to save a copy of the page when the VM system needs to reuse it for other purposes. This is where SWAP comes in. SWAP is allocated to create backing store for memory that does not otherwise have it. FreeBSD allocates the swap management structure for a VM Object only when it is actually needed. However, the swap management structure has had problems historically: * Under FreeBSD 3.X the swap management structure preallocates an array that encompasses the entire object requiring swap backing store-even if only a few pages of that object are swap-backed. This creates a kernel memory fragmentation problem when large objects are mapped, or processes with large runsizes (RSS) fork. * Also, in order to keep track of swap space, a "list of holes" is kept in kernel memory, and this tends to get severely fragmented as well. Since the "list of holes" is a linear list, the swap allocation and freeing performance is a non-optimal O(n)-per-page. * It requires kernel memory allocations to take place during the swap freeing process, and that creates low memory deadlock problems. * The problem is further exacerbated by holes created due to the interleaving algorithm. * Also, the swap block map can become fragmented fairly easily resulting in non-contiguous allocations. * Kernel memory must also be allocated on the fly for additional swap management structures when a swapout occurs. It is evident from that list that there was plenty of room for improvement. For FreeBSD 4.X, I completely rewrote the swap subsystem: * Swap management structures are allocated through a hash table rather than a linear array giving them a fixed allocation size and much finer granularity. * Rather then using a linearly linked list to keep track of swap space reservations, it now uses a bitmap of swap blocks arranged in a radix tree structure with free-space hinting in the radix node structures. This effectively makes swap allocation and freeing an O(1) operation. * The entire radix tree bitmap is also preallocated in order to avoid having to allocate kernel memory during critical low memory swapping operations. After all, the system tends to swap when it is low on memory so we should avoid allocating kernel memory at such times in order to avoid potential deadlocks. * To reduce fragmentation the radix tree is capable of allocating large contiguous chunks at once, skipping over smaller fragmented chunks. I did not take the final step of having an "allocating hint pointer" that would trundle through a portion of swap as allocations were made in order to further guarantee contiguous allocations or at least locality of reference, but I ensured that such an addition could be made. [[freeing-pages]] == When to free a page Since the VM system uses all available memory for disk caching, there are usually very few truly-free pages. The VM system depends on being able to properly choose pages which are not in use to reuse for new allocations. Selecting the optimal pages to free is possibly the single-most important function any VM system can perform because if it makes a poor selection, the VM system may be forced to unnecessarily retrieve pages from disk, seriously degrading system performance. How much overhead are we willing to suffer in the critical path to avoid freeing the wrong page? Each wrong choice we make will cost us hundreds of thousands of CPU cycles and a noticeable stall of the affected processes, so we are willing to endure a significant amount of overhead in order to be sure that the right page is chosen. This is why FreeBSD tends to outperform other systems when memory resources become stressed. The free page determination algorithm is built upon a history of the use of memory pages. To acquire this history, the system takes advantage of a page-used bit feature that most hardware page tables have. In any case, the page-used bit is cleared and at some later point the VM system comes across the page again and sees that the page-used bit has been set. This indicates that the page is still being actively used. If the bit is still clear it is an indication that the page is not being actively used. By testing this bit periodically, a use history (in the form of a counter) for the physical page is developed. When the VM system later needs to free up some pages, checking this history becomes the cornerstone of determining the best candidate page to reuse. For those platforms that do not have this feature, the system actually emulates a page-used bit. It unmaps or protects a page, forcing a page fault if the page is accessed again. When the page fault is taken, the system simply marks the page as having been used and unprotects the page so that it may be used. While taking such page faults just to determine if a page is being used appears to be an expensive proposition, it is much less expensive than reusing the page for some other purpose only to find that a process needs it back and then have to go to disk. FreeBSD makes use of several page queues to further refine the selection of pages to reuse as well as to determine when dirty pages must be flushed to their backing store. Since page tables are dynamic entities under FreeBSD, it costs virtually nothing to unmap a page from the address space of any processes using it. When a page candidate has been chosen based on the page-use counter, this is precisely what is done. The system must make a distinction between clean pages which can theoretically be freed up at any time, and dirty pages which must first be written to their backing store before being reusable. When a page candidate has been found it is moved to the inactive queue if it is dirty, or the cache queue if it is clean. A separate algorithm based on the dirty-to-clean page ratio determines when dirty pages in the inactive queue must be flushed to disk. Once this is accomplished, the flushed pages are moved from the inactive queue to the cache queue. At this point, pages in the cache queue can still be reactivated by a VM fault at relatively low cost. However, pages in the cache queue are considered to be "immediately freeable" and will be reused in an LRU (least-recently used) fashion when the system needs to allocate new memory. It is important to note that the FreeBSD VM system attempts to separate clean and dirty pages for the express reason of avoiding unnecessary flushes of dirty pages (which eats I/O bandwidth), nor does it move pages between the various page queues gratuitously when the memory subsystem is not being stressed. This is why you will see some systems with very low cache queue counts and high active queue counts when doing a `systat -vm` command. As the VM system becomes more stressed, it makes a greater effort to maintain the various page queues at the levels determined to be the most effective. An urban myth has circulated for years that Linux did a better job avoiding swapouts than FreeBSD, but this in fact is not true. What was actually occurring was that FreeBSD was proactively paging out unused pages in order to make room for more disk cache while Linux was keeping unused pages in core and leaving less memory available for cache and process pages. I do not know whether this is still true today. [[prefault-optimizations]] == Pre-Faulting and Zeroing Optimizations Taking a VM fault is not expensive if the underlying page is already in core and can simply be mapped into the process, but it can become expensive if you take a whole lot of them on a regular basis. A good example of this is running a program such as man:ls[1] or man:ps[1] over and over again. If the program binary is mapped into memory but not mapped into the page table, then all the pages that will be accessed by the program will have to be faulted in every time the program is run. This is unnecessary when the pages in question are already in the VM Cache, so FreeBSD will attempt to pre-populate a process's page tables with those pages that are already in the VM Cache. One thing that FreeBSD does not yet do is pre-copy-on-write certain pages on exec. For example, if you run the man:ls[1] program while running `vmstat 1` you will notice that it always takes a certain number of page faults, even when you run it over and over again. These are zero-fill faults, not program code faults (which were pre-faulted in already). Pre-copying pages on exec or fork is an area that could use more study. A large percentage of page faults that occur are zero-fill faults. You can usually see this by observing the `vmstat -s` output. These occur when a process accesses pages in its BSS area. The BSS area is expected to be initially zero but the VM system does not bother to allocate any memory at all until the process actually accesses it. When a fault occurs the VM system must not only allocate a new page, it must zero it as well. To optimize the zeroing operation the VM system has the ability to pre-zero pages and mark them as such, and to request pre-zeroed pages when zero-fill faults occur. The pre-zeroing occurs whenever the CPU is idle but the number of pages the system pre-zeros is limited in order to avoid blowing away the memory caches. This is an excellent example of adding complexity to the VM system in order to optimize the critical path. [[page-table-optimizations]] == Page Table Optimizations The page table optimizations make up the most contentious part of the FreeBSD VM design and they have shown some strain with the advent of serious use of `mmap()`. I think this is actually a feature of most BSDs though I am not sure when it was first introduced. There are two major optimizations. The first is that hardware page tables do not contain persistent state but instead can be thrown away at any time with only a minor amount of management overhead. The second is that every active page table entry in the system has a governing `pv_entry` structure which is tied into the `vm_page` structure. FreeBSD can simply iterate through those mappings that are known to exist while Linux must check all page tables that _might_ contain a specific mapping to see if it does, which can achieve O(n^2) overhead in certain situations. It is because of this that FreeBSD tends to make better choices on which pages to reuse or swap when memory is stressed, giving it better performance under load. However, FreeBSD requires kernel tuning to accommodate large-shared-address-space situations such as those that can occur in a news system because it may run out of `pv_entry` structures. Both Linux and FreeBSD need work in this area. FreeBSD is trying to maximize the advantage of a potentially sparse active-mapping model (not all processes need to map all pages of a shared library, for example), whereas Linux is trying to simplify its algorithms. FreeBSD generally has the performance advantage here at the cost of wasting a little extra memory, but FreeBSD breaks down in the case where a large file is massively shared across hundreds of processes. Linux, on the other hand, breaks down in the case where many processes are sparsely-mapping the same shared library and also runs non-optimally when trying to determine whether a page can be reused or not. [[page-coloring-optimizations]] == Page Coloring We will end with the page coloring optimizations. Page coloring is a performance optimization designed to ensure that accesses to contiguous pages in virtual memory make the best use of the processor cache. In ancient times (i.e. 10+ years ago) processor caches tended to map virtual memory rather than physical memory. This led to a huge number of problems including having to clear the cache on every context switch in some cases, and problems with data aliasing in the cache. Modern processor caches map physical memory precisely to solve those problems. This means that two side-by-side pages in a processes address space may not correspond to two side-by-side pages in the cache. In fact, if you are not careful side-by-side pages in virtual memory could wind up using the same page in the processor cache-leading to cacheable data being thrown away prematurely and reducing CPU performance. This is true even with multi-way set-associative caches (though the effect is mitigated somewhat). FreeBSD's memory allocation code implements page coloring optimizations, which means that the memory allocation code will attempt to locate free pages that are contiguous from the point of view of the cache. For example, if page 16 of physical memory is assigned to page 0 of a process's virtual memory and the cache can hold 4 pages, the page coloring code will not assign page 20 of physical memory to page 1 of a process's virtual memory. It would, instead, assign page 21 of physical memory. The page coloring code attempts to avoid assigning page 20 because this maps over the same cache memory as page 16 and would result in non-optimal caching. This code adds a significant amount of complexity to the VM memory allocation subsystem as you can well imagine, but the result is well worth the effort. Page Coloring makes VM memory as deterministic as physical memory in regards to cache performance. [[conclusion]] == Conclusion Virtual memory in modern operating systems must address a number of different issues efficiently and for many different usage patterns. The modular and algorithmic approach that BSD has historically taken allows us to study and understand the current implementation as well as relatively cleanly replace large sections of the code. There have been a number of improvements to the FreeBSD VM system in the last several years, and work is ongoing. [[allen-briggs-qa]] == Bonus QA session by Allen Briggs === What is the interleaving algorithm that you refer to in your listing of the ills of the FreeBSD 3.X swap arrangements? FreeBSD uses a fixed swap interleave which defaults to 4. This means that FreeBSD reserves space for four swap areas even if you only have one, two, or three. Since swap is interleaved the linear address space representing the "four swap areas" will be fragmented if you do not actually have four swap areas. For example, if you have two swap areas A and B FreeBSD's address space representation for that swap area will be interleaved in blocks of 16 pages: .... A B C D A B C D A B C D A B C D .... FreeBSD 3.X uses a "sequential list of free regions" approach to accounting for the free swap areas. The idea is that large blocks of free linear space can be represented with a single list node ([.filename]#kern/subr_rlist.c#). But due to the fragmentation the sequential list winds up being insanely fragmented. In the above example, completely unused swap will have A and B shown as "free" and C and D shown as "all allocated". Each A-B sequence requires a list node to account for because C and D are holes, so the list node cannot be combined with the next A-B sequence. Why do we interleave our swap space instead of just tack swap areas onto the end and do something fancier? It is a whole lot easier to allocate linear swaths of an address space and have the result automatically be interleaved across multiple disks than it is to try to put that sophistication elsewhere. The fragmentation causes other problems. Being a linear list under 3.X, and having such a huge amount of inherent fragmentation, allocating and freeing swap winds up being an O(N) algorithm instead of an O(1) algorithm. Combined with other factors (heavy swapping) and you start getting into O(N^2) and O(N^3) levels of overhead, which is bad. The 3.X system may also need to allocate KVM during a swap operation to create a new list node which can lead to a deadlock if the system is trying to pageout pages in a low-memory situation. Under 4.X we do not use a sequential list. Instead we use a radix tree and bitmaps of swap blocks rather than ranged list nodes. We take the hit of preallocating all the bitmaps required for the entire swap area up front but it winds up wasting less memory due to the use of a bitmap (one bit per block) instead of a linked list of nodes. The use of a radix tree instead of a sequential list gives us nearly O(1) performance no matter how fragmented the tree becomes. === How is the separation of clean and dirty (inactive) pages related to the situation where you see low cache queue counts and high active queue counts in systat -vm? Do the systat stats roll the active and dirty pages together for the active queue count? Yes, that is confusing. The relationship is "goal" verses "reality". Our goal is to separate the pages but the reality is that if we are not in a memory crunch, we do not really have to. What this means is that FreeBSD will not try very hard to separate out dirty pages (inactive queue) from clean pages (cache queue) when the system is not being stressed, nor will it try to deactivate pages (active queue -> inactive queue) when the system is not being stressed, even if they are not being used. === In man:ls[1] the / vmstat 1 example, would not some of the page faults be data page faults (COW from executable file to private page)? I.e., I would expect the page faults to be some zero-fill and some program data. Or are you implying that FreeBSD does do pre-COW for the program data? A COW fault can be either zero-fill or program-data. The mechanism is the same either way because the backing program-data is almost certainly already in the cache. I am indeed lumping the two together. FreeBSD does not pre-COW program data or zero-fill, but it _does_ pre-map pages that exist in its cache. === In your section on page table optimizations, can you give a little more detail about pv_entry and vm_page (or should vm_page be vm_pmap-as in 4.4, cf. pp. 180-181 of McKusick, Bostic, Karel, Quarterman)? Specifically, what kind of operation/reaction would require scanning the mappings? A `vm_page` represents an (object,index#) tuple. A `pv_entry` represents a hardware page table entry (pte). If you have five processes sharing the same physical page, and three of those processes's page tables actually map the page, that page will be represented by a single `vm_page` structure and three `pv_entry` structures. `pv_entry` structures only represent pages mapped by the MMU (one `pv_entry` represents one pte). This means that when we need to remove all hardware references to a `vm_page` (in order to reuse the page for something else, page it out, clear it, dirty it, and so forth) we can simply scan the linked list of pv_entry's associated with that vm_page to remove or modify the pte's from their page tables. Under Linux there is no such linked list. In order to remove all the hardware page table mappings for a `vm_page` linux must index into every VM object that _might_ have mapped the page. For example, if you have 50 processes all mapping the same shared library and want to get rid of page X in that library, you need to index into the page table for each of those 50 processes even if only 10 of them have actually mapped the page. So Linux is trading off the simplicity of its design against performance. Many VM algorithms which are O(1) or (small N) under FreeBSD wind up being O(N), O(N^2), or worse under Linux. Since the pte's representing a particular page in an object tend to be at the same offset in all the page tables they are mapped in, reducing the number of accesses into the page tables at the same pte offset will often avoid blowing away the L1 cache line for that offset, which can lead to better performance. FreeBSD has added complexity (the `pv_entry` scheme) in order to increase performance (to limit page table accesses to _only_ those pte's that need to be modified). But FreeBSD has a scaling problem that Linux does not in that there are a limited number of `pv_entry` structures and this causes problems when you have massive sharing of data. In this case you may run out of `pv_entry` structures even though there is plenty of free memory available. This can be fixed easily enough by bumping up the number of `pv_entry` structures in the kernel config, but we really need to find a better way to do it. In regards to the memory overhead of a page table verses the `pv_entry` scheme: Linux uses "permanent" page tables that are not throw away, but does not need a `pv_entry` for each potentially mapped pte. FreeBSD uses "throw away" page tables but adds in a `pv_entry` structure for each actually-mapped pte. I think memory utilization winds up being about the same, giving FreeBSD an algorithmic advantage with its ability to throw away page tables at will with very low overhead. === Finally, in the page coloring section, it might help to have a little more description of what you mean here. I did not quite follow it. Do you know how an L1 hardware memory cache works? I will explain: Consider a machine with 16MB of main memory but only 128K of L1 cache. Generally the way this cache works is that each 128K block of main memory uses the _same_ 128K of cache. If you access offset 0 in main memory and then offset 128K in main memory you can wind up throwing away the cached data you read from offset 0! Now, I am simplifying things greatly. What I just described is what is called a "direct mapped" hardware memory cache. Most modern caches are what are called 2-way-set-associative or 4-way-set-associative caches. The set-associatively allows you to access up to N different memory regions that overlap the same cache memory without destroying the previously cached data. But only N. So if I have a 4-way set associative cache I can access offset 0, offset 128K, 256K and offset 384K and still be able to access offset 0 again and have it come from the L1 cache. If I then access offset 512K, however, one of the four previously cached data objects will be thrown away by the cache. It is extremely important... _extremely_ important for most of a processor's memory accesses to be able to come from the L1 cache, because the L1 cache operates at the processor frequency. The moment you have an L1 cache miss and have to go to the L2 cache or to main memory, the processor will stall and potentially sit twiddling its fingers for _hundreds_ of instructions worth of time waiting for a read from main memory to complete. Main memory (the dynamic ram you stuff into a computer) is __slow__, when compared to the speed of a modern processor core. Ok, so now onto page coloring: All modern memory caches are what are known as _physical_ caches. They cache physical memory addresses, not virtual memory addresses. This allows the cache to be left alone across a process context switch, which is very important. But in the UNIX(R) world you are dealing with virtual address spaces, not physical address spaces. Any program you write will see the virtual address space given to it. The actual _physical_ pages underlying that virtual address space are not necessarily physically contiguous! In fact, you might have two pages that are side by side in a processes address space which wind up being at offset 0 and offset 128K in _physical_ memory. A program normally assumes that two side-by-side pages will be optimally cached. That is, that you can access data objects in both pages without having them blow away each other's cache entry. But this is only true if the physical pages underlying the virtual address space are contiguous (insofar as the cache is concerned). This is what Page coloring does. Instead of assigning _random_ physical pages to virtual addresses, which may result in non-optimal cache performance, Page coloring assigns _reasonably-contiguous_ physical pages to virtual addresses. Thus programs can be written under the assumption that the characteristics of the underlying hardware cache are the same for their virtual address space as they would be if the program had been run directly in a physical address space. Note that I say "reasonably" contiguous rather than simply "contiguous". From the point of view of a 128K direct mapped cache, the physical address 0 is the same as the physical address 128K. So two side-by-side pages in your virtual address space may wind up being offset 128K and offset 132K in physical memory, but could also easily be offset 128K and offset 4K in physical memory and still retain the same cache performance characteristics. So page-coloring does _not_ have to assign truly contiguous pages of physical memory to contiguous pages of virtual memory, it just needs to make sure it assigns contiguous pages from the point of view of cache performance and operation. diff --git a/documentation/content/en/books/arch-handbook/_index.adoc b/documentation/content/en/books/arch-handbook/_index.adoc index a5cefc4616..59041a32d6 100644 --- a/documentation/content/en/books/arch-handbook/_index.adoc +++ b/documentation/content/en/books/arch-handbook/_index.adoc @@ -1,60 +1,61 @@ --- title: FreeBSD Architecture Handbook authors: - author: The FreeBSD Documentation Project copyright: Copyright © 2000-2006, 2012-2021 The FreeBSD Documentation Project -description: FreeBSD Architecture Handbook Index +description: For FreeBSD system developers. This book covers the architectural details of many important FreeBSD kernel subsystems trademarks: ["freebsd", "apple", "microsoft", "unix", "general"] next: books/arch-handbook/parti +weight: 50 --- = FreeBSD Architecture Handbook :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: ifeval::["{backend}" == "html5"] include::shared/mirrors.adoc[] include::shared/authors.adoc[] include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/teams.adoc[] include::shared/en/urls.adoc[] :chapters-path: content/en/books/arch-handbook/ endif::[] ifeval::["{backend}" == "pdf"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] :chapters-path: endif::[] ifeval::["{backend}" == "epub3"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] :chapters-path: endif::[] [.abstract-title] Abstract Welcome to the FreeBSD Architecture Handbook. This manual is a _work in progress_ and is the work of many individuals. Many sections do not yet exist and some of those that do exist need to be updated. If you are interested in helping with this project, send email to the {freebsd-doc}. The latest version of this document is always available from the link:https://www.FreeBSD.org/[FreeBSD World Wide Web server]. It may also be downloaded in a variety of formats and compression options from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server] or one of the numerous link:{handbook}#mirrors-ftp/[mirror sites]. ''' include::content/en/books/arch-handbook/toc.adoc[] diff --git a/documentation/content/en/books/design-44bsd/_index.adoc b/documentation/content/en/books/design-44bsd/_index.adoc index bab937e2fc..596a138529 100644 --- a/documentation/content/en/books/design-44bsd/_index.adoc +++ b/documentation/content/en/books/design-44bsd/_index.adoc @@ -1,507 +1,508 @@ --- title: The Design and Implementation of the 4.4BSD Operating System authors: - author: Marshall Kirk McKusick - author: Keith Bostic - author: Michael J. Karels - author: John S. Quarterman copyright: 1996 Addison-Wesley Longman, Inc -description: The Design and Implementation of the 4.4BSD Operating System. Second chapter. +description: Donated by Addison-Wesley, provides a design overview of 4.4BSD, from which FreeBSD was originally derived trademarks: ["design-44bsd"] +weight: 60 --- = The Design and Implementation of the 4.4BSD Operating System :doctype: book :toc: macro :toclevels: 2 :icons: font :xrefstyle: basic :relfileprefix: ../ :outfilesuffix: :sectnums: :sectnumlevels: 6 :partnums: :chapter-signifier: Chapter :part-signifier: Part :source-highlighter: rouge :experimental: :skip-front-matter: :sectnumoffset: 2 ifeval::["{backend}" == "html5"] include::shared/mirrors.adoc[] include::shared/authors.adoc[] include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/teams.adoc[] include::shared/en/urls.adoc[] :imagesdir: ../../images/books/design-44bsd/ endif::[] ifeval::["{backend}" == "pdf"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../static/images/books/design-44bsd/ endif::[] ifeval::["{backend}" == "epub3"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../static/images/books/design-44bsd/ endif::[] ''' toc::[] [[overview]] == Design Overview of 4.4BSD [[overview-facilities]] === 4.4BSD Facilities and the Kernel The 4.4BSD kernel provides four basic facilities: processes, a filesystem, communications, and system startup. This section outlines where each of these four basic services is described in this book. . Processes constitute a thread of control in an address space. Mechanisms for creating, terminating, and otherwise controlling processes are described in Chapter 4. The system multiplexes separate virtual-address spaces for each process; this memory management is discussed in Chapter 5. . The user interface to the filesystem and devices is similar; common aspects are discussed in Chapter 6. The filesystem is a set of named files, organized in a tree-structured hierarchy of directories, and of operations to manipulate them, as presented in Chapter 7. Files reside on physical media such as disks. 4.4BSD supports several organizations of data on the disk, as set forth in Chapter 8. Access to files on remote machines is the subject of Chapter 9. Terminals are used to access the system; their operation is the subject of Chapter 10. . Communication mechanisms provided by traditional UNIX systems include simplex reliable byte streams between related processes (see pipes, Section 11.1), and notification of exceptional events (see signals, Section 4.7). 4.4BSD also has a general interprocess-communication facility. This facility, described in Chapter 11, uses access mechanisms distinct from those of the filesystem, but, once a connection is set up, a process can access it as though it were a pipe. There is a general networking framework, discussed in Chapter 12, that is normally used as a layer underlying the IPC facility. Chapter 13 describes a particular networking implementation in detail. . Any real operating system has operational issues, such as how to start it running. Startup and operational issues are described in Chapter 14. Sections 2.3 through 2.14 present introductory material related to Chapters 3 through 14. We shall define terms, mention basic system calls, and explore historical developments. Finally, we shall give the reasons for many major design decisions. ==== The Kernel The _kernel_ is the part of the system that runs in protected mode and mediates access by all user programs to the underlying hardware (e.g., CPU, disks, terminals, network links) and software constructs (e.g., filesystem, network protocols). The kernel provides the basic system facilities; it creates and manages processes, and provides functions to access the filesystem and communication facilities. These functions, called _system calls_ appear to user processes as library subroutines. These system calls are the only interface that processes have to these facilities. Details of the system-call mechanism are given in Chapter 3, as are descriptions of several kernel mechanisms that do not execute as the direct result of a process doing a system call. A _kernel_ in traditional operating-system terminology, is a small nucleus of software that provides only the minimal facilities necessary for implementing additional operating-system services. In contemporary research operating systems -- such as Chorus <>, Mach <>, Tunis <>, and the V Kernel <> -- this division of functionality is more than just a logical one. Services such as filesystems and networking protocols are implemented as client application processes of the nucleus or kernel. The 4.4BSD kernel is not partitioned into multiple processes. This basic design decision was made in the earliest versions of UNIX. The first two implementations by Ken Thompson had no memory mapping, and thus made no hardware-enforced distinction between user and kernel space <>. A message-passing system could have been implemented as readily as the actually implemented model of kernel and user processes. The monolithic kernel was chosen for simplicity and performance. And the early kernels were small; the inclusion of facilities such as networking into the kernel has increased its size. The current trend in operating-systems research is to reduce the kernel size by placing such services in user space. Users ordinarily interact with the system through a command-language interpreter, called a _shell_, and perhaps through additional user application programs. Such programs and the shell are implemented with processes. Details of such programs are beyond the scope of this book, which instead concentrates almost exclusively on the kernel. Sections 2.3 and 2.4 describe the services provided by the 4.4BSD kernel, and give an overview of the latter's design. Later chapters describe the detailed design and implementation of these services as they appear in 4.4BSD. [[overview-kernel-organization]] === Kernel Organization In this section, we view the organization of the 4.4BSD kernel in two ways: [arabic] . As a static body of software, categorized by the functionality offered by the modules that make up the kernel . By its dynamic operation, categorized according to the services provided to users The largest part of the kernel implements the system services that applications access through system calls. In 4.4BSD, this software has been organized according to the following: * Basic kernel facilities: timer and system-clock handling, descriptor management, and process management * Memory-management support: paging and swapping * Generic system interfaces: the I/O, control, and multiplexing operations performed on descriptors * The filesystem: files, directories, pathname translation, file locking, and I/O buffer management * Terminal-handling support: the terminal-interface driver and terminal line disciplines * Interprocess-communication facilities: sockets * Support for network communication: communication protocols and generic network facilities, such as routing .Machine-independent software in the 4.4BSD kernel [[table-mach-indep]] [cols=",,",options="header",] |=== |Category |Lines of code |Percentage of kernel |headers |9,393 |4.6 |initialization |1,107 |0.6 |kernel facilities |8,793 |4.4 |generic interfaces |4,782 |2.4 |interprocess communication |4,540 |2.2 |terminal handling |3,911 |1.9 |virtual memory |11,813 |5.8 |vnode management |7,954 |3.9 |filesystem naming |6,550 |3.2 |fast filestore |4,365 |2.2 |log-structure filestore |4,337 |2.1 |memory-based filestore |645 |0.3 |cd9660 filesystem |4,177 |2.1 |miscellaneous filesystems (10) |12,695 |6.3 |network filesystem |17,199 |8.5 |network communication |8,630 |4.3 |internet protocols |11,984 |5.9 |ISO protocols |23,924 |11.8 |X.25 protocols |10,626 |5.3 |XNS protocols |5,192 |2.6 |=== Most of the software in these categories is machine independent and is portable across different hardware architectures. The machine-dependent aspects of the kernel are isolated from the mainstream code. In particular, none of the machine-independent code contains conditional code for specific architecture. When an architecture-dependent action is needed, the machine-independent code calls an architecture-dependent function that is located in the machine-dependent code. The software that is machine dependent includes * Low-level system-startup actions * Trap and fault handling * Low-level manipulation of the run-time context of a process * Configuration and initialization of hardware devices * Run-time support for I/O devices .Machine-dependent software for the HP300 in the 4.4BSD kernel [[table-mach-dep]] [cols=",,",options="header",] |=== |Category |Lines of code |Percentage of kernel |machine dependent headers |1,562 |0.8 |device driver headers |3,495 |1.7 |device driver source |17,506 |8.7 |virtual memory |3,087 |1.5 |other machine dependent |6,287 |3.1 |routines in assembly language |3,014 |1.5 |HP/UX compatibility |4,683 |2.3 |=== <> summarizes the machine-independent software that constitutes the 4.4BSD kernel for the HP300. The numbers in column 2 are for lines of C source code, header files, and assembly language. Virtually all the software in the kernel is written in the C programming language; less than 2 percent is written in assembly language. As the statistics in <> show, the machine-dependent software, excluding HP/UX and device support, accounts for a minuscule 6.9 percent of the kernel. Only a small part of the kernel is devoted to initializing the system. This code is used when the system is _bootstrapped_ into operation and is responsible for setting up the kernel hardware and software environment (see Chapter 14). Some operating systems (especially those with limited physical memory) discard or _overlay_ the software that performs these functions after that software has been executed. The 4.4BSD kernel does not reclaim the memory used by the startup code because that memory space is barely 0.5 percent of the kernel resources used on a typical machine. Also, the startup code does not appear in one place in the kernel -- it is scattered throughout, and it usually appears in places logically associated with what is being initialized. [[overview-kernel-service]] === Kernel Services The boundary between the kernel- and user-level code is enforced by hardware-protection facilities provided by the underlying hardware. The kernel operates in a separate address space that is inaccessible to user processes. Privileged operations -- such as starting I/O and halting the central processing unit (CPU) -- are available to only the kernel. Applications request services from the kernel with _system calls_. System calls are used to cause the kernel to execute complicated operations, such as writing data to secondary storage, and simple operations, such as returning the current time of day. All system calls appear _synchronous_ to applications: The application does not run while the kernel does the actions associated with a system call. The kernel may finish some operations associated with a system call after it has returned. For example, a _write_ system call will copy the data to be written from the user process to a kernel buffer while the process waits, but will usually return from the system call before the kernel buffer is written to the disk. A system call usually is implemented as a hardware trap that changes the CPU's execution mode and the current address-space mapping. Parameters supplied by users in system calls are validated by the kernel before being used. Such checking ensures the integrity of the system. All parameters passed into the kernel are copied into the kernel's address space, to ensure that validated parameters are not changed as a side effect of the system call. System-call results are returned by the kernel, either in hardware registers or by their values being copied to user-specified memory addresses. Like parameters passed into the kernel, addresses used for the return of results must be validated to ensure that they are part of an application's address space. If the kernel encounters an error while processing a system call, it returns an error code to the user. For the C programming language, this error code is stored in the global variable _errno_, and the function that executed the system call returns the value -1. User applications and the kernel operate independently of each other. 4.4BSD does not store I/O control blocks or other operating-system-related data structures in the application's address space. Each user-level application is provided an independent address space in which it executes. The kernel makes most state changes, such as suspending a process while another is running, invisible to the processes involved. [[overview-process-management]] === Process Management 4.4BSD supports a multitasking environment. Each task or thread of execution is termed a _process_. The _context_ of a 4.4BSD process consists of user-level state, including the contents of its address space and the run-time environment, and kernel-level state, which includes scheduling parameters, resource controls, and identification information. The context includes everything used by the kernel in providing services for the process. Users can create processes, control the processes' execution, and receive notification when the processes' execution status changes. Every process is assigned a unique value, termed a _process identifier_ (PID). This value is used by the kernel to identify a process when reporting status changes to a user, and by a user when referencing a process in a system call. The kernel creates a process by duplicating the context of another process. The new process is termed a _child process_ of the original _parent process_ The context duplicated in process creation includes both the user-level execution state of the process and the process's system state managed by the kernel. Important components of the kernel state are described in Chapter 4. [[fig-process-lifecycle]] .Process lifecycle image:fig1.png[Process lifecycle] The process lifecycle is depicted in <>. A process may create a new process that is a copy of the original by using the _fork_ system call. The _fork_ call returns twice: once in the parent process, where the return value is the process identifier of the child, and once in the child process, where the return value is 0. The parent-child relationship induces a hierarchical structure on the set of processes in the system. The new process shares all its parent's resources, such as file descriptors, signal-handling status, and memory layout. Although there are occasions when the new process is intended to be a copy of the parent, the loading and execution of a different program is a more useful and typical action. A process can overlay itself with the memory image of another program, passing to the newly created image a set of parameters, using the system call _execve_. One parameter is the name of a file whose contents are in a format recognized by the system -- either a binary-executable file or a file that causes the execution of a specified interpreter program to process its contents. A process may terminate by executing an _exit_ system call, sending 8 bits of exit status to its parent. If a process wants to communicate more than a single byte of information with its parent, it must either set up an interprocess-communication channel using pipes or sockets, or use an intermediate file. Interprocess communication is discussed extensively in Chapter 11. A process can suspend execution until any of its child processes terminate using the _wait_ system call, which returns the PID and exit status of the terminated child process. A parent process can arrange to be notified by a signal when a child process exits or terminates abnormally. Using the _wait4_ system call, the parent can retrieve information about the event that caused termination of the child process and about resources consumed by the process during its lifetime. If a process is orphaned because its parent exits before it is finished, then the kernel arranges for the child's exit status to be passed back to a special system process _init_: see Sections 3.1 and 14.6). The details of how the kernel creates and destroys processes are given in Chapter 5. Processes are scheduled for execution according to a _process-priority_ parameter. This priority is managed by a kernel-based scheduling algorithm. Users can influence the scheduling of a process by specifying a parameter (_nice_) that weights the overall scheduling priority, but are still obligated to share the underlying CPU resources according to the kernel's scheduling policy. ==== Signals The system defines a set of _signals_ that may be delivered to a process. Signals in 4.4BSD are modeled after hardware interrupts. A process may specify a user-level subroutine to be a _handler_ to which a signal should be delivered. When a signal is generated, it is blocked from further occurrence while it is being _caught_ by the handler. Catching a signal involves saving the current process context and building a new one in which to run the handler. The signal is then delivered to the handler, which can either abort the process or return to the executing process (perhaps after setting a global variable). If the handler returns, the signal is unblocked and can be generated (and caught) again. Alternatively, a process may specify that a signal is to be _ignored_, or that a default action, as determined by the kernel, is to be taken. The default action of certain signals is to terminate the process. This termination may be accompanied by creation of a _core file_ that contains the current memory image of the process for use in postmortem debugging. Some signals cannot be caught or ignored. These signals include _SIGKILL_, which kills runaway processes, and the job-control signal _SIGSTOP_. A process may choose to have signals delivered on a special stack so that sophisticated software stack manipulations are possible. For example, a language supporting coroutines needs to provide a stack for each coroutine. The language run-time system can allocate these stacks by dividing up the single stack provided by 4.4BSD. If the kernel does not support a separate signal stack, the space allocated for each coroutine must be expanded by the amount of space required to catch a signal. All signals have the same _priority_. If multiple signals are pending simultaneously, the order in which signals are delivered to a process is implementation specific. Signal handlers execute with the signal that caused their invocation to be blocked, but other signals may yet occur. Mechanisms are provided so that processes can protect critical sections of code against the occurrence of specified signals. The detailed design and implementation of signals is described in Section 4.7. ==== Process Groups and Sessions Processes are organized into _process groups_. Process groups are used to control access to terminals and to provide a means of distributing signals to collections of related processes. A process inherits its process group from its parent process. Mechanisms are provided by the kernel to allow a process to alter its process group or the process group of its descendents. Creating a new process group is easy; the value of a new process group is ordinarily the process identifier of the creating process. The group of processes in a process group is sometimes referred to as a _job_ and is manipulated by high-level system software, such as the shell. A common kind of job created by a shell is a _pipeline_ of several processes connected by pipes, such that the output of the first process is the input of the second, the output of the second is the input of the third, and so forth. The shell creates such a job by forking a process for each stage of the pipeline, then putting all those processes into a separate process group. A user process can send a signal to each process in a process group, as well as to a single process. A process in a specific process group may receive software interrupts affecting the group, causing the group to suspend or resume execution, or to be interrupted or terminated. A terminal has a process-group identifier assigned to it. This identifier is normally set to the identifier of a process group associated with the terminal. A job-control shell may create a number of process groups associated with the same terminal; the terminal is the _controlling terminal_ for each process in these groups. A process may read from a descriptor for its controlling terminal only if the terminal's process-group identifier matches that of the process. If the identifiers do not match, the process will be blocked if it attempts to read from the terminal. By changing the process-group identifier of the terminal, a shell can arbitrate a terminal among several different jobs. This arbitration is called _job control_ and is described, with process groups, in Section 4.8. Just as a set of related processes can be collected into a process group, a set of process groups can be collected into a _session_. The main uses for sessions are to create an isolated environment for a daemon process and its children, and to collect together a user's login shell and the jobs that that shell spawns. [[overview-memory-management]] === Memory Management Each process has its own private address space. The address space is initially divided into three logical segments: _text_, _data_, and _stack_. The text segment is read-only and contains the machine instructions of a program. The data and stack segments are both readable and writable. The data segment contains the initialized and uninitialized data portions of a program, whereas the stack segment holds the application's run-time stack. On most machines, the stack segment is extended automatically by the kernel as the process executes. A process can expand or contract its data segment by making a system call, whereas a process can change the size of its text segment only when the segment's contents are overlaid with data from the filesystem, or when debugging takes place. The initial contents of the segments of a child process are duplicates of the segments of a parent process. The entire contents of a process address space do not need to be resident for a process to execute. If a process references a part of its address space that is not resident in main memory, the system _pages_ the necessary information into memory. When system resources are scarce, the system uses a two-level approach to maintain available resources. If a modest amount of memory is available, the system will take memory resources away from processes if these resources have not been used recently. Should there be a severe resource shortage, the system will resort to _swapping_ the entire context of a process to secondary storage. The _demand paging_ and _swapping_ done by the system are effectively transparent to processes. A process may, however, advise the system about expected future memory utilization as a performance aid. ==== BSD Memory-Management Design Decisions The support of large sparse address spaces, mapped files, and shared memory was a requirement for 4.2BSD. An interface was specified, called _mmap_, that allowed unrelated processes to request a shared mapping of a file into their address spaces. If multiple processes mapped the same file into their address spaces, changes to the file's portion of an address space by one process would be reflected in the area mapped by the other processes, as well as in the file itself. Ultimately, 4.2BSD was shipped without the _mmap_ interface, because of pressure to make other features, such as networking, available. Further development of the _mmap_ interface continued during the work on 4.3BSD. Over 40 companies and research groups participated in the discussions leading to the revised architecture that was described in the Berkeley Software Architecture Manual <>. Several of the companies have implemented the revised interface <>. Once again, time pressure prevented 4.3BSD from providing an implementation of the interface. Although the latter could have been built into the existing 4.3BSD virtual-memory system, the developers decided not to put it in because that implementation was nearly 10 years old. Furthermore, the original virtual-memory design was based on the assumption that computer memories were small and expensive, whereas disks were locally connected, fast, large, and inexpensive. Thus, the virtual-memory system was designed to be frugal with its use of memory at the expense of generating extra disk traffic. In addition, the 4.3BSD implementation was riddled with VAX memory-management hardware dependencies that impeded its portability to other computer architectures. Finally, the virtual-memory system was not designed to support the tightly coupled multiprocessors that are becoming increasingly common and important today. Attempts to improve the old implementation incrementally seemed doomed to failure. A completely new design, on the other hand, could take advantage of large memories, conserve disk transfers, and have the potential to run on multiprocessors. Consequently, the virtual-memory system was completely replaced in 4.4BSD. The 4.4BSD virtual-memory system is based on the Mach 2.0 VM system <>. with updates from Mach 2.5 and Mach 3.0. It features efficient support for sharing, a clean separation of machine-independent and machine-dependent features, as well as (currently unused) multiprocessor support. Processes can map files anywhere in their address space. They can share parts of their address space by doing a shared mapping of the same file. Changes made by one process are visible in the address space of the other process, and also are written back to the file itself. Processes can also request private mappings of a file, which prevents any changes that they make from being visible to other processes mapping the file or being written back to the file itself. Another issue with the virtual-memory system is the way that information is passed into the kernel when a system call is made. 4.4BSD always copies data from the process address space into a buffer in the kernel. For read or write operations that are transferring large quantities of data, doing the copy can be time consuming. An alternative to doing the copying is to remap the process memory into the kernel. The 4.4BSD kernel always copies the data for several reasons: * Often, the user data are not page aligned and are not a multiple of the hardware page length. * If the page is taken away from the process, it will no longer be able to reference that page. Some programs depend on the data remaining in the buffer even after those data have been written. * If the process is allowed to keep a copy of the page (as it is in current 4.4BSD semantics), the page must be made _copy-on-write_. A copy-on-write page is one that is protected against being written by being made read-only. If the process attempts to modify the page, the kernel gets a write fault. The kernel then makes a copy of the page that the process can modify. Unfortunately, the typical process will immediately try to write new data to its output buffer, forcing the data to be copied anyway. * When pages are remapped to new virtual-memory addresses, most memory-management hardware requires that the hardware address-translation cache be purged selectively. The cache purges are often slow. The net effect is that remapping is slower than copying for blocks of data less than 4 to 8 Kbyte. The biggest incentives for memory mapping are the needs for accessing big files and for passing large quantities of data between processes. The _mmap_ interface provides a way for both of these tasks to be done without copying. ==== Memory Management Inside the Kernel The kernel often does allocations of memory that are needed for only the duration of a single system call. In a user process, such short-term memory would be allocated on the run-time stack. Because the kernel has a limited run-time stack, it is not feasible to allocate even moderate-sized blocks of memory on it. Consequently, such memory must be allocated through a more dynamic mechanism. For example, when the system must translate a pathname, it must allocate a 1-Kbyte buffer to hold the name. Other blocks of memory must be more persistent than a single system call, and thus could not be allocated on the stack even if there was space. An example is protocol-control blocks that remain throughout the duration of a network connection. Demands for dynamic memory allocation in the kernel have increased as more services have been added. A generalized memory allocator reduces the complexity of writing code inside the kernel. Thus, the 4.4BSD kernel has a single memory allocator that can be used by any part of the system. It has an interface similar to the C library routines _malloc_ and _free_ that provide memory allocation to application programs <>. Like the C library interface, the allocation routine takes a parameter specifying the size of memory that is needed. The range of sizes for memory requests is not constrained; however, physical memory is allocated and is not paged. The free routine takes a pointer to the storage being freed, but does not require the size of the piece of memory being freed. [[overview-io-system]] === I/O System The basic model of the UNIX I/O system is a sequence of bytes that can be accessed either randomly or sequentially. There are no _access methods_ and no _control blocks_ in a typical UNIX user process. Different programs expect various levels of structure, but the kernel does not impose structure on I/O. For instance, the convention for text files is lines of ASCII characters separated by a single newline character (the ASCII line-feed character), but the kernel knows nothing about this convention. For the purposes of most programs, the model is further simplified to being a stream of data bytes, or an _I/O stream_. It is this single common data form that makes the characteristic UNIX tool-based approach work <>. An I/O stream from one program can be fed as input to almost any other program. (This kind of traditional UNIX I/O stream should not be confused with the Eighth Edition stream I/O system or with the System V, Release 3 STREAMS, both of which can be accessed as traditional I/O streams.) ==== Descriptors and I/O UNIX processes use _descriptors_ to reference I/O streams. Descriptors are small unsigned integers obtained from the _open_ and _socket_ system calls. The _open_ system call takes as arguments the name of a file and a permission mode to specify whether the file should be open for reading or for writing, or for both. This system call also can be used to create a new, empty file. A _read_ or _write_ system call can be applied to a descriptor to transfer data. The _close_ system call can be used to deallocate any descriptor. Descriptors represent underlying objects supported by the kernel, and are created by system calls specific to the type of object. In 4.4BSD, three kinds of objects can be represented by descriptors: files, pipes, and sockets. * A _file_ is a linear array of bytes with at least one name. A file exists until all its names are deleted explicitly and no process holds a descriptor for it. A process acquires a descriptor for a file by opening that file's name with the _open_ system call. I/O devices are accessed as files. * A _pipe_ is a linear array of bytes, as is a file, but it is used solely as an I/O stream, and it is unidirectional. It also has no name, and thus cannot be opened with _open_. Instead, it is created by the _pipe_ system call, which returns two descriptors, one of which accepts input that is sent to the other descriptor reliably, without duplication, and in order. The system also supports a named pipe or FIFO. A FIFO has properties identical to a pipe, except that it appears in the filesystem; thus, it can be opened using the _open_ system call. Two processes that wish to communicate each open the FIFO: One opens it for reading, the other for writing. * A _socket_ is a transient object that is used for interprocess communication; it exists only as long as some process holds a descriptor referring to it. A socket is created by the _socket_ system call, which returns a descriptor for it. There are different kinds of sockets that support various communication semantics, such as reliable delivery of data, preservation of message ordering, and preservation of message boundaries. In systems before 4.2BSD, pipes were implemented using the filesystem; when sockets were introduced in 4.2BSD, pipes were reimplemented as sockets. The kernel keeps for each process a _descriptor table_, which is a table that the kernel uses to translate the external representation of a descriptor into an internal representation. (The descriptor is merely an index into this table.) The descriptor table of a process is inherited from that process's parent, and thus access to the objects to which the descriptors refer also is inherited. The main ways that a process can obtain a descriptor are by opening or creation of an object, and by inheritance from the parent process. In addition, socket IPC allows passing of descriptors in messages between unrelated processes on the same machine. Every valid descriptor has an associated _file offset_ in bytes from the beginning of the object. Read and write operations start at this offset, which is updated after each data transfer. For objects that permit random access, the file offset also may be set with the _lseek_ system call. Ordinary files permit random access, and some devices do, as well. Pipes and sockets do not. When a process terminates, the kernel reclaims all the descriptors that were in use by that process. If the process was holding the final reference to an object, the object's manager is notified so that it can do any necessary cleanup actions, such as final deletion of a file or deallocation of a socket. ==== Descriptor Management Most processes expect three descriptors to be open already when they start running. These descriptors are 0, 1, 2, more commonly known as _standard input_, _standard output_, and _standard error_, respectively. Usually, all three are associated with the user's terminal by the login process (see Section 14.6) and are inherited through _fork_ and _exec_ by processes run by the user. Thus, a program can read what the user types by reading standard input, and the program can send output to the user's screen by writing to standard output. The standard error descriptor also is open for writing and is used for error output, whereas standard output is used for ordinary output. These (and other) descriptors can be mapped to objects other than the terminal; such mapping is called _I/O redirection_, and all the standard shells permit users to do it. The shell can direct the output of a program to a file by closing descriptor 1 (standard output) and opening the desired output file to produce a new descriptor 1. It can similarly redirect standard input to come from a file by closing descriptor 0 and opening the file. Pipes allow the output of one program to be input to another program without rewriting or even relinking of either program. Instead of descriptor 1 (standard output) of the source program being set up to write to the terminal, it is set up to be the input descriptor of a pipe. Similarly, descriptor 0 (standard input) of the sink program is set up to reference the output of the pipe, instead of the terminal keyboard. The resulting set of two processes and the connecting pipe is known as a _pipeline_. Pipelines can be arbitrarily long series of processes connected by pipes. The _open_, _pipe_, and _socket_ system calls produce new descriptors with the lowest unused number usable for a descriptor. For pipelines to work, some mechanism must be provided to map such descriptors into 0 and 1. The _dup_ system call creates a copy of a descriptor that points to the same file-table entry. The new descriptor is also the lowest unused one, but if the desired descriptor is closed first, _dup_ can be used to do the desired mapping. Care is required, however: If descriptor 1 is desired, and descriptor 0 happens also to have been closed, descriptor 0 will be the result. To avoid this problem, the system provides the _dup2_ system call; it is like _dup_, but it takes an additional argument specifying the number of the desired descriptor (if the desired descriptor was already open, _dup2_ closes it before reusing it). ==== Devices Hardware devices have filenames, and may be accessed by the user via the same system calls used for regular files. The kernel can distinguish a _device special file_ or _special file_, and can determine to what device it refers, but most processes do not need to make this determination. Terminals, printers, and tape drives are all accessed as though they were streams of bytes, like 4.4BSD disk files. Thus, device dependencies and peculiarities are kept in the kernel as much as possible, and even in the kernel most of them are segregated in the device drivers. Hardware devices can be categorized as either _structured_ or _unstructured_; they are known as _block_ or _character_ devices, respectively. Processes typically access devices through _special files_ in the filesystem. I/O operations to these files are handled by kernel-resident software modules termed _device drivers_. Most network-communication hardware devices are accessible through only the interprocess-communication facilities, and do not have special files in the filesystem name space, because the _raw-socket_ interface provides a more natural interface than does a special file. Structured or block devices are typified by disks and magnetic tapes, and include most random-access devices. The kernel supports read-modify-write-type buffering actions on block-oriented structured devices to allow the latter to be read and written in a totally random byte-addressed fashion, like regular files. Filesystems are created on block devices. Unstructured devices are those devices that do not support a block structure. Familiar unstructured devices are communication lines, raster plotters, and unbuffered magnetic tapes and disks. Unstructured devices typically support large block I/O transfers. Unstructured files are called _character devices_ because the first of these to be implemented were terminal device drivers. The kernel interface to the driver for these devices proved convenient for other devices that were not block structured. Device special files are created by the _mknod_ system call. There is an additional system call, _ioctl_, for manipulating the underlying device parameters of special files. The operations that can be done differ for each device. This system call allows the special characteristics of devices to be accessed, rather than overloading the semantics of other system calls. For example, there is an _ioctl_ on a tape drive to write an end-of-tape mark, instead of there being a special or modified version of _write_. ==== Socket IPC The 4.2BSD kernel introduced an IPC mechanism more flexible than pipes, based on _sockets_. A socket is an endpoint of communication referred to by a descriptor, just like a file or a pipe. Two processes can each create a socket, and then connect those two endpoints to produce a reliable byte stream. Once connected, the descriptors for the sockets can be read or written by processes, just as the latter would do with a pipe. The transparency of sockets allows the kernel to redirect the output of one process to the input of another process residing on another machine. A major difference between pipes and sockets is that pipes require a common parent process to set up the communications channel. A connection between sockets can be set up by two unrelated processes, possibly residing on different machines. System V provides local interprocess communication through FIFOs (also known as _named pipes_). FIFOs appear as an object in the filesystem that unrelated processes can open and send data through in the same way as they would communicate through a pipe. Thus, FIFOs do not require a common parent to set them up; they can be connected after a pair of processes are up and running. Unlike sockets, FIFOs can be used on only a local machine; they cannot be used to communicate between processes on different machines. FIFOs are implemented in 4.4BSD only because they are required by the POSIX.1 standard. Their functionality is a subset of the socket interface. The socket mechanism requires extensions to the traditional UNIX I/O system calls to provide the associated naming and connection semantics. Rather than overloading the existing interface, the developers used the existing interfaces to the extent that the latter worked without being changed, and designed new interfaces to handle the added semantics. The _read_ and _write_ system calls were used for byte-stream type connections, but six new system calls were added to allow sending and receiving addressed messages such as network datagrams. The system calls for writing messages include _send_, _sendto_, and _sendmsg_. The system calls for reading messages include _recv_, _recvfrom_, and _recvmsg_. In retrospect, the first two in each class are special cases of the others; _recvfrom_ and _sendto_ probably should have been added as library interfaces to _recvmsg_ and _sendmsg_, respectively. ==== Scatter/Gather I/O In addition to the traditional _read_ and _write_ system calls, 4.2BSD introduced the ability to do scatter/gather I/O. Scatter input uses the _readv_ system call to allow a single read to be placed in several different buffers. Conversely, the _writev_ system call allows several different buffers to be written in a single atomic write. Instead of passing a single buffer and length parameter, as is done with _read_ and _write_, the process passes in a pointer to an array of buffers and lengths, along with a count describing the size of the array. This facility allows buffers in different parts of a process address space to be written atomically, without the need to copy them to a single contiguous buffer. Atomic writes are necessary in the case where the underlying abstraction is record based, such as tape drives that output a tape block on each write request. It is also convenient to be able to read a single request into several different buffers (such as a record header into one place and the data into another). Although an application can simulate the ability to scatter data by reading the data into a large buffer and then copying the pieces to their intended destinations, the cost of memory-to-memory copying in such cases often would more than double the running time of the affected application. Just as _send_ and _recv_ could have been implemented as library interfaces to _sendto_ and _recvfrom_, it also would have been possible to simulate _read_ with _readv_ and _write_ with _writev_. However, _read_ and _write_ are used so much more frequently that the added cost of simulating them would not have been worthwhile. ==== Multiple Filesystem Support With the expansion of network computing, it became desirable to support both local and remote filesystems. To simplify the support of multiple filesystems, the developers added a new virtual node or _vnode_ interface to the kernel. The set of operations exported from the vnode interface appear much like the filesystem operations previously supported by the local filesystem. However, they may be supported by a wide range of filesystem types: * Local disk-based filesystems * Files imported using a variety of remote filesystem protocols * Read-only CD-ROM filesystems * Filesystems providing special-purpose interfaces -- for example, the `/proc` filesystem A few variants of 4.4BSD, such as FreeBSD, allow filesystems to be loaded dynamically when the filesystems are first referenced by the _mount_ system call. The vnode interface is described in Section 6.5; its ancillary support routines are described in Section 6.6; several of the special-purpose filesystems are described in Section 6.7. [[overview-filesystem]] === Filesystems A regular file is a linear array of bytes, and can be read and written starting at any byte in the file. The kernel distinguishes no record boundaries in regular files, although many programs recognize line-feed characters as distinguishing the ends of lines, and other programs may impose other structure. No system-related information about a file is kept in the file itself, but the filesystem stores a small amount of ownership, protection, and usage information with each file. A _filename_ component is a string of up to 255 characters. These filenames are stored in a type of file called a _directory_. The information in a directory about a file is called a _directory entry_ and includes, in addition to the filename, a pointer to the file itself. Directory entries may refer to other directories, as well as to plain files. A hierarchy of directories and files is thus formed, and is called a _filesystem_; .A small filesystem [[fig-small-fs]] image:fig2.png[A small filesystem] a small one is shown in <>. Directories may contain subdirectories, and there is no inherent limitation to the depth with which directory nesting may occur. To protect the consistency of the filesystem, the kernel does not permit processes to write directly into directories. A filesystem may include not only plain files and directories, but also references to other objects, such as devices and sockets. The filesystem forms a tree, the beginning of which is the _root directory_, sometimes referred to by the name _slash_, spelled with a single solidus character (/). The root directory contains files; in our example in Fig 2.2, it contains `vmunix`, a copy of the kernel-executable object file. It also contains directories; in this example, it contains the `usr` directory. Within the `usr` directory is the `bin` directory, which mostly contains executable object code of programs, such as the files `ls` and `vi`. A process identifies a file by specifying that file's _pathname_, which is a string composed of zero or more filenames separated by slash (/) characters. The kernel associates two directories with each process for use in interpreting pathnames. A process's _root directory_ is the topmost point in the filesystem that the process can access; it is ordinarily set to the root directory of the entire filesystem. A pathname beginning with a slash is called an _absolute pathname_, and is interpreted by the kernel starting with the process's root directory. A pathname that does not begin with a slash is called a _relative pathname_, and is interpreted relative to the _current working directory_ of the process. (This directory also is known by the shorter names _current directory_ or _working directory_.) The current directory itself may be referred to directly by the name _dot_, spelled with a single period (`.`). The filename _dot-dot_ (`..`) refers to a directory's parent directory. The root directory is its own parent. A process may set its root directory with the _chroot_ system call, and its current directory with the _chdir_ system call. Any process may do _chdir_ at any time, but _chroot_ is permitted only a process with superuser privileges. _Chroot_ is normally used to set up restricted access to the system. Using the filesystem shown in Fig. 2.2, if a process has the root of the filesystem as its root directory, and has `/usr` as its current directory, it can refer to the file `vi` either from the root with the absolute pathname `/usr/bin/vi`, or from its current directory with the relative pathname `bin/vi`. System utilities and databases are kept in certain well-known directories. Part of the well-defined hierarchy includes a directory that contains the _home directory_ for each user -- for example, `/usr/staff/mckusick` and `/usr/staff/karels` in Fig. 2.2. When users log in, the current working directory of their shell is set to the home directory. Within their home directories, users can create directories as easily as they can regular files. Thus, a user can build arbitrarily complex subhierarchies. The user usually knows of only one filesystem, but the system may know that this one virtual filesystem is really composed of several physical filesystems, each on a different device. A physical filesystem may not span multiple hardware devices. Since most physical disk devices are divided into several logical devices, there may be more than one filesystem per physical device, but there will be no more than one per logical device. One filesystem -- the filesystem that anchors all absolute pathnames -- is called the _root filesystem_, and is always available. Others may be mounted; that is, they may be integrated into the directory hierarchy of the root filesystem. References to a directory that has a filesystem mounted on it are converted transparently by the kernel into references to the root directory of the mounted filesystem. The _link_ system call takes the name of an existing file and another name to create for that file. After a successful _link_, the file can be accessed by either filename. A filename can be removed with the _unlink_ system call. When the final name for a file is removed (and the final process that has the file open closes it), the file is deleted. Files are organized hierarchically in _directories_. A directory is a type of file, but, in contrast to regular files, a directory has a structure imposed on it by the system. A process can read a directory as it would an ordinary file, but only the kernel is permitted to modify a directory. Directories are created by the _mkdir_ system call and are removed by the _rmdir_ system call. Before 4.2BSD, the _mkdir_ and _rmdir_ system calls were implemented by a series of _link_ and _unlink_ system calls being done. There were three reasons for adding systems calls explicitly to create and delete directories: [arabic] . The operation could be made atomic. If the system crashed, the directory would not be left half-constructed, as could happen when a series of link operations were used. . When a networked filesystem is being run, the creation and deletion of files and directories need to be specified atomically so that they can be serialized. . When supporting non-UNIX filesystems, such as an MS-DOS filesystem, on another partition of the disk, the other filesystem may not support link operations. Although other filesystems might support the concept of directories, they probably would not create and delete the directories with links, as the UNIX filesystem does. Consequently, they could create and delete directories only if explicit directory create and delete requests were presented. The _chown_ system call sets the owner and group of a file, and _chmod_ changes protection attributes. _Stat_ applied to a filename can be used to read back such properties of a file. The _fchown_, _fchmod_, and _fstat_ system calls are applied to a descriptor, instead of to a filename, to do the same set of operations. The _rename_ system call can be used to give a file a new name in the filesystem, replacing one of the file's old names. Like the directory-creation and directory-deletion operations, the _rename_ system call was added to 4.2BSD to provide atomicity to name changes in the local filesystem. Later, it proved useful explicitly to export renaming operations to foreign filesystems and over the network. The _truncate_ system call was added to 4.2BSD to allow files to be shortened to an arbitrary offset. The call was added primarily in support of the Fortran run-time library, which has the semantics such that the end of a random-access file is set to be wherever the program most recently accessed that file. Without the _truncate_ system call, the only way to shorten a file was to copy the part that was desired to a new file, to delete the old file, then to rename the copy to the original name. As well as this algorithm being slow, the library could potentially fail on a full filesystem. Once the filesystem had the ability to shorten files, the kernel took advantage of that ability to shorten large empty directories. The advantage of shortening empty directories is that it reduces the time spent in the kernel searching them when names are being created or deleted. Newly created files are assigned the user identifier of the process that created them and the group identifier of the directory in which they were created. A three-level access-control mechanism is provided for the protection of files. These three levels specify the accessibility of a file to [arabic] . The user who owns the file . The group that owns the file . Everyone else Each level of access has separate indicators for read permission, write permission, and execute permission. Files are created with zero length, and may grow when they are written. While a file is open, the system maintains a pointer into the file indicating the current location in the file associated with the descriptor. This pointer can be moved about in the file in a random-access fashion. Processes sharing a file descriptor through a _fork_ or _dup_ system call share the current location pointer. Descriptors created by separate _open_ system calls have separate current location pointers. Files may have _holes_ in them. Holes are void areas in the linear extent of the file where data have never been written. A process can create these holes by positioning the pointer past the current end-of-file and writing. When read, holes are treated by the system as zero-valued bytes. Earlier UNIX systems had a limit of 14 characters per filename component. This limitation was often a problem. For example, in addition to the natural desire of users to give files long descriptive names, a common way of forming filenames is as `basename.extension`, where the extension (indicating the kind of file, such as `.c` for C source or `.o` for intermediate binary object) is one to three characters, leaving 10 to 12 characters for the basename. Source-code-control systems and editors usually take up another two characters, either as a prefix or a suffix, for their purposes, leaving eight to 10 characters. It is easy to use 10 or 12 characters in a single English word as a basename (e.g., ``multiplexer''). It is possible to keep within these limits, but it is inconvenient or even dangerous, because other UNIX systems accept strings longer than the limit when creating files, but then _truncate_ to the limit. A C language source file named `multiplexer.c` (already 13 characters) might have a source-code-control file with `s.` prepended, producing a filename `s.multiplexer` that is indistinguishable from the source-code-control file for `multiplexer.ms`, a file containing `troff` source for documentation for the C program. The contents of the two original files could easily get confused with no warning from the source-code-control system. Careful coding can detect this problem, but the long filenames first introduced in 4.2BSD practically eliminate it. [[overview-filestore]] === Filestores The operations defined for local filesystems are divided into two parts. Common to all local filesystems are hierarchical naming, locking, quotas, attribute management, and protection. These features are independent of how the data will be stored. 4.4BSD has a single implementation to provide these semantics. The other part of the local filesystem is the organization and management of the data on the storage media. Laying out the contents of files on the storage media is the responsibility of the filestore. 4.4BSD supports three different filestore layouts: * The traditional Berkeley Fast Filesystem * The log-structured filesystem, based on the Sprite operating-system design <> * A memory-based filesystem Although the organizations of these filestores are completely different, these differences are indistinguishable to the processes using the filestores. The Fast Filesystem organizes data into cylinder groups. Files that are likely to be accessed together, based on their locations in the filesystem hierarchy, are stored in the same cylinder group. Files that are not expected to accessed together are moved into different cylinder groups. Thus, files written at the same time may be placed far apart on the disk. The log-structured filesystem organizes data as a log. All data being written at any point in time are gathered together, and are written at the same disk location. Data are never overwritten; instead, a new copy of the file is written that replaces the old one. The old files are reclaimed by a garbage-collection process that runs when the filesystem becomes full and additional free space is needed. The memory-based filesystem is designed to store data in virtual memory. It is used for filesystems that need to support fast but temporary data, such as `/tmp`. The goal of the memory-based filesystem is to keep the storage packed as compactly as possible to minimize the usage of virtual-memory resources. [[overview-nfs]] === Network Filesystem Initially, networking was used to transfer data from one machine to another. Later, it evolved to allowing users to log in remotely to another machine. The next logical step was to bring the data to the user, instead of having the user go to the data -- and network filesystems were born. Users working locally do not experience the network delays on each keystroke, so they have a more responsive environment. Bringing the filesystem to a local machine was among the first of the major client-server applications. The _server_ is the remote machine that exports one or more of its filesystems. The _client_ is the local machine that imports those filesystems. From the local client's point of view, a remotely mounted filesystem appears in the file-tree name space just like any other locally mounted filesystem. Local clients can change into directories on the remote filesystem, and can read, write, and execute binaries within that remote filesystem identically to the way that they can do these operations on a local filesystem. When the local client does an operation on a remote filesystem, the request is packaged and is sent to the server. The server does the requested operation and returns either the requested information or an error indicating why the request was denied. To get reasonable performance, the client must cache frequently accessed data. The complexity of remote filesystems lies in maintaining cache consistency between the server and its many clients. Although many remote-filesystem protocols have been developed over the years, the most pervasive one in use among UNIX systems is the Network Filesystem (NFS), whose protocol and most widely used implementation were done by Sun Microsystems. The 4.4BSD kernel supports the NFS protocol, although the implementation was done independently from the protocol specification <>. The NFS protocol is described in Chapter 9. [[overview-terminal]] === Terminals Terminals support the standard system I/O operations, as well as a collection of terminal-specific operations to control input-character editing and output delays. At the lowest level are the terminal device drivers that control the hardware terminal ports. Terminal input is handled according to the underlying communication characteristics, such as baud rate, and according to a set of software-controllable parameters, such as parity checking. Layered above the terminal device drivers are line disciplines that provide various degrees of character processing. The default line discipline is selected when a port is being used for an interactive login. The line discipline is run in _canonical mode_; input is processed to provide standard line-oriented editing functions, and input is presented to a process on a line-by-line basis. Screen editors and programs that communicate with other computers generally run in _noncanonical mode_ (also commonly referred to as _raw mode_ or _character-at-a-time mode_). In this mode, input is passed through to the reading process immediately and without interpretation. All special-character input processing is disabled, no erase or other line editing processing is done, and all characters are passed to the program that is reading from the terminal. It is possible to configure the terminal in thousands of combinations between these two extremes. For example, a screen editor that wanted to receive user interrupts asynchronously might enable the special characters that generate signals and enable output flow control, but otherwise run in noncanonical mode; all other characters would be passed through to the process uninterpreted. On output, the terminal handler provides simple formatting services, including * Converting the line-feed character to the two-character carriage-return-line-feed sequence * Inserting delays after certain standard control characters * Expanding tabs * Displaying echoed nongraphic ASCII characters as a two-character sequence of the form ``^C'' (i.e., the ASCII caret character followed by the ASCII character that is the character's value offset from the ASCII ``@'' character). Each of these formatting services can be disabled individually by a process through control requests. [[overview-ipc]] === Interprocess Communication Interprocess communication in 4.4BSD is organized in _communication domains_. Domains currently supported include the _local domain_, for communication between processes executing on the same machine; the _internet domain_, for communication between processes using the TCP/IP protocol suite (perhaps within the Internet); the ISO/OSI protocol family for communication between sites required to run them; and the _XNS domain_, for communication between processes using the XEROX Network Systems (XNS) protocols. Within a domain, communication takes place between communication endpoints known as _sockets_. As mentioned in Section 2.6, the _socket_ system call creates a socket and returns a descriptor; other IPC system calls are described in Chapter 11. Each socket has a type that defines its communications semantics; these semantics include properties such as reliability, ordering, and prevention of duplication of messages. Each socket has associated with it a _communication protocol_. This protocol provides the semantics required by the socket according to the latter's type. Applications may request a specific protocol when creating a socket, or may allow the system to select a protocol that is appropriate for the type of socket being created. Sockets may have addresses bound to them. The form and meaning of socket addresses are dependent on the communication domain in which the socket is created. Binding a name to a socket in the local domain causes a file to be created in the filesystem. Normal data transmitted and received through sockets are untyped. Data-representation issues are the responsibility of libraries built on top of the interprocess-communication facilities. In addition to transporting normal data, communication domains may support the transmission and reception of specially typed data, termed _access rights_. The local domain, for example, uses this facility to pass descriptors between processes. Networking implementations on UNIX before 4.2BSD usually worked by overloading the character-device interfaces. One goal of the socket interface was for naive programs to be able to work without change on stream-style connections. Such programs can work only if the _read_ and _write_ systems calls are unchanged. Consequently, the original interfaces were left intact, and were made to work on stream-type sockets. A new interface was added for more complicated sockets, such as those used to send datagrams, with which a destination address must be presented with each _send_ call. Another benefit is that the new interface is highly portable. Shortly after a test release was available from Berkeley, the socket interface had been ported to System III by a UNIX vendor (although AT&T did not support the socket interface until the release of System V Release 4, deciding instead to use the Eighth Edition stream mechanism). The socket interface was also ported to run in many Ethernet boards by vendors, such as Excelan and Interlan, that were selling into the PC market, where the machines were too small to run networking in the main processor. More recently, the socket interface was used as the basis for Microsoft's Winsock networking interface for Windows. [[overview-network-communication]] === Network Communication Some of the communication domains supported by the _socket_ IPC mechanism provide access to network protocols. These protocols are implemented as a separate software layer logically below the socket software in the kernel. The kernel provides many ancillary services, such as buffer management, message routing, standardized interfaces to the protocols, and interfaces to the network interface drivers for the use of the various network protocols. At the time that 4.2BSD was being implemented, there were many networking protocols in use or under development, each with its own strengths and weaknesses. There was no clearly superior protocol or protocol suite. By supporting multiple protocols, 4.2BSD could provide interoperability and resource sharing among the diverse set of machines that was available in the Berkeley environment. Multiple-protocol support also provides for future changes. Today's protocols designed for 10- to 100-Mbit-per-second Ethernets are likely to be inadequate for tomorrow's 1- to 10-Gbit-per-second fiber-optic networks. Consequently, the network-communication layer is designed to support multiple protocols. New protocols are added to the kernel without the support for older protocols being affected. Older applications can continue to operate using the old protocol over the same physical network as is used by newer applications running with a newer network protocol. [[overview-network-implementation]] === Network Implementation The first protocol suite implemented in 4.2BSD was DARPA's Transmission Control Protocol/Internet Protocol (TCP/IP). The CSRG chose TCP/IP as the first network to incorporate into the socket IPC framework, because a 4.1BSD-based implementation was publicly available from a DARPA-sponsored project at Bolt, Beranek, and Newman (BBN). That was an influential choice: The 4.2BSD implementation is the main reason for the extremely widespread use of this protocol suite. Later performance and capability improvements to the TCP/IP implementation have also been widely adopted. The TCP/IP implementation is described in detail in Chapter 13. The release of 4.3BSD added the Xerox Network Systems (XNS) protocol suite, partly building on work done at the University of Maryland and at Cornell University. This suite was needed to connect isolated machines that could not communicate using TCP/IP. The release of 4.4BSD added the ISO protocol suite because of the latter's increasing visibility both within and outside the United States. Because of the somewhat different semantics defined for the ISO protocols, some minor changes were required in the socket interface to accommodate these semantics. The changes were made such that they were invisible to clients of other existing protocols. The ISO protocols also required extensive addition to the two-level routing tables provided by the kernel in 4.3BSD. The greatly expanded routing capabilities of 4.4BSD include arbitrary levels of routing with variable-length addresses and network masks. [[overview-operation]] === System Operation Bootstrapping mechanisms are used to start the system running. First, the 4.4BSD kernel must be loaded into the main memory of the processor. Once loaded, it must go through an initialization phase to set the hardware into a known state. Next, the kernel must do autoconfiguration, a process that finds and configures the peripherals that are attached to the processor. The system begins running in single-user mode while a start-up script does disk checks and starts the accounting and quota checking. Finally, the start-up script starts the general system services and brings up the system to full multiuser operation. During multiuser operation, processes wait for login requests on the terminal lines and network ports that have been configured for user access. When a login request is detected, a login process is spawned and user validation is done. When the login validation is successful, a login shell is created from which the user can run additional processes. :sectnums!: [bibliography] [[references]] == References [[biblio-accetta]] Accetta et al, 1986 Mach: A New Kernel Foundation for UNIX Development" M.Accetta R.Baron W.Bolosky D.Golub R.Rashid A.Tevanian M.Young 93-113 USENIX Association Conference Proceedings USENIX Association June 1986 [[biblio-cheriton]] Cheriton, 1988 The V Distributed System D. R.Cheriton 314-333 Comm ACM, 31, 3 March 1988 [[biblio-ewens]] Ewens et al, 1985 Tunis: A Distributed Multiprocessor Operating System P.Ewens D. R.Blythe M.Funkenhauser R. C.Holt 247-254 USENIX Assocation Conference Proceedings USENIX Association June 1985 [[biblio-gingell]] Gingell et al, 1987 Virtual Memory Architecture in SunOS R.Gingell J.Moran W.Shannon 81-94 USENIX Association Conference Proceedings USENIX Association June 1987 [[biblio-kernighan]] Kernighan & Pike, 1984 The UNIX Programming Environment B. W.Kernighan R.Pike Prentice-Hall Englewood Cliffs NJ 1984 [[biblio-macklem]] Macklem, 1994 The 4.4BSD NFS Implementation R.Macklem 6:1-14 4.4BSD System Manager's Manual O'Reilly & Associates, Inc. Sebastopol CA 1994 [[biblio-mckusick-2]] McKusick & Karels, 1988 Design of a General Purpose Memory Allocator for the 4.3BSD UNIX Kernel M. K.McKusick M. J.Karels 295-304 USENIX Assocation Conference Proceedings USENIX Assocation June 1998 [[biblio-mckusick-1]] McKusick et al, 1994 Berkeley Software Architecture Manual, 4.4BSD Edition M. K.McKusick M. J.Karels S. J.Leffler W. N.Joy R. S.Faber 5:1-42 4.4BSD Programmer's Supplementary Documents O'Reilly & Associates, Inc. Sebastopol CA 1994 [[biblio-ritchie]] Ritchie, 1988 Early Kernel Design private communication D. M.Ritchie March 1988 [[biblio-rosenblum]] Rosenblum & Ousterhout, 1992 The Design and Implementation of a Log-Structured File System M.Rosenblum K.Ousterhout 26-52 ACM Transactions on Computer Systems, 10, 1 Association for Computing Machinery February 1992 [[biblio-rozier]] Rozier et al, 1988 Chorus Distributed Operating Systems M.Rozier V.Abrossimov F.Armand I.Boule M.Gien M.Guillemont F.Herrmann C.Kaiser S.Langlois P.Leonard W.Neuhauser 305-370 USENIX Computing Systems, 1, 4 Fall 1988 [[biblio-tevanian]] Tevanian, 1987 Architecture-Independent Virtual Memory Management for Parallel and Distributed Environments: The Mach Approach Technical Report CMU-CS-88-106, A.Tevanian Department of Computer Science, Carnegie-Mellon University Pittsburgh PA December 1987 diff --git a/documentation/content/en/books/dev-model/_index.adoc b/documentation/content/en/books/dev-model/_index.adoc index c5b4061159..8d3b9ed4db 100644 --- a/documentation/content/en/books/dev-model/_index.adoc +++ b/documentation/content/en/books/dev-model/_index.adoc @@ -1,1000 +1,1001 @@ --- title: A project model for the FreeBSD Project authors: - author: Niklas Saers copyright: Copyright © 2002-2005 Niklas Saers -description: A project model for the FreeBSD Project +description: A formal study of the organization of the FreeBSD project trademarks: ["freebsd", "ibm", "ieee", "adobe", "intel", "linux", "microsoft", "opengroup", "sun", "netbsd", "general"] +weight: 45 --- = A project model for the FreeBSD Project :doctype: book :toc: macro :toclevels: 2 :icons: font :xrefstyle: basic :relfileprefix: ../ :outfilesuffix: :sectnums: :sectnumlevels: 6 :partnums: :chapter-signifier: Chapter :part-signifier: Part :source-highlighter: rouge :experimental: :skip-front-matter: ifeval::["{backend}" == "html5"] include::shared/mirrors.adoc[] include::shared/authors.adoc[] include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/teams.adoc[] include::shared/en/urls.adoc[] :imagesdir: ../../../../images/books/dev-model/ endif::[] ifeval::["{backend}" == "pdf"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../../static/images/books/dev-model/ endif::[] ifeval::["{backend}" == "epub3"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] :imagesdir: ../../../../static/images/books/dev-model/ endif::[] ''' toc::[] [[foreword]] [.abstract-title] Foreword Up until now, the FreeBSD project has released a number of described techniques to do different parts of work. However, a project model summarising how the project is structured is needed because of the increasing amount of project members. footnote:[This goes hand-in-hand with Brooks' law that adding another person to a late project will make it later since it will increase the communication needs . A project model is a tool to reduce the communication needs.] This paper will provide such a project model and is donated to the FreeBSD Documentation project where it can evolve together with the project so that it can at any point in time reflect the way the project works. It is based on [<>]. I would like to thank the following people for taking the time to explain things that were unclear to me and for proofreading the document. * Andrey A. Chernov mailto:ache@freebsd.org[ache@freebsd.org] * Bruce A. Mah mailto:bmah@freebsd.org[bmah@freebsd.org] * Dag-Erling Smørgrav mailto:des@freebsd.org[des@freebsd.org] * Giorgos Keramidas mailto:keramida@freebsd.org[keramida@freebsd.org] * Ingvil Hovig mailto:ingvil.hovig@skatteetaten.no[ingvil.hovig@skatteetaten.no] * Jesper Holck mailto:jeh.inf@cbs.dk[jeh.inf@cbs.dk] * John Baldwin mailto:jhb@freebsd.org[jhb@freebsd.org] * John Polstra mailto:jdp@freebsd.org[jdp@freebsd.org] * Kirk McKusick mailto:mckusick@freebsd.org[mckusick@freebsd.org] * Mark Linimon mailto:linimon@freebsd.org[linimon@freebsd.org] * Marleen Devos * Niels Jørgenssen mailto:nielsj@ruc.dk[nielsj@ruc.dk] * Nik Clayton mailto:nik@freebsd.org[nik@freebsd.org] * Poul-Henning Kamp mailto:phk@freebsd.org[phk@freebsd.org] * Simon L. Nielsen mailto:simon@freebsd.org[simon@freebsd.org] [[overview]] == Overview A project model is a means to reduce the communications overhead in a project. As shown by [<>], increasing the number of project participants increases the communication in the project exponentionally. FreeBSD has during the past few years increased both its mass of active users and committers, and the communication in the project has risen accordingly. This project model will serve to reduce this overhead by providing an up-to-date description of the project. During the Core elections in 2002, Mark Murray stated "I am opposed to a long rule-book, as that satisfies lawyer-tendencies, and is counter to the technocentricity that the project so badly needs." [<>]. This project model is not meant to be a tool to justify creating impositions for developers, but as a tool to facilitate coordination. It is meant as a description of the project, with an overview of how the different processes are executed. It is an introduction to how the FreeBSD project works. The FreeBSD project model will be described as of July 1st, 2004. It is based on the Niels Jørgensen's paper [<>], FreeBSD's official documents, discussions on FreeBSD mailing lists and interviews with developers. After providing definitions of terms used, this document will outline the organisational structure (including role descriptions and communication lines), discuss the methodology model and after presenting the tools used for process control, it will present the defined processes. Finally it will outline major sub-projects of the FreeBSD project. [<>] Section 1.2 and 1.3 give the vision and the architectural guidelines for the project. The vision is "To produce the best UNIX-like operating system package possible, with due respect to the original software tools ideology as well as usability, performance and stability." The architectural guidelines help determine whether a problem that someone wants to be solved is within the scope of the project [[definitions]] == Definitions [[ref-activity]] === Activity An "activity" is an element of work performed during the course of a project [<>]. It has an output and leads towards an outcome. Such an output can either be an input to another activity or a part of the process' delivery. [[def-process]] === Process A "process" is a series of activities that lead towards a particular outcome. A process can consist of one or more sub-processes. An example of a process is software design. [[ref-hat]] === Hat A "hat" is synonymous with role. A hat has certain responsibilities in a process and for the process outcome. The hat executes activities. It is well defined what issues the hat should be contacted about by the project members and people outside the project. [[ref-outcome]] === Outcome An "outcome" is the final output of the process. This is synonymous with deliverable, that is defined as "any measurable, tangible, verifiable outcome, result or item that must be produced to complete a project or part of a project. Often used more narrowly in reference to an external deliverable, which is a deliverable that is subject to approval by the project sponsor or customer" by [<>]. Examples of outcomes are a piece of software, a decision made or a report written. [[ref-freebsd]] === FreeBSD When saying "FreeBSD" we will mean the BSD derivative UNIX-like operating system FreeBSD, whereas when saying "the FreeBSD Project" we will mean the project organisation. [[model-orgstruct]] == Organisational structure While no-one takes ownership of FreeBSD, the FreeBSD organisation is divided into core, committers and contributors and is part of the FreeBSD community that lives around it. The FreeBSD Project's structure (in order of descending authority) [.informaltable] [cols="1,1", options="header"] |=== | Group | Number of people |Core members |9 |Committers |269 |Contributors |~3000 |=== Number of committers has been determined by going through CVS logs from January 1st, 2004 to December 31st, 2004 and contributors by going through the list of contributions and problem reports. The main resource in the FreeBSD community is its developers: the committers and contributors. It is with their contributions that the project can move forward. Regular developers are referred to as contributors. As of January 1st, 2003, there are an estimated 5500 contributors on the project. Committers are developers with the privilege of being able to commit changes. These are usually the most active developers who are willing to spend their time not only integrating their own code but integrating code submitted by the developers who do not have this privilege. They are also the developers who elect the core team, and they have access to closed discussions. The project can be grouped into four distinct separate parts, and most developers will focus their involvement in one part of FreeBSD. The four parts are kernel development, userland development, ports and documentation. When referring to the base system, both kernel and userland is meant. This split changes our table to look like this: The FreeBSD Project's structure with committers in categories [.informaltable] [cols="1,1,1", options="header"] |=== | Group | Category | Number of people |Core members | |9 |Committers |Kernel |56 | |Userland |50 | |Docs |9 | |Ports |120 | |Total |269 |Contributors | |~3000 |=== Number of committers per area has been determined by going through CVS logs from January 1st, 2004 to December 31st, 2004. Note that many committers work in multiple areas, making the total number higher than the real number of committers. The total number of committers at that time was 269. Committers fall into three groups: committers who are only concerned with one area of the project (for instance file systems), committers who are involved only with one sub-project, and committers who commit to different parts of the code, including sub-projects. Because some committers work on different parts, the total number in the committers section of the table is higher than in the above table. The kernel is the main building block of FreeBSD. While the userland applications are protected against faults in other userland applications, the entire system is vulnerable to errors in the kernel. This, combined with the vast amount of dependencies in the kernel and that it is not easy to see all the consequences of a kernel change, demands developers with a relative full understanding of the kernel. Multiple development efforts in the kernel also require a closer coordination than userland applications do. The core utilities, known as userland, provide the interface that identifies FreeBSD, both user interface, shared libraries and external interfaces to connecting clients. Currently, 162 people are involved in userland development and maintenance, many being maintainers for their own part of the code. Maintainership will be discussed in the <> section. Documentation is handled by <> and includes all documents surrounding the FreeBSD project, including the web pages. There were during 2004 101 people making commits to the FreeBSD Documentation Project. Ports is the collection of meta-data that is needed to make software packages build correctly on FreeBSD. An example of a port is the port for the web-browser Mozilla. It contains information about where to fetch the source, what patches to apply and how, and how the package should be installed on the system. This allows automated tools to fetch, build and install the package. As of this writing, there are more than 12600 ports available. footnote:[Statistics are generated by counting the number of entries in the file fetched by portsdb by April 1st, 2005. portsdb is a part of the port sysutils/portupgrade.] , ranging from web servers to games, programming languages and most of the application types that are in use on modern computers. Ports will be discussed further in the section <>. [[methodology-model]] == Methodology model [[development-model]] === Development model There is no defined model for how people write code in FreeBSD. However, Niels Jørgenssen has suggested a model of how written code is integrated into the project. Jørgenssen's model for change integration [.informaltable] [cols="1,1,1", options="header"] |=== | Stage | Next if successful | Next if unsuccessful |code |review | |review |pre-commit test |code |pre-commit test |development release |code |development release |parallel debugging |code |parallel debugging |production release |code |production release | |code |=== The "development release" is the FreeBSD-CURRENT ("-CURRENT") branch and the "production release" is the FreeBSD-STABLE branch ("-STABLE") [<>]. This is a model for one change, and shows that after coding, developers seek community review and try integrating it with their own systems. After integrating the change into the development release, called FreeBSD-CURRENT, it is tested by many users and developers in the FreeBSD community. After it has gone through enough testing, it is merged into the production release, called FreeBSD-STABLE. Unless each stage is finished successfully, the developer needs to go back and make modifications in the code and restart the process. To integrate a change with either -CURRENT or -STABLE is called making a commit. Jørgensen found that most FreeBSD developers work individually, meaning that this model is used in parallel by many developers on the different ongoing development efforts. A developer can also be working on multiple changes, so that while they are waiting for review or people to test one or more of their changes, they may be writing another change. As each commit represents an increment, this is a massively incremental model. The commits are in fact so frequent that during one year footnote:[The period from January 1st, 2004 to December 31st, 2004 was examined to find this number.] , 85427 commits were made, making a daily average of 233 commits. Within the "code" bracket in Jørgensen's model, each programmer has their own working style and follows their own development models. The bracket could very well have been called "development" as it includes requirements gathering and analysis, system and detailed design, implementation and verification. However, the only output from these stages is the source code or system documentation. From a stepwise model's perspective (such as the waterfall model), the other brackets can be seen as further verification and system integration. This system integration is also important to see if a change is accepted by the community. Up until the code is committed, the developer is free to choose how much to communicate about it to the rest of the project. In order for -CURRENT to work as a buffer (so that bright ideas that had some undiscovered drawbacks can be backed out) the minimum time a commit should be in -CURRENT before merging it to -STABLE is 3 days. Such a merge is referred to as an MFC (Merge From Current). It is important to notice the word "change". Most commits do not contain radical new features, but are maintenance updates. The only exceptions from this model are security fixes and changes to features that are deprecated in the -CURRENT branch. In these cases, changes can be committed directly to the -STABLE branch. In addition to many people working on the project, there are many related projects to the FreeBSD Project. These are either projects developing brand new features, sub-projects or projects whose outcome is incorporated into FreeBSD footnote:[For instance, the development of the Bluetooth stack started as a sub-project until it was deemed stable enough to be merged into the -CURRENT branch. Now it is a part of the core FreeBSD system.]. These projects fit into the FreeBSD Project just like regular development efforts: they produce code that is integrated with the FreeBSD Project. However, some of them (like Ports and Documentation) have the privilege of being applicable to both branches or commit directly to both -CURRENT and -STABLE. There is no standards to how design should be done, nor is design collected in a centralised repository. The main design is that of 4.4BSD. footnote:[According to Kirk McKusick, after 20 years of developing UNIX operating systems, the interfaces are for the most part figured out. There is therefore no need for much design. However, new applications of the system and new hardware leads to some implementations being more beneficial than those that used to be preferred. One example is the introduction of web browsing that made the normal TCP/IP connection a short burst of data rather than a steady stream over a longer period of time.] As design is a part of the "Code" bracket in Jørgenssen's model, it is up to every developer or sub-project how this should be done. Even if the design should be stored in a central repository, the output from the design stages would be of limited use as the differences of methodologies would make them poorly if at all interoperable. For the overall design of the project, the project relies on the sub-projects to negotiate fit interfaces between each other rather than to dictate interfacing. [[release-branches]] === Release branches The releases of FreeBSD are best illustrated by a tree with many branches where each major branch represents a major version. Minor versions are represented by branches of the major branches. In the following release tree, arrows that follow one-another in a particular direction represent a branch. Boxes with full lines and diamonds represent official releases. Boxes with dotted lines represent the development branch at that time. Security branches are represented by ovals. Diamonds differ from boxes in that they represent a fork, meaning a place where a branch splits into two branches where one of the branches becomes a sub-branch. For example, at 4.0-RELEASE the 4.0-CURRENT branch split into 4-STABLE and 5.0-CURRENT. At 4.5-RELEASE, the branch forked off a security branch called RELENG_4_5. .The FreeBSD release tree image::branches.png[Refer to table below for a screen-reader friendly version.] [.informaltable] [cols="1,1,1", options="header"] |=== | Major release | Forked from | Following minor releases |... | | |3.0 Current (development branch) | |Releng 3 branches: 3.0 Release to 3.5 Release, leading to 3.5.1 Release and the subsequent 3 Stable branch |4.0 Current (development branch) |3.1 Release |Releng 4 branches: 4.1 Release to 4.6 Release (and 4.6.2 Release), then 4.7 Release to 4.11 Release (all starting at 4.3 Release also leading to a Releng_4_n branch), and the subsequent 4 Release branch |5.0 Current (development branch) |4.0 Release |Releng 5 branches: 5.0 Release to 5.4 Release (all except 5.0 and 5.3 also leading to a Releng_5_n branch), and the subsequent 5 Release branch |6.0 Current (development branch) |5.3 Release | |... | | |=== The latest -CURRENT version is always referred to as -CURRENT, while the latest -STABLE release is always referred to as -STABLE. In this figure, -STABLE refers to 4-STABLE while -CURRENT refers to 5.0-CURRENT following 5.0-RELEASE. [<>] A "major release" is always made from the -CURRENT branch. However, the -CURRENT branch does not need to fork at that point in time, but can focus on stabilising. An example of this is that following 3.0-RELEASE, 3.1-RELEASE was also a continuation of the -CURRENT-branch, and -CURRENT did not become a true development branch until this version was released and the 3-STABLE branch was forked. When -CURRENT returns to becoming a development branch, it can only be followed by a major release. 5-STABLE is predicted to be forked off 5.0-CURRENT at around 5.3-RELEASE. It is not until 5-STABLE is forked that the development branch will be branded 6.0-CURRENT. A "minor release" is made from the -CURRENT branch following a major release, or from the -STABLE branch. Following and including, 4.3-RELEASEfootnote:[The first release this actually happened for was 4.5-RELEASE, but security branches were at the same time created for 4.3-RELEASE and 4.4-RELEASE.], when a minor release has been made, it becomes a "security branch". This is meant for organisations that do not want to follow the -STABLE branch and the potential new/changed features it offers, but instead require an absolutely stable environment, only updating to implement security updates. footnote:[There is a terminology overlap with respect to the word "stable", which leads to some confusion. The -STABLE branch is still a development branch, whose goal is to be useful for most people. If it is never acceptable for a system to get changes that are not announced at the time it is deployed, that system should run a security branch.] Each update to a security branch is called a "patchlevel". For every security enhancement that is done, the patchlevel number is increased, making it easy for people tracking the branch to see what security enhancements they have implemented. In cases where there have been especially serious security flaws, an entire new release can be made from a security branch. An example of this is 4.6.2-RELEASE. [[model-summary]] === Model summary To summarise, the development model of FreeBSD can be seen as the following tree: .The overall development model image::freebsd-code-model.png[Refer to paragraphs below for a screen-reader friendly version.] The tree of the FreeBSD development with ongoing development efforts and continuous integration. The tree symbolises the release versions with major versions spawning new main branches and minor versions being versions of the main branch. The top branch is the -CURRENT branch where all new development is integrated, and the -STABLE branch is the branch directly below it. Below the -STABLE branch are old, unsupported versions. Clouds of development efforts hang over the project where developers use the development models they see fit. The product of their work is then integrated into -CURRENT where it undergoes parallel debugging and is finally merged from -CURRENT into -STABLE. Security fixes are merged from -STABLE to the security branches. Many committers have a special area of responsibility. These roles are called hats. These hats can be either project roles, such as public relations officer, or maintainer for a certain area of the code. Because this is a project where people give voluntarily of their spare time, people with assigned hats are not always available. They must therefore appoint a deputy that can perform the hat's role in their absence. The other option is to have the role held by a group. Many of these hats are not formalised. Formalised hats have a charter stating the exact purpose of the hat along with its privileges and responsibilities. The writing of such charters is a new part of the project, and has thus yet to be completed for all hats. These hat descriptions are not such a formalisation, rather a summary of the role with links to the charter where available and contact addresses. [[sect-hats]] == Hats [[general-hats]] === General Hats [[role-contributor]] ==== Contributor A Contributor contributes to the FreeBSD project either as a developer, as an author, by sending problem reports, or in other ways contributing to the progress of the project. A contributor has no special privileges in the FreeBSD project. [<>] [[role-committer]] ==== Committer A person who has the required privileges to add their code or documentation to the repository. A committer has made a commit within the past 12 months. [<>] An active committer is a committer who has made an average of one commit per month during that time. It is worth noting that there are no technical barriers to prevent someone, once having gained commit privileges to the main- or a sub-project, to make commits in parts of that project's source the committer did not specifically get permission to modify. However, when wanting to make modifications to parts a committer has not been involved in before, they should read the logs to see what has happened in this area before, and also read the MAINTAINERS file to see if the maintainer of this part has any special requests on how changes in the code should be made. [[role-core]] ==== Core Team The core team is elected by the committers from the pool of committers and serves as the board of directors of the FreeBSD project. It promotes active contributors to committers, assigns people to well-defined hats, and is the final arbiter of decisions involving which way the project should be heading. As of July 1st, 2004, core consisted of 9 members. Elections are held every two years. [[role-maintainer]] ==== Maintainership Maintainership means that that person is responsible for what is allowed to go into that area of the code and has the final say should disagreements over the code occur. This involves proactive work aimed at stimulating contributions and reactive work in reviewing commits. With the FreeBSD source comes the MAINTAINERS file that contains a one-line summary of how each maintainer would like contributions to be made. Having this notice and contact information enables developers to focus on the development effort rather than being stuck in a slow correspondence should the maintainer be unavailable for some time. If the maintainer is unavailable for an unreasonably long period of time, and other people do a significant amount of work, maintainership may be switched without the maintainer's approval. This is based on the stance that maintainership should be demonstrated, not declared. Maintainership of a particular piece of code is a hat that is not held as a group. [[official-hats]] === Official Hats The official hats in the FreeBSD Project are hats that are more or less formalised and mainly administrative roles. They have the authority and responsibility for their area. The following list shows the responsibility lines and gives a description of each hat, including who it is held by. [[role-doc-manager]] ==== Documentation project manager <> architect is responsible for defining and following up documentation goals for the committers in the Documentation project, which they supervise. Hat held by: The DocEng team mailto:doceng@FreeBSD.org[doceng@FreeBSD.org]. The https://www.freebsd.org/internal/doceng/[ DocEng Charter]. [[role-postmaster]] ==== Postmaster The Postmaster is responsible for mail being correctly delivered to the committers' email address. They are also responsible for ensuring that the mailing lists work and should take measures against possible disruptions of mail such as having troll-, spam- and virus-filters. Hat currently held by: the Postmaster Team mailto:postmaster@FreeBSD.org[postmaster@FreeBSD.org]. [[role-release-coordination]] ==== Release Coordination The responsibilities of the Release Engineering Team are * Setting, publishing and following a release schedule for official releases * Documenting and formalising release engineering procedures * Creation and maintenance of code branches * Coordinating with the Ports and Documentation teams to have an updated set of packages and documentation released with the new releases * Coordinating with the Security team so that pending releases are not affected by recently disclosed vulnerabilities. Further information about the development process is available in the <> section. [[role-releng]] Hat held by: the Release Engineering team mailto:re@FreeBSD.org[re@FreeBSD.org]. The https://www.freebsd.org/releng/charter/[ Release Engineering Charter]. [[role-pr-cr]] ==== Public Relations & Corporate Liaison The Public Relations & Corporate Liaison's responsibilities are: * Making press statements when happenings that are important to the FreeBSD Project happen. * Being the official contact person for corporations that are working close with the FreeBSD Project. * Take steps to promote FreeBSD within both the Open Source community and the corporate world. * Handle the "freebsd-advocacy" mailing list. This hat is currently not occupied. [[role-security-officer]] ==== Security Officer The Security Officer's main responsibility is to coordinate information exchange with others in the security community and in the FreeBSD project. The Security Officer is also responsible for taking action when security problems are reported and promoting proactive development behavior when it comes to security. Because of the fear that information about vulnerabilities may leak out to people with malicious intent before a patch is available, only the Security Officer, consisting of an officer, a deputy and two <> members, receive sensitive information about security issues. However, to create or implement a patch, the Security Officer has the Security Officer Team mailto:security-team@FreeBSD.org[security-team@FreeBSD.org] to help do the work. [[role-repo-manager]] ==== Source Repository Manager The Source Repository Manager is the only one who is allowed to directly modify the repository without using the <> tool. It is their responsibility to ensure that technical problems that arise in the repository are resolved quickly. The source repository manager has the authority to back out commits if this is necessary to resolve a SVN technical problem. Hat held by: the Source Repository Manager mailto:clusteradm@FreeBSD.org[clusteradm@FreeBSD.org]. [[role-election-manager]] ==== Election Manager The Election Manager is responsible for the <> process. The manager is responsible for running and maintaining the election system, and is the final authority should minor unforeseen events happen in the election process. Major unforeseen events have to be discussed with the <> Hat held only during elections. [[role-webmaster]] ==== Web site Management The Web site Management hat is responsible for coordinating the rollout of updated web pages on mirrors around the world, for the overall structure of the primary web site and the system it is running upon. The management needs to coordinate the content with <> and acts as maintainer for the "www" tree. Hat held by: the FreeBSD Webmasters mailto:www@FreeBSD.org[www@FreeBSD.org]. [[role-ports-manager]] ==== Ports Manager The Ports Manager acts as a liaison between <> and the core project, and all requests from the project should go to the ports manager. Hat held by: the Ports Management Team mailto:portmgr@FreeBSD.org[portmgr@FreeBSD.org]. The https://www.freebsd.org/portmgr/charter/[Portmgr charter]. [[role-standards]] ==== Standards The Standards hat is responsible for ensuring that FreeBSD complies with the standards it is committed to , keeping up to date on the development of these standards and notifying FreeBSD developers of important changes that allows them to take a proactive role and decrease the time between a standards update and FreeBSD's compliancy. Hat currently held by: Garrett Wollman mailto:wollman@FreeBSD.org[wollman@FreeBSD.org]. [[role-core-secretary]] ==== Core Secretary The Core Secretary's main responsibility is to write drafts to and publish the final Core Reports. The secretary also keeps the core agenda, thus ensuring that no balls are dropped unresolved. Hat currently held by: {bofh}. [[role-bugmeister]] ==== Bugmeister The Bugmeister is responsible for ensuring that the maintenance database is in working order, that the entries are correctly categorised and that there are no invalid entries. They supervise bugbusters. Hat currently held by: the Bugmeister Team mailto:bugmeister@FreeBSD.org[bugmeister@FreeBSD.org]. [[role-donations]] ==== Donations Liaison Officer The task of the donations liaison officer is to match the developers with needs with people or organisations willing to make a donation. Hat held by: the Donations Liaison Office mailto:donations@FreeBSD.org[donations@FreeBSD.org]. The https://www.freebsd.org/donations/[ Donations Liaison Charter]. [[role-admin]] ==== Admin (Also called "FreeBSD Cluster Admin") The admin team consists of the people responsible for administrating the computers that the project relies on for its distributed work and communication to be synchronised. It consists mainly of those people who have physical access to the servers. Hat held by: the Admin team mailto:admin@FreeBSD.org[admin@FreeBSD.org]. [[proc-depend-hats]] === Process dependent hats [[role-problem-originator]] ==== Report originator The person originally responsible for filing a Problem Report. [[role-bugbuster]] ==== Bugbuster A person who will either find the right person to solve the problem, or close the PR if it is a duplicate or otherwise not an interesting one. [[role-mentor]] ==== Mentor A mentor is a committer who takes it upon them to introduce a new committer to the project, both in terms of ensuring the new committer's setup is valid, that the new committer knows the available tools required in their work, and that the new committer knows what is expected of them in terms of behavior. [[role-vendor]] ==== Vendor The person(s) or organisation whom external code comes from and whom patches are sent to. [[role-reviewer]] ==== Reviewers People on the mailing list where the request for review is posted. The following section will describe the defined project processes. Issues that are not handled by these processes happen on an ad-hoc basis based on what has been customary to do in similar cases. [[model-processes]] == Processes [[proc-addrem-committer]] === Adding new and removing old committers The Core team has the responsibility of giving and removing commit privileges to contributors. This can only be done through a vote on the core mailing list. The ports and documentation sub-projects can give commit privileges to people working on these projects, but have to date not removed such privileges. Normally a contributor is recommended to core by a committer. For contributors or outsiders to contact core asking to be a committer is not well thought of and is usually rejected. If the area of particular interest for the developer potentially overlaps with other committers' area of maintainership, the opinion of those maintainers is sought. However, it is frequently this committer that recommends the developer. When a contributor is given committer status, they are assigned a mentor. The committer who recommended the new committer will, in the general case, take it upon themselves to be the new committers mentor. When a contributor is given their commit bit, a <>-signed email is sent from either <>, <>, or nik@freebsd.org to both admins@freebsd.org, the assigned mentor, the new committer, and core confirming the approval of a new account. The mentor then gathers a password line, <> public key, and PGP key from the new committer and sends them to <>. When the new account is created, the mentor activates the commit bit and guides the new committer through the rest of the initial process. .Process summary: adding a new committer image::proc-add-committer.png[Refer to paragraph below for a screen-reader friendly version.] When a contributor sends a piece of code, the receiving committer may choose to recommend that the contributor is given commit privileges. If they recommend this to core, core will vote on this recommendation. If the vote is in favour, a mentor is assigned the new committer and the new committer has to email their details to the administrators for an account to be created. After this, the new committer is all set to make their first commit. By tradition, this is by adding their name to the committers list. Recall that a committer is considered to be someone who has committed code during the past 12 months. However, it is not until after 18 months of inactivity have passed that commit privileges are eligible to be revoked. [<>] There are, however, no automatic procedures for doing this. For reactions concerning commit privileges not triggered by time, see <>. .Process summary: removing a committer image::proc-rm-committer.png[Refer to paragraph below for a screen-reader friendly version.] When Core decides to clean up the committers list, they check who has not made a commit for the past 18 months. Committers who have not done so have their commit bits revoked and their account removed by the administrators. It is also possible for committers to request that their commit bit be retired if for some reason they are no longer going to be actively committing to the project. In this case, it can also be restored at a later time by core, should the committer ask. Roles in this process: . <> . <> . <> . <> . <> [<>] [<>] [<>] [[committing]] === Committing code The committing of new or modified code is one of the most frequent processes in the FreeBSD project and will usually happen many times a day. Committing of code can only be done by a "committer". Committers commit either code written by themselves, code submitted to them, or code submitted through a <>. When code is written by the developer that is non-trivial, they should seek a code review from the community. This is done by sending mail to the relevant list asking for review. Before submitting the code for review, they should ensure it compiles correctly with the entire tree and that all relevant tests run. This is called "pre-commit test". When contributed code is received, it should be reviewed by the committer and tested the same way. When a change is committed to a part of the source that has been contributed from an outside <>, the maintainer should ensure that the patch is contributed back to the vendor. This is in line with the open source philosophy and makes it easier to stay in sync with outside projects as the patches do not have to be reapplied every time a new release is made. After the code has been available for review and no further changes are necessary, the code is committed into the development branch, -CURRENT. If the change applies for the -STABLE branch or the other branches as well, a "Merge From Current" ("MFC") countdown is set by the committer. After the number of days the committer chose when setting the MFC have passed, an email will automatically be sent to the committer reminding them to commit it to the -STABLE branch (and possibly security branches as well). Only security critical changes should be merged to security branches. Delaying the commit to -STABLE and other branches allows for "parallel debugging" where the committed code is tested on a wide range of configurations. This makes changes to -STABLE to contain fewer faults and thus giving the branch its name. .Process summary: A committer commits code image::proc-commit.png[Refer to paragraph below for a screen-reader friendly version.] When a committer has written a piece of code and wants to commit it, they first need to determine if it is trivial enough to go in without prior review or if it should first be reviewed by the developer community. If the code is trivial or has been reviewed and the committer is not the maintainer, they should consult the maintainer before proceeding. If the code is contributed by an outside vendor, the maintainer should create a patch that is sent back to the vendor. The code is then committed and then deployed by the users. Should they find problems with the code, this will be reported and the committer can go back to writing a patch. If a vendor is affected, they can choose to implement or ignore the patch. .Process summary: A contributor commits code image::proc-contrib.png[Refer to paragraphs below and above for a screen-reader friendly version.] The difference when a contributor makes a code contribution is that they submit the code through the Bugzilla interface. This report is picked up by the maintainer who reviews the code and commits it. Hats included in this process are: . <> . <> . <> . <> [<>] [<>] [[process-core-election]] === Core election Core elections are held at least every two years. footnote:[The first Core election was held September 2000] Nine core members are elected. New elections are held if the number of core members drops below seven. New elections can also be held should at least 1/3 of the active committers demand this. When an election is to take place, core announces this at least 6 weeks in advance, and appoints an election manager to run the elections. Only committers can be elected into core. The candidates need to submit their candidacy at least one week before the election starts, but can refine their statements until the voting starts. They are presented in the http://election.uk.freebsd.org/candidates.html[candidates list]. When writing their election statements, the candidates must answer a few standard questions submitted by the election manager. During elections, the rule that a committer must have committed during the 12 past months is followed strictly. Only these committers are eligible to vote. When voting, the committer may vote once in support of up to nine nominees. The voting is done over a period of four weeks with reminders being posted on "developers" mailing list that is available to all committers. The election results are released one week after the election ends, and the new core team takes office one week after the results have been posted. Should there be a voting tie, this will be resolved by the new, unambiguously elected core members. Votes and candidate statements are archived, but the archives are not publicly available. .Process summary: Core elections image::proc-elections.png[Refer to paragraph below for a screen-reader friendly version.] Core announces the election and selects an election manager who prepares the elections, and when ready, candidates can announce their candidacies through submitting their statements. The committers then vote. After the vote is over, the election results are announced and the new core team takes office. Hats in core elections are: * <> * <> * <> [<>] [<>] [<>] [[new-features]] === Development of new features Within the project there are sub-projects that are working on new features. These projects are generally done by one person [<>]. Every project is free to organise development as it sees fit. However, when the project is merged to the -CURRENT branch it must follow the project guidelines. When the code has been well tested in the -CURRENT branch and deemed stable enough and relevant to the -STABLE branch, it is merged to the -STABLE branch. The requirements of the project are given by developer wishes, requests from the community in terms of direct requests by mail, Problem Reports, commercial funding for the development of features, or contributions by the scientific community. The wishes that come within the responsibility of a developer are given to that developer who prioritises their time between the request and their wishes. A common way to do this is maintain a TODO-list maintained by the project. Items that do not come within someone's responsibility are collected on TODO-lists unless someone volunteers to take the responsibility. All requests, their distribution and follow-up are handled by the <> tool. Requirements analysis happens in two ways. The requests that come in are discussed on mailing lists, both within the main project and in the sub-project that the request belongs to or is spawned by the request. Furthermore, individual developers on the sub-project will evaluate the feasibility of the requests and determine the prioritisation between them. Other than archives of the discussions that have taken place, no outcome is created by this phase that is merged into the main project. As the requests are prioritised by the individual developers on the basis of doing what they find interesting, necessary, or are funded to do, there is no overall strategy or prioritisation of what requests to regard as requirements and following up their correct implementation. However, most developers have some shared vision of what issues are more important, and they can ask for guidelines from the release engineering team. The verification phase of the project is two-fold. Before committing code to the current-branch, developers request their code to be reviewed by their peers. This review is for the most part done by functional testing, but also code review is important. When the code is committed to the branch, a broader functional testing will happen, that may trigger further code review and debugging should the code not behave as expected. This second verification form may be regarded as structural verification. Although the sub-projects themselves may write formal tests such as unit tests, these are usually not collected by the main project and are usually removed before the code is committed to the current branch. footnote:[More and more tests are however performed when building the system (make world). These tests are however a very new addition and no systematic framework for these tests have yet been created.] [[model-maintenance]] === Maintenance It is an advantage to the project to for each area of the source have at least one person that knows this area well. Some parts of the code have designated maintainers. Others have de-facto maintainers, and some parts of the system do not have maintainers. The maintainer is usually a person from the sub-project that wrote and integrated the code, or someone who has ported it from the platform it was written for. footnote:[sendmail and named are examples of code that has been merged from other platforms.] The maintainer's job is to make sure the code is in sync with the project the code comes from if it is contributed code, and apply patches submitted by the community or write fixes to issues that are discovered. The main bulk of work that is put into the FreeBSD project is maintenance. [<>] has made a figure showing the life cycle of changes. Jørgenssen's model for change integration [.informaltable] [cols="1,1,1", options="header"] |=== | Stage | Next if successful | Next if unsuccessful |code |review | |review |pre-commit test |code |pre-commit test |development release |code |development release |parallel debugging |code |parallel debugging |production release |code |production release | |code |=== Here "development release" refers to the -CURRENT branch while "production release" refers to the -STABLE branch. The "pre-commit test" is the functional testing by peer developers when asked to do so or trying out the code to determine the status of the sub-project. "Parallel debugging" is the functional testing that can trigger more review, and debugging when the code is included in the -CURRENT branch. As of this writing, there were 269 committers in the project. When they commit a change to a branch, that constitutes a new release. It is very common for users in the community to track a particular branch. The immediate existence of a new release makes the changes widely available right away and allows for rapid feedback from the community. This also gives the community the response time they expect on issues that are of importance to them. This makes the community more engaged, and thus allows for more and better feedback that again spurs more maintenance and ultimately should create a better product. Before making changes to code in parts of the tree that has a history unknown to the committer, the committer is required to read the commit logs to see why certain features are implemented the way they are in order not to make mistakes that have previously either been thought through or resolved. [[model-pr]] === Problem reporting Before FreeBSD 10, FreeBSD included a problem reporting tool called `send-pr`. Problems include bug reports, feature requests, feature enhancements and notices of new versions of external software that are included in the project. Although `send-pr` is available, users and developers are encouraged to submit issues using our https://bugs.freebsd.org/submit/[ problem report form]. Problem reports are sent to an email address where it is inserted into the Problem Reports maintenance database. A <> classifies the problem and sends it to the correct group or maintainer within the project. After someone has taken responsibility for the report, the report is being analysed. This analysis includes verifying the problem and thinking out a solution for the problem. Often feedback is required from the report originator or even from the FreeBSD community. Once a patch for the problem is made, the originator may be asked to try it out. Finally, the working patch is integrated into the project, and documented if applicable. It there goes through the regular maintenance cycle as described in section <>. These are the states a problem report can be in: open, analyzed, feedback, patched, suspended and closed. The suspended state is for when further progress is not possible due to the lack of information or for when the task would require so much work that nobody is working on it at the moment. .Process summary: problem reporting image::proc-pr.png[Refer to paragraph below for a screen-reader friendly version.] A problem is reported by the report originator. It is then classified by a bugbuster and handed to the correct maintainer. They verify the problem and discuss the problem with the originator until they have enough information to create a working patch. This patch is then committed and the problem report is closed. The roles included in this process are: . <> . <> . <> [<>]. [<>] [[process-reactions]] === Reacting to misbehavior [<>] has a number of rules that committers should follow. However, it happens that these rules are broken. The following rules exist in order to be able to react to misbehavior. They specify what actions will result in how long a suspension of the committer's commit privileges. * Committing during code freezes without the approval of the Release Engineering team - 2 days * Committing to a security branch without approval - 2 days * Commit wars - 5 days to all participating parties * Impolite or inappropriate behavior - 5 days [<>] For the suspensions to be efficient, any single core member can implement a suspension before discussing it on the "core" mailing list. Repeat offenders can, with a 2/3 vote by core, receive harsher penalties, including permanent removal of commit privileges. (However, the latter is always viewed as a last resort, due to its inherent tendency to create controversy.) All suspensions are posted to the "developers" mailing list, a list available to committers only. It is important that you cannot be suspended for making technical errors. All penalties come from breaking social etiquette. Hats involved in this process: * <> * <> [[process-release-engineering]] === Release engineering The FreeBSD project has a Release Engineering team with a principal release engineer that is responsible for creating releases of FreeBSD that can be brought out to the user community via the net or sold in retail outlets. Since FreeBSD is available on multiple platforms and releases for the different architectures are made available at the same time, the team has one person in charge of each architecture. Also, there are roles in the team responsible for coordinating quality assurance efforts, building a package set and for having an updated set of documents. When referring to the release engineer, a representative for the release engineering team is meant. When a release is coming, the FreeBSD project changes shape somewhat. A release schedule is made containing feature- and code-freezes, release of interim releases and the final release. A feature-freeze means no new features are allowed to be committed to the branch without the release engineers' explicit consent. Code-freeze means no changes to the code (like bugs-fixes) are allowed to be committed without the release engineers' explicit consent. This feature- and code-freeze is known as stabilising. During the release process, the release engineer has the full authority to revert to older versions of code and thus "back out" changes should they find that the changes are not suitable to be included in the release. There are three different kinds of releases: . .0 releases are the first release of a major version. These are branched of the -CURRENT branch and have a significantly longer release engineering cycle due to the unstable nature of the -CURRENT branch . .X releases are releases of the -STABLE branch. They are scheduled to come out every 4 months. . .X.Y releases are security releases that follow the .X branch. These come out only when sufficient security fixes have been merged since the last release on that branch. New features are rarely included, and the security team is far more involved in these than in regular releases. For releases of the -STABLE-branch, the release process starts 45 days before the anticipated release date. During the first phase, the first 15 days, the developers merge what changes they have had in -CURRENT that they want to have in the release to the release branch. When this period is over, the code enters a 15 day code freeze in which only bug fixes, documentation updates, security-related fixes and minor device driver changes are allowed. These changes must be approved by the release engineer in advance. At the beginning of the last 15 day period a release candidate is created for widespread testing. Updates are less likely to be allowed during this period, except for important bug fixes and security updates. In this final period, all releases are considered release candidates. At the end of the release process, a release is created with the new version number, including binary distributions on web sites and the creation of CD-ROM images. However, the release is not considered "really released" until a <>-signed message stating exactly that, is sent to the mailing list freebsd-announce; anything labelled as a "release" before that may well be in-process and subject to change before the PGP-signed message is sent. footnote:[Many commercial vendors use these images to create CD-ROMs that are sold in retail outlets.]. The releases of the -CURRENT-branch (that is, all releases that end with ".0") are very similar, but with twice as long timeframe. It starts 8 weeks prior to the release with announcement of the release time line. Two weeks into the release process, the feature freeze is initiated and performance tweaks should be kept to a minimum. Four weeks prior to the release, an official beta version is made available. Two weeks prior to release, the code is officially branched into a new version. This version is given release candidate status, and as with the release engineering of -STABLE, the code freeze of the release candidate is hardened. However, development on the main development branch can continue. Other than these differences, the release engineering processes are alike. *.0 releases go into their own branch and are aimed mainly at early adopters. The branch then goes through a period of stabilisation, and it is not until the <> decides the demands to stability have been satisfied that the branch becomes -STABLE and -CURRENT targets the next major version. While this for the majority has been with *.1 versions, this is not a demand. Most releases are made when a given date that has been deemed a long enough time since the previous release comes. A target is set for having major releases every 18 months and minor releases every 4 months. The user community has made it very clear that security and stability cannot be sacrificed by self-imposed deadlines and target release dates. For slips of time not to become too long with regards to security and stability issues, extra discipline is required when committing changes to -STABLE. . Make release schedule . Feature freeze . Code freeze . Make branch . Release candidate . Stabilize release (loop back to previous step as many times as necessary; when release is considered stable, proceed with next step) . Build packages . Warn mirrors . Publish release [<>] [[tools]] == Tools The major support tools for supporting the development process are Bugzilla, Mailman, and OpenSSH. These are externally developed tools and are commonly used in the open source world. [[tool-svn]] === Subversion (SVN) Subversion ("SVN") is a system to handle multiple versions of text files and tracking who committed what changes and why. A project lives within a "repository" and different versions are considered different "branches". [[tool-bugzilla]] === Bugzilla Bugzilla is a maintenance database consisting of a set of tools to track bugs at a central site. It supports the bug tracking process for sending and handling bugs as well as querying and updating the database and editing bug reports. The project uses its web interface to send "Problem Reports" to the project's central Bugzilla server. The committers also have web and command-line clients. [[model-mailman]] === Mailman Mailman is a program that automates the management of mailing lists. The FreeBSD Project uses it to run 16 general lists, 60 technical lists, 4 limited lists and 5 lists with SVN commit logs. It is also used for many mailing lists set up and used by other people and projects in the FreeBSD community. General lists are lists for the general public, technical lists are mainly for the development of specific areas of interest, and closed lists are for internal communication not intended for the general public. The majority of all the communication in the project goes through these 85 lists [<>, Appendix C]. [[tool-pgp]] === Pretty Good Privacy Pretty Good Privacy, better known as PGP, is a cryptosystem using a public key architecture to allow people to digitally sign and/or encrypt information in order to ensure secure communication between two parties. A signature is used when sending information out to many recipients, enabling them to verify that the information has not been tampered with before they received it. In the FreeBSD Project this is the primary means of ensuring that information has been written by the person who claims to have written it, and not altered in transit. [[tool-ssh2]] === Secure Shell Secure Shell is a standard for securely logging into a remote system and for executing commands on the remote system. It allows other connections, called tunnels, to be established and protected between the two involved systems. This standard exists in two primary versions, and only version two is used for the FreeBSD Project. The most common implementation of the standard is OpenSSH that is a part of the project's main distribution. Since its source is updated more often than FreeBSD releases, the latest version is also available in the ports tree. [[sub-projects]] == Sub-projects Sub-projects are formed to reduce the amount of communication needed to coordinate the group of developers. When a problem area is sufficiently isolated, most communication would be within the group focusing on the problem, requiring less communication with the groups they communicate with than were the group not isolated. [[sub-project-ports]] === The Ports Subproject A "port" is a set of meta-data and patches that are needed to fetch, compile and install correctly an external piece of software on a FreeBSD system. The amount of ports has grown at a tremendous rate, as shown by the following figure. .Number of ports added between 1996 and 2008 [[fig-ports]] image::portsstatus.png[Refer to tables below for a screen-reader friendly version.] <> shows the number of ports available to FreeBSD in the period 1995 to 2008. It looks like the curve has first grown exponentially, and then from the middle of 2001 to the middle of 2007 grown linearly at a rate of about 2000 ports/year, before its growth rate gets lower. Approximate dates each multiple of 1000 ports is reached [.informaltable] [cols="1,1", options="header"] |=== | Number of ports | Approximate date |1000 |Late 1997 |2000 |Late 1998 |3000 |Early 2000 |4000 |Late 2000 |5000 |Mid 2001 |6000 |4th quarter of 2001 |7000 |Mid 2002 |8000 |4th quarter of 2002 |9000 |Mid 2003 |10000 |End of 2003 |11000 |Mid 2004 |12000 |End of 2004 |13000 |Mid 2005 |14000 |Early 2006 |15000 |Mid 2006 |16000 |3rd quarter 2006 |17000 |2nd quarter 2007 |=== Approximate number of ports at the start of each year [.informaltable] [cols="1,1", options="header"] |=== | Year | Approximate number of ports |1995 |100 |1996 |300 |1997 |700 |1998 |1200 |1999 |2000 |2000 |2900 |2001 |4300 |2002 |6200 |2003 |8100 |2004 |10050 |2005 |12100 |2006 |14000 |2007 |16200 |2008 |17900 |=== As the external software described by the port often is under continued development, the amount of work required to maintain the ports is already large, and increasing. This has led to the ports part of the FreeBSD project gaining a more empowered structure, and is more and more becoming a sub-project of the FreeBSD project. Ports has its own core team with the <> as its leader, and this team can appoint committers without FreeBSD Core's approval. Unlike in the FreeBSD Project, where a lot of maintenance frequently is rewarded with a commit bit, the ports sub-project contains many active maintainers that are not committers. Unlike the main project, the ports tree is not branched. Every release of FreeBSD follows the current ports collection and has thus available updated information on where to find programs and how to build them. This, however, means that a port that makes dependencies on the system may need to have variations depending on what version of FreeBSD it runs on. With an unbranched ports repository it is not possible to guarantee that any port will run on anything other than -CURRENT and -STABLE, in particular older, minor releases. There is neither the infrastructure nor volunteer time needed to guarantee this. For efficiency of communication, teams depending on Ports, such as the release engineering team, have their own ports liaisons. [[sub-project-documentation]] === The FreeBSD Documentation Project The FreeBSD Documentation project was started January 1995. From the initial group of a project leader, four team leaders and 16 members, they are now a total of 44 committers. The documentation mailing list has just under 300 members, indicating that there is quite a large community around it. The goal of the Documentation project is to provide good and useful documentation of the FreeBSD project, thus making it easier for new users to get familiar with the system and detailing advanced features for the users. The main tasks in the Documentation project are to work on current projects in the "FreeBSD Documentation Set", and translate the documentation to other languages. Like the FreeBSD Project, documentation is split in the same branches. This is done so that there is always an updated version of the documentation for each version. Only documentation errors are corrected in the security branches. Like the ports sub-project, the Documentation project can appoint documentation committers without FreeBSD Core's approval. [<>]. The Documentation project has link:{fdp-primer}[a primer]. This is used both to introduce new project members to the standard tools and syntaxes and to act as a reference when working on the project. :sectnums!: [bibliography] [[bibliography]] == References [[brooks]] [Brooks, 1995] Frederick P. Brooks. Copyright © 1975, 1995 Pearson Education Limited. 0201835959. Addison-Wesley Pub Co. The Mythical Man-Month. Essays on Software Engineering, Anniversary Edition (2nd Edition). [[thesis]] [Saers, 2003] Niklas Saers. Copyright © 2003. A project model for the FreeBSD Project. Candidatus Scientiarum thesis. http://niklas.saers.com/thesis. [[jorgensen2001]] [Jørgensen, 2001] Niels Jørgensen. Copyright © 2001. Putting it All in the Trunk. Incremental Software Development in the FreeBSD Open Source Project. http://www.dat.ruc.dk/~nielsj/research/papers/freebsd.pdf. [[ref-pmbok]] [PMI, 2000] Project Management Institute. Copyright © 1996, 2000 Project Management Institute. 1-880410-23-0. Project Management Institute. Newtown Square Pennsylvania USA . PMBOK Guide. A Guide to the Project Management Body of Knowledge, 2000 Edition. [[freebsd-bylaws]] [FreeBSD, 2000A] Copyright © 2002 The FreeBSD Project. Core Bylaws. https://www.freebsd.org/internal/bylaws/. [[freebsd-developer-handbook]] [FreeBSD, 2002A] Copyright © 2002 The FreeBSD Documentation Project. FreeBSD Developer's Handbook. link:{developers-handbook}[Developers Handbook]. [[bsd-election2002]] [FreeBSD, 2002B] Copyright © 2002 The FreeBSD Project. Core team election 2002. http://election.uk.freebsd.org/candidates.html. [[freebsd-handle-pr]] [FreeBSD, 2002C] Dag-Erling Smørgrav and Hiten Pandya. Copyright © 2002 The FreeBSD Documentation Project. The FreeBSD Documentation Project. Problem Report Handling Guidelines. link:{pr-guidelines}[Problem Report Handling Guidelines]. [[freebsd-send-pr]] [FreeBSD, 2002D] Dag-Erling Smørgrav. Copyright © 2002 The FreeBSD Documentation Project. The FreeBSD Documentation Project. Writing FreeBSD Problem Reports. link:{problem-reports}[Writing FreeBSD Problem Reports]. [[freebsd-committer]] [FreeBSD, 2001] Copyright © 2001 The FreeBSD Documentation Project. The FreeBSD Documentation Project. Committers Guide. link:{committers-guide}[Committer's Guide]. [[freebsd-releng]] [FreeBSD, 2002E] Murray Stokely. Copyright © 2002 The FreeBSD Documentation Project. The FreeBSD Documentation Project. FreeBSD Release Engineering. https://link:{releng}[FreeBSD Release Engineering]. [[ref-bsd-handbook]] [FreeBSD, 2003A] The FreeBSD Documentation Project. FreeBSD Handbook. link:{handbook}[FreeBSD Handbook]. [[freebsd-contributors]] [FreeBSD, 2002F] Copyright © 2002 The FreeBSD Documentation Project. The FreeBSD Documentation Project. Contributors to FreeBSD. link:{contributors}[Contributors to FreeBSD]. [[freebsd-election]] [FreeBSD, 2002G] Copyright © 2002 The FreeBSD Project. The FreeBSD Project. Core team elections 2002. http://election.uk.freebsd.org. [[freebsd-expiration-policy]] [FreeBSD, 2002H] Copyright © 2002 The FreeBSD Project. The FreeBSD Project. Commit Bit Expiration Policy. 2002/04/06 15:35:30. https://www.freebsd.org/internal/expire-bits/. [[freebsd-new-account]] [FreeBSD, 2002I] Copyright © 2002 The FreeBSD Project. The FreeBSD Project. New Account Creation Procedure. 2002/08/19 17:11:27. https://www.freebsd.org/internal/new-account/. [[freebsd-doceng-charter]] [FreeBSD, 2003B] Copyright © 2002 The FreeBSD Documentation Project. The FreeBSD Documentation Project. FreeBSD DocEng Team Charter. 2003/03/16 12:17. https://www.freebsd.org/internal/doceng/. [[ref-freebsd-trenches]] [Lehey, 2002] Greg Lehey. Copyright © 2002 Greg Lehey. Greg Lehey. Two years in the trenches. The evolution of a software project. http://www.lemis.com/grog/In-the-trenches.pdf. diff --git a/documentation/content/en/books/developers-handbook/_index.adoc b/documentation/content/en/books/developers-handbook/_index.adoc index c7407e78d1..30c5d158f8 100644 --- a/documentation/content/en/books/developers-handbook/_index.adoc +++ b/documentation/content/en/books/developers-handbook/_index.adoc @@ -1,58 +1,59 @@ --- title: FreeBSD Developers' Handbook authors: - author: The FreeBSD Documentation Project copyright: 1995-2021 The FreeBSD Documentation Project -description: FreeBSD Developers' Handbook Index +description: For people who want to develop software for FreeBSD (and not just people who are developing FreeBSD itself) trademarks: ["freebsd", "apple", "ibm", "ieee", "intel", "linux", "microsoft", "opengroup", "sun", "general"] next: books/developers-handbook/parti +weight: 25 --- = FreeBSD Developers' Handbook :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: ifeval::["{backend}" == "html5"] include::shared/mirrors.adoc[] include::shared/authors.adoc[] include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/teams.adoc[] include::shared/en/urls.adoc[] endif::[] ifeval::["{backend}" == "pdf"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] endif::[] ifeval::["{backend}" == "epub3"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] endif::[] [.abstract-title] [abstract] Abstract Welcome to the Developers' Handbook. This manual is a _work in progress_ and is the work of many individuals. Many sections do not yet exist and some of those that do exist need to be updated. If you are interested in helping with this project, send email to the {freebsd-doc}. The latest version of this document is always available from the link:https://www.FreeBSD.org[FreeBSD World Wide Web server]. It may also be downloaded in a variety of formats and compression options from the link:https://download.freebsd.org/ftp/doc/[FreeBSD FTP server] or one of the numerous link:{handbook}#mirrors-ftp/[mirror sites]. ''' include::content/en/books/developers-handbook/toc.adoc[] diff --git a/documentation/content/en/books/faq/_index.adoc b/documentation/content/en/books/faq/_index.adoc index cf3d7379df..70690a9837 100644 --- a/documentation/content/en/books/faq/_index.adoc +++ b/documentation/content/en/books/faq/_index.adoc @@ -1,2642 +1,2643 @@ --- title: Frequently Asked Questions for FreeBSD 11.X, 12.X, and 13.X authors: - author: The FreeBSD Documentation Project copyright: 1995-2021 The FreeBSD Documentation Project -description: Frequently Asked Questions (FAQ) for FreeBSD 11.X, 12.X, and 13.X +description: Frequently Asked Questions, and answers, covering all aspects of FreeBSD trademarks: ["freebsd", "ibm", "ieee", "adobe", "intel", "linux", "microsoft", "opengroup", "sun", "netbsd", "general"] +weight: 5 --- = Frequently Asked Questions for FreeBSD {rel2-relx} and {rel-relx} :doctype: book :toc: macro :toclevels: 1 :icons: font :xrefstyle: basic :relfileprefix: ../ :outfilesuffix: :sectnums: :sectnumlevels: 6 :partnums: :chapter-signifier: Chapter :part-signifier: Part :source-highlighter: rouge :experimental: :skip-front-matter: :rel-numbranch: 4 :rel-head: 14-CURRENT :rel-head-relx: 14.X :rel-head-releng: head/ :rel-relx: 13.X :rel-stable: 13-STABLE :rel-releng: stable/13/ :rel-relengdate: December 2018 :rel2-relx: 12.X :rel2-stable: 12-STABLE :rel2-releng: stable/12/ :rel2-relengdate: December 2018 :rel3-relx: 11.X :rel3-stable: 11-STABLE :rel3-releng: stable/11/ :rel3-relengdate: October 2016 ifeval::["{backend}" == "html5"] include::shared/mirrors.adoc[] include::shared/authors.adoc[] include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] include::shared/en/teams.adoc[] include::shared/en/urls.adoc[] endif::[] ifeval::["{backend}" == "pdf"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] endif::[] ifeval::["{backend}" == "epub3"] include::../../../../shared/mirrors.adoc[] include::../../../../shared/authors.adoc[] include::../../../../shared/releases.adoc[] include::../../../../shared/en/mailing-lists.adoc[] include::../../../../shared/en/teams.adoc[] include::../../../../shared/en/urls.adoc[] endif::[] [.abstract-title] Abstract This is the Frequently Asked Questions (FAQ) for FreeBSD versions {rel-relx}, {rel2-relx}, and {rel3-relx}. Every effort has been made to make this FAQ as informative as possible; if you have any suggestions as to how it may be improved, send them to the {freebsd-doc}. The latest version of this document is always available from the link:{faq}[FreeBSD website]. It may also be downloaded as one large link:.[HTML] file with HTTP or as a variety of other formats from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server]. ''' toc::[] [[introduction]] == Introduction [[what-is-FreeBSD]] === What is FreeBSD? FreeBSD is a modern operating system for desktops, laptops, servers, and embedded systems with support for a large number of https://www.FreeBSD.org/platforms/[platforms]. It is based on U.C. Berkeley's "4.4BSD-Lite" release, with some "4.4BSD-Lite2" enhancements. It is also based indirectly on William Jolitz's port of U.C. Berkeley's "Net/2" to the i386(TM), known as "386BSD", though very little of the 386BSD code remains. FreeBSD is used by companies, Internet Service Providers, researchers, computer professionals, students and home users all over the world in their work, education and recreation. For more detailed information on FreeBSD, refer to the link:{handbook}[FreeBSD Handbook]. [[FreeBSD-goals]] === What is the goal of the FreeBSD Project? The goal of the FreeBSD Project is to provide a stable and fast general purpose operating system that may be used for any purpose without strings attached. [[bsd-license-restrictions]] === Does the FreeBSD license have any restrictions? Yes. Those restrictions do not control how the code is used, but how to treat the FreeBSD Project itself. The license itself is available at https://www.FreeBSD.org/copyright/freebsd-license/[license] and can be summarized like this: * Do not claim that you wrote this. * Do not sue us if it breaks. * Do not remove or modify the license. Many of us have a significant investment in the project and would certainly not mind a little financial compensation now and then, but we definitely do not insist on it. We believe that our first and foremost "mission" is to provide code to any and all comers, and for whatever purpose, so that the code gets the widest possible use and provides the widest possible benefit. This, we believe, is one of the most fundamental goals of Free Software and one that we enthusiastically support. Code in our source tree which falls under the https://www.FreeBSD.org/copyright/COPYING[GNU General Public License (GPL)] or https://www.FreeBSD.org/copyright/COPYING.LIB[GNU Library General Public License (LGPL)] comes with slightly more strings attached, though at least on the side of enforced access rather than the usual opposite. Due to the additional complexities that can evolve in the commercial use of GPL software, we do, however, endeavor to replace such software with submissions under the more relaxed https://www.FreeBSD.org/copyright/freebsd-license/[FreeBSD license] whenever possible. [[replace-current-OS]] === Can FreeBSD replace my current operating system? For most people, yes. But this question is not quite that cut-and-dried. Most people do not actually use an operating system. They use applications. The applications are what really use the operating system. FreeBSD is designed to provide a robust and full-featured environment for applications. It supports a wide variety of web browsers, office suites, email readers, graphics programs, programming environments, network servers, and much more. Most of these applications can be managed through the https://www.FreeBSD.org/ports/[Ports Collection]. If an application is only available on one operating system, that operating system cannot just be replaced. Chances are, there is a very similar application on FreeBSD, however. As a solid office or Internet server or a reliable workstation, FreeBSD will almost certainly do everything you need. Many computer users across the world, including both novices and experienced UNIX(R) administrators, use FreeBSD as their only desktop operating system. Users migrating to FreeBSD from another UNIX(R)-like environment will find FreeBSD to be similar. Windows(R) and Mac OS(R) users may be interested in instead using https://www.ghostbsd.org/[GhostBSD], https://www.midnightbsd.org/[MidnightBSD] or https://www.nomadbsd.org/[NomadBSD] three FreeBSD-based desktop distributions. Non-UNIX(R) users should expect to invest some additional time learning the UNIX(R) way of doing things. This FAQ and the link:{handbook}[FreeBSD Handbook] are excellent places to start. [[why-called-FreeBSD]] === Why is it called FreeBSD? * It may be used free of charge, even by commercial users. * Full source for the operating system is freely available, and the minimum possible restrictions have been placed upon its use, distribution and incorporation into other work (commercial or non-commercial). * Anyone who has an improvement or bug fix is free to submit their code and have it added to the source tree (subject to one or two obvious provisions). It is worth pointing out that the word "free" is being used in two ways here: one meaning "at no cost" and the other meaning "do whatever you like". Apart from one or two things you _cannot_ do with the FreeBSD code, for example pretending you wrote it, you can really do whatever you like with it. [[differences-to-other-bsds]] === What are the differences between FreeBSD and NetBSD, OpenBSD, and other open source BSD operating systems? James Howard wrote a good explanation of the history and differences between the various projects, called https://jameshoward.us/archive/bsd-family-tree/[The BSD Family Tree] which goes a fair way to answering this question. Some of the information is out of date, but the history portion in particular remains accurate. Most of the BSDs share patches and code, even today. All of the BSDs have common ancestry. The design goals of FreeBSD are described in <>, above. The design goals of the other most popular BSDs may be summarized as follows: * OpenBSD aims for operating system security above all else. The OpenBSD team wrote man:ssh[1] and man:pf[4], which have both been ported to FreeBSD. * NetBSD aims to be easily ported to other hardware platforms. * DragonFly BSD is a fork of FreeBSD 4.8 that has since developed many interesting features of its own, including the HAMMER file system and support for user-mode "vkernels". [[latest-version]] === What is the latest version of FreeBSD? At any point in the development of FreeBSD, there can be multiple parallel branches. {rel-relx} releases are made from the {rel-stable} branch, and {rel2-relx} releases are made from the {rel2-stable} branch. Up until the release of 12.0, the {rel2-relx} series was the one known as _-STABLE_. However, as of {rel-head-relx}, the {rel2-relx} branch will be designated for an "extended support" status and receive only fixes for major problems, such as security-related fixes. Releases are made <>. While many people stay more up-to-date with the FreeBSD sources (see the questions on <> and <>) than that, doing so is more of a commitment, as the sources are a moving target. More information on FreeBSD releases can be found on the https://www.FreeBSD.org/releng/#release-build[Release Engineering page] and in man:release[7]. [[current]] === What is _FreeBSD-CURRENT_? link:{handbook}#current[FreeBSD-CURRENT] is the development version of the operating system, which will in due course become the new FreeBSD-STABLE branch. As such, it is really only of interest to developers working on the system and die-hard hobbyists. See the link:{handbook}#current[relevant section] in the link:{handbook}[Handbook] for details on running _-CURRENT_. Users not familiar with FreeBSD should not use FreeBSD-CURRENT. This branch sometimes evolves quite quickly and due to mistake can be un-buildable at times. People that use FreeBSD-CURRENT are expected to be able to analyze, debug, and report problems. [[stable]] === What is the FreeBSD-STABLE concept? _FreeBSD-STABLE_ is the development branch from which major releases are made. Changes go into this branch at a slower pace and with the general assumption that they have first been tested in FreeBSD-CURRENT. However, at any given time, the sources for FreeBSD-STABLE may or may not be suitable for general use, as it may uncover bugs and corner cases that were not yet found in FreeBSD-CURRENT. Users who do not have the resources to perform testing should instead run the most recent release of FreeBSD. _FreeBSD-CURRENT_, on the other hand, has been one unbroken line since 2.0 was released. For more detailed information on branches see "link:{releng}#rel-branch[FreeBSD Release Engineering: Creating the Release Branch]", the status of the branches and the upcoming release schedule can be found on the https://www.FreeBSD.org/releng[Release Engineering Information] page. Version https://download.FreeBSD.org/ftp/releases/amd64/amd64/{rel121-current}-RELEASE/[{rel121-current}] is the latest release from the {rel-stable} branch; it was released in {rel121-current-date}. Version https://download.FreeBSD.org/ftp/releases/amd64/amd64/{rel113-current}-RELEASE/[{rel113-current}] is the latest release from the {rel2-stable} branch; it was released in {rel113-current-date}. [[release-freq]] === When are FreeBSD releases made? The {re} releases a new major version of FreeBSD about every 18 months and a new minor version about every 8 months, on average. Release dates are announced well in advance, so that the people working on the system know when their projects need to be finished and tested. A testing period precedes each release, to ensure that the addition of new features does not compromise the stability of the release. Many users regard this caution as one of the best things about FreeBSD, even though waiting for all the latest goodies to reach _-STABLE_ can be a little frustrating. More information on the release engineering process (including a schedule of upcoming releases) can be found on the https://www.FreeBSD.org/releng/[release engineering] pages on the FreeBSD Web site. For people who need or want a little more excitement, binary snapshots are made weekly as discussed above. [[snapshot-freq]] === When are FreeBSD snapshots made? FreeBSD link:https://www.FreeBSD.org/snapshots/[snapshot] releases are made based on the current state of the _-CURRENT_ and _-STABLE_ branches. The goals behind each snapshot release are: * To test the latest version of the installation software. * To give people who would like to run _-CURRENT_ or _-STABLE_ but who do not have the time or bandwidth to follow it on a day-to-day basis an easy way of bootstrapping it onto their systems. * To preserve a fixed reference point for the code in question, just in case we break something really badly later. (Although Subversion normally prevents anything horrible like this happening.) * To ensure that all new features and fixes in need of testing have the greatest possible number of potential testers. No claims are made that any _-CURRENT_ snapshot can be considered "production quality" for any purpose. If a stable and fully tested system is needed, stick to full releases. Snapshot releases are directly available from link:https://www.FreeBSD.org/snapshots/[snapshot]. Official snapshots are generated on a regular basis for all actively developed branches. [[responsible]] === Who is responsible for FreeBSD? The key decisions concerning the FreeBSD project, such as the overall direction of the project and who is allowed to add code to the source tree, are made by a link:https://www.FreeBSD.org/administration#t-core[core team] of 9 people. There is a much larger team of more than 350 link:{contributors}#staff-committers[committers] who are authorized to make changes directly to the FreeBSD source tree. However, most non-trivial changes are discussed in advance in the <>, and there are no restrictions on who may take part in the discussion. [[where-get]] === Where can I get FreeBSD? Every significant release of FreeBSD is available via anonymous FTP from the https://download.FreeBSD.org/ftp/releases/[FreeBSD FTP site]: * The latest {rel-stable} release, {rel121-current}-RELEASE can be found in the https://download.FreeBSD.org/ftp/releases/amd64/amd64/{rel121-current}-RELEASE/[{rel121-current}-RELEASE directory]. * link:https://www.FreeBSD.org/snapshots/[Snapshot] releases are made monthly for the <> and <> branch, these being of service purely to bleeding-edge testers and developers. * The latest {rel2-stable} release, {rel113-current}-RELEASE can be found in the https://download.FreeBSD.org/ftp/releases/amd64/amd64/{rel113-current}-RELEASE/[{rel113-current}-RELEASE directory]. Information about obtaining FreeBSD on CD, DVD, and other media can be found in link:{handbook}#mirrors/[the Handbook]. [[access-pr]] === How do I access the Problem Report database? The Problem Report database of all user change requests may be queried by using our web-based PR https://bugs.FreeBSD.org/search/[query] interface. The link:https://www.FreeBSD.org/support/bugreports[web-based problem report submission interface] can be used to submit problem reports through a web browser. Before submitting a problem report, read link:{problem-reports}[Writing FreeBSD Problem Reports], an article on how to write good problem reports. [[support]] == Documentation and Support [[books]] === What good books are there about FreeBSD? The project produces a wide range of documentation, available online from this link: https://www.FreeBSD.org/docs/[https://www.FreeBSD.org/docs/]. [[doc-formats]] === Is the documentation available in other formats, such as plain text (ASCII), or PDF? Yes. The documentation is available in a number of different formats and compression schemes on the FreeBSD FTP site, in the https://download.freebsd.org/ftp/doc/[/ftp/doc/] directory. The documentation is categorized in a number of different ways. These include: * The document's name, such as `faq`, or `handbook`. * The document's language and encoding. These are based on the locale names found under [.filename]#/usr/share/locale# on a FreeBSD system. The current languages and encodings are as follows: + [.informaltable] [cols="1,1", frame="none", options="header"] |=== | Name | Meaning |`en_US.ISO8859-1` |English (United States) |`bn_BD.ISO10646-1` |Bengali or Bangla (Bangladesh) |`da_DK.ISO8859-1` |Danish (Denmark) |`de_DE.ISO8859-1` |German (Germany) |`el_GR.ISO8859-7` |Greek (Greece) |`es_ES.ISO8859-1` |Spanish (Spain) |`fr_FR.ISO8859-1` |French (France) |`hu_HU.ISO8859-2` |Hungarian (Hungary) |`it_IT.ISO8859-15` |Italian (Italy) |`ja_JP.eucJP` |Japanese (Japan, EUC encoding) |`ko_KR.UTF-8` |Korean (Korea, UTF-8 encoding) |`mn_MN.UTF-8` |Mongolian (Mongolia, UTF-8 encoding) |`nl_NL.ISO8859-1` |Dutch (Netherlands) |`pl_PL.ISO8859-2` |Polish (Poland) |`pt_BR.ISO8859-1` |Portuguese (Brazil) |`ru_RU.KOI8-R` |Russian (Russia, KOI8-R encoding) |`tr_TR.ISO8859-9` |Turkish (Turkey) |`zh_CN.UTF-8` |Simplified Chinese (China, UTF-8 encoding) |`zh_TW.UTF-8` |Traditional Chinese (Taiwan, UTF-8 encoding) |=== + [NOTE] ==== Some documents may not be available in all languages. ==== * The document's format. We produce the documentation in a number of different output formats. Each format has its own advantages and disadvantages. Some formats are better suited for online reading, while others are meant to be aesthetically pleasing when printed on paper. Having the documentation available in any of these formats ensures that our readers will be able to read the parts they are interested in, either on their monitor, or on paper after printing the documents. The currently available formats are: + [.informaltable] [cols="1,1", frame="none", options="header"] |=== | Format | Meaning |`html-split` |A collection of small, linked, HTML files. |`html` |One large HTML file containing the entire document |`pdf` |Adobe's Portable Document Format |`txt` |Plain text |=== * The compression and packaging scheme. .. Where the format is `html-split`, the files are bundled up using man:tar[1]. The resulting [.filename]#.tar# is then compressed using the compression schemes detailed in the next point. .. All the other formats generate one file. For example, [.filename]#article.pdf#, [.filename]#book.html#, and so on. + These files are then compressed using either the `zip` or `bz2` compression schemes. man:tar[1] can be used to uncompress these files. + So the PDF version of the Handbook, compressed using `bzip2` will be stored in a file called [.filename]#book.pdf.bz2# in the [.filename]#handbook/# directory. After choosing the format and compression mechanism, download the compressed files, uncompress them, and then copy the appropriate documents into place. For example, the split HTML version of the FAQ, compressed using man:bzip2[1], can be found in [.filename]#doc/en_US.ISO8859-1/books/faq/book.html-split.tar.bz2# To download and uncompress that file, type: [source,shell] .... # fetch https://download.freebsd.org/ftp/doc/en_US.ISO8859-1/books/faq/book.html-split.tar.bz2 # tar xvf book.html-split.tar.bz2 .... If the file is compressed, tar will automatically detect the appropriate format and decompress it correctly, resulting in a collection of [.filename]#.html# files. The main one is called [.filename]#index.html#, which will contain the table of contents, introductory material, and links to the other parts of the document. [[mailing]] === Where do I find info on the FreeBSD mailing lists? What FreeBSD news groups are available? Refer to the link:{handbook}#eresources-mail[Handbook entry on mailing-lists] and the link:{handbook}#eresources-news/[Handbook entry on newsgroups]. [[irc]] === Are there FreeBSD IRC (Internet Relay Chat) channels? Yes, most major IRC networks host a FreeBSD chat channel: * Channel `#FreeBSDhelp` on http://www.efnet.org/index.php[EFNet] is a channel dedicated to helping FreeBSD users. * Channel `#FreeBSD` on http://freenode.net/[Freenode] is a general help channel with many users at any time. The conversations have been known to run off-topic for a while, but priority is given to users with FreeBSD questions. Other users can help with the basics, referring to the Handbook whenever possible and providing links for learning more about a particular topic. This is primarily an English speaking channel, though it does have users from all over the world. Non-native English speakers should try to ask the question in English first and then relocate to `##freebsd-lang` as appropriate. * Channel `#FreeBSD` on http://www.dal.net/[DALNET] is available at `irc.dal.net` in the US and `irc.eu.dal.net` in Europe. * Channel `#FreeBSD` on http://www.undernet.org/[UNDERNET] is available at `us.undernet.org` in the US and `eu.undernet.org` in Europe. Since it is a help channel, be prepared to read the documents you are referred to. * Channel `#FreeBSD` on http://www.rusnet.org.ru/[RUSNET] is a Russian language channel dedicated to helping FreeBSD users. This is also a good place for non-technical discussions. * Channel `#bsdchat` on http://freenode.net/[Freenode] is a Traditional Chinese (UTF-8 encoding) language channel dedicated to helping FreeBSD users. This is also a good place for non-technical discussions. The FreeBSD wiki has a https://wiki.freebsd.org/IRC/Channels[good list] of IRC channels. Each of these channels are distinct and are not connected to each other. Since their chat styles differ, try each to find one suited to your chat style. [[forums]] === Are there any web based forums to discuss FreeBSD? The official FreeBSD forums are located at https://forums.FreeBSD.org/[https://forums.FreeBSD.org/]. [[training]] === Where can I get commercial FreeBSD training and support? http://www.ixsystems.com[iXsystems, Inc.], parent company of the http://www.freebsdmall.com/[FreeBSD Mall], provides commercial FreeBSD and TrueOS software http://www.ixsystems.com/support[support], in addition to FreeBSD development and tuning solutions. BSD Certification Group, Inc. provides system administration certifications for DragonFly BSD, FreeBSD, NetBSD, and OpenBSD. Refer to http://www.BSDCertification.org[their site] for more information. Any other organizations providing training and support should contact the Project to be listed here. [[install]] == Installation [[which-architecture]] === Which platform should I download? I have a 64 bit capable Intel(R) CPU, but I only see amd64. amd64 is the term FreeBSD uses for 64-bit compatible x86 architectures (also known as "x86-64" or "x64"). Most modern computers should use amd64. Older hardware should use i386. When installing on a non-x86-compatible architecture, select the platform which best matches the hardware. [[floppy-download]] === Which file do I download to get FreeBSD? On the https://www.freebsd.org/where/[Getting FreeBSD] page, select `[iso]` next to the architecture that matches the hardware. Any of the following can be used: [.informaltable] [cols="1,1", frame="none", options="header"] |=== | file | description |[.filename]#disc1.iso# |Contains enough to install FreeBSD and a minimal set of packages. |[.filename]#dvd1.iso# |Similar to [.filename]#disc1.iso# but with additional packages. |[.filename]#memstick.img# |A bootable image sufficient for writing to a USB stick. |[.filename]#bootonly.iso# |A minimal image that requires network access during installation to completely install FreeBSD. |=== Full instructions on this procedure and a little bit more about installation issues in general can be found in the link:{handbook}#bsdinstall/[Handbook entry on installing FreeBSD]. [[floppy-image-too-large]] === What do I do if the install image does not boot? This can be caused by not downloading the image in _binary_ mode when using FTP. Some FTP clients default their transfer mode to _ascii_ and attempt to change any end-of-line characters received to match the conventions used by the client's system. This will almost invariably corrupt the boot image. Check the SHA-256 checksum of the downloaded boot image: if it is not _exactly_ that on the server, then the download process is suspect. When using a command line FTP client, type _binary_ at the FTP command prompt after getting connected to the server and before starting the download of the image. [[install-instructions-location]] === Where are the instructions for installing FreeBSD? Installation instructions can be found at link:{handbook}#bsdinstall/[Handbook entry on installing FreeBSD]. [[custom-boot-floppy]] === How can I make my own custom release or install disk? Customized FreeBSD installation media can be created by building a custom release. Follow the instructions in the link:{releng}[Release Engineering] article. [[windows-coexist]] === Can Windows(R) co-exist with FreeBSD? (x86-specific) If Windows(R) is installed first, then yes. FreeBSD's boot manager will then manage to boot Windows(R) and FreeBSD. If Windows(R) is installed afterwards, it will overwrite the boot manager. If that happens, see the next section. [[bootmanager-restore]] === Another operating system destroyed my Boot Manager. How do I get it back? (x86-specific) This depends upon the boot manager. The FreeBSD boot selection menu can be reinstalled using man:boot0cfg[8]. For example, to restore the boot menu onto the disk _ada0_: [source,shell] .... # boot0cfg -B ada0 .... The non-interactive MBR bootloader can be installed using man:gpart[8]: [source,shell] .... # gpart bootcode -b /boot/mbr ada0 .... For more complex situations, including GPT disks, see man:gpart[8]. [[need-complete-sources]] === Do I need to install the source? In general, no. There is nothing in the base system which requires the presence of the source to operate. Some ports, like package:sysutils/lsof[], will not build unless the source is installed. In particular, if the port builds a kernel module or directly operates on kernel structures, the source must be installed. [[need-kernel]] === Do I need to build a kernel? Usually not. The supplied `GENERIC` kernel contains the drivers an ordinary computer will need. man:freebsd-update[8], the FreeBSD binary upgrade tool, cannot upgrade custom kernels, another reason to stick with the `GENERIC` kernel when possible. For computers with very limited RAM, such as embedded systems, it may be worthwhile to build a smaller custom kernel containing just the required drivers. [[password-encryption]] === Should I use DES, Blowfish, or MD5 passwords and how do I specify which form my users receive? FreeBSD uses _SHA512_ by default. DES passwords are still available for backwards compatibility with operating systems that still use the less secure password format. FreeBSD also supports the Blowfish and MD5 password formats. Which password format to use for new passwords is controlled by the `passwd_format` login capability in [.filename]#/etc/login.conf#, which takes values of `des`, `blf` (if these are available) or `md5`. See the man:login.conf[5] manual page for more information about login capabilities. [[ffs-limits]] === What are the limits for FFS file systems? For FFS file systems, the largest file system is practically limited by the amount of memory required to man:fsck[8] the file system. man:fsck[8] requires one bit per fragment, which with the default fragment size of 4 KB equates to 32 MB of memory per TB of disk. This does mean that on architectures which limit userland processes to 2 GB (e.g., i386(TM)), the maximum man:fsck[8]'able filesystem is ~60 TB. If there was not a man:fsck[8] memory limit the maximum filesystem size would be 2 ^ 64 (blocks) * 32 KB => 16 Exa * 32 KB => 512 ZettaBytes. The maximum size of a single FFS file is approximately 2 PB with the default block size of 32 KB. Each 32 KB block can point to 4096 blocks. With triple indirect blocks, the calculation is 32 KB * 12 + 32 KB * 4096 + 32 KB * 4096^2 + 32 KB * 4096^3. Increasing the block size to 64 KB will increase the max file size by a factor of 16. [[archsw-readin-failed-error]] === Why do I get an error message, readin failed after compiling and booting a new kernel? The world and kernel are out of sync. This is not supported. Be sure to use `make buildworld` and `make buildkernel` to update the kernel. Boot the system by specifying the kernel directly at the second stage, pressing any key when the `|` shows up before loader is started. [[general-configuration-tool]] === Is there a tool to perform post-installation configuration tasks? Yes. bsdconfig provides a nice interface to configure FreeBSD post-installation. [[hardware]] == Hardware Compatibility [[compatibility-general]] === General [[which-hardware-to-get]] ==== I want to get a piece of hardware for my FreeBSD system. Which model/brand/type is best? This is discussed continually on the FreeBSD mailing lists but is to be expected since hardware changes so quickly. Read through the Hardware Notes for FreeBSD link:{u-rel121-hardware}[{rel121-current}] or link:{u-rel113-hardware}[{rel113-current}] and search the mailing list https://www.FreeBSD.org/search/#mailinglists[archives] before asking about the latest and greatest hardware. Chances are a discussion about that type of hardware took place just last week. Before purchasing a laptop, check the archives for {freebsd-questions}, or possibly a specific mailing list for a particular hardware type. [[memory-upper-limitation]] ==== What are the limits for memory? FreeBSD as an operating system generally supports as much physical memory (RAM) as the platform it is running on does. Keep in mind that different platforms have different limits for memory; for example i386(TM) without PAE supports at most 4 GB of memory (and usually less than that because of PCI address space) and i386(TM) with PAE supports at most 64 GB memory. As of FreeBSD 10, AMD64 platforms support up to 4 TB of physical memory. [[memory-i386-over-4gb]] ==== Why does FreeBSD report less than 4 GB memory when installed on an i386(TM) machine? The total address space on i386(TM) machines is 32-bit, meaning that at most 4 GB of memory is addressable (can be accessed). Furthermore, some addresses in this range are reserved by hardware for different purposes, for example for using and controlling PCI devices, for accessing video memory, and so on. Therefore, the total amount of memory usable by the operating system for its kernel and applications is limited to significantly less than 4 GB. Usually, 3.2 GB to 3.7 GB is the maximum usable physical memory in this configuration. To access more than 3.2 GB to 3.7 GB of installed memory (meaning up to 4 GB but also more than 4 GB), a special tweak called PAE must be used. PAE stands for Physical Address Extension and is a way for 32-bit x86 CPUs to address more than 4 GB of memory. It remaps the memory that would otherwise be overlaid by address reservations for hardware devices above the 4 GB range and uses it as additional physical memory (see man:pae[4]). Using PAE has some drawbacks; this mode of memory access is a little bit slower than the normal (without PAE) mode and loadable modules (see man:kld[4]) are not supported. This means all drivers must be compiled into the kernel. The most common way to enable PAE is to build a new kernel with the special ready-provided kernel configuration file called [.filename]#PAE#, which is already configured to build a safe kernel. Note that some entries in this kernel configuration file are too conservative and some drivers marked as unready to be used with PAE are actually usable. A rule of thumb is that if the driver is usable on 64-bit architectures (like AMD64), it is also usable with PAE. When creating a custom kernel configuration file, PAE can be enabled by adding the following line: [.programlisting] .... options PAE .... PAE is not much used nowadays because most new x86 hardware also supports running in 64-bit mode, known as AMD64 or Intel(R) 64. It has a much larger address space and does not need such tweaks. FreeBSD supports AMD64 and it is recommended that this version of FreeBSD be used instead of the i386(TM) version if 4 GB or more memory is required. [[compatibility-processors]] === Architectures and Processors [[architectures]] ==== Does FreeBSD support architectures other than the x86? Yes. FreeBSD divides support into multiple tiers. Tier 1 architectures, such as i386 or amd64; are fully supported. Tiers 2 and 3 are supported on a best-effort basis. A full explanation of the tier system is available in the link:{committers-guide}#archs/[Committer's Guide.] A complete list of supported architectures can be found on the https://www.FreeBSD.org/platforms/[platforms page.] [[smp-support]] ==== Does FreeBSD support Symmetric Multiprocessing (SMP)? FreeBSD supports symmetric multi-processor (SMP) on all non-embedded platforms (e.g, i386, amd64, etc.). SMP is also supported in arm and MIPS kernels, although some CPUs may not support this. FreeBSD's SMP implementation uses fine-grained locking, and performance scales nearly linearly with number of CPUs. man:smp[4] has more details. [[microcode]] ==== What is microcode? How do I install Intel(R) CPU microcode updates? Microcode is a method of programmatically implementing hardware level instructions. This allows for CPU bugs to be fixed without replacing the on board chip. Install package:sysutils/devcpu-data[], then add: [.programlisting] .... microcode_update_enable="YES" .... to [.filename]#/etc/rc.conf# [[compatibility-peripherals]] === Peripherals [[supported-peripherals]] ==== What kind of peripherals does FreeBSD support? See the complete list in the Hardware Notes for FreeBSD link:{u-rel121-hardware}[{rel121-current}] or link:{u-rel113-hardware}[{rel113-current}]. [[compatibility-kbd-mice]] === Keyboards and Mice [[moused]] ==== Is it possible to use a mouse outside the X Window system? The default console driver, man:vt[4], provides the ability to use a mouse pointer in text consoles to cut & paste text. Run the mouse daemon, man:moused[8], and turn on the mouse pointer in the virtual console: [source,shell] .... # moused -p /dev/xxxx -t yyyy # vidcontrol -m on .... Where _xxxx_ is the mouse device name and _yyyy_ is a protocol type for the mouse. The mouse daemon can automatically determine the protocol type of most mice, except old serial mice. Specify the `auto` protocol to invoke automatic detection. If automatic detection does not work, see the man:moused[8] manual page for a list of supported protocol types. For a PS/2 mouse, add `moused_enable="YES"` to [.filename]#/etc/rc.conf# to start the mouse daemon at boot time. Additionally, to use the mouse daemon on all virtual terminals instead of just the console, add `allscreens_flags="-m on"` to [.filename]#/etc/rc.conf#. When the mouse daemon is running, access to the mouse must be coordinated between the mouse daemon and other programs such as X Windows. Refer to the FAQ<> for more details on this issue. [[text-mode-cut-paste]] ==== How do I cut and paste text with a mouse in the text console? It is not possible to remove data using the mouse. However, it is possible to copy and paste. Once the mouse daemon is running as described in the <>, hold down button 1 (left button) and move the mouse to select a region of text. Then, press button 2 (middle button) to paste it at the text cursor. Pressing button 3 (right button) will "extend" the selected region of text. If the mouse does not have a middle button, it is possible to emulate one or remap buttons using mouse daemon options. See the man:moused[8] manual page for details. [[mouse-wheel-buttons]] ==== My mouse has a fancy wheel and buttons. Can I use them in FreeBSD? The answer is, unfortunately, "It depends". These mice with additional features require specialized driver in most cases. Unless the mouse device driver or the user program has specific support for the mouse, it will act just like a standard two, or three button mouse. For the possible usage of wheels in the X Window environment, refer to <>. [[keyboard-delete-key]] ==== How do I use my delete key in sh and csh? For the Bourne Shell, add the following lines to [.filename]#~/.shrc#. See man:sh[1] and man:editrc[5]. [.programlisting] .... bind ^[[3~ ed-delete-next-char # for xterm .... For the C Shell, add the following lines to [.filename]#~/.cshrc#. See man:csh[1]. [.programlisting] .... bindkey ^[[3~ delete-char # for xterm .... [[compatibility-other]] === Other Hardware [[es1370-silent-pcm]] ==== Workarounds for no sound from my man:pcm[4] sound card? Some sound cards set their output volume to 0 at every boot. Run the following command every time the machine boots: [source,shell] .... # mixer pcm 100 vol 100 cd 100 .... [[power-management-support]] ==== Does FreeBSD support power management on my laptop? FreeBSD supports the ACPI features found in modern hardware. Further information can be found in man:acpi[4]. [[troubleshoot]] == Troubleshooting [[pae]] === Why is FreeBSD finding the wrong amount of memory on i386(TM) hardware? The most likely reason is the difference between physical memory addresses and virtual addresses. The convention for most PC hardware is to use the memory area between 3.5 GB and 4 GB for a special purpose (usually for PCI). This address space is used to access PCI hardware. As a result real, physical memory cannot be accessed by that address space. What happens to the memory that should appear in that location is hardware dependent. Unfortunately, some hardware does nothing and the ability to use that last 500 MB of RAM is entirely lost. Luckily, most hardware remaps the memory to a higher location so that it can still be used. However, this can cause some confusion when watching the boot messages. On a 32-bit version of FreeBSD, the memory appears lost, since it will be remapped above 4 GB, which a 32-bit kernel is unable to access. In this case, the solution is to build a PAE enabled kernel. See the entry on memory limits for more information. On a 64-bit version of FreeBSD, or when running a PAE-enabled kernel, FreeBSD will correctly detect and remap the memory so it is usable. During boot, however, it may seem as if FreeBSD is detecting more memory than the system really has, due to the described remapping. This is normal and the available memory will be corrected as the boot process completes. [[signal11]] === Why do my programs occasionally die with Signal 11 errors? Signal 11 errors are caused when a process has attempted to access memory which the operating system has not granted it access to. If something like this is happening at seemingly random intervals, start investigating the cause. These problems can usually be attributed to either: . If the problem is occurring only in a specific custom application, it is probably a bug in the code. . If it is a problem with part of the base FreeBSD system, it may also be buggy code, but more often than not these problems are found and fixed long before us general FAQ readers get to use these bits of code (that is what -CURRENT is for). It is probably not a FreeBSD bug if the problem occurs compiling a program, but the activity that the compiler is carrying out changes each time. For example, if `make buildworld` fails while trying to compile [.filename]#ls.c# into [.filename]#ls.o# and, when run again, it fails in the same place, this is a broken build. Try updating source and try again. If the compile fails elsewhere, it is almost certainly due to hardware. In the first case, use a debugger such as man:gdb[1] to find the point in the program which is attempting to access a bogus address and fix it. In the second case, verify which piece of hardware is at fault. Common causes of this include: . The hard disks might be overheating: Check that the fans are still working, as the disk and other hardware might be overheating. . The processor running is overheating: This might be because the processor has been overclocked, or the fan on the processor might have died. In either case, ensure that the hardware is running at what it is specified to run at, at least while trying to solve this problem. If it is not, clock it back to the default settings.) + Regarding overclocking, it is far cheaper to have a slow system than a fried system that needs replacing! Also the community is not sympathetic to problems on overclocked systems. . Dodgy memory: if multiple memory SIMMS/DIMMS are installed, pull them all out and try running the machine with each SIMM or DIMM individually to narrow the problem down to either the problematic DIMM/SIMM or perhaps even a combination. . Over-optimistic motherboard settings: the BIOS settings, and some motherboard jumpers, provide options to set various timings. The defaults are often sufficient, but sometimes setting the wait states on RAM too low, or setting the "RAM Speed: Turbo" option will cause strange behavior. A possible idea is to set to BIOS defaults, after noting the current settings first. . Unclean or insufficient power to the motherboard. Remove any unused I/O boards, hard disks, or CD-ROMs, or disconnect the power cable from them, to see if the power supply can manage a smaller load. Or try another power supply, preferably one with a little more power. For instance, if the current power supply is rated at 250 Watts, try one rated at 300 Watts. Read the section on <> for a further explanation and a discussion on how memory testing software or hardware can still pass faulty memory. There is an extensive FAQ on this at http://www.bitwizard.nl/sig11/[the SIG11 problem FAQ]. Finally, if none of this has helped, it is possibly a bug in FreeBSD. Follow <> to send a problem report. [[trap-12-panic]] === My system crashes with either Fatal trap 12: page fault in kernel mode, or panic:, and spits out a bunch of information. What should I do? The FreeBSD developers are interested in these errors, but need more information than just the error message. Copy the full crash message. Then consult the FAQ section on <>, build a debugging kernel, and get a backtrace. This might sound difficult, but does not require any programming skills. Just follow the instructions. [[proc-table-full]] === What is the meaning of the error maxproc limit exceeded by uid %i, please see tuning(7) and login.conf(5)? The FreeBSD kernel will only allow a certain number of processes to exist at one time. The number is based on the `kern.maxusers` man:sysctl[8] variable. `kern.maxusers` also affects various other in-kernel limits, such as network buffers. If the machine is heavily loaded, increase `kern.maxusers`. This will increase these other system limits in addition to the maximum number of processes. To adjust the `kern.maxusers` value, see the link:{handbook}#kern-maxfiles[File/Process Limits] section of the Handbook. While that section refers to open files, the same limits apply to processes. If the machine is lightly loaded but running a very large number of processes, adjust the `kern.maxproc` tunable by defining it in [.filename]#/boot/loader.conf#. The tunable will not get adjusted until the system is rebooted. For more information about tuning tunables, see man:loader.conf[5]. If these processes are being run by a single user, adjust `kern.maxprocperuid` to be one less than the new `kern.maxproc` value. It must be at least one less because one system program, man:init[8], must always be running. [[remote-fullscreen]] === Why do full screen applications on remote machines misbehave? The remote machine may be setting the terminal type to something other than `xterm` which is required by the FreeBSD console. Alternatively the kernel may have the wrong values for the width and height of the terminal. Check the value of the `TERM` environment variable is `xterm`. If the remote machine does not support that try `vt100`. Run `stty -a` to check what the kernel thinks the terminal dimensions are. If they are incorrect, they can be changed by running `stty rows _RR_ cols _CC_`. Alternatively, if the client machine has package:x11/xterm[] installed, then running `resize` will query the terminal for the correct dimensions and set them. [[connection-delay]] === Why does it take so long to connect to my computer via ssh or telnet? The symptom: there is a long delay between the time the TCP connection is established and the time when the client software asks for a password (or, in man:telnet[1]'s case, when a login prompt appears). The problem: more likely than not, the delay is caused by the server software trying to resolve the client's IP address into a hostname. Many servers, including the Telnet and SSH servers that come with FreeBSD, do this to store the hostname in a log file for future reference by the administrator. The remedy: if the problem occurs whenever connecting the client computer to any server, the problem is with the client. If the problem only occurs when someone connects to the server computer, the problem is with the server. If the problem is with the client, the only remedy is to fix the DNS so the server can resolve it. If this is on a local network, consider it a server problem and keep reading. If this is on the Internet, contact your ISP. If the problem is with the server on a local network, configure the server to resolve address-to-hostname queries for the local address range. See man:hosts[5] and man:named[8] for more information. If this is on the Internet, the problem may be that the local server's resolver is not functioning correctly. To check, try to look up another host such as `www.yahoo.com`. If it does not work, that is the problem. Following a fresh install of FreeBSD, it is also possible that domain and name server information is missing from [.filename]#/etc/resolv.conf#. This will often cause a delay in SSH, as the option `UseDNS` is set to `yes` by default in [.filename]#/etc/ssh/sshd_config#. If this is causing the problem, either fill in the missing information in [.filename]#/etc/resolv.conf# or set `UseDNS` to `no` in [.filename]#sshd_config# as a temporary workaround. [[file-table-full]] === Why does file: table is full show up repeatedly in man:dmesg[8]? This error message indicates that the number of available file descriptors have been exhausted on the system. Refer to the link:{handbook}#kern-maxfiles[kern.maxfiles] section of the link:{handbook}#configtuning-kernel-limits/[Tuning Kernel Limits] section of the Handbook for a discussion and solution. [[computer-clock-skew]] === Why does the clock on my computer keep incorrect time? The computer has two or more clocks, and FreeBSD has chosen to use the wrong one. Run man:dmesg[8], and check for lines that contain `Timecounter`. The one with the highest quality value that FreeBSD chose. [source,shell] .... # dmesg | grep Timecounter Timecounter "i8254" frequency 1193182 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 Timecounter "TSC" frequency 2998570050 Hz quality 800 Timecounters tick every 1.000 msec .... Confirm this by checking the `kern.timecounter.hardware` man:sysctl[3]. [source,shell] .... # sysctl kern.timecounter.hardware kern.timecounter.hardware: ACPI-fast .... It may be a broken ACPI timer. The simplest solution is to disable the ACPI timer in [.filename]#/boot/loader.conf#: [.programlisting] .... debug.acpi.disabled="timer" .... Or the BIOS may modify the TSC clock-perhaps to change the speed of the processor when running from batteries, or going into a power saving mode, but FreeBSD is unaware of these adjustments, and appears to gain or lose time. In this example, the `i8254` clock is also available, and can be selected by writing its name to the `kern.timecounter.hardware` man:sysctl[3]. [source,shell] .... # sysctl kern.timecounter.hardware=i8254 kern.timecounter.hardware: TSC -> i8254 .... The computer should now start keeping more accurate time. To have this change automatically run at boot time, add the following line to [.filename]#/etc/sysctl.conf#: [.programlisting] .... kern.timecounter.hardware=i8254 .... [[indefinite-wait-buffer]] === What does the error swap_pager: indefinite wait buffer: mean? This means that a process is trying to page memory from disk, and the page attempt has hung trying to access the disk for more than 20 seconds. It might be caused by bad blocks on the disk drive, disk wiring, cables, or any other disk I/O-related hardware. If the drive itself is bad, disk errors will appear in [.filename]#/var/log/messages# and in the output of `dmesg`. Otherwise, check the cables and connections. [[lock-order-reversal]] === What is a lock order reversal? The FreeBSD kernel uses a number of resource locks to arbitrate contention for certain resources. When multiple kernel threads try to obtain multiple resource locks, there's always the potential for a deadlock, where two threads have each obtained one of the locks and blocks forever waiting for the other thread to release one of the other locks. This sort of locking problem can be avoided if all threads obtain the locks in the same order. A run-time lock diagnostic system called man:witness[4], enabled in FreeBSD-CURRENT and disabled by default for stable branches and releases, detects the potential for deadlocks due to locking errors, including errors caused by obtaining multiple resource locks with a different order from different parts of the kernel. The man:witness[4] framework tries to detect this problem as it happens, and reports it by printing a message to the system console about a `lock order reversal` (often referred to also as LOR). It is possible to get false positives, as man:witness[4] is conservative. A true positive report _does not_ mean that a system is dead-locked; instead it should be understood as a warning that a deadlock could have happened here. [NOTE] ==== Problematic LORs tend to get fixed quickly, so check the {freebsd-current} before posting to it. ==== [[called-with-non-sleepable-locks-held]] === What does Called ... with the following non-sleepable locks held mean? This means that a function that may sleep was called while a mutex (or other unsleepable) lock was held. The reason this is an error is because mutexes are not intended to be held for long periods of time; they are supposed to only be held to maintain short periods of synchronization. This programming contract allows device drivers to use mutexes to synchronize with the rest of the kernel during interrupts. Interrupts (under FreeBSD) may not sleep. Hence it is imperative that no subsystem in the kernel block for an extended period while holding a mutex. To catch such errors, assertions may be added to the kernel that interact with the man:witness[4] subsystem to emit a warning or fatal error (depending on the system configuration) when a potentially blocking call is made while holding a mutex. In summary, such warnings are non-fatal, however with unfortunate timing they could cause undesirable effects ranging from a minor blip in the system's responsiveness to a complete system lockup. For additional information about locking in FreeBSD see man:locking[9]. [[touch-not-found]] === Why does buildworld/installworld die with the message touch: not found? This error does not mean that the man:touch[1] utility is missing. The error is instead probably due to the dates of the files being set sometime in the future. If the CMOS clock is set to local time, run `adjkerntz -i` to adjust the kernel clock when booting into single-user mode. [[applications]] == User Applications [[user-apps]] === Where are all the user applications? Refer to link:https://www.FreeBSD.org/ports/[the ports page] for info on software packages ported to FreeBSD. Most ports should work on all supported versions of FreeBSD. Those that do not are specifically marked as such. Each time a FreeBSD release is made, a snapshot of the ports tree at the time of release is also included in the [.filename]#ports/# directory. FreeBSD supports compressed binary packages to easily install and uninstall ports. Use man:pkg[7] to control the installation of packages. [[how-do-download-ports-tree]] === How do I download the Ports tree? Should I be using Git? See crossref:handbook[ports-using-installation-methods,Installing the Ports Collection]. [[ports-4x]] === Why can I not build this port on my {rel2-relx} -, or {rel-relx} -STABLE machine? If the installed FreeBSD version lags significantly behind _-CURRENT_ or _-STABLE_, update the Ports Collection using the instructions in link:{handbook}#ports-using/[Using the Ports Collection]. If the system is up-to-date, someone might have committed a change to the port which works for _-CURRENT_ but which broke the port for _-STABLE_. https://bugs.FreeBSD.org/submit/[Submit] a bug report, since the Ports Collection is supposed to work for both the _-CURRENT_ and _-STABLE_ branches. [[make-index]] === I just tried to build INDEX using make index, and it failed. Why? First, make sure that the Ports Collection is up-to-date. Errors that affect building [.filename]#INDEX# from an up-to-date copy of the Ports Collection are high-visibility and are thus almost always fixed immediately. There are rare cases where [.filename]#INDEX# will not build due to odd cases involving `OPTIONS_SET` being set in [.filename]#make.conf#. If you suspect that this is the case, try to make [.filename]#INDEX# with those variables turned off before reporting it to {freebsd-ports}. [[ports-update]] === I updated the sources, now how do I update my installed ports? FreeBSD does not include a port upgrading tool, but it does have some tools to make the upgrade process somewhat easier. Additional tools are available to simplify port handling and are described the link:{handbook}#ports-using/[Upgrading Ports] section in the FreeBSD Handbook. [[ports-major-upgrade]] === Do I need to recompile every port each time I perform a major version update? Yes! While a recent system will run with software compiled under an older release, things will randomly crash and fail to work once other ports are installed or updated. When the system is upgraded, various shared libraries, loadable modules, and other parts of the system will be replaced with newer versions. Applications linked against the older versions may fail to start or, in other cases, fail to function properly. For more information, see link:{handbook}#freebsdupdate-upgrade[the section on upgrades] in the FreeBSD Handbook. [[ports-minor-upgrade]] === Do I need to recompile every port each time I perform a minor version update? In general, no. FreeBSD developers do their utmost to guarantee binary compatibility across all releases with the same major version number. Any exceptions will be documented in the Release Notes, and advice given there should be followed. [[minimal-sh]] === Why is /bin/sh so minimal? Why does FreeBSD not use bash or another shell? Many people need to write shell scripts which will be portable across many systems. That is why POSIX(R) specifies the shell and utility commands in great detail. Most scripts are written in Bourne shell (man:sh[1]), and because several important programming interfaces (man:make[1], man:system[3], man:popen[3], and analogues in higher-level scripting languages like Perl and Tcl) are specified to use the Bourne shell to interpret commands. As the Bourne shell is so often and widely used, it is important for it to be quick to start, be deterministic in its behavior, and have a small memory footprint. The existing implementation is our best effort at meeting as many of these requirements simultaneously as we can. To keep `/bin/sh` small, we have not provided many of the convenience features that other shells have. That is why other more featureful shells like `bash`, `scsh`, man:tcsh[1], and `zsh` are available. Compare the memory utilization of these shells by looking at the "VSZ" and "RSS" columns in a `ps -u` listing. [[kernelconfig]] == Kernel Configuration [[make-kernel]] === I would like to customize my kernel. Is it difficult? Not at all! Check out the link:{handbook}#kernelconfig/[kernel config section of the Handbook]. [NOTE] ==== The new [.filename]#kernel# will be installed to the [.filename]#/boot/kernel# directory along with its modules, while the old kernel and its modules will be moved to the [.filename]#/boot/kernel.old# directory. If a mistake is made in the configuration, simply boot the previous version of the kernel. ==== [[why-kernel-big]] === Why is my kernel so big? `GENERIC` kernels shipped with FreeBSD are compiled in _debug mode_. Kernels built in debug mode contain debug data in separate files that are used for debugging. FreeBSD releases prior to 11.0 store these debug files in the same directory as the kernel itself, [.filename]#/boot/kernel/#. In FreeBSD 11.0 and later the debug files are stored in [.filename]#/usr/lib/debug/boot/kernel/#. Note that there will be little or no performance loss from running a debug kernel, and it is useful to keep one around in case of a system panic. When running low on disk space, there are different options to reduce the size of [.filename]#/boot/kernel/# and [.filename]#/usr/lib/debug/#. To not install the symbol files, make sure the following line exists in [.filename]#/etc/src.conf#: [.programlisting] .... WITHOUT_KERNEL_SYMBOLS=yes .... For more information see man:src.conf[5]. If you want to avoid building debug files altogether, make sure that both of the following are true: * This line does not exist in the kernel configuration file: + [.programlisting] .... makeoptions DEBUG=-g .... * Do not run man:config[8] with `-g`. Either of the above settings will cause the kernel to be built in debug mode. To build and install only the specified modules, list them in [.filename]#/etc/make.conf#: [.programlisting] .... MODULES_OVERRIDE= accf_http ipfw .... Replace _accf_httpd ipfw_ with a list of needed modules. Only the listed modules will be built. This reduces the size of the kernel directory and decreases the amount of time needed to build the kernel. For more information, read [.filename]#/usr/share/examples/etc/make.conf#. Unneeded devices can be removed from the kernel to further reduce the size. See <> for more information. To put any of these options into effect, follow the instructions to link:{handbook}#kernelconfig-building/[build and install] the new kernel. For reference, the FreeBSD 11 amd64 kernel ([.filename]#/boot/kernel/kernel#) is approximately 25 MB. [[generic-kernel-build-failure]] === Why does every kernel I try to build fail to compile, even GENERIC? There are a number of possible causes for this problem: * The source tree is different from the one used to build the currently running system. When attempting an upgrade, read [.filename]#/usr/src/UPDATING#, paying particular attention to the "COMMON ITEMS" section at the end. * The `make buildkernel` did not complete successfully. The `make buildkernel` target relies on files generated by the `make buildworld` target to complete its job correctly. * Even when building <>, it is possible that the source tree was fetched at a time when it was either being modified or it was broken. Only releases are guaranteed to be buildable, although <> builds fine the majority of the time. Try re-fetching the source tree and see if the problem goes away. Try using a different mirror in case the previous one is having problems. [[scheduler-in-use]] === Which scheduler is in use on a running system? The name of the scheduler currently being used is directly available as the value of the `kern.sched.name` sysctl: [source,shell] .... % sysctl kern.sched.name kern.sched.name: ULE .... [[scheduler-kern-quantum]] === What is kern.sched.quantum? `kern.sched.quantum` is the maximum number of ticks a process can run without being preempted in the 4BSD scheduler. [[disks]] == Disks, File Systems, and Boot Loaders [[adding-disks]] === How can I add my new hard disk to my FreeBSD system? See the link:{handbook}#disks-adding/[Adding Disks] section in the FreeBSD Handbook. [[new-huge-disk]] === How do I move my system over to my huge new disk? The best way is to reinstall the operating system on the new disk, then move the user data over. This is highly recommended when tracking _-STABLE_ for more than one release or when updating a release instead of installing a new one. Install booteasy on both disks with man:boot0cfg[8] and dual boot until you are happy with the new configuration. Skip the next paragraph to find out how to move the data after doing this. Alternatively, partition and label the new disk with either man:sade[8] or man:gpart[8]. If the disks are MBR-formatted, booteasy can be installed on both disks with man:boot0cfg[8] so that the computer can dual boot to the old or new system after the copying is done. Once the new disk set up, the data cannot just be copied. Instead, use tools that understand device files and system flags, such as man:dump[8]. Although it is recommended to move the data while in single-user mode, it is not required. When the disks are formatted with UFS, never use anything but man:dump[8] and man:restore[8] to move the root file system. These commands should also be used when moving a single partition to another empty partition. The sequence of steps to use `dump` to move the data from one UFS partitions to a new partition is: [.procedure] ==== . `newfs` the new partition. . `mount` it on a temporary mount point. . `cd` to that directory. . `dump` the old partition, piping output to the new one. ==== For example, to move [.filename]#/dev/ada1s1a# with [.filename]#/mnt# as the temporary mount point, type: [source,shell] .... # newfs /dev/ada1s1a # mount /dev/ada1s1a /mnt # cd /mnt # dump 0af - / | restore rf - .... Rearranging partitions with `dump` takes a bit more work. To merge a partition like [.filename]#/var# into its parent, create the new partition large enough for both, move the parent partition as described above, then move the child partition into the empty directory that the first move created: [source,shell] .... # newfs /dev/ada1s1a # mount /dev/ada1s1a /mnt # cd /mnt # dump 0af - / | restore rf - # cd var # dump 0af - /var | restore rf - .... To split a directory from its parent, say putting [.filename]#/var# on its own partition when it was not before, create both partitions, then mount the child partition on the appropriate directory in the temporary mount point, then move the old single partition: [source,shell] .... # newfs /dev/ada1s1a # newfs /dev/ada1s1d # mount /dev/ada1s1a /mnt # mkdir /mnt/var # mount /dev/ada1s1d /mnt/var # cd /mnt # dump 0af - / | restore rf - .... The man:cpio[1] and man:pax[1] utilities are also available for moving user data. These are known to lose file flag information, so use them with caution. [[safe-softupdates]] === Which partitions can safely use Soft Updates? I have heard that Soft Updates on / can cause problems. What about Journaled Soft Updates? Short answer: Soft Updates can usually be safely used on all partitions. Long answer: Soft Updates has two characteristics that may be undesirable on certain partitions. First, a Soft Updates partition has a small chance of losing data during a system crash. The partition will not be corrupted as the data will simply be lost. Second, Soft Updates can cause temporary space shortages. When using Soft Updates, the kernel can take up to thirty seconds to write changes to the physical disk. When a large file is deleted the file still resides on disk until the kernel actually performs the deletion. This can cause a very simple race condition. Suppose one large file is deleted and another large file is immediately created. The first large file is not yet actually removed from the physical disk, so the disk might not have enough room for the second large file. This will produce an error that the partition does not have enough space, even though a large chunk of space has just been released. A few seconds later, the file creation works as expected. If a system should crash after the kernel accepts a chunk of data for writing to disk, but before that data is actually written out, data could be lost. This risk is extremely small, but generally manageable. These issues affect all partitions using Soft Updates. So, what does this mean for the root partition? Vital information on the root partition changes very rarely. If the system crashed during the thirty-second window after such a change is made, it is possible that data could be lost. This risk is negligible for most applications, but be aware that it exists. If the system cannot tolerate this much risk, do not use Soft Updates on the root file system! [.filename]#/# is traditionally one of the smallest partitions. If [.filename]#/tmp# is on [.filename]#/#, there may be intermittent space problems. Symlinking [.filename]#/tmp# to [.filename]#/var/tmp# will solve this problem. Finally, man:dump[8] does not work in live mode (-L) on a filesystem, with Journaled Soft Updates (SU+J). [[mount-foreign-fs]] === Can I mount other foreign file systems under FreeBSD? FreeBSD supports a variety of other file systems. UFS:: UFS CD-ROMs can be mounted directly on FreeBSD. Mounting disk partitions from Digital UNIX and other systems that support UFS may be more complex, depending on the details of the disk partitioning for the operating system in question. ext2/ext3:: FreeBSD supports `ext2fs`, `ext3fs`, and `ext4fs` partitions. See man:ext2fs[5] for more information. NTFS:: FUSE based NTFS support is available as a port (package:sysutils/fusefs-ntfs[]). For more information, see man:ntfs-3g[8]. FAT:: FreeBSD includes a read-write FAT driver. For more information, see man:mount_msdosfs[8]. ZFS:: FreeBSD includes a port of Sun(TM)'s ZFS driver. The current recommendation is to use it only on amd64 platforms with sufficient memory. For more information, see man:zfs[8]. FreeBSD includes the Network File System NFS and the FreeBSD Ports Collection provides several FUSE applications to support many other file systems. [[mount-dos]] === How do I mount a secondary DOS partition? The secondary DOS partitions are found after _all_ the primary partitions. For example, if `E` is the second DOS partition on the second SCSI drive, there will be a device file for "slice 5" in [.filename]#/dev#. To mount it: [source,shell] .... # mount -t msdosfs /dev/da1s5 /dos/e .... [[crypto-file-system]] === Is there a cryptographic file system for FreeBSD? Yes, man:gbde[8] and man:geli[8]. See the link:{handbook}#disks-encrypting/[Encrypting Disk Partitions] section of the FreeBSD Handbook. [[grub-loader]] === How do I boot FreeBSD and Linux(R) using GRUB? To boot FreeBSD using GRUB, add the following to either [.filename]#/boot/grub/menu.lst# or [.filename]#/boot/grub/grub.conf#, depending upon which is used by the Linux(R) distribution. [.programlisting] .... title FreeBSD 9.1 root (hd0,a) kernel /boot/loader .... Where _hd0,a_ points to the root partition on the first disk. To specify the slice number, use something like this _(hd0,2,a)_. By default, if the slice number is omitted, GRUB searches the first slice which has the `a` partition. [[booteasy-loader]] === How do I boot FreeBSD and Linux(R) using BootEasy? Install LILO at the start of the Linux(R) boot partition instead of in the Master Boot Record. Then boot LILO from BootEasy. This is recommended when running Windows(R) and Linux(R) as it makes it simpler to get Linux(R) booting again if Windows(R) is reinstalled. [[changing-bootprompt]] === How do I change the boot prompt from ??? to something more meaningful? This cannot be accomplished with the standard boot manager without rewriting it. There are a number of other boot managers in the [.filename]#sysutils# category of the Ports Collection. [[removable-drives]] === How do I use a new removable drive? If the drive already has a file system on it, use a command like this: [source,shell] .... # mount -t msdosfs /dev/da0s1 /mnt .... If the drive will only be used with FreeBSD systems, partition it with UFS or ZFS. This will provide long filename support, improvement in performance, and stability. If the drive will be used by other operating systems, a more portable choice, such as msdosfs, is better. [source,shell] .... # dd if=/dev/zero of=/dev/da0 count=2 # gpart create -s GPT /dev/da0 # gpart add -t freebsd-ufs /dev/da0 .... Finally, create a new file system: [source,shell] .... # newfs /dev/da0p1 .... and mount it: [source,shell] .... # mount /dev/da0s1 /mnt .... It is a good idea to add a line to [.filename]#/etc/fstab# (see man:fstab[5]) so you can just type `mount /mnt` in the future: [.programlisting] .... /dev/da0p1 /mnt ufs rw,noauto 0 0 .... [[mount-cd-superblock]] === Why do I get Incorrect super block when mounting a CD? The type of device to mount must be specified. This is described in the Handbook section on link:{handbook}#mounting-cd[Using Data CDs]. [[cdrom-not-configured]] === Why do I get Device not configured when mounting a CD? This generally means that there is no CD in the drive, or the drive is not visible on the bus. Refer to the link:{handbook}#mounting-cd[Using Data CDs] section of the Handbook for a detailed discussion of this issue. [[cdrom-unicode-filenames]] === Why do all non-English characters in filenames show up as ? on my CDs when mounted in FreeBSD? The CD probably uses the "Joliet" extension for storing information about files and directories. This is discussed in the Handbook section on link:{handbook}#mounting-cd[Using Data CD-ROMs]. [[burncd-isofs]] === A CD burned under FreeBSD cannot be read under any other operating system. Why? This means a raw file was burned to the CD, rather than creating an ISO 9660 file system. Take a look at the Handbook section on link:{handbook}#mounting-cd[Using Data CDs]. [[copy-cd]] === How can I create an image of a data CD? This is discussed in the Handbook section on link:{handbook}#mkisofs[Writing Data to an ISO File System]. For more on working with CD-ROMs, see the link:{handbook}#creating-cds/[Creating CDs Section] in the Storage chapter in the Handbook. [[mount-audio-CD]] === Why can I not mount an audio CD? Trying to mount an audio CD will produce an error like `cd9660: /dev/cd0: Invalid argument`. This is because `mount` only works on file systems. Audio CDs do not have file systems; they just have data. Instead, use a program that reads audio CDs, such as the package:audio/xmcd[] package or port. [[multi-session-CD]] === How do I mount a multi-session CD? By default, man:mount[8] will attempt to mount the last data track (session) of a CD. To load an earlier session, use the `-s` command line argument. Refer to man:mount_cd9660[8] for specific examples. [[user-floppymount]] === How do I let ordinary users mount CD-ROMs, DVDs, USB drives, and other removable media? As `root` set the sysctl variable `vfs.usermount` to `1`. [source,shell] .... # sysctl vfs.usermount=1 .... To make this persist across reboots, add the line `vfs.usermount=1` to [.filename]#/etc/sysctl.conf# so that it is reset at system boot time. Users can only mount devices they have read permissions to. To allow users to mount a device permissions must be set in [.filename]#/etc/devfs.conf#. For example, to allow users to mount the first USB drive add: [.programlisting] .... # Allow all users to mount a USB drive. own /dev/da0 root:operator perm /dev/da0 0666 .... All users can now mount devices they could read onto a directory that they own: [source,shell] .... % mkdir ~/my-mount-point % mount -t msdosfs /dev/da0 ~/my-mount-point .... Unmounting the device is simple: [source,shell] .... % umount ~/my-mount-point .... Enabling `vfs.usermount`, however, has negative security implications. A better way to access MS-DOS(R) formatted media is to use the package:emulators/mtools[] package in the Ports Collection. [NOTE] ==== The device name used in the previous examples must be changed according to the configuration. ==== [[du-vs-df]] === The du and df commands show different amounts of disk space available. What is going on? This is due to how these commands actually work. `du` goes through the directory tree, measures how large each file is, and presents the totals. `df` just asks the file system how much space it has left. They seem to be the same thing, but a file without a directory entry will affect `df` but not `du`. When a program is using a file, and the file is deleted, the file is not really removed from the file system until the program stops using it. The file is immediately deleted from the directory listing, however. As an example, consider a file large enough to affect the output of `du` and `df`. A file being viewed with `more` can be deleted without causing an error. The entry is removed from the directory so no other program or user can access it. However, `du` shows that it is gone as it has walked the directory tree and the file is not listed. `df` shows that it is still there, as the file system knows that `more` is still using that space. Once the `more` session ends, `du` and `df` will agree. This situation is common on web servers. Many people set up a FreeBSD web server and forget to rotate the log files. The access log fills up [.filename]#/var#. The new administrator deletes the file, but the system still complains that the partition is full. Stopping and restarting the web server program would free the file, allowing the system to release the disk space. To prevent this from happening, set up man:newsyslog[8]. Note that Soft Updates can delay the freeing of disk space and it can take up to 30 seconds for the change to be visible. [[add-swap-space]] === How can I add more swap space? This section link:{handbook}#adding-swap-space/[of the Handbook] describes how to do this. [[manufacturer-disk-size]] === Why does FreeBSD see my disk as smaller than the manufacturer says it is? Disk manufacturers calculate gigabytes as a billion bytes each, whereas FreeBSD calculates them as 1,073,741,824 bytes each. This explains why, for example, FreeBSD's boot messages will report a disk that supposedly has 80 GB as holding 76,319 MB. Also note that FreeBSD will (by default) <> 8% of the disk space. [[disk-more-than-full]] === How is it possible for a partition to be more than 100% full? A portion of each UFS partition (8%, by default) is reserved for use by the operating system and the `root` user. man:df[1] does not count that space when calculating the `Capacity` column, so it can exceed 100%. Notice that the `Blocks` column is always greater than the sum of the `Used` and `Avail` columns, usually by a factor of 8%. For more details, look up `-m` in man:tunefs[8]. [[all-about-zfs]] == ZFS [[how-much-ram-for-zfs]] === What is the minimum amount of RAM one should have to run ZFS? A minimum of 4GB of RAM is required for comfortable usage, but individual workloads can vary widely. [[what-is-zil]] === What is the ZIL and when does it get used? The ZIL (ZFS intent log) is a write log used to implement posix write commitment semantics across crashes. Normally writes are bundled up into transaction groups and written to disk when filled ("Transaction Group Commit"). However syscalls like man:fsync[2] require a commitment that the data is written to stable storage before returning. The ZIL is needed for writes that have been acknowledged as written but which are not yet on disk as part of a transaction. The transaction groups are timestamped. In the event of a crash the last valid timestamp is found and missing data is merged in from the ZIL. [[need-ssd-for-zil]] === Do I need a SSD for ZIL? By default, ZFS stores the ZIL in the pool with all the data. If an application has a heavy write load, storing the ZIL in a separate device that has very fast synchronous, sequential write performance can improve overall system performance. For other workloads, a SSD is unlikely to make much of an improvement. [[what-is-l2arc]] === What is the L2ARC? The L2ARC is a read cache stored on a fast device such as an SSD. This cache is not persistent across reboots. Note that RAM is used as the first layer of cache and the L2ARC is only needed if there is insufficient RAM. L2ARC needs space in the ARC to index it. So, perversely, a working set that fits perfectly in the ARC will not fit perfectly any more if a L2ARC is used because part of the ARC is holding the L2ARC index, pushing part of the working set into the L2ARC which is slower than RAM. [[should-enable-dedup]] === Is enabling deduplication advisable? Generally speaking, no. Deduplication takes up a significant amount of RAM and may slow down read and write disk access times. Unless one is storing data that is very heavily duplicated, such as virtual machine images or user backups, it is possible that deduplication will do more harm than good. Another consideration is the inability to revert deduplication status. If data is written when deduplication is enabled, disabling dedup will not cause those blocks which were deduplicated to be replicated until they are next modified. Deduplication can also lead to some unexpected situations. In particular, deleting files may become much slower. [[zpool-fully-full]] === I cannot delete or create files on my ZFS pool. How can I fix this? This could happen because the pool is 100% full. ZFS requires space on the disk to write transaction metadata. To restore the pool to a usable state, truncate the file to delete: [source,shell] .... % truncate -s 0 unimportant-file .... File truncation works because a new transaction is not started, new spare blocks are created instead. [NOTE] ==== On systems with additional ZFS dataset tuning, such as deduplication, the space may not be immediately available ==== [[zfs-ssd-trim]] === Does ZFS support TRIM for Solid State Drives? ZFS TRIM support was added to FreeBSD 10-CURRENT with revision link:https://svnweb.freebsd.org/changeset/base/240868[r240868]. ZFS TRIM support was added to all FreeBSD-STABLE branches in link:https://svnweb.freebsd.org/changeset/base/252162[r252162] and link:https://svnweb.freebsd.org/changeset/base/251419[r251419], respectively. ZFS TRIM is enabled by default, and can be turned off by adding this line to [.filename]#/etc/sysctl.conf#: [.programlisting] .... vfs.zfs.trim.enabled=0 .... [NOTE] ==== ZFS TRIM support was added to GELI as of link:https://svnweb.freebsd.org/changeset/base/286444[r286444]. Please see man:geli[8] and the `-T` switch. ==== [[admin]] == System Administration [[startup-config-files]] === Where are the system start-up configuration files? The primary configuration file is [.filename]#/etc/defaults/rc.conf# which is described in man:rc.conf[5]. System startup scripts such as [.filename]#/etc/rc# and [.filename]#/etc/rc.d#, which are described in man:rc[8], include this file. _Do not edit this file!_ Instead, to edit an entry in [.filename]#/etc/defaults/rc.conf#, copy the line into [.filename]#/etc/rc.conf# and change it there. For example, to start man:sshd[8], the included OpenSSH daemon: [source,shell] .... # echo 'sshd_enable="YES"' >> /etc/rc.conf .... Alternatively, use man:sysrc[8] to modify [.filename]#/etc/rc.conf#: [source,shell] .... # sysrc sshd_enable="YES" .... To start up local services, place shell scripts in the [.filename]#/usr/local/etc/rc.d# directory. These shell scripts should be set executable, the default file mode is `555`. [[adding-users]] === How do I add a user easily? Use the man:adduser[8] command, or the man:pw[8] command for more complicated situations. To remove the user, use the man:rmuser[8] command or, if necessary, man:pw[8]. [[root-not-found-cron-errors]] === Why do I keep getting messages like root: not found after editing /etc/crontab? This is normally caused by editing the system crontab. This is not the correct way to do things as the system crontab has a different format to the per-user crontabs. The system crontab has an extra field, specifying which user to run the command as. man:cron[8] assumes this user is the first word of the command to execute. Since no such command exists, this error message is displayed. To delete the extra, incorrect crontab: [source,shell] .... # crontab -r .... [[su-wheel-group]] === Why do I get the error, you are not in the correct group to su root when I try to su to root? This is a security feature. In order to `su` to `root`, or any other account with superuser privileges, the user account must be a member of the `wheel` group. If this feature were not there, anybody with an account on a system who also found out ``root``'s password would be able to gain superuser level access to the system. To allow someone to `su` to `root`, put them in the `wheel` group using `pw`: [source,shell] .... # pw groupmod wheel -m lisa .... The above example will add user `lisa` to the group `wheel`. [[rcconf-readonly]] === I made a mistake in rc.conf, or another startup file, and now I cannot edit it because the file system is read-only. What should I do? Restart the system using `boot -s` at the loader prompt to enter single-user mode. When prompted for a shell pathname, press kbd:[Enter] and run `mount -urw /` to re-mount the root file system in read/write mode. You may also need to run `mount -a -t ufs` to mount the file system where your favorite editor is defined. If that editor is on a network file system, either configure the network manually before mounting the network file systems, or use an editor which resides on a local file system, such as man:ed[1]. In order to use a full screen editor such as man:vi[1] or man:emacs[1], run `export TERM=xterm` so that these editors can load the correct data from the man:termcap[5] database. After performing these steps, edit [.filename]#/etc/rc.conf# to fix the syntax error. The error message displayed immediately after the kernel boot messages should indicate the number of the line in the file which is at fault. [[printer-setup]] === Why am I having trouble setting up my printer? See the link:{handbook}#printing/[Handbook entry on printing] for troubleshooting tips. [[keyboard-mappings]] === How can I correct the keyboard mappings for my system? Refer to the Handbook section on link:{handbook}#using-localization/[using localization], specifically the section on link:{handbook}#setting-console[console setup]. [[user-quotas]] === Why can I not get user quotas to work properly? . It is possible that the kernel is not configured to use quotas. In this case, add the following line to the kernel configuration file and recompile the kernel: + [.programlisting] .... options QUOTA .... + Refer to the link:{handbook}#quotas/[Handbook entry on quotas] for full details. . Do not turn on quotas on [.filename]#/#. . Put the quota file on the file system that the quotas are to be enforced on: + [.informaltable] [cols="1,1", frame="none", options="header"] |=== | File System | Quota file |[.filename]#/usr# |[.filename]#/usr/admin/quotas# |[.filename]#/home# |[.filename]#/home/admin/quotas# |... |... |=== [[sysv-ipc]] === Does FreeBSD support System V IPC primitives? Yes, FreeBSD supports System V-style IPC, including shared memory, messages and semaphores, in the [.filename]#GENERIC# kernel. With a custom kernel, support may be loaded with the [.filename]#sysvshm.ko#, [.filename]#sysvsem.ko# and [.filename]#sysvmsg.ko# kernel modules, or enabled in the custom kernel by adding the following lines to the kernel configuration file: [.programlisting] .... options SYSVSHM # enable shared memory options SYSVSEM # enable for semaphores options SYSVMSG # enable for messaging .... Recompile and install the kernel. [[sendmail-alternative]] === What other mail-server software can I use instead of Sendmail? The http://www.sendmail.org/[Sendmail] server is the default mail-server software for FreeBSD, but it can be replaced with another MTA installed from the Ports Collection. Available ports include package:mail/exim[], package:mail/postfix[], and package:mail/qmail[]. Search the mailing lists for discussions regarding the advantages and disadvantages of the available MTAs. [[forgot-root-pw]] === I have forgotten the root password! What do I do? Do not panic! Restart the system, type `boot -s` at the `Boot:` prompt to enter single-user mode. At the question about the shell to use, hit kbd:[Enter] which will display a # prompt. Enter `mount -urw /` to remount the root file system read/write, then run `mount -a` to remount all the file systems. Run `passwd root` to change the `root` password then run man:exit[1] to continue booting. [NOTE] ==== If you are still prompted to give the `root` password when entering the single-user mode, it means that the console has been marked as `insecure` in [.filename]#/etc/ttys#. In this case, it will be required to boot from a FreeBSD installation disk, choose the [.guimenuitem]#Live CD# or [.guimenuitem]#Shell# at the beginning of the install process and issue the commands mentioned above. Mount the specific partition in this case and then chroot to it. For example, replace `mount -urw /` with `mount /dev/ada0p1 /mnt; chroot /mnt` for a system on _ada0p1_. ==== [NOTE] ==== If the root partition cannot be mounted from single-user mode, it is possible that the partitions are encrypted and it is impossible to mount them without the access keys. For more information see the section about encrypted disks in the FreeBSD link:{handbook}#disks-encrypting/[Handbook]. ==== [[CAD-reboot]] === How do I keep kbd:[Control] + kbd:[Alt] + kbd:[Delete] from rebooting the system? When using man:vt[4], the default console driver, this can be done by setting the following man:sysctl[8]: [source,shell] .... # sysctl kern.vt.kbd_reboot=0 .... [[dos-to-unix-txt]] === How do I reformat DOS text files to UNIX(R) ones? Use this man:perl[1] command: [source,shell] .... % perl -i.bak -npe 's/\r\n/\n/g' file(s) .... where _file(s)_ is one or more files to process. The modification is done in-place, with the original file stored with a [.filename]#.bak# extension. Alternatively, use man:tr[1]: [source,shell] .... % tr -d '\r' < dos-text-file > unix-file .... _dos-text-file_ is the file containing DOS text while _unix-file_ will contain the converted output. This can be quite a bit faster than using `perl`. Yet another way to reformat DOS text files is to use the package:converters/dosunix[] port from the Ports Collection. Consult its documentation about the details. [[reread-rc]] === How do I re-read [.filename]#/etc/rc.conf# and re-start [.filename]#/etc/rc# without a reboot? Go into single-user mode and then back to multi-user mode: [source,shell] .... # shutdown now # return # exit .... [[release-candidate]] === I tried to update my system to the latest _-STABLE_, but got _-BETAx_, _-RC_ or __-PRERELEASE__! What is going on? Short answer: it is just a name. _RC_ stands for "Release Candidate". It signifies that a release is imminent. In FreeBSD, _-PRERELEASE_ is typically synonymous with the code freeze before a release. (For some releases, the _-BETA_ label was used in the same way as _-PRERELEASE_.) Long answer: FreeBSD derives its releases from one of two places. Major, dot-zero, releases, such as 9.0-RELEASE are branched from the head of the development stream, commonly referred to as <>. Minor releases, such as 6.3-RELEASE or 5.2-RELEASE, have been snapshots of the active <> branch. Starting with 4.3-RELEASE, each release also now has its own branch which can be tracked by people requiring an extremely conservative rate of development (typically only security advisories). When a release is about to be made, the branch from which it will be derived from has to undergo a certain process. Part of this process is a code freeze. When a code freeze is initiated, the name of the branch is changed to reflect that it is about to become a release. For example, if the branch used to be called 6.2-STABLE, its name will be changed to 6.3-PRERELEASE to signify the code freeze and signify that extra pre-release testing should be happening. Bug fixes can still be committed to be part of the release. When the source code is in shape for the release the name will be changed to 6.3-RC to signify that a release is about to be made from it. Once in the RC stage, only the most critical bugs found can be fixed. Once the release (6.3-RELEASE in this example) and release branch have been made, the branch will be renamed to 6.3-STABLE. For more information on version numbers and the various Subversion branches, refer to the link:{releng}[Release Engineering] article. [[kernel-chflag-failure]] === I tried to install a new kernel, and the man:chflags[1] failed. How do I get around this? Short answer: the security level is greater than 0. Reboot directly to single-user mode to install the kernel. Long answer: FreeBSD disallows changing system flags at security levels greater than 0. To check the current security level: [source,shell] .... # sysctl kern.securelevel .... The security level cannot be lowered in multi-user mode, so boot to single-user mode to install the kernel, or change the security level in [.filename]#/etc/rc.conf# then reboot. See the man:init[8] manual page for details on `securelevel`, and see [.filename]#/etc/defaults/rc.conf# and the man:rc.conf[5] manual page for more information on [.filename]#rc.conf#. [[kernel-securelevel-time]] === I cannot change the time on my system by more than one second! How do I get around this? Short answer: the system is at a security level greater than 1. Reboot directly to single-user mode to change the date. Long answer: FreeBSD disallows changing the time by more that one second at security levels greater than 1. To check the security level: [source,shell] .... # sysctl kern.securelevel .... The security level cannot be lowered in multi-user mode. Either boot to single-user mode to change the date or change the security level in [.filename]#/etc/rc.conf# and reboot. See the man:init[8] manual page for details on `securelevel`, and see [.filename]#/etc/defaults/rc.conf# and the man:rc.conf[5] manual page for more information on [.filename]#rc.conf#. [[statd-mem-leak]] === Why is rpc.statd using 256 MB of memory? No, there is no memory leak, and it is not using 256 MB of memory. For convenience, `rpc.statd` maps an obscene amount of memory into its address space. There is nothing terribly wrong with this from a technical standpoint; it just throws off things like man:top[1] and man:ps[1]. man:rpc.statd[8] maps its status file (resident on [.filename]#/var#) into its address space; to save worrying about remapping the status file later when it needs to grow, it maps the status file with a generous size. This is very evident from the source code, where one can see that the length argument to man:mmap[2] is `0x10000000`, or one sixteenth of the address space on an IA32, or exactly 256 MB. [[unsetting-schg]] === Why can I not unset the schg file flag? The system is running at securelevel greater than 0. Lower the securelevel and try again. For more information, see <> and the man:init[8] manual page. [[vnlru]] === What is vnlru? `vnlru` flushes and frees vnodes when the system hits the `kern.maxvnodes` limit. This kernel thread sits mostly idle, and only activates when there is a huge amount of RAM and users are accessing tens of thousands of tiny files. [[top-memory-states]] === What do the various memory states displayed by top mean? * `Active`: pages recently statistically used. * `Inactive`: pages recently statistically unused. * `Laundry`: pages recently statistically unused but known to be dirty, that is, whose contents needs to be paged out before they can be reused. * `Free`: pages without data content, which can be immediately reused. * `Wired`: pages that are fixed into memory, usually for kernel purposes, but also sometimes for special use in processes. Pages are most often written to disk (sort of a VM sync) when they are in the laundry state, but active or inactive pages can also be synced. This depends upon the CPU tracking of the modified bit being available, and in certain situations there can be an advantage for a block of VM pages to be synced, regardless of the queue they belong to. In most common cases, it is best to think of the laundry queue as a queue of relatively unused pages that might or might not be in the process of being written to disk. The inactive queue contains a mix of clean and dirty pages; clean pages near the head of the queue are reclaimed immediately to alleviate a free page shortage, and dirty pages are moved to the laundry queue for deferred processing. There are some other flags (e.g., busy flag or busy count) that might modify some of the described rules. [[free-memory-amount]] === How much free memory is available? There are a couple of kinds of "free memory". The most common is the amount of memory immediately available without reclaiming memory already in use. That is the size of the free pages queue plus some other reserved pages. This amount is exported by the `vm.stats.vm.v_free_count` man:sysctl[8], shown, for instance, by man:top[1]. Another kind of "free memory" is the total amount of virtual memory available to userland processes, which depends on the sum of swap space and usable memory. Other kinds of "free memory" descriptions are also possible, but it is relatively useless to define these, but rather it is important to make sure that the paging rate is kept low, and to avoid running out of swap space. [[var-empty]] === What is [.filename]#/var/empty#? [.filename]#/var/empty# is a directory that the man:sshd[8] program uses when performing privilege separation. The [.filename]#/var/empty# directory is empty, owned by `root` and has the `schg` flag set. This directory should not be deleted. [[newsyslog-expectations]] === I just changed [.filename]#/etc/newsyslog.conf#. How can I check if it does what I expect? To see what man:newsyslog[8] will do, use the following: [source,shell] .... % newsyslog -nrvv .... [[timezone]] === My time is wrong, how can I change the timezone? Use man:tzsetup[8]. [[X11]] == The X Window System and Virtual Consoles [[whatis-X]] === What is the X Window System? The X Window System (commonly `X11`) is the most widely available windowing system capable of running on UNIX(R) or UNIX(R) like systems, including FreeBSD. http://www.x.org/wiki/[The X.Org Foundation] administers the http://en.wikipedia.org/wiki/X_Window_System_core_protocol[X protocol standards], with the current reference implementation, version 11 release 7.7, so references are often shortened to `X11`. Many implementations are available for different architectures and operating systems. An implementation of the server-side code is properly known as an `X server`. [[running-X]] === I want to run Xorg, how do I go about it? To install Xorg do one of the following: Use the package:x11/xorg[] meta-port, which builds and installs every Xorg component. Use package:x11/xorg-minimal[], which builds and installs only the necessary Xorg components. Install Xorg from FreeBSD packages: [source,shell] .... # pkg install xorg .... After the installation of Xorg, follow the instructions from the link:{handbook}#x-config/[X11 Configuration] section of the FreeBSD Handbook. [[running-X-securelevels]] === I tried to run X, but I get a No devices detected. error when I type startx. What do I do now? The system is probably running at a raised `securelevel`. It is not possible to start X at a raised `securelevel` because X requires write access to man:io[4]. For more information, see at the man:init[8] manual page. There are two solutions to the problem: set the `securelevel` back down to zero or run man:xdm[1] (or an alternative display manager) at boot time before the `securelevel` is raised. See <> for more information about running man:xdm[1] at boot time. [[x-and-moused]] === Why does my mouse not work with X? When using man:vt[4], the default console driver, FreeBSD can be configured to support a mouse pointer on each virtual screen. To avoid conflicting with X, man:vt[4] supports a virtual device called [.filename]#/dev/sysmouse#. All mouse events received from the real mouse device are written to the man:sysmouse[4] device via man:moused[8]. To use the mouse on one or more virtual consoles, _and_ use X, see <> and set up man:moused[8]. Then edit [.filename]#/etc/X11/xorg.conf# and make sure the following lines exist: [.programlisting] .... Section "InputDevice" Option "Protocol" "SysMouse" Option "Device" "/dev/sysmouse" ..... .... Starting with Xorg version 7.4, the `InputDevice` sections in [.filename]#xorg.conf# are ignored in favor of autodetected devices. To restore the old behavior, add the following line to the `ServerLayout` or `ServerFlags` section: [.programlisting] .... Option "AutoAddDevices" "false" .... Some people prefer to use [.filename]#/dev/mouse# under X. To make this work, [.filename]#/dev/mouse# should be linked to [.filename]#/dev/sysmouse# (see man:sysmouse[4]) by adding the following line to [.filename]#/etc/devfs.conf# (see man:devfs.conf[5]): [.programlisting] .... link sysmouse mouse .... This link can be created by restarting man:devfs[5] with the following command (as `root`): [source,shell] .... # service devfs restart .... [[x-and-wheel]] === My mouse has a fancy wheel. Can I use it in X? Yes, if X is configured for a 5 button mouse. To do this, add the lines `Buttons 5` and `ZAxisMapping 4 5` to the "InputDevice" section of [.filename]#/etc/X11/xorg.conf#, as seen in this example: [.programlisting] .... Section "InputDevice" Identifier "Mouse1" Driver "mouse" Option "Protocol" "auto" Option "Device" "/dev/sysmouse" Option "Buttons" "5" Option "ZAxisMapping" "4 5" EndSection .... The mouse can be enabled in Emacs by adding these lines to [.filename]#~/.emacs#: [.programlisting] .... ;; wheel mouse (global-set-key [mouse-4] 'scroll-down) (global-set-key [mouse-5] 'scroll-up) .... [[x-and-synaptic]] === My laptop has a Synaptics touchpad. Can I use it in X? Yes, after configuring a few things to make it work. In order to use the Xorg synaptics driver, first remove `moused_enable` from [.filename]#rc.conf#. To enable synaptics, add the following line to [.filename]#/boot/loader.conf#: [.programlisting] .... hw.psm.synaptics_support="1" .... Add the following to [.filename]#/etc/X11/xorg.conf#: [.programlisting] .... Section "InputDevice" Identifier "Touchpad0" Driver "synaptics" Option "Protocol" "psm" Option "Device" "/dev/psm0" EndSection .... And be sure to add the following into the "ServerLayout" section: [.programlisting] .... InputDevice "Touchpad0" "SendCoreEvents" .... [[no-remote-x11]] === How do I use remote X displays? For security reasons, the default setting is to not allow a machine to remotely open a window. To enable this feature, start X with the optional `-listen_tcp` argument: [source,shell] .... % startx -listen_tcp .... [[virtual-console]] === What is a virtual console and how do I make more? Virtual consoles provide several simultaneous sessions on the same machine without doing anything complicated like setting up a network or running X. When the system starts, it will display a login prompt on the monitor after displaying all the boot messages. Type in your login name and password to start working on the first virtual console. To start another session, perhaps to look at documentation for a program or to read mail while waiting for an FTP transfer to finish, hold down kbd:[Alt] and press kbd:[F2]. This will display the login prompt for the second virtual console. To go back to the original session, press kbd:[Alt+F1]. The default FreeBSD installation has eight virtual consoles enabled. kbd:[Alt+F1], kbd:[Alt+F2], kbd:[Alt+F3], and so on will switch between these virtual consoles. To enable more of virtual consoles, edit [.filename]#/etc/ttys# (see man:ttys[5]) and add entries for [.filename]#ttyv8# to [.filename]#ttyvc#, after the comment on "Virtual terminals": [.programlisting] .... # Edit the existing entry for ttyv8 in /etc/ttys and change # "off" to "on". ttyv8 "/usr/libexec/getty Pc" xterm on secure ttyv9 "/usr/libexec/getty Pc" xterm on secure ttyva "/usr/libexec/getty Pc" xterm on secure ttyvb "/usr/libexec/getty Pc" xterm on secure .... The more virtual terminals, the more resources that are used. This can be problematic on systems with 8 MB RAM or less. Consider changing `secure` to `insecure`. [IMPORTANT] ==== In order to run an X server, at least one virtual terminal must be left to `off` for it to use. This means that only eleven of the Alt-function keys can be used as virtual consoles so that one is left for the X server. ==== For example, to run X and eleven virtual consoles, the setting for virtual terminal 12 should be: [.programlisting] .... ttyvb "/usr/libexec/getty Pc" xterm off secure .... The easiest way to activate the virtual consoles is to reboot. [[vty-from-x]] === How do I access the virtual consoles from X? Use kbd:[Ctrl+Alt+Fn] to switch back to a virtual console. Press kbd:[Ctrl+Alt+F1] to return to the first virtual console. Once at a text console, use kbd:[Alt+Fn] to move between them. To return to the X session, switch to the virtual console running X. If X was started from the command line using `startx`, the X session will attach to the next unused virtual console, not the text console from which it was invoked. For eight active virtual terminals, X will run on the ninth, so use kbd:[Alt+F9]. [[xdm-boot]] === How do I start XDM on boot? There are two schools of thought on how to start man:xdm[1]. One school starts `xdm` from [.filename]#/etc/ttys# (see man:ttys[5]) using the supplied example, while the other sets `xdm_enable=yes` in [.filename]#/etc/rc.conf#. Both are equally valid, and one may work in situations where the other does not. In both cases the result is the same: X will pop up a graphical login prompt. The man:ttys[5] method has the advantage of documenting which vty X will start on and passing the responsibility of restarting the X server on logout to man:init[8]. The man:rc[8] method makes it easy to `kill xdm` if there is a problem starting the X server. When using the man:rc[8] method, `xdm_tty` (default `ttyv8`) can be set in [.filename]#/etc/rc.conf# to choose which vty man:xdm[1] opens on. [[xconsole-failure]] === Why do I get Couldn't open console when I run xconsole? When X is started with `startx`, the permissions on [.filename]#/dev/console# will _not_ get changed, resulting in things like `xterm -C` and `xconsole` not working. This is because of the way console permissions are set by default. On a multi-user system, one does not necessarily want just any user to be able to write on the system console. For users who are logging directly onto a machine with a VTY, the man:fbtab[5] file exists to solve such problems. In a nutshell, make sure an uncommented line of the form is in [.filename]#/etc/fbtab# (see man:fbtab[5]): [.programlisting] .... /dev/ttyv0 0600 /dev/console .... It will ensure that whomever logs in on [.filename]#/dev/ttyv0# will own the console. [[ps2-x]] === Why does my PS/2 mouse misbehave under X? The mouse and the mouse driver may have become out of synchronization. In rare cases, the driver may also erroneously report synchronization errors: [.programlisting] .... psmintr: out of sync (xxxx != yyyy) .... If this happens, disable the synchronization check code by setting the driver flags for the PS/2 mouse driver to `0x100`. This can be easiest achieved by adding `hint.psm.0.flags="0x100"` to [.filename]#/boot/loader.conf# and rebooting. [[mouse-button-reverse]] === How do I reverse the mouse buttons? Type `xmodmap -e "pointer = 3 2 1"`. Add this command to [.filename]#~/.xinitrc# or [.filename]#~/.xsession# to make it happen automatically. [[install-splash]] === How do I install a splash screen and where do I find them? The detailed answer for this question can be found in the link:{handbook}#boot-splash/[Boot Time Splash Screens] section of the FreeBSD Handbook. [[windows-keys]] === Can I use the kbd:[Windows] keys on my keyboard in X? Yes. Use man:xmodmap[1] to define which functions the keys should perform. Assuming all Windows keyboards are standard, the keycodes for these three keys are the following: * 115 - kbd:[Windows] key, between the left-hand kbd:[Ctrl] and kbd:[Alt] keys * 116 - kbd:[Windows] key, to the right of kbd:[AltGr] * 117 - kbd:[Menu], to the left of the right-hand kbd:[Ctrl] To have the left kbd:[Windows] key print a comma, try this. [source,shell] .... # xmodmap -e "keycode 115 = comma" .... To have the kbd:[Windows] key-mappings enabled automatically every time X is started, either put the `xmodmap` commands in [.filename]#~/.xinitrc# or, preferably, create a [.filename]#~/.xmodmaprc# and include the `xmodmap` options, one per line, then add the following line to [.filename]#~/.xinitrc#: [.programlisting] .... xmodmap $HOME/.xmodmaprc .... For example, to map the 3 keys to be kbd:[F13], kbd:[F14], and kbd:[F15], respectively. This would make it easy to map them to useful functions within applications or the window manager. To do this, put the following in [.filename]#~/.xmodmaprc#. [.programlisting] .... keycode 115 = F13 keycode 116 = F14 keycode 117 = F15 .... For the package:x11-wm/fvwm2[] desktop manager, one could map the keys so that kbd:[F13] iconifies or de-iconifies the window the cursor is in, kbd:[F14] brings the window the cursor is in to the front or, if it is already at the front, pushes it to the back, and kbd:[F15] pops up the main Workplace menu even if the cursor is not on the desktop, which is useful when no part of the desktop is visible. The following entries in [.filename]#~/.fvwmrc# implement the aforementioned setup: [.programlisting] .... Key F13 FTIWS A Iconify Key F14 FTIWS A RaiseLower Key F15 A A Menu Workplace Nop .... [[x-3d-acceleration]] === How can I get 3D hardware acceleration for OpenGL(R)? The availability of 3D acceleration depends on the version of Xorg and the type of video chip. For an nVidia chip, use the binary drivers provided for FreeBSD by installing one of the following ports: The latest versions of nVidia cards are supported by the package:x11/nvidia-driver[] port. Older drivers are available as package:x11/nvidia-driver-[] nVidia provides detailed information on which card is supported by which driver on their web site: http://www.nvidia.com/object/IO_32667.html[http://www.nvidia.com/object/IO_32667.html]. For Matrox G200/G400, check the package:x11-drivers/xf86-video-mga[] port. For ATI Rage 128 and Radeon see man:ati[4], man:r128[4] and man:radeon[4]. [[networking]] == Networking [[diskless-booting]] === Where can I get information on diskless booting? "Diskless booting" means that the FreeBSD box is booted over a network, and reads the necessary files from a server instead of its hard disk. For full details, see link:{handbook}#network-diskless/[the Handbook entry on diskless booting]. [[router]] === Can a FreeBSD box be used as a dedicated network router? Yes. Refer to the Handbook entry on link:{handbook}#advanced-networking/[advanced networking], specifically the section on link:{handbook}#network-routing/[routing and gateways]. [[natd]] === Does FreeBSD support NAT or Masquerading? Yes. For instructions on how to use NAT over a PPP connection, see the link:{handbook}#userppp/[Handbook entry on PPP]. To use NAT over some other sort of network connection, look at the link:{handbook}#network-natd[natd] section of the Handbook. [[ethernet-aliases]] === How can I set up Ethernet aliases? If the alias is on the same subnet as an address already configured on the interface, add `netmask 0xffffffff` to this command: [source,shell] .... # ifconfig ed0 alias 192.0.2.2 netmask 0xffffffff .... Otherwise, specify the network address and netmask as usual: [source,shell] .... # ifconfig ed0 alias 172.16.141.5 netmask 0xffffff00 .... More information can be found in the FreeBSD link:{handbook}#configtuning-virtual-hosts/[Handbook]. [[nfs-linux]] === Why can I not NFS-mount from a Linux(R) box? Some versions of the Linux(R) NFS code only accept mount requests from a privileged port; try to issue the following command: [source,shell] .... # mount -o -P linuxbox:/blah /mnt .... [[exports-errors]] === Why does mountd keep telling me it can't change attributes and that I have a bad exports list on my FreeBSD NFS server? The most frequent problem is not understanding the correct format of [.filename]#/etc/exports#. Review man:exports[5] and the link:{handbook}#network-nfs/[NFS] entry in the Handbook, especially the section on link:{handbook}#configuring-nfs[configuring NFS]. [[ip-multicast]] === How do I enable IP multicast support? Install the package:net/mrouted[] package or port and add `mrouted_enable="YES"` to [.filename]#/etc/rc.conf# start this service at boot time. [[fqdn-hosts]] === Why do I have to use the FQDN for hosts on my site? See the answer in the FreeBSD link:{handbook}#mail-trouble/[Handbook]. [[network-permission-denied]] === Why do I get an error, Permission denied, for all networking operations? If the kernel is compiled with the `IPFIREWALL` option, be aware that the default policy is to deny all packets that are not explicitly allowed. If the firewall is unintentionally misconfigured, restore network operability by typing the following as `root`: [source,shell] .... # ipfw add 65534 allow all from any to any .... Consider setting `firewall_type="open"` in [.filename]#/etc/rc.conf#. For further information on configuring this firewall, see the link:{handbook}#firewalls-ipfw/[Handbook chapter]. [[ipfw-fwd]] === Why is my `ipfw` “fwd” rule to redirect a service to another machine not working? Possibly because network address translation (NAT) is needed instead of just forwarding packets. A "fwd" rule only forwards packets, it does not actually change the data inside the packet. Consider this rule: [source,shell] .... 01000 fwd 10.0.0.1 from any to foo 21 .... When a packet with a destination address of _foo_ arrives at the machine with this rule, the packet is forwarded to _10.0.0.1_, but it still has the destination address of _foo_. The destination address of the packet is not changed to _10.0.0.1_. Most machines would probably drop a packet that they receive with a destination address that is not their own. Therefore, using a "fwd" rule does not often work the way the user expects. This behavior is a feature and not a bug. See the <>, the man:natd[8] manual, or one of the several port redirecting utilities in the link:https://www.FreeBSD.org/ports/[Ports Collection] for a correct way to do this. [[service-redirect]] === How can I redirect service requests from one machine to another? FTP and other service requests can be redirected with the package:sysutils/socket[] package or port. Replace the entry for the service in [.filename]#/etc/inetd.conf# to call `socket`, as seen in this example for ftpd: [.programlisting] .... ftp stream tcp nowait nobody /usr/local/bin/socket socket ftp.example.com ftp .... where _ftp.example.com_ and _ftp_ are the host and port to redirect to, respectively. [[bandwidth-mgr-tool]] === Where can I get a bandwidth management tool? There are three bandwidth management tools available for FreeBSD. man:dummynet[4] is integrated into FreeBSD as part of man:ipfw[4]. http://www.sonycsl.co.jp/person/kjc/programs.html[ALTQ] has been integrated into FreeBSD as part of man:pf[4]. Bandwidth Manager from http://www.etinc.com/[Emerging Technologies] is a commercial product. [[bpf-not-configured]] === Why do I get /dev/bpf0: device not configured? The running application requires the Berkeley Packet Filter (man:bpf[4]), but it was removed from a custom kernel. Add this to the kernel config file and build a new kernel: [.programlisting] .... device bpf # Berkeley Packet Filter .... [[mount-smb-share]] === How do I mount a disk from a Windows(R) machine that is on my network, like smbmount in Linux(R)? Use the SMBFS toolset. It includes a set of kernel modifications and a set of userland programs. The programs and information are available as man:mount_smbfs[8] in the base system. [[icmp-response-bw-limit]] === What are these messages about: Limiting icmp/open port/closed port response in my log files? This kernel message indicates that some activity is provoking it to send a large amount of ICMP or TCP reset (RST) responses. ICMP responses are often generated as a result of attempted connections to unused UDP ports. TCP resets are generated as a result of attempted connections to unopened TCP ports. Among others, these are the kinds of activities which may cause these messages: * Brute-force denial of service (DoS) attacks (as opposed to single-packet attacks which exploit a specific vulnerability). * Port scans which attempt to connect to a large number of ports (as opposed to only trying a few well-known ports). The first number in the message indicates how many packets the kernel would have sent if the limit was not in place, and the second indicates the limit. This limit is controlled using `net.inet.icmp.icmplim`. This example sets the limit to `300` packets per second: [source,shell] .... # sysctl net.inet.icmp.icmplim=300 .... To disable these messages without disabling response limiting, use `net.inet.icmp.icmplim_output` to disable the output: [source,shell] .... # sysctl net.inet.icmp.icmplim_output=0 .... Finally, to disable response limiting completely, set `net.inet.icmp.icmplim` to `0`. Disabling response limiting is discouraged for the reasons listed above. [[unknown-hw-addr-format]] === What are these arp: unknown hardware address format error messages? This means that some device on the local Ethernet is using a MAC address in a format that FreeBSD does not recognize. This is probably caused by someone experimenting with an Ethernet card somewhere else on the network. This is most commonly seen on cable modem networks. It is harmless, and should not affect the performance of the FreeBSD system. [[arp-wrong-iface]] === Why do I keep seeing messages like: 192.168.0.10 is on fxp1 but got reply from 00:15:17:67:cf:82 on rl0, and how do I disable it? A packet is coming from outside the network unexpectedly. To disable them, set `net.link.ether.inet.log_arp_wrong_iface` to `0`. [[ipv6-only]] === How do I compile an IPv6 only kernel? Configure your kernel with these settings: [source,shell] .... include GENERIC ident GENERIC-IPV6ONLY makeoptions MKMODULESENV+="WITHOUT_INET_SUPPORT=" nooptions INET nodevice gre .... [[security]] == Security [[sandbox]] === What is a sandbox? "Sandbox" is a security term. It can mean two things: * A process which is placed inside a set of virtual walls that are designed to prevent someone who breaks into the process from being able to break into the wider system. + The process is only able to run inside the walls. Since nothing the process does in regards to executing code is supposed to be able to breach the walls, a detailed audit of its code is not needed in order to be able to say certain things about its security. + The walls might be a user ID, for example. This is the definition used in the man:security[7] and man:named[8] man pages. + Take the `ntalk` service, for example (see man:inetd[8]). This service used to run as user ID `root`. Now it runs as user ID `tty`. The `tty` user is a sandbox designed to make it more difficult for someone who has successfully hacked into the system via `ntalk` from being able to hack beyond that user ID. * A process which is placed inside a simulation of the machine. It means that someone who is able to break into the process may believe that he can break into the wider machine but is, in fact, only breaking into a simulation of that machine and not modifying any real data. + The most common way to accomplish this is to build a simulated environment in a subdirectory and then run the processes in that directory chrooted so that [.filename]#/# for that process is this directory, not the real [.filename]#/# of the system). + Another common use is to mount an underlying file system read-only and then create a file system layer on top of it that gives a process a seemingly writeable view into that file system. The process may believe it is able to write to those files, but only the process sees the effects - other processes in the system do not, necessarily. + An attempt is made to make this sort of sandbox so transparent that the user (or hacker) does not realize that he is sitting in it. UNIX(R) implements two core sandboxes. One is at the process level, and one is at the userid level. Every UNIX(R) process is completely firewalled off from every other UNIX(R) process. One process cannot modify the address space of another. A UNIX(R) process is owned by a particular userid. If the user ID is not the `root` user, it serves to firewall the process off from processes owned by other users. The user ID is also used to firewall off on-disk data. [[securelevel]] === What is securelevel? `securelevel` is a security mechanism implemented in the kernel. When the securelevel is positive, the kernel restricts certain tasks; not even the superuser (`root`) is allowed to do them. The securelevel mechanism limits the ability to: * Unset certain file flags, such as `schg` (the system immutable flag). * Write to kernel memory via [.filename]#/dev/mem# and [.filename]#/dev/kmem#. * Load kernel modules. * Alter firewall rules. To check the status of the securelevel on a running system: [source,shell] .... # sysctl -n kern.securelevel .... The output contains the current value of the securelevel. If it is greater than 0, at least some of the securelevel's protections are enabled. The securelevel of a running system cannot be lowered as this would defeat its purpose. If a task requires that the securelevel be non-positive, change the `kern_securelevel` and `kern_securelevel_enable` variables in [.filename]#/etc/rc.conf# and reboot. For more information on securelevel and the specific things all the levels do, consult man:init[8]. [WARNING] ==== Securelevel is not a silver bullet; it has many known deficiencies. More often than not, it provides a false sense of security. One of its biggest problems is that in order for it to be at all effective, all files used in the boot process up until the securelevel is set must be protected. If an attacker can get the system to execute their code prior to the securelevel being set (which happens quite late in the boot process since some things the system must do at start-up cannot be done at an elevated securelevel), its protections are invalidated. While this task of protecting all files used in the boot process is not technically impossible, if it is achieved, system maintenance will become a nightmare since one would have to take the system down, at least to single-user mode, to modify a configuration file. This point and others are often discussed on the mailing lists, particularly the {freebsd-security}. Search the archives link:https://www.FreeBSD.org/search/[here] for an extensive discussion. A more fine-grained mechanism is preferred. ==== [[toor-account]] === What is this UID 0 toor account? Have I been compromised? Do not worry. `toor` is an "alternative" superuser account, where toor is root spelled backwards. It is intended to be used with a non-standard shell so the default shell for `root` does not need to change. This is important as shells which are not part of the base distribution, but are instead installed from ports or packages, are installed in [.filename]#/usr/local/bin# which, by default, resides on a different file system. If ``root``'s shell is located in [.filename]#/usr/local/bin# and the file system containing [.filename]#/usr/local/bin#) is not mounted, `root` will not be able to log in to fix a problem and will have to reboot into single-user mode in order to enter the path to a shell. Some people use `toor` for day-to-day `root` tasks with a non-standard shell, leaving `root`, with a standard shell, for single-user mode or emergencies. By default, a user cannot log in using `toor` as it does not have a password, so log in as `root` and set a password for `toor` before using it to login. [[serial]] == Serial Communications This section answers common questions about serial communications with FreeBSD. [[serial-console-prompt]] === How do I get the boot: prompt to show on the serial console? See link:{handbook}#serialconsole-setup/[this section of the Handbook]. [[found-serial]] === How do I tell if FreeBSD found my serial ports or modem cards? As the FreeBSD kernel boots, it will probe for the serial ports for which the kernel is configured. Either watch the boot messages closely or run this command after the system is up and running: [source,shell] .... % grep -E '^(sio|uart)[0-9]' < /var/run/dmesg.boot sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A .... This example shows two serial ports. The first is on IRQ4, port address `0x3f8`, and has a 16550A-type UART chip. The second uses the same kind of chip but is on IRQ3 and is at port address `0x2f8`. Internal modem cards are treated just like serial ports, except that they always have a modem attached to the port. The [.filename]#GENERIC# kernel includes support for two serial ports using the same IRQ and port address settings in the above example. If these settings are not right for the system, or if there are more modem cards or serial ports than the kernel is configured for, reconfigure using the instructions in <> for more details. [[access-serial-ports]] === How do I access the serial ports on FreeBSD? (x86-specific) The third serial port, [.filename]#sio2#, or [.filename]#COM3#, is on [.filename]#/dev/cuad2# for dial-out devices, and on [.filename]#/dev/ttyd2# for dial-in devices. What is the difference between these two classes of devices? When opening [.filename]#/dev/ttydX# in blocking mode, a process will wait for the corresponding [.filename]#cuadX# device to become inactive, and then wait for the carrier detect line to go active. When the [.filename]#cuadX# device is opened, it makes sure the serial port is not already in use by the [.filename]#ttydX# device. If the port is available, it steals it from the [.filename]#ttydX# device. Also, the [.filename]#cuadX# device does not care about carrier detect. With this scheme and an auto-answer modem, remote users can log in and local users can still dial out with the same modem and the system will take care of all the conflicts. [[enable-multiport-serial]] === How do I enable support for a multi-port serial card? The section on kernel configuration provides information about configuring the kernel. For a multi-port serial card, place an man:sio[4] line for each serial port on the card in the man:device.hints[5] file. But place the IRQ specifiers on only one of the entries. All of the ports on the card should share one IRQ. For consistency, use the last serial port to specify the IRQ. Also, specify the following option in the kernel configuration file: [.programlisting] .... options COM_MULTIPORT .... The following [.filename]#/boot/device.hints# example is for an AST 4-port serial card on IRQ 12: [.programlisting] .... hint.sio.4.at="isa" hint.sio.4.port="0x2a0" hint.sio.4.flags="0x701" hint.sio.5.at="isa" hint.sio.5.port="0x2a8" hint.sio.5.flags="0x701" hint.sio.6.at="isa" hint.sio.6.port="0x2b0" hint.sio.6.flags="0x701" hint.sio.7.at="isa" hint.sio.7.port="0x2b8" hint.sio.7.flags="0x701" hint.sio.7.irq="12" .... The flags indicate that the master port has minor number `7` (`0x700`), and all the ports share an IRQ (`0x001`). [[default-serial-params]] === Can I set the default serial parameters for a port? See the link:{handbook}#serial/#serial-hw-config[Serial Communications] section in the FreeBSD Handbook. [[cannot-tip]] === Why can I not run tip or cu? The built-in man:tip[1] and man:cu[1] utilities can only access the [.filename]#/var/spool/lock# directory via user `uucp` and group `dialer`. Use the `dialer` group to control who has access to the modem or remote systems by adding user accounts to `dialer`. Alternatively, everyone can be configured to run man:tip[1] and man:cu[1] by typing: [source,shell] .... # chmod 4511 /usr/bin/cu # chmod 4511 /usr/bin/tip .... [[misc]] == Miscellaneous Questions [[more-swap]] === FreeBSD uses a lot of swap space even when the computer has free memory left. Why? FreeBSD will proactively move entirely idle, unused pages of main memory into swap in order to make more main memory available for active use. This heavy use of swap is balanced by using the extra free memory for caching. Note that while FreeBSD is proactive in this regard, it does not arbitrarily decide to swap pages when the system is truly idle. Thus, the system will not be all paged out after leaving it idle overnight. [[top-freemem]] === Why does top show very little free memory even when I have very few programs running? The simple answer is that free memory is wasted memory. Any memory that programs do not actively allocate is used within the FreeBSD kernel as disk cache. The values shown by man:top[1] labeled as `Inact` and `Laundry` are cached data at different aging levels. This cached data means the system does not have to access a slow disk again for data it has accessed recently, thus increasing overall performance. In general, a low value shown for `Free` memory in man:top[1] is good, provided it is not _very_ low. [[chmod-symlinks]] === Why will `chmod` not change the permissions on symlinks? Symlinks do not have permissions, and by default, man:chmod[1] will follow symlinks to change the permissions on the source file, if possible. For the file, [.filename]#foo# with a symlink named [.filename]#bar#, this command will always succeed. [source,shell] .... % chmod g-w bar .... However, the permissions on [.filename]#bar# will not have changed. When changing modes of the file hierarchies rooted in the files instead of the files themselves, use either `-H` or `-L` together with `-R` to make this work. See man:chmod[1] and man:symlink[7] for more information. [WARNING] ==== `-R` does a _recursive_ man:chmod[1]. Be careful about specifying directories or symlinks to directories to man:chmod[1]. To change the permissions of a directory referenced by a symlink, use man:chmod[1] without any options and follow the symlink with a trailing slash ([.filename]#/#). For example, if [.filename]#foo# is a symlink to directory [.filename]#bar#, to change the permissions of [.filename]#foo# (actually [.filename]#bar#), do something like: [source,shell] .... % chmod 555 foo/ .... With the trailing slash, man:chmod[1] will follow the symlink, [.filename]#foo#, to change the permissions of the directory, [.filename]#bar#. ==== [[dos-binaries]] === Can I run DOS binaries under FreeBSD? Yes. A DOS emulation program, package:emulators/doscmd[], is available in the FreeBSD Ports Collection. If doscmd will not suffice, package:emulators/pcemu[] emulates an 8088 and enough BIOS services to run many DOS text-mode applications. It requires the X Window System. The Ports Collection also has package:emulators/dosbox[]. The main focus of this application is emulating old DOS games using the local file system for files. [[translation]] === What do I need to do to translate a FreeBSD document into my native language? See the link:{fdp-primer}#translations/[Translation FAQ] in the FreeBSD Documentation Project Primer. [[freebsd-mail-bounces]] === Why does my email to any address at FreeBSD.org bounce? The `FreeBSD.org` mail system implements some Postfix checks on incoming mail and rejects mail that is either from misconfigured relays or otherwise appears likely to be spam. Some of the specific requirements are: * The IP address of the SMTP client must "reverse-resolve" to a forward confirmed hostname. * The fully-qualified hostname given in the SMTP conversation (either HELO or EHLO) must resolve to the IP address of the client. Other advice to help mail reach its destination include: * Mail should be sent in plain text, and messages sent to mailing lists should generally be no more than 200KB in length. * Avoid excessive cross posting. Choose _one_ mailing list which seems most relevant and send it there. If you still have trouble with email infrastructure at `FreeBSD.org`, send a note with the details to mailto:postmaster@freebsd.org[postmaster@freebsd.org]; Include a date/time interval so that logs may be reviewed - and note that we only keep one week's worth of mail logs. (Be sure to specify the time zone or offset from UTC.) [[free-account]] === Where can I find a free FreeBSD account? While FreeBSD does not provide open access to any of their servers, others do provide open access UNIX(R) systems. The charge varies and limited services may be available. http://www.arbornet.org/[Arbornet, Inc], also known as _M-Net_, has been providing open access to UNIX(R) systems since 1983. Starting on an Altos running System III, the site switched to BSD/OS in 1991. In June of 2000, the site switched again to FreeBSD. _M-Net_ can be accessed via telnet and SSH and provides basic access to the entire FreeBSD software suite. However, network access is limited to members and patrons who donate to the system, which is run as a non-profit organization. _M-Net_ also provides an bulletin board system and interactive chat. [[daemon-name]] === What is the cute little red guy's name? He does not have one, and is just called "the BSD daemon". If you insist upon using a name, call him "beastie". Note that "beastie" is pronounced "BSD". More about the BSD daemon is available on his http://www.mckusick.com/beastie/index.html[home page]. [[use-beastie]] === Can I use the BSD daemon image? Perhaps. The BSD daemon is copyrighted by Marshall Kirk McKusick. Check his http://www.mckusick.com/beastie/mainpage/copyright.html[Statement on the Use of the BSD Daemon Figure] for detailed usage terms. In summary, the image can be used in a tasteful manner, for personal use, so long as appropriate credit is given. Before using the logo commercially, contact {mckusick} for permission. More details are available on the http://www.mckusick.com/beastie/index.html[BSD Daemon's home page]. [[daemon-images]] === Do you have any BSD daemon images I could use? Xfig and eps drawings are available under [.filename]#/usr/share/examples/BSD_daemon/#. [[glossary]] === I have seen an acronym or other term on the mailing lists and I do not understand what it means. Where should I look? Refer to the link:{handbook}#freebsd-glossary/[FreeBSD Glossary]. [[bikeshed-painting]] === Why should I care what color the bikeshed is? The really, really short answer is that you should not. The somewhat longer answer is that just because you are capable of building a bikeshed does not mean you should stop others from building one just because you do not like the color they plan to paint it. This is a metaphor indicating that you need not argue about every little feature just because you know enough to do so. Some people have commented that the amount of noise generated by a change is inversely proportional to the complexity of the change. The longer and more complete answer is that after a very long argument about whether man:sleep[1] should take fractional second arguments, {phk} posted a long message entitled link:http://www.bikeshed.com[A bike shed (any color will do) on greener grass...]. The appropriate portions of that message are quoted below. **** “What is it about this bike shed?” Some of you have asked me. It is a long story, or rather it is an old story, but it is quite short actually. C. Northcote Parkinson wrote a book in the early 1960s, called “Parkinson's Law”, which contains a lot of insight into the dynamics of management. [snip a bit of commentary on the book] In the specific example involving the bike shed, the other vital component is an atomic power-plant, I guess that illustrates the age of the book. Parkinson shows how you can go into the board of directors and get approval for building a multi-million or even billion dollar atomic power plant, but if you want to build a bike shed you will be tangled up in endless discussions. Parkinson explains that this is because an atomic plant is so vast, so expensive and so complicated that people cannot grasp it, and rather than try, they fall back on the assumption that somebody else checked all the details before it got this far. Richard P. Feynmann gives a couple of interesting, and very much to the point, examples relating to Los Alamos in his books. A bike shed on the other hand. Anyone can build one of those over a weekend, and still have time to watch the game on TV. So no matter how well prepared, no matter how reasonable you are with your proposal, somebody will seize the chance to show that he is doing his job, that he is paying attention, that he is here. In Denmark we call it “setting your fingerprint”. It is about personal pride and prestige, it is about being able to point somewhere and say “There! I did that.” It is a strong trait in politicians, but present in most people given the chance. Just think about footsteps in wet cement. --Poul-Henning Kamp on freebsd-hackers, October 2, 1999 **** [[funnies]] == The FreeBSD Funnies [[very-very-cool]] === How cool is FreeBSD? [qanda] Q::Has anyone done any temperature testing while running FreeBSD? I know Linux(R) runs cooler than DOS, but have never seen a mention of FreeBSD. It seems to run really hot. A:: No, but we have done numerous taste tests on blindfolded volunteers who have also had 250 micrograms of LSD-25 administered beforehand. 35% of the volunteers said that FreeBSD tasted sort of orange, whereas Linux(R) tasted like purple haze. Neither group mentioned any significant variances in temperature. We eventually had to throw the results of this survey out entirely anyway when we found that too many volunteers were wandering out of the room during the tests, thus skewing the results. We think most of the volunteers are at Apple now, working on their new "scratch and sniff" GUI. It is a funny old business we are in! Seriously, FreeBSD uses the HLT (halt) instruction when the system is idle thus lowering its energy consumption and therefore the heat it generates. Also if you have ACPI (Advanced Configuration and Power Interface) configured, then FreeBSD can also put the CPU into a low power mode. [[letmeoutofhere]] === Who is scratching in my memory banks?? [qanda] Q:: Is there anything "odd" that FreeBSD does when compiling the kernel which would cause the memory to make a scratchy sound? When compiling (and for a brief moment after recognizing the floppy drive upon startup, as well), a strange scratchy sound emanates from what appears to be the memory banks. A:: Yes! You will see frequent references to "daemons" in the BSD documentation, and what most people do not know is that this refers to genuine, non-corporeal entities that now possess your computer. The scratchy sound coming from your memory is actually high-pitched whispering exchanged among the daemons as they best decide how to deal with various system administration tasks. If the noise gets to you, a good `fdisk /mbr` from DOS will get rid of them, but do not be surprised if they react adversely and try to stop you. In fact, if at any point during the exercise you hear the satanic voice of Bill Gates coming from the built-in speaker, take off running and do not ever look back! Freed from the counterbalancing influence of the BSD daemons, the twin demons of DOS and Windows(R) are often able to re-assert total control over your machine to the eternal damnation of your soul. Now that you know, given a choice you would probably prefer to get used to the scratchy noises, no? [[changing-lightbulbs]] === How many FreeBSD hackers does it take to change a lightbulb? One thousand, one hundred and sixty-nine: Twenty-three to complain to -CURRENT about the lights being out; Four to claim that it is a configuration problem, and that such matters really belong on -questions; Three to submit PRs about it, one of which is misfiled under doc and consists only of "it's dark"; One to commit an untested lightbulb which breaks buildworld, then back it out five minutes later; Eight to flame the PR originators for not including patches in their PRs; Five to complain about buildworld being broken; Thirty-one to answer that it works for them, and they must have updated at a bad time; One to post a patch for a new lightbulb to -hackers; One to complain that he had patches for this three years ago, but when he sent them to -CURRENT they were just ignored, and he has had bad experiences with the PR system; besides, the proposed new lightbulb is non-reflexive; Thirty-seven to scream that lightbulbs do not belong in the base system, that committers have no right to do things like this without consulting the Community, and WHAT IS -CORE DOING ABOUT IT!? Two hundred to complain about the color of the bicycle shed; Three to point out that the patch breaks man:style[9]; Seventeen to complain that the proposed new lightbulb is under GPL; Five hundred and eighty-six to engage in a flame war about the comparative advantages of the GPL, the BSD license, the MIT license, the NPL, and the personal hygiene of unnamed FSF founders; Seven to move various portions of the thread to -chat and -advocacy; One to commit the suggested lightbulb, even though it shines dimmer than the old one; Two to back it out with a furious flame of a commit message, arguing that FreeBSD is better off in the dark than with a dim lightbulb; Forty-six to argue vociferously about the backing out of the dim lightbulb and demanding a statement from -core; Eleven to request a smaller lightbulb so it will fit their Tamagotchi if we ever decide to port FreeBSD to that platform; Seventy-three to complain about the SNR on -hackers and -chat and unsubscribe in protest; Thirteen to post "unsubscribe", "How do I unsubscribe?", or "Please remove me from the list", followed by the usual footer; One to commit a working lightbulb while everybody is too busy flaming everybody else to notice; Thirty-one to point out that the new lightbulb would shine 0.364% brighter if compiled with TenDRA (although it will have to be reshaped into a cube), and that FreeBSD should therefore switch to TenDRA instead of GCC; One to complain that the new lightbulb lacks fairings; Nine (including the PR originators) to ask "what is MFC?"; Fifty-seven to complain about the lights being out two weeks after the bulb has been changed. _{nik} adds:_ _I was laughing quite hard at this._ _And then I thought, "Hang on, shouldn't there be '1 to document it.' in that list somewhere?"_ _And then I was enlightened :-)_ _{tabthorpe}_ says: "None, _real_ FreeBSD hackers are not afraid of the dark!" [[dev-null]] === Where does data written to [.filename]#/dev/null# go? It goes into a special data sink in the CPU where it is converted to heat which is vented through the heatsink / fan assembly. This is why CPU cooling is increasingly important; as people get used to faster processors, they become careless with their data and more and more of it ends up in [.filename]#/dev/null#, overheating their CPUs. If you delete [.filename]#/dev/null# (which effectively disables the CPU data sink) your CPU may run cooler but your system will quickly become constipated with all that excess data and start to behave erratically. If you have a fast network connection you can cool down your CPU by reading data out of [.filename]#/dev/random# and sending it off somewhere; however you run the risk of overheating your network connection and [.filename]#/# or angering your ISP, as most of the data will end up getting converted to heat by their equipment, but they generally have good cooling, so if you do not overdo it you should be OK. _Paul Robinson adds:_ There are other methods. As every good sysadmin knows, it is part of standard practice to send data to the screen of interesting variety to keep all the pixies that make up your picture happy. Screen pixies (commonly mis-typed or re-named as "pixels") are categorized by the type of hat they wear (red, green or blue) and will hide or appear (thereby showing the color of their hat) whenever they receive a little piece of food. Video cards turn data into pixie-food, and then send them to the pixies - the more expensive the card, the better the food, so the better behaved the pixies are. They also need constant stimulation - this is why screen savers exist. To take your suggestions further, you could just throw the random data to console, thereby letting the pixies consume it. This causes no heat to be produced at all, keeps the pixies happy and gets rid of your data quite quickly, even if it does make things look a bit messy on your screen. Incidentally, as an ex-admin of a large ISP who experienced many problems attempting to maintain a stable temperature in a server room, I would strongly discourage people sending the data they do not want out to the network. The fairies who do the packet switching and routing get annoyed by it as well. [[punk-my-friend]] === My colleague sits at the computer too much, how can I prank her? Install package:games/sl[] and wait for her to mistype `sl` for `ls`. [[advanced]] == Advanced Topics [[learn-advanced]] === How can I learn more about FreeBSD's internals? See the link:{arch-handbook}[FreeBSD Architecture Handbook]. Additionally, much general UNIX(R) knowledge is directly applicable to FreeBSD. [[how-to-contribute]] === How can I contribute to FreeBSD? What can I do to help? We accept all types of contributions: documentation, code, and even art. See the article on link:{contributing}[Contributing to FreeBSD] for specific advice on how to do this. And thanks for the thought! [[define-snap-release]] === What are snapshots and releases? There are currently {rel-numbranch} active/semi-active branches in the FreeBSD http://svnweb.FreeBSD.org/base/[Subversion Repository]. (Earlier branches are only changed very rarely, which is why there are only {rel-numbranch} active branches of development): * {rel2-releng} AKA {rel2-stable} * {rel-releng} AKA {rel-stable} * {rel-head-releng} AKA _-CURRENT_ AKA {rel-head} `HEAD` is not an actual branch tag. It is a symbolic constant for the current, non-branched development stream known as _-CURRENT_. Right now, _-CURRENT_ is the {rel-head-relx} development stream; the {rel-stable} branch, {rel-releng}, forked off from _-CURRENT_ in {rel-relengdate} and the {rel2-stable} branch, {rel2-releng}, forked off from _-CURRENT_ in {rel2-relengdate}. [[kernel-panic-troubleshooting]] === How can I make the most of the data I see when my kernel panics? Here is typical kernel panic: [.programlisting] .... Fatal trap 12: page fault while in kernel mode fault virtual address = 0x40 fault code = supervisor read, page not present instruction pointer = 0x8:0xf014a7e5 stack pointer = 0x10:0xf4ed6f24 frame pointer = 0x10:0xf4ed6f28 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 80 (mount) interrupt mask = trap number = 12 panic: page fault .... This message is not enough. While the instruction pointer value is important, it is also configuration dependent as it varies depending on the kernel image. If it is a [.filename]#GENERIC# kernel image from one of the snapshots, it is possible for somebody else to track down the offending function, but for a custom kernel, only you can tell us where the fault occurred. To proceed: [.procedure] ==== . Write down the instruction pointer value. Note that the `0x8:` part at the beginning is not significant in this case: it is the `0xf0xxxxxx` part that we want. . When the system reboots, do the following: + [source,shell] .... % nm -n kernel.that.caused.the.panic | grep f0xxxxxx .... + where `f0xxxxxx` is the instruction pointer value. The odds are you will not get an exact match since the symbols in the kernel symbol table are for the entry points of functions and the instruction pointer address will be somewhere inside a function, not at the start. If you do not get an exact match, omit the last digit from the instruction pointer value and try again: + [source,shell] .... % nm -n kernel.that.caused.the.panic | grep f0xxxxx .... + If that does not yield any results, chop off another digit. Repeat until there is some sort of output. The result will be a possible list of functions which caused the panic. This is a less than exact mechanism for tracking down the point of failure, but it is better than nothing. ==== However, the best way to track down the cause of a panic is by capturing a crash dump, then using man:kgdb[1] to generate a stack trace on the crash dump. In any case, the method is this: [.procedure] ==== . Make sure that the following line is included in the kernel configuration file: + [.programlisting] .... makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols .... . Change to the [.filename]#/usr/src# directory: + [source,shell] .... # cd /usr/src .... . Compile the kernel: + [source,shell] .... # make buildkernel KERNCONF=MYKERNEL .... . Wait for man:make[1] to finish compiling. + [source,shell] .... # make installkernel KERNCONF=MYKERNEL .... . Reboot. ==== [NOTE] ==== If `KERNCONF` is not included, the [.filename]#GENERIC# kernel will instead be built and installed. ==== The man:make[1] process will have built two kernels. [.filename]#/usr/obj/usr/src/sys/MYKERNEL/kernel# and [.filename]#/usr/obj/usr/src/sys/MYKERNEL/kernel.debug#. [.filename]#kernel# was installed as [.filename]#/boot/kernel/kernel#, while [.filename]#kernel.debug# can be used as the source of debugging symbols for man:kgdb[1]. To capture a crash dump, edit [.filename]#/etc/rc.conf# and set `dumpdev` to point to either the swap partition or `AUTO`. This will cause the man:rc[8] scripts to use the man:dumpon[8] command to enable crash dumps. This command can also be run manually. After a panic, the crash dump can be recovered using man:savecore[8]; if `dumpdev` is set in [.filename]#/etc/rc.conf#, the man:rc[8] scripts will run man:savecore[8] automatically and put the crash dump in [.filename]#/var/crash#. [NOTE] ==== FreeBSD crash dumps are usually the same size as physical RAM. Therefore, make sure there is enough space in [.filename]#/var/crash# to hold the dump. Alternatively, run man:savecore[8] manually and have it recover the crash dump to another directory with more room. It is possible to limit the size of the crash dump by using `options MAXMEM=N` where _N_ is the size of kernel's memory usage in KBs. For example, for 1 GB of RAM, limit the kernel's memory usage to 128 MB, so that the crash dump size will be 128 MB instead of 1 GB. ==== Once the crash dump has been recovered , get a stack trace as follows: [source,shell] .... % kgdb /usr/obj/usr/src/sys/MYKERNEL/kernel.debug /var/crash/vmcore.0 (kgdb) backtrace .... Note that there may be several screens worth of information. Ideally, use man:script[1] to capture all of them. Using the unstripped kernel image with all the debug symbols should show the exact line of kernel source code where the panic occurred. The stack trace is usually read from the bottom up to trace the exact sequence of events that lead to the crash. man:kgdb[1] can also be used to print out the contents of various variables or structures to examine the system state at the time of the crash. [TIP] ==== If a second computer is available, man:kgdb[1] can be configured to do remote debugging, including setting breakpoints and single-stepping through the kernel code. ==== [NOTE] ==== If `DDB` is enabled and the kernel drops into the debugger, a panic and a crash dump can be forced by typing `panic` at the `ddb` prompt. It may stop in the debugger again during the panic phase. If it does, type `continue` and it will finish the crash dump. ==== [[dlsym-failure]] === Why has dlsym() stopped working for ELF executables? The ELF toolchain does not, by default, make the symbols defined in an executable visible to the dynamic linker. Consequently `dlsym()` searches on handles obtained from calls to `dlopen(NULL, flags)` will fail to find such symbols. To search, using `dlsym()`, for symbols present in the main executable of a process, link the executable using the `--export-dynamic` option to the ELF linker (man:ld[1]). [[change-kernel-address-space]] === How can I increase or reduce the kernel address space on i386? By default, the kernel address space is 1 GB (2 GB for PAE) for i386. When running a network-intensive server or using ZFS, this will probably not be enough. Add the following line to the kernel configuration file to increase available space and rebuild the kernel: [.programlisting] .... options KVA_PAGES=N .... To find the correct value of _N_, divide the desired address space size (in megabytes) by four. (For example, it is `512` for 2 GB.) [[acknowledgments]] == Acknowledgments This innocent little Frequently Asked Questions document has been written, rewritten, edited, folded, spindled, mutilated, eviscerated, contemplated, discombobulated, cogitated, regurgitated, rebuilt, castigated, and reinvigorated over the last decade, by a cast of hundreds if not thousands. Repeatedly. We wish to thank every one of the people responsible, and we encourage you to link:{contributing}[join them] in making this FAQ even better. diff --git a/documentation/content/en/books/fdp-primer/_index.adoc b/documentation/content/en/books/fdp-primer/_index.adoc index 403e249f11..fb703561d5 100644 --- a/documentation/content/en/books/fdp-primer/_index.adoc +++ b/documentation/content/en/books/fdp-primer/_index.adoc @@ -1,33 +1,34 @@ --- title: FreeBSD Documentation Project Primer for New Contributors authors: - author: The FreeBSD Documentation Project copyright: 1998-2021 DocEng trademarks: ["general"] -description: FreeBSD Documentation Project Primer for New Contributors Index +description: Everything you need to know in order to start contributing to the FreeBSD Documentation Project next: books/fdp-primer/preface +weight: 10 --- = FreeBSD Documentation Project Primer for New Contributors :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :experimental: [.abstract-title] Abstract Thank you for becoming a part of the FreeBSD Documentation Project. Your contribution is extremely valuable, and we appreciate it. This primer covers details needed to start contributing to the FreeBSD Documentation Project, or FDP, including tools, software, and the philosophy behind the Documentation Project. This is a work in progress. Corrections and additions are always welcome. ''' include::content/en/books/fdp-primer/toc.adoc[] diff --git a/documentation/content/en/books/handbook/_index.adoc b/documentation/content/en/books/handbook/_index.adoc index 0fdd49727d..f1f246dd93 100644 --- a/documentation/content/en/books/handbook/_index.adoc +++ b/documentation/content/en/books/handbook/_index.adoc @@ -1,39 +1,40 @@ --- title: FreeBSD Handbook authors: - author: The FreeBSD Documentation Project copyright: 1995-2021 The FreeBSD Documentation Project -description: FreeBSD Handbook Index +description: A constantly evolving, comprehensive resource for FreeBSD users trademarks: ["freebsd", "ibm", "ieee", "redhat", "3com", "adobe", "apple", "intel", "linux", "microsoft", "opengroup", "sun", "realnetworks", "oracle", "3ware", "arm", "adaptec", "google", "heidelberger", "intuit", "lsilogic", "themathworks", "thomson", "vmware", "wolframresearch", "xiph", "xfree86", "general"] next: books/handbook/preface +weight: 1 --- = FreeBSD Handbook :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: include::shared/releases.adoc[] include::shared/en/mailing-lists.adoc[] [.abstract-title] Abstract Welcome to FreeBSD! This handbook covers the installation and day to day use of _FreeBSD {rel130-current}-RELEASE_, _FreeBSD {rel122-current}-RELEASE_ and _FreeBSD {rel114-current}-RELEASE_. This book is the result of ongoing work by many individuals. Some sections might be outdated. Those interested in helping to update and expand this document should send email to the {freebsd-doc}. The latest version of this book is available from the https://www.FreeBSD.org/[FreeBSD web site]. Previous versions can be obtained from https://docs.FreeBSD.org/doc/[https://docs.FreeBSD.org/doc/]. The book can be downloaded in a variety of formats and compression options from the https://download.freebsd.org/ftp/doc/[FreeBSD FTP server] or one of the numerous link:./mirrors#mirrors-ftp[mirror sites]. Printed copies can be purchased at the https://www.freebsdmall.com/[FreeBSD Mall]. Searches can be performed on the handbook and other documents on the link:https://www.FreeBSD.org/search/[search page]. ''' include::content/en/books/handbook/toc.adoc[] include::content/en/books/handbook/toc-figures.adoc[] include::content/en/books/handbook/toc-tables.adoc[] include::content/en/books/handbook/toc-examples.adoc[] diff --git a/documentation/content/en/books/porters-handbook/_index.adoc b/documentation/content/en/books/porters-handbook/_index.adoc index 60a9df7f8a..a82db8f052 100644 --- a/documentation/content/en/books/porters-handbook/_index.adoc +++ b/documentation/content/en/books/porters-handbook/_index.adoc @@ -1,23 +1,24 @@ --- title: FreeBSD Porter's Handbook authors: - author: The FreeBSD Documentation Project copyright: 2000-2021 The FreeBSD Documentation Project -description: FreeBSD Porter's Handbook Index +description: Essential reading if you plan on providing a port of a third party piece of software trademarks: ["freebsd", "sun", "unix", "general"] next: books/porters-handbook/porting-why +weight: 15 --- = FreeBSD Porter's Handbook :doctype: book :toc: macro :toclevels: 1 :icons: font :sectnums: :sectnumlevels: 6 :source-highlighter: rouge :experimental: ''' include::content/en/books/porters-handbook/toc.adoc[] diff --git a/documentation/themes/beastie/layouts/index.html b/documentation/themes/beastie/layouts/index.html index 1ba57c710a..8b56f7c484 100644 --- a/documentation/themes/beastie/layouts/index.html +++ b/documentation/themes/beastie/layouts/index.html @@ -1,63 +1,67 @@ {{ define "main" }}
{{ $home := .Site.Home }}

{{ i18n "books" }} - {{ $home.Language.LanguageName }}

{{ $books := slice}} {{ range where .Site.Pages "Section" "books" }} {{ $subDirsNumer := .File.Path | strings.Count "/" }} {{ if eq $subDirsNumer 2 }} {{ $books = $books | append (dict "page" . "path" .File.Path) }} {{ end }} {{ end }}
    - {{ range sort $books "path" }} + {{ range sort $books "weight" "path" }} {{ if in .path "_index.adoc" }} {{ $bookPath := replaceRE "/_index.adoc" "" .path }} {{ $bookName := replaceRE "books/" "" $bookPath }} -
  • {{ .page.Title }}
    • +
  • {{ .page.Title }} +

    {{ .page.Params.description }}

    +
    • {{ end }} {{ end }}

{{ i18n "articles" }} - {{ $home.Language.LanguageName }}

{{ $articles := slice}} {{ range where .Site.Pages "Section" "articles" }} {{ $subDirsNumer := .File.Path | strings.Count "/" }} {{ if eq $subDirsNumer 2 }} {{ $articles = $articles | append (dict "page" . "path" .File.Path) }} {{ end }} {{ end }}
    - {{ range sort $articles "path" }} + {{ range sort $articles "weight" "path" }} {{ if in .path "_index.adoc" }} {{ $articlePath := replaceRE "/_index.adoc" "" .path }} {{ $articleName := replaceRE "articles/" "" $articlePath }} -
  • {{ .page.Title }}
    • +
  • {{ .page.Title }} +

    {{ .page.Params.description }}

    +
    • {{ end }} {{ end }}
{{ end }}