diff --git a/en/news/2000/press.xml b/en/news/2000/press.xml
index 3c4c5fbb4a..04eaf59594 100644
--- a/en/news/2000/press.xml
+++ b/en/news/2000/press.xml
@@ -1,610 +1,610 @@
How the FreeBSD Ports collection works. How to recover files off of FreeBSD system. A geek's appraisal of the Apple OS X from Jordan Hubbard, one
of the lead developers on the FreeBSD project. Using a Linux install under FreeBSD's Linux compatibility mode. Using FreeBSD on a laptop. Using RCS for file revision control. This column gives an overview of the different versions of BSD,
with links for more information. TeraSolutions, Inc. and Lightning Internet Services announce that
the OpenSource archive at
ftp.freesoftware.com has surpassed the download milestone of
two trillion bytes per day from a single server machine. A short article on using the FreeBSD Memory Filesystem. An interview with Robert Watson, one of the lead developers in the
TrustedBSD project. An early review of FreeBSD's SMB support. An article on a developers experience porting software from
Linux to FreeBSD. An informative article on BSD, and where it is going. An article evaluating various commercial OSes that contains a
blurb about BSDI and FreeBSD. With the recent hype surrounding open source software, an
important project has gone unnoticed in the media. This project,
FreeBSD, aims to create a rock-solid UNIX clone based on the 4BSD
work from the University of California at Berkeley. FreeBSD, a relatively unknown operating system is playing a big
role on the Internet. How Berkeley hackers built the Net's most fabled free operating
system on the ashes of the '60s---and then lost the lead to
Linux. A short guide to installing FreeBSD 4.0. Compares the merged Walnut Creek/BSDI OS offering to Linux. Open-source software sometimes provides a better solution than
- expensive commerical, closed software.
PolyServe, a provider of software-based, distributed server clustering technology, announced co-marketing agreement with FreeBSD, Inc. to ship PolyServe's Understudy (TM) software program with all new versions of FreeBSD 4.0 operating system software.
Talks of the lack of awareness in the market of the strengths of the BSD operating system and of the plans afoot to change this.
BSDi will be offering technical support contracts for FreeBSD beginning in May 2000.
Commentary on the BSDI/FreeBSD merger.
The second part of a review of FreeBSD v3.4.
An interview with three BSD veterans on the past and future of BSD.
An interview with BSD veteran Keith Bostic on the BSDI/FreeBSD merger. ``BSD has always had the best technology'', says Keith.
Michael Lucas presents a guide to customizing the FreeBSD kernel, written for the Linux oriented.
This article attempts to give a System V or Linux administrator a basic grounding in FreeBSD configuration and usage.
A good description of the FreeBSD Ports collection.
Praise for FreeBSD from this article: ``FreeBSD is the system of choice because it is fast, stable, and can handle large volumes of traffic.''
An article on the BSD License.
Promotes the BSD OSes as better alternatives to Linux in the areas of performance, reliability and security.
A Linux user writes about his experiences with the FreeBSD ports system.
A review of FreeBSD 3.4.
A Jordan Hubbard Interview on Improvements, New Platforms and What's to Come.
A Linux user writes about his experiences with FreeBSD.
A report that looks at and debunks some of the myths associated with Open Source development.
About 17 percent of enterprises plan to deploy FreeBSD or Linux as a primary platform for e-commerce within two years.
Steve Jobs' Macworld Expo keynote speech mentions FreeBSD as one of the components in the new Darwin OS from Apple.
In an article on the next generation Darwin OS, Apple Inc., refers to FreeBSD as one of the ``most acclaimed OS projects of the modern era.''
FreeBSD has several options for using software from other platforms such as Linux. This article examines Linux emulation under FreeBSD.
BSDCon will be hosted by the Usenix Association this year. The conference will run from February 11-14 in San Francisco. The FreeBSD Project will have a broad presence at this conference, and anyone interested in learning more about specific technologies or the FreeBSD Project in general is encouraged to attend.
New committer: Doug Ambrisko (Aironet)
New committer: Sergey Skvortsov (Ports)
The FreeBSD-stable branch of the source tree has now been frozen in preparation for the release of FreeBSD 4.5. This means that any new commits to the -stable source tree must be approved by the release engineering team first. Our expected "ship" date for 4.5 is January 20th, 2002.
The November 2001 status report is now available; see the status reports Web page for more information.
The FreeBSD Backports Collection is a new site created by D J Hawkey Jr. It contains patches that appeared in FreeBSD-stable that have not yet been merged in to older releases. Wider testing of these patches makes it more likely that they will be committed to earlier FreeBSD releases. So if your site relies on earlier releases of FreeBSD, and, for whatever reason, you do not wish to update to the most recent release, you are encouraged to visit this site frequently.
New committer: Martin Blapp (Ports)
BSDFreak is a new site that provides tutorials, articles, and journals covering BSD operating systems from a user's perspective.
New committer: Michael Lucas (Documentation Project)
New committer: Prafulla Deuskar (Intel gigabit device driver)
New committer: Yoichi NAKAYAMA (Ports)
Dag-Erling Smørgrav has written an article about writing FreeBSD problem reports.
New committer: Ernst de Haan (Ports)
The FreeBSD Core Team has appointed Dag-Erling Smørgrav as Bugmeister.
New committer: Patrick Li (Ports)
New committer: Cy Schubert (Ports)
New committer: Anders Nordby (Ports)
New committer: Christian Weisgerber (Ports)
Soren Schmidt, author of the ATA driver, tested a new 160 GB ATA harddisk provided kindly by Maxtor Corporation with the new ATA specification's 48-bit addressing mode support in FreeBSD 5.0-CURRENT. The results show that the code is stable and functions as it should and will be backported to STABLE.
New committer: MANTANI Nobutaka (Ports)
New committer: Andrew R. Reiter (SMPng, TrustedBSD)
"The FreeBSD Handbook, 2nd Edition" is now available in printed form! This is the primary source of documentation produced by the FreeBSD Documentation Project and is available now from, amongst other places, The FreeBSD Mall. For a complete list of changes in this edition, see the announce message from the editors. The ISBN for this book is 1571763031. 653 pages.
New committer: Makoto Matsushita (release building)
Bob Van Valzah has submitted an article introducing failure-resilient servers and step-by-step instructions for building one with Vinum.
Doug Rabson and Peter Wemm have been working non-stop on the FreeBSD/ia64 port in the past few weeks and said today that it boots into multi-user mode - without any operator attendence. This is indeed a major + without any operator attendance. This is indeed a major milestone in continued FreeBSD porting efforts. Right now most work is concentrating on fixing any problems in the sourcetree which become exposed by this platform's porting effort.
Jake Burkholder and Thomas Moestl have been porting FreeBSD to the ultra sparc for the past few months and first booted a machine into single user mode on the 18th of October. The log from the serial console can be found at http://people.FreeBSD.org/~jake/tip.single_user.
New committer: Akio Morita (PC98)
Benno Rice has committed a mega-patch which added support for OpenFirmware to the FreeBSD loader. The loader can now load a kernel over the network and execute it on an Apple iMac.
After a few months of development Doug Rabson and Peter Wemm have committed patches which extends the FreeBSD/ia64 port's functionality and adds the possibility to boot on real hardware.
New committer: Giorgos Keramidas (Docs)
The August 2001 Status Report is now available; see the Status Reports Web Page.
Many people, upon hearing of 5.0's untimely delay for a full year, have asked what they can do to help 5.0 get back on track. This email gives a list of outstanding projects for 5.0 and as much information as possible about how to get involved.
FreeBSD 5.0 has been delayed until November 2002. The complete announcement from Jordan is available here.
New committer: Robert Drehmel
New committer: Pete Fritchman (Ports)
Annelise Anderson, a frequent contributor to the FreeBSD mailing lists, has written "FreeBSD: An Open-Source Operating System for Your PC", an introduction to FreeBSD aimed at the new user. Published by The Bit Tree Press, the ISBN is 0971204500, and it can be ordered from, amongst other places, the DaemonNews Mall.
The Installing FreeBSD section of the Handbook has been substantially improved and updated. The new documentation features "screenshots" of almost every stage of the installation process, and expanded text detailing what each stage of the install covers. The bulk of the work was carried out by Randy Pratt.
New committer: Max Khon
New committer: Randy Pratt (Docs)
The July 2001 status report is now available; see the Status Reports Web page.
An RDF file of the last 10 news headlines on the FreeBSD site is now available. The URL is http://www.FreeBSD.org/news/news.rdf. You can use this file to syndicate FreeBSD news headlines on to your own web site (as Daily DaemonNews and the FreeBSD Diary do), or on to your desktop, using applications such as KNewsTicker.
Pandaemonium, the BSD Users Group of Western Australia, has been added to the Support page.
New committer: Valentino Vaschetto (Docs)
A second edition of The FreeBSD Handbook will be in production shortly. A task list has been published for anyone who wants to help contribute to the state of available printed documentation about FreeBSD.
New committer: Mark Peek
New committer: Rob Braun
New committer: Dave Zarzycki
New committer: Mike Barcroft
New committer: Chern Lee (Docs)
New committer: Brooks Davis (Networking, Mobile Computing)
John Kozubik has submitted an article explaining How to use FreeBSD with solid state media.
Benno Rice has completed enough work to allow FreeBSD to reach the mountroot prompt on the PowerPC processor. Please see the PowerPC platform page and mailing list for more information.
New committer: Mike Silbersack (Networking)
New committer: Johann Visagie (Ports)
Robert Watson has compiled a status report for the FreeBSD Project. These reports are scheduled to continue on a monthly basis.
New committer: Anton Berezin (all things Perl)
New committer: David W. Chapman Jr.(Ports)
New committer: Mark Pulford (Ports)
New committer: Crist J. Clark (Networking, security)
The famous ftp site, ftp.FreeBSD.org, is now back in full operation. Many thanks to Tele Danmark, who are supplying the machine as well as the network connection.
New committer: Jim Pirzyk
The first May 2001 issue of the FreeBSD 'zine is now available. Starting this month, there will be two issues per month; one on the 1st, and one on the 15th.
FreeBSD now works with multiple processors on Alpha systems, thanks to the efforts of John Baldwin, Andrew Gallatin, and Doug Rabson.
New committer: Takuya SHIOZAKI (Internationalization)
New committer: Udo Erdelhoff (Docs)
The Developer's Handbook is now available on the web site. This is an evolving resource for people wanting to develop software for FreeBSD (and not just for the committers who are developing FreeBSD). Don't forget that a complete list of documentation available from this site is also available.
Addison Wesley have allowed us to republish Chapter 8 of the FreeBSD Corporate Networker's Guide. Chapter 8 provides an in-depth look at providing printing services to Windows, NT, and Novell clients using FreeBSD.
Yet another new committer: Jens Schweikhardt (Standards compliance)
The April issue of The FreeBSD 'zine is now available.
The Ports Collection now contains more than 5,000 individual entries!
New committer: George Reid (Sound support, ports)
Wind River to Acquire BSDi Software Assets. Please read the Wind River Press Release, the announcement from Jordan K. Hubbard, and the FreeBSD Core Team statement.
New committer: Yar Tikhiy (Networking)
New committer: Eric Melville (System tools)
The March issue of The FreeBSD 'zine is now available.
New committer: Dima Dorfman (Docs)
New committer: Michael C. Wu (Internationalization, porting efforts)
New committer: Thomas Möstl (POSIX.1e extensions)
New committer: Orion Hodson (Sound support)
New committer: Jesper Skriver
The February issue of The FreeBSD 'zine is now available.
New committer: Mike Heffner (Audit project)
New committer: Jimmy Olgeni (Ports)
New committer: Dirk Meyer (Ports)
New committer: Ying-chieh Liao (Ports)
Nearly four years after it was acquired by Microsoft, and in spite of a well-publicized effort to migrate it to Windows and IIS, Hotmail is still partly based on FreeBSD and Apache.
An article discussing FreeBSD as an workstation OS for new Unix users.
A brief introduction to portupgrade.
A short article on accessing a Windows(R) share from a FreeBSD workstation.
Byte's Moshe Bar does a comparison, through informal benchmarks, of FreeBSD 4.3 to Linux 2.4.10 running sendmail, procmail, MySQL, and Apache. The emphasis of the article is examination of the newly rewritten VM system in Linux, so the tests are conducted with only 512 MB of RAM.
An interview with Matt Dillon, a key developer in FreeBSD on the upcoming features in FreeBSD 5.0.
A short article on running Windows(R) applications under WINE in FreeBSD.
A short article on dealing with the all too common full disk.
A short article on ripping CDs on FreeBSD.
This is a review of Kaspersky Anti-Virus for FreeBSD, a product which can protect a network of Microsoft Windows hosts by scanning e-mail and SMB file shares.
How to mirror the FreeBSD CVS repository.
An article on FreeBSD's CVSup infrastructure used to distribute its source code worldwide.
An short interview with Jordan Hubbard, one of the founders of the FreeBSD project.
Using DUMMYNET to control bandwidth allocation
Linux, Solaris, FreeBSD and Windows 2000 are benchmarked for network applications. This article has a sequel where the tests were redone after tuning FreeBSD.
NAI Labs, a division of Network Associates, Inc., announced a $1.2 million contract awarded by the U.S. Navy's Space and Warfare Systems Command to develop security extensions to the Open Source FreeBSD operating system.
An article describing the ways to control user access to your FreeBSD system.
Using the functionality of newsyslog in FreeBSD.
A report on Microsoft's venture to port its C# programming language to FreeBSD.
Apple (http://www.apple.com/) has recruited FreeBSD founder Jordan Hubbard to its team, in a bid to steer its Mac OS X BSD (Berkeley Software Distribution) efforts.
An article which states that open-source software connected with the FreeBSD operating system is used in several places deep inside several versions of Microsoft's Windows software, and on numerous server computers that manage major functions at Microsoft's free e-mail service, Hotmail.
A report on the backfiring of the Microsoft effort to vilify open source software.
System logging in FreeBSD using syslogd.
Using CVS in client-mode.
Configuring FreeBSD for wireless operation.
A survey of the games available in the FreeBSD ports collection.
Submitting change requests to the FreeBSD project using send-pr.
A mini tutorial on DocBook and its use by the FreeBSD Documentation Project.
An introduction to the FreeBSD project.
BYTE's Linux guru finds himself wondering why he isn't running - FreeBSD --- a comparision (with informal benchmarks) of FreeBSD + FreeBSD --- a comparison (with informal benchmarks) of FreeBSD 4.1.1 and a Linux based distribution running the v2.4.0 Linux kernel.
Howto modify a FreeBSD port.
A system administrator's view of the Ports system.
A reviewer finds FreeBSD 4.1 to be better suited for web serving than a Red Hat Linux distribution.
A report from a roundtable at the recent USENIX Security Symposium 2000, involving several prominent developers in the BSD world.
Last month's status report was apparently a great success: I received countless e-mails with comments, questions, and suggestions. I've tried to incorporate any suggestions and address any problems from these e-mails in this month's report, which captures a far more extensive snapshot of FreeBSD activity in the last month. Unlike last month's report, it does a better job of reflecting non-development activity, such as on-going conference planning, documentation, and so on. This is a trend I hope to see improve in future months as well.
On the topic of conferences, in the future I'd like to report more on publication activities relating to FreeBSD, including online journals with articles relating to FreeBSD, paper journals, conference papers, and so on. Likewise, I would be interested in including references to Call for Papers relating to FreeBSD. I'll take this opportunity to plug both registration and paper submission for BSDCon Europe in November, which has status included in this report, and for the general BSD Conference being hosted by USENIX in February. Your attendance and submissions make these conferences "happen", and promote FreeBSD as a platform for new research, feature development, and application products. Work of extremely high calibre is performed on FreeBSD, and we need to get the word out.
Next month, we're maintaining much the same submission requirements: reports should be one or two paragraphs long, sent by e-mail, and approximate the layout of the entries this month (Project, Contact, URL, and text). I'll send out reminders again over the week before the deadline, with more specific instructions. An area where I'd like to explore improvement lies in the coordination of related status reports for larger projects, such as new architectural work or platform ports. This might even have the effect of encouraging communication within these projects :-). I'd like to continue to focus on pulling in a broader range of groups and their activities, including the Security Officer, Release Engineer, and Core Team.
-- Robert Watson < rwatson@FreeBSD.org >
ACPI (Advanced Configuration and Power Interface) is an industry standard which obsoletes APM, Intel MPS, PnPBIOS, and other Intel PC firmware interface standards. It is also used on the IA64 platform. More information on ACPI is available at
http://developer.intel.com/technology/iapc/acpiThe FreeBSD ACPI subsystem project is based heavily on the Intel ACPI Component Architecture. This status report outlines the current state of the project; future updates will focus on changes as they occur.
The Intel ACPI interpreter is fully integrated, although bugs are still coming out of the woodwork occasionally.
Work is ongoing in the following areas:
The ARM port is currently going pretty well. The kernel is compiling and is able to boot to the point where it panics trying to initialize the network subsystem. The current reference platform is the Netwinder but this may change as many people expressed interest in a more broadly available platform. Things that need to be done before it can get further includes adding footbridge, timer and interrupt supports. The pmap module is not completed yet either.
Now that BIND 8.2.4 is finally imported the time has come to look at getting BIND 9 imported into CURRENT. The current idea is to have it imported alongside BIND 8 so that people can play with either one until all import problems have been taken care of and people have tested it a bit.
Although gaining a new name, the project has been at a standstill due to both resource availability during the move between BSDi and Wind River, and other commitments of the developers. The project should obtain an official mailing list, as well as return to an active state after the dust settles.
The conference will take place at the Thistle Hotel, Brighton, UK from 9-11 November 2001.
The aim of the conference is to provide a focal point for European users and developers of all the BSD derived operating systems. The format will be similar to other conferences, with 2 days of technical sessions over the Saturday and Sunday.
We'll be finalizing the schedule towards the end of the month and anybody who is interested in doing a talk should contact us ASAP. There are no restrictions on the use of talks; if it's been done before we may still be interested in having it presented to an European audience, and we make no claims to the talks so speakers are free to present the talks again at other conferences.
We're also still looking for sponsors.
We had 80 pre-registrations in the first week so we're expecting a good turnout.
The new CAM transport code is starting to get supported in more HBAs and to get refined so that it does the intended per-protocol support. No progress on doing any SMPng work for CAM has been made yet. This is a fairly high priority.
Thanks to various outstanding individual efforts, we are now down to just below 2300 open bug-reports. This means that we have fought our way back to the level we had around march 2000.
Work continues (in large part sponsored by WRS) on updating the Handbook ready for the second print edition. There has been a flurry of activity in this area recently, and the ToDo list can be seen at
http://www.FreeBSD.org/docproj/handbook.html
Dima and others are doing a stellar job of keeping up with the steady flow of incoming PRs relating to the documentation project.
The Developers' Handbook,
http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/developers-handbook/index.html
is a year old; it contains a wealth of useful content for developers developing on, or for, FreeBSD. As ever, more contributions are always required, not only for the developers' handbook, but for all of the FreeBSD documentation set.
The basic design hasn't changed and this project mainly is in the phase of continued hardening and test case development. The next major feature will be to fully integrate into the new CAM TRAN code and to fully support on the fly device addition and removal. The only HBA supported is QLogic at this time. Future support for the QLogic line is planned to have 2300 (2Gb) and IP support before October.
Hardware watchpoints are now available for kernel debugging on the IA32 (i386) architecture. One can now set hardware watchpoints using the new ddb command 'hwatch', which is analogous to the existing 'watch' command. Alternatively, if greater flexibility is required, direct access to the debug registers is available using the ddb 'set' command which allows complete control over the processor hardware debug facilities. Hardware watchpoints are very useful in tracking down those elusive memory overwrite bugs in the kernel. Hardware watchpoints can even be used to set a code breakpoint in ROM, which is commonly found in embedded systems.
Support for configuring IEEE 802.11 wireless devices via ifconfig has been committed to -current and -stable. It contains most of the functionality needed to configure an wireless device. Some missing features are being worked on including integrated support for DHCP so a single entry in /etc/rc.conf can be used to fully configure a wireless device on a DHCP lan and setting the CTS/RTS threshold. Currently the an(4) and wi(4) drivers are supported in -current and -stable with the awi(4) device supported in -current. Further work is needed to support Frequency Hopping devices such as ray(4).
jailNG is a from-scratch rewrite of the popular jail(8) service, focusing on improved management functions, as well as more fine-grained configurability. An initial prototype has been written, based on explicitly named and configured jails, and work is proceeding on userland integration. Currently, it's not clear if the timeline for this will be 5.0-RELEASE, or 5.1-RELEASE.
The main development in the FreeBSD Java Project over the last month was the release of an initial "Developers Only" patchset for the JDK 1.3.1. Since that release progress had been made towards a much more usable alpha quality patchset which is likely to be turned into a port, as per the current JDK 1.2.2 patchset. This new patchset will feature a number of bugfixes, which essentially get the JDK to a working state for early adopters, and an initial implementation of "native threads" based on FreeBSD's userland pthreads. Unfortunately this implementation isn't fully functional, but is included in the hope of getting more eyeballs on the code (particularly experienced pthread programmers). We'd also like to welcome Fuyuhiko Maruyama-san as a new committer, the usual punishment for too many good patches.
We have been working to provide Japanese version of FreeBSD online manuals, since 1996. Currently, RELENG_4 manuals are based. Translated versions are placed on doc/ja_JP.eucJP/man and provided to users using ports/japanese/man-doc. Also, we discuss about related commands (e.g. ports/japanese/man and ports/japanese/groff).
The first FreeBSD kernel summit meeting was held June 29-30, 2001 in Boston, MA at the Usenix 2001 Annual Technical Conference. Links to a variety of files are posted on the web site.
Note: I (jhb) am still working on writing up a general summary of the meeting. When that is completed it will be posted here and mailed to the -hackers mailing list.
I'm working on multithreading the kernel. So far I have over - 400KB of diffs relative to todays -current (I'm keeping my tree + 400KB of diffs relative to today's -current (I'm keeping my tree updated with changes as they occur rather than get hit with a big update at the end).
I have split the proc structure and am changing most of the kernel to pass around a thread identifier instead of a proc structure.
The following interfaces have been changed so far:
I have still a lot of work to go with a lot of "dumb editing" (s/struct proc \*p/struct thread \*td/) usually I change a few items and then fix everything that breaks when I try compile it. I'd like to check it in on a branch so others can help the editing but haven't worked out the best way to do it yet.
I have implemented changes to the scheduler so that KSE's are scheduled instead of processes, and threads sleep, letting the KSE pick up a new thread. but it's not anywhere ready yet (heck it doesn't compile yet :-)
Note that I have not yet updated the document listed above.. everywhere it mentions "ksec" or "KSE-context", the code uses the word "thread". I will update it soon as Jason has sent me the source.
The FreeBSD Monthly Development Status Report aims to keep users and developers up-to-date on the latest goings-on in the FreeBSD project by providing summaries of each project and its status. At the time of this writing, the July 2001 status report is being prepared and is very near release. The FreeBSD Web site now has a Status Reports section, which, when the July 2001 report is released, will be updated to include a link to an HTML-ified version.
The NetBSD rc.d port aims to improve the FreeBSD startup process by porting Luke Mewburn's rc.d work from NetBSD to FreeBSD. This will score FreeBSD startup and shutdown dependencies without losing the traditional and much loved monolithic configuration filesystem.
Luke Mewburn's USENIX paper and slides on the system as implemented in NetBSD are available here:
http://groups.yahoo.com/group/FreeBSD-rc/message/3
Interested parties are urged to study this material before joining the discussion list.
The intention at this stage is to decide on an approach that will ensure that the differences between the NetBSD rc.d system and the system as ported to FreeBSD will be kept to a minimum. This will probably involve discussions with Luke around those areas of the system that are identified as areas for potential improvement.
The goal of this project is the implementation of ATM signalling and other ATM protocols by means of the netgraph(4) framework. This should provide an easily extensible architecture for using ATM on FreeBSD. Currently the full UNI4.0 stack (except for the LIJ capability) has been implemented, including ILMI and a first version of the ATM Forum API for UNI. An implementation of Classical IP over ATM is also available. Drivers have been implemented for the Fore PCA200E and Fore HE-155 cards.
Network device cloning support has been imported from NetBSD. This allows virtual devices to be allocated on demand rather then being statically allocated at compile time. Our implementation differs slightly from that of NetBSD's in that we allow both the creation of specific devices (i.e. gif0) and arbitrary devices instead of just allowing specific devices. Currently, the only device in the tree which has been converted is the gif(4) device which has been converted in both -current and -stable. Work is ongoing to convert all other virtual network devices with work in progress on faith, stf, and vlan interfaces. In general this conversion is accompanied by appropriate modifications to make these devices fully modular.
NGPT is an effort led by IBM engineers to implement MxN threads (also known as many user threads to one kernel thread mapping) on Linux. I have ported it to FreeBSD to use rfork(2).
The port is right here:
Funded by: Monzoon Networking, LLC
This month has been a month of conventration and consolidation. Much of the changes from current have been migrating into stable. I've improved power support, suspend/resume interactions, interrupt handling, and ability to work after windows/NEWCARD has run. Interrupt routing continues to be a locking issue for a complete MFC. Current patches are available at the above website. I'm racing to get this done before 4.4 is released.
Information on Intel ORP - a BSD licensed Java VM is right here:
http://www.intel.com/research/mrl/orp/
A FreeBSD patch has been tested to work with NGPT and submitted to the ORP project. The patch is available here:
http://www.sharma-home.net/~adsharma/projects/orp/orp-freebsd-1.0.5.patch.txt.gz
There are some issues to be ironed out to make it work with FreeBSD's default (user level) pthread implementation.
OpenPackages intends to create a software packaging system that will allow third-party programs to be installed, without operating system dependent changes, on as many platforms as are feasible. OpenPackages was originally based on code from the BSD ports systems, and has been improved and extended by developers of many heritages.
The OpenPackages Project is pleased to release the Milestone 2 codebase. This release contains a working package building system and a single test package. OP currently is known to build on certain instances of the following operating systems: FreeBSD, HP/UX, IRIX, Linux (Debian, Red Hat, Suse, Mandrake, TurboLinux, Caldera, etc.), NetBSD, OpenBSD, Solaris
(First report)
Large cleanup and extension of FreeBSD PAM modules. All modules are to be documented, consistent in style (style(9) used) and as complete as possible WRT functionality. Mostly done.
We now have the rudiments of device support. We have a nexus driver for OpenFirmware machines, along with support for the Apple UniNorth PCI/AGP host bridge. I'm currently trying to get the USB hardware working so that I can get closer to having a console driver independent of OpenFirmware, then I'll be trying to get the system to get to single-user mode using NFS.
Work has begun, but nothing has yet been committed. The NCP addresses used by ppp have been abstracted and initial support has been added to the filter set for ipv6 addresses. NCP negotiation hasn't yet been started.
Patches have been submitted to get ppp working under HURD, and mostly under Linux. There are GPL copyright problems that need to be addressed.
Making pppoed function in a production environment. Most of the work is complete and committed. Additional work includes adding a -l option where ``-l label'' is shorthand for ``-e exec ppp -direct label'' and discovering why rogue child processes are being left around.
PRFW is a set of hooks which I have integrated into the FreeBSD kernel. This allows modules to easily intercept system calls with less overhead. It also supports per-pid restrictions, which means, one process may not be able to use X function in Y manner, but another process may.
Progress: I was working on this in 4.3-RELEASE, but now I'm merging it into current. I will be submitting a patch to the mailing lists in about a week.
This driver is currently not working well under -current and is undergoing some work at this time. No major design or feature changes are planned. There was some notion of adding TapeAlert support, but HP supports that as a binary product via a user library and it was felt that it'd be more politically prudent to leave it alone.
In the 'smpng' p4 branch there is code to make the ast() function loop to close the race when an AST is triggered while we are handling previously triggered AST's.
In the 'jhb_preemption' p4 branch work is being done to make the kernel fully preemptive. It is reportedly stable on UP x86, but SMP x86 locks up, UP alpha has problems during shutdown and can recurse indefinitely until it exhausts its stack.
We are using a perforce repository for live development work, which can track multiple separate long-lived works-in-progress and collaborate between multiple developers at the same time on the same change set.
FreeBSD-current is being imported into p4 hourly, for easy tracking of the moving -current tree.
I haven't written up a good primer yet, but we're able to open this up to the general developer community. NEWCARD work looks like it will be done here too. Perforce is ideal for tracking this sort of long-lived project without having to resort to passing patches around.
KSE work is now being checked into a kse p4 branch - thanks Julian!
KSE work is focusing on getting the main API changes into the base tree well before 5.0.
mb_alloc is a specialized allocator for mbufs and mbuf clusters. It offers various important advantages over the old mbuf allocator, particularly for MP machines. Additionally, it is designed with the possibility of important future enhancements in mind.
The mb_alloc code has been committed to -CURRENT a month ago and appears to be holding up well. Prior to committing it, preliminary performance measurements were done merely to ensure that it is not significantly worse than the old allocator, even with Giant still in place. Results were promising [http://people.FreeBSD.org/~bmilekic/code/mb_alloc/results.html] - also see jlemon's results (link at the bottom of accompanying text). Since the commit, Matt Jacob has provided useful feedback and bugfixes. Work is now being done to re-enable mbtypes statistics and make appropriate changes to netstat(1) and systat(1).
The sparc64 port has been committed to the FreeBSD repository. As such further development will occur in cvs, rather than as a separately maintained patch set. Significant progress has been made since the last status report, including; support for kernel debugging with ddb, much more complete pmap support, support for context switching and process creation, and filling out of important machine dependent data structures. Thomas Moestl has shown a strong interest in working on the port and is in the process of implementing support for saving and restoring a process's floating point context. I look forward to working with him and any other developers that happen to fall out of the wood works.
The sparc64 loader is functional enough to boot an ELF binary from an UFS filesystem using the existent openfirmware library, which has been revised to work flawlessly on 32-bit and 64-bit architectures. Support for netbooting and modules will be implemented next, followed by a better openfirmware mapping strategy.
This project brings a SYN cache implementation to FreeBSD, in order to make it more robust to DoS attacks. A SYN cookie approach was considered, but ultimately rejected because it does not conform to the TCP protocol. The SYN cache will work with T/TCP, IPV6 and IPSEC, and the size of each cache element is currently is less than 1/5th the size of a normal TCP control block.
It's been a busy month, with a number of relevant news items. Not least important is that NAI Labs was awarded a $1.2M contract from the US Defense Advanced Research Projects Agency (DARPA) to work on a variety of components relevant to the TrustedBSD Project, including support for pluggable security models, and supporting features such as improving the extended attributes implementation, simple crypto support for swap and filesystems, documentation, and much more.
On the features side, progress continues on Mandatory Access Control, object labeling, and improving the consistency of kernel access control mechanisms--in particular, with regard to inter-process authorization and credential management. Work has begun on porting LOMAC, NAI Labs' Low-Watermark Mandatory Access Control scheme, from Linux to FreeBSD, and it has been re-licensed under a BSD license. We hope to have an initial port complete in time for 5.0-RELEASE later this year.
At long last, FreeBSD 5.0 is here. Along with putting the final polish on the tree, FreeBSD developers somehow found the time to work on other things too. IA64 took some major steps towards working on the Itanium2 platform, an effort was started to convert all drivers to use busdma and ban vtophys(), hardware crypto support and DEVD hit the tree, NewReno was fixed and effort began on locking down the network layer of the kernel. Also high performance, modular scheduler started taking shape and will be a welcome addition to the kernel soon.
Looking forward, the focus will be on stabilizing and improving the performance of 5.0. The RELENG_5 (aka 5-STABLE) branch will be created once we've reached our goals in this area, so hopefully we will get there quickly. Meanwhile, preparations for the next release from the 4.x series, 4.8, will begin soon. Of course, the best way to get 5.x to stabilize os to install and run it!
Thanks,
Scott Long, Robert Watson
I'm very pleased to announce that all kernel modules and few userland tools made it to the FreeBSD source tree. Many thanks to Julian Elischer.
Unfortunately no big changes since the last report. Some minor problems have been discovered and patches are available on request. I will prepare all the patches and submit them to Julian for review.
OBEX server and client (based on OpenOBEX library) is almost complete. I'm currently doing interoperability testing. If anyone has hardware and time please contact me. The HCI security daemon has been implemented and tested with Sony Ericsson T68i cell phone and Windows stack. It is now possible to setup secure Bluetooth connections.
A few people have complained about RFCOMM daemon. These individuals want to use GPRS and Bluetooth enabled cell phone to access Internet. If you have this problem please contact me for possible workaround. My next goal is to get robust RFCOMM implementation to address all these issues.
Largely bug-fixing and userland application tweaks; new interfaces were added to manipulate ACLs on extended attributes; bugs were fixed in ls relating to ACL flagging. Patches to teach cp, mv, gzip, bzip, and other apps about ACL preservation are in testing and review. tunefs flags were added to ease configuration of ACLs, especially on UFS2 file systems.
Possible changes to make use of Linux/Solaris umask semantics are under consideration: right now we implement verbatim POSIX.1e/IRIX merging of the umask, ACL mask, and requested creation mode during file, device, fifo, and directory creation. Solaris and the most recent Linux patches ignore the umask in the context of a default ACL; this requires some rearrangement of umask handling in our VFS, although the results would be quite useful. We're exploring how to do this in a low impact way.
Framework changes:
Instrument KLD system calls (module and kld load, unload, stat) Instrument NFSd system call. Instrument swapoff(2). Instrument per-architecture privileged parts of sysarch(). Make use of condition variables to allow callers to wait for the framework to "unbusy" when loading/unloading policies, rather than returning EBUSY. Store mount pointer in devfs_mount structure for use by policies. Improve handling of labels in loopback interface "re-align" packet copy case. Provide full paths on devfs object creations to help policies label them properly (not merged). Experimentation with moving MAC labels into m_tags (not merged). NFS server now uses real ucreds, not hacked up ucreds, meaning we can start laying the groundwork for enforcement on NFS operations. (not merged)
Policy changes
LOMAC: mac_lomac replaces lomac (LOMAC now uses the MAC Framework), SEBSD: Improved support for devfs labeling based on SELinux genfs. Handling of hard link checks. Support export of process transition information for login and others using sysctl. Login now prompts for roles. Allow policy reload. TTY labeling. Locking adaptation from Linux. Many, many policy adaptations and fixes. We can now boot in enforcing mode! mac_bsdextended: fix a bug in which VAPPEND wasn't mapped to VWRITE, so opens with the O_APPEND bug failed improperly.
Userland changes
setfmac(8) now supports a setfsmac(8) execution mode, which accepts initial labeling specification files. Supports an SELinux compatibility mode so it can accept SELinux label specfiles using the SEBSD module. sendmail(8) now sets user labels as part of the context switch for mail delivery.
Documentation changes
Man page updates for MAC command line tools, modules, admin hints, etc. Updates to the FreeBSD Developer's Handbook chapter on MAC policies and entry points. MAC section in FreeBSD Handbook.
This project has been coming along pretty well. The amd(4) and xl(4) drivers have now been converted to use the busdma API, sparc64 got the bus_dmamap_load_mbuf() and bus_dmamap_load_uio() functions, and the gem(4) and hme(4) drivers have been updated to use bus_dmamap_load_mbuf() instead of bus_dmamap_load().
A lot more still needs to be done, as shown on the project's page. A fair number of conversions are on their way though, and we can expect a fair number of drivers to be converted soon, thanks to all the developers who are working on this project.
The POSIX Utility Conformance in FreeBSD list (link above) has been updated to reflect current reality. Not much work remains to complete base utility conformance.
On the API front, grantpt(), posix_openpt(), unlockpt(), wordexp(), and wordfree() were implemented. The header <wordexp.h> was added.
There are currently about 40 unassigned tasks on our project's status board ranging from documentation, utilities, to kernel hacking. We would encourage any developers looking for something to work on to check out the status board and see if anything interests them.
The goal of this project is to import the OpenBSD kernel-level crypto subsystem. This facility provides kernel- and user-level access to hardware crypto devices for the calculation of cryptographic hashes, ciphers, and public key operations. The main clients of this facility are the kernel RNG (/dev/random), network protocols (e.g. IPsec), and OpenSSL (through the /dev/crypto device).
This work will be part of the 5.0 release and has been committed to the -stable source tree for inclusion in the 4.8 release.
Recent work has focused on improving performance. System statistics are now maintained and an optional profiling facility was added for analyzing performance. Using this facility the overhead for using the crypto API has been significantly reduced.
The ubsec (Broadcom) driver was changed to significantly improve performance under load. In addition several memory leaks were fixed in the driver and the public key support was enabled for use.
Upcoming work will focus on load-balancing requests across multiple crypto devices and integrating OpenSSL 0.9.7 which will automatically enable application use of crypto hardware.
Devd has been integrated into FreeBSD 5.0-RELEASE. The integrated code supports a range of configuration options. The config files are fully parsed now and their actions are performed.
Future work in this area is likely to be limited to improving the devctl interface. /dev/devctl likely will be a cloneable device in future versions. Individual device control via devctl is also planned.
The Donations project expedited several dozen donations during 2002, and was able to place most of what was offered. We still are in dire need of SMP and Sparc systems. You can see information on our needs and donations that have been handled by the team on the donations web page.
We are relying increasingly upon the developer wantlist to place items offered to the Project, and using the commit statistics to help place items. As such, active committers who ask for what they want beforehand have a decent chance of getting it. Less active committers, and committers who do not ask for what they want, will be lower in our priorities but will not be excluded.
We are in the process of streamlining the tax deduction process for donations, and hope to have news on that shortly. We are also always working to accelerate and reduce our internal processes, to get the most equipment in the hands of the most people as quickly as possible.
I especially want to thank David O'Brien and Tom Rhodes for stepping up and making the team far more successful. Also, the FreeBSD Foundation has been quite helpful in handling tax-deductible contributions.
The main goal of this project is to modify the IPsec protocols to use the kernel-level crypto subsystem imported from OpenBSD (see elsewhere). A secondary goal is to do general performance tuning of the IPsec protocols.
This work will be part of the 5.0 release. Performance has been improved due to work on the crypto subsystem.
The goal of the project is to use a small amount of space in the FFS superblock to store a volume label of the user's choice. A GEOM module will then expose the volume labels into a namespace in devfs. The idea is to make it easier to manage filesystems across disk swaps and movement from system to system.
At this point, everything pretty much works. I've submitted parts of the patch to respective subsystem maintainers for review. There are some issues with namespace collision that I haven't addressed yet, but the basic functionality is there
Most of the articles are translated too. Marc is still translating the handbook, 60% is currently translated. Stéphane has began the integration of our French localization web site in the US CVS Tree. Sébastien is still maintaining the Release Notes.
We launched a new site, www.FreeBSD-fr.info, consisting in a French Daemon News like site. Netasq have donated our new server; we will install it in a new hosting provider in the few next weeks. One of the big job now is the translation of the FAQ, and the big project will be the manual pages.
Since the ports tree has been frozen for most of this reporting period, there have not been too many GNOME updates going into the official CVS tree. However, development has not stopped. GNOME 2.2 is nearing completion, and quite a few FreeBSD users have stepped up to test the GNOME 2.1 port sources from the MarcusCom CVS repository. If anyone else is interested, follow the instructions on the aforementioned cvsweb URL, and checkout the "ports" module.
The upcoming FreeBSD 5.0-RELEASE will be the first release to have the GNOME 2.0 desktop as the default GNOME desktop choice. During the previously mentioned ports freeze, all the GNOME 2 ports were fixed up so that they build and package on both i386 and Alpha platforms. Alas, the one port that will not make the cut for Alpha is Mozilla. There are still problems with the xpcom code, but work is ongoing to get a working Alpha port.
Finally, the FreeBSD Mono (an OpenSource C# runtime) port has also received some new life. Mono has been updated to 0.17 (the latest released version), and Juli Mallett has ported gtk-sharp (GTK+ bindings for C#).
The ia64 port is up and running on the new Itanium2 based hp machines thanks to a lot of hard work by Marcel Moolenaar. So far we are running on the hp rx2600 as these were the machines graciously donated by Hewlett-Packard and Intel. We had a prototype Intel Tiger4 system for a while, but we had to return the machine and we do not know if it currently runs. Most of the changes necessary to run these are sitting in the perforce tree and are not in the -current or RELENG_5 cvs tree. As a result, the cvs derived builds (-current and the 5.0-RC series and presumably 5.0-RELEASE) are only usable on obsolete Itanium1 systems.
Lots of other stability and functionality fixes have been made over the last few months, including initial libc_r support. The OS appears to be stable enough for sustained workloads - it is building packages now, for example. We still do not have gdb support, even for reading core files.
We have been updating our Japanese translated manual pages to RELENG_5 based. All existing entries have been updated, but 15 exceptions are not, most of which require massive update. We will also need to add translations which did not exist on RELENG_4.
KGI (Kernel Graphic Interface) is a kernel infrastructure providing user applications with means to access hardware graphic resources (dma, irqs, mmio). KGI is already available under Linux as a separate standalone project. The KGI/FreeBSD project aims at integrating KGI in the FreeBSD kernel.
KGI/FreeBSD has been recently donated 2 PCI graphic cards (Matrox - Millenium II and a coming Mach64) and other have been proposed. + Millennium II and a coming Mach64) and other have been proposed. Please see the FreeBSD web pages for details. Thanks to donation@ for organizing and promoting donations. Thanks to the donators for their contribution to KGI/FreeBSD.
KGI/FreeBSD progressed fine the last months. Most of the VM issues for mapping HW resources in user space have been addressed and a first attempt of coding was made. This prototyping raised some API compatibility problems with the current Linux implementation and was discussed heavily on the kgi devel lists. Ask if you're interested in such issues, I'll be pleased to share them.
Most of coding is now done. Let's start debugging!
Work is ongoing to continue to lock up the network stack. Recently, the focus has been on the IP stack. The plan there involves a series of inter-related pieces to lock up the ifaddr ref count, the inet list, the ifaddr uses, the ARP code, the routing tree, and the routing entries. We are over 3/5 of the way done down this path.
In addition to TCP and UDP, the other networking protocols such as raw IP, IPv6, AppleTalk, and XNS need to be locked up. Around 1/4 these remaining protocols have been locked and will be committed after the IP stack is locked.
The protocol independent socket layer needs to be locked and operating correctly with the protocol dependent locks. This part is mostly done save for much needed testing and code cleanup.
Finally, a pass will be need to be made to lock up the devices drivers and various statistics counters.
This effort fixes some outstanding problems in our TCP stack with regard to congestion control. The first item is to fix our NewReno implementation. Following that, the next urgent correction is to fix a problem involving window updates and dupack counts. When that stabilizes, we will then change the recovery code to make use of SACK information. Eventually, this project will update the BSD stack to add Limited Transmit and other new internet standards and standards-track improvements.
The 3 FreeBSD package clusters (i386, alpha, sparc64) have been unified to run from the same master machine, instead of using 3 separate masters. This has freed up some machine resources to use as additional client machine, as well as simplifying administrative overheads. Build logs for all 3 architectures can now be found on the http://bento.FreeBSD.org webpage. The sparc64 package cluster now has 3 build machines (an u5 and two u10s), and an ia64 cluster is about to be created.
Package builds now keep track of how many sequential times a port has failed to build (html summaries are available on the bento website). This allows tracking of ports which have suddenly become broken (e.g. due to a bad upgrade, or due to changes in the FreeBSD source tree), and in the future will be used to send out notifications to port maintainers when their port fails to build 5 times in a row. This feature is currently experimental, and further code changes will be needed to stabilize it.
The goal of this project is to improve the wireless networking support in the system. By the time of this report the 802.11 link layer code should be committed. A version of the wi driver that uses this code should be committed shortly. Conversion of other drivers is planned as are drivers for new devices.
Support for 802.1x/EAP is the next planned milestone (both as a supplicant and authenticator).
November and December were especially busy for the release engineering team. Scott Long joined the team to help with secretary and communications tasks while Brian Somers bowed out to focus on other projects.
FreeBSD 5.0-DP2 was released in November after much delay and anticipation, and marked the final milestone needed for 5.0 to become a reality. Shortly after that, we imposed a code freeze on the HEAD branch of CVS and released 5.0-RC1. Creation of the RELENG_5_0 branch came next, followed by the release of 5.0-RC2 from this branch. At this point, enough critical problems still existed that we scheduled an RC3 release for the new year, and pushed the final 5.0-RELEASE date to mid-January. By the time this is published, FreeBSD 5.0-RELEASE should be a reality.
For the time being, there will not be a RELENG_5 (aka 5-STABLE) branch. FreeBSD 4.x releases will continue, with 4.8 being scheduled for March 2003. Release in the 4.x series will be lead by Murray Stokely, and releases in the 5.x series will be lead by Scott Long. Once HEAD has reached acceptable performance and stability goals, the RELENG_5 branch will be created and HEAD will move towards 6.0 development. We hope to reach this with the 5.1 release this spring.
A new scheduler will be available as an optional component along side the current scheduler in the 5.1 release. It has been designed to work well with KSE and SMP. Some ideas have been borrowed from solaris and linux along with many novel approaches. It has O(1) performance with regard to the number of processes in the system. It also has cpu affinity which should provide a speed boost for many applications.
The scheduler has a few loose ends and lots of tuning before it is production quality although it is quite stable. Please see the post to arch and subsequent discussion for more details.
The FreeBSD Bi-monthly status reports are back! In this edition, we catch up on seven highly productive months and look forward to the end of 2003.
As always, the FreeBSD development crew has been hard at work. Support for the AMD64 platform quickly sprang up and is nearly complete. KSE has improved greatly since the 5.1 release and will soon become the default threading package in FreeBSD. Many other projects are in the works to improve performance, enhance the user experience, and expand FreeBSD into new areas. Take a look below at the impressive summary of work!
Scott Long, Robert Watson
Still in the planning stage. Working on creating an extensible interface that is usable for both userland and kernel implementations for device drivers. Deciding on how to interface userland implemented device drivers with applications.
KSE seems to be working well on x86, amd64, and ia64. The alpha userland bits are done, but a couple of functions are unimplemented in the kernel. For sparc64, the necessary functions are implemented in the kernel, but the userland context switching functions need more attention.
Since 5.1, efficient scope system threads (no upcalls when they block) have been implemented, and KSE based pthread library can have both POSIX scope process threads and scope system threads. It is also possible that KSE based pthread library can implement pthread both in 1:1 and M:N mode, I know Dan has such Makefile file patch for libkse not yet committed.
KSE program now can work under ULE scheduler, its efficient should be improved under the new scheduler in future. BSD scheduler is still the best scheduler for current KSE implement.
Much has happened since the last bi-monthly report, which was more than half a year ago. FreeBSD 5.0 and FreeBSD 5.1 have been released for example. With FreeBSD 5.2 approaching quickly, we're not going to look back too far when it comes to our achievements. There's too much ahead of us...
Two milestones have been reached after FreeBSD 5.1. The first is the ability to support both Intel and HP machines with sources in CVS. This due to a whole new driver for serial ports, or UARTs. Unfortunately this still implies that syscons is not configured. That's another task for another time, but keep an eye on KGI/FreeBSD... The second milestone is the completion of KSE support. Both M:N and 1:1 threading is functional on ia64 and the old libc_r library has been obsoleted. Testing has shown that KSE (i.e. M:N) may well become the default threading model. It's looking good.
The ABI hasn't changed after 5.1 and the expectation is that it won't change much. This means that we can think about becoming a tier 1 platform. This also means we need gdb(1) support. Work on it has been started but the road is bumpy and long. Kernel stability also has improved significantly and we typically have one kernel panic remaining: VM fault on no fault entry. This will be addressed with the long awaited PMAP overhaul (see below).
Most work for FreeBSD 5.2 will be "sharpening the saw". Get those loose ends tied. This is a slight change of plan made possible by a slip in the release schedule. The 5.2 release is not going to be the start of the -stable branch; it has been moved to 5.3. So, we use the extra time to prepare the ground for 5.3.
The planned PMAP overhaul will probably be finished after 5.2. This should address all known issues with SMP and fix those last panics. As a side-effect, major performance improvements can be expected. More news about this in the next status reports.
The following items are in progress in the Disk I/O area: Turn scsi_cd.c into a GEOM driver. (Patch out for review). Turn atapi-cd.c into a GEOM driver. Turn fd.c into a GEOM driver. Move softupdates and snapshot processing from SPECFS to UFS/FFS. Move userland access to device drivers out of vnodes.
Once these preliminaries are dealt with, scatter/gather and mapped/unmapped support will be added to struct bio/GEOM.
FreeBSD Update is a system for tracking the FreeBSD release (security) branches. In addition to being faster and more convenient than source updates, FreeBSD Update also requires less bandwidth and is more secure than source updates via CVSup. However, FreeBSD Update is limited; it can only update files which were installed from an official RELEASE image and not recompiled locally. Right now I'm publishing binary updates for 4.7-RELEASE and 4.8-RELEASE; since my only available box takes 3.5 hours to buildworld, I don't have enough resources to do any more than that.
In the near future, I'd like to: Find someone who is willing to donate a faster buildbox; start building updates for other releases (at a minimum, for all "supported" FreeBSD releases); add warnings if a file would have been updated but can't be updated because it was recompiled locally; add code to compare the local system against a list of "valid" MD5 hashes for intrusion detection purposes; and add support for cross-signing, whereby several machines could build updates independently to protect against buildbox compromise.
The project started this spring and released version 1.0 with a port installation (security/pf) in may 2003. Version 2.0 is on the doorstep as OpenBSD 3.4 will be released. Due to the porting efforts we were able to reveal some bugs in the OpenBSD code and provided locking for the PFIL_HOOKS, which we utilize. Tarball installation of a loadable kernel module for testing can be found on the project homepage, a patchset is in the making.
PF was started at OpenBSD as a substitute for ipfilter and provides the same function set. However, in the two years it exists now, it has gained many superior features that no other packet filter has. For a impression take a look at the pf FAQ.
We hope to be eventually integrated into the base system. Before that we have to resolve some issues with tcpdump and kame.
I'm very pleased to announce that another release is available for download at http://www.geocities.com/m_evmenkin/ngbt-fbsd-20030908.tar.gz. I have also prepared patch for the FreeBSD source tree. The patch was submitted for review to the committers.
Fixed few bugs in kernel modules. The ng_hci(4) and ng_l2cap(4) modules were changed to fix issue with Netgraph timeouts. The ng_ubt(4) module was changed to fix compilation issue on -current.
Improved user-space utilities. Implemented new libsdp(3). Added new sdpcontrol(8) utility. The rfcomm_sppd(1), rfcomm_pppd(8) and obexapp(1) were changed and now can obtain RFCOMM channel via SDP from the server. The hccontorol(8) utility now has four new commands. The hcsecd(8) daemon now saves link keys on the disk.
I've been recently contacted by few individuals who whould like to port current FreeBSD Bluetooth code to other BSD systems (OpenBSD and NetBSD). The work is slowly progressing towards un-Netgraph'ing current code. In the mean time Netgraph version will be the primary supported version of the code.
The rescue build infrastructure has been committed. There is one known issue with make using both the '-s' and '-j' flags that appears to be a bug in make. Anyone interested in tracking down should contact us.
Support for a dynamically linked /bin and /sbin has been committed, although it is not turned on by default. Adventurous users can try it out by building /bin and /sbin using the WITH_DYNAMICROOT make flag. More testing is needed to determine if this is going to be default for 5.2-RELEASE. If anyone would like to benchmark worldstones with and without dynamically linked /bin and /sbin, please feel free to do so and submit the results.
Work is continuing on updating ACPI with new features as well as bugfixing. A new embedded controller driver was written in July with support for the ACPI 2.0 ECDT as well as more robust polling support. Also, a buffer overflow in the ACPICA resource list handling that caused panics for some users was fixed. Marcel helped get acpidump(8) tested and basically working on ia64.
Upcoming work includes integrating ACPI notifies with devd(8), committing user-submitted drivers for ASUS and Toshiba hotkeys, Cx processor sleep states (so my laptop doesn't burn my lap), and power resource support for intelligently powering down unused or idle devices.
Users who have problems with ACPI are encouraged to submit a PR and email its number to acpi-jp@jp.FreeBSD.org. Bug reports of panics or crashes have first priority and non-working features or missing devices (except suspend/resume problems) second. Reports of failed suspend/resume should NOT be submitted as PRs at this time due to most of them being a result of incomplete device support that is being addressed. However, feel free to mail them to the list as any information is helpful.
The uart(4) project was born out of the need to have a working serial interface (i.e. an RS-232-C interface) in a legacy-free configuration and after an unsuccessful attempt to convert sio(4). The biggest problem with sio(4) is that it has been intertwined in many ugly ways into the kernel's core. Conversion could not happen without breaking something that invariably affects some group of people negatively. With sio(4) as a good bad example and a strong desire to solve multiple problems at once, the idea of an UART (Universal Asynchronuous Receiver/Transmitter) device that, given its generic name, could handle different flavors of UART hardware started to settle firmly in the authors mind.
The biggest challenge was of course solving the problem of the low-level console access prior to the initialization of the bus infrastructure and still have a driver that uses the bus access exclusively. Along the way the problem of having an UART function as the keyboard on sparc64 was solved with the introduction of system devices, which also encapsulated the console as a system device.
The uart(4) driver can be enhanced to support the various UART hardware on pc98 and this is currently being worked on. Keyboard support on sparc64 is underway as well. Plans exist for a rewrite of the remote gdb support that uses a generic interface to allow various drivers, including uart(4), to register itself as a communications channel. And since uart(4) does not support multi- port cards by itself, we likely need to either enhance puc(4) or otherwise introduce other umbrella drivers
Since I ported icc to FreeBSD I wanted to build FreeBSD with icc. Now with icc 7.1 (and some patches) it is possible. There are still some bugs, e.g. NFS doesn't work with an icc compiled kernel, IP seems to be fragile, and some advanced optimizations trigger an ICE (Intel is working on it). At the moment I'm waiting for our admins to install icc on the FreeBSD cluster (we got a commercial license from Intel, so we are allowed to distribute binaries which are compiled with icc), after that I will try to convince some people with more knowledge of the IP and NFS parts of the kernel to debug the remaining problems. When the icc compiled kernel seems to work mostly bugfree the userland will get the porting focus. Interested people may try to do a build of the ports tree with icc independently from the status of the porting of the userland... if this happens at the FreeBSD cluster, we would also be allowed to distribute the binaries.
Benefits include: another set of compiler errors (debugging help), more portable source, and code which is better optimized for a P4 (gcc has some drawbacks in this area)
The FreeBSD ports were updated to KDE 3.1.4, another bug- and security-fixes release. With this update, the QT port was updated to version 3.2. Both will be included in FreeBSD 4.9. Significant work was spent to fix KDE on FreeBSD-CURRENT after the removal of the gcc -pthread Option. Automatic package builds from KDE CVS continued to ensure and improve the quality of the upcoming KDE 3.2 release.
Future: Work is in progress to setup a new server for hosting the KDE-FreeBSD Website, Repository and another KDE CVS mirror. With help from Marcel Moolenaar the project will try to make KDE compile and working on the Intel IA64. And last but not least efforts are being made to fix the currently broken kdesu program.
WifiBSD is a miniture version of FreeBSD for wireless applications. Originally for the Soekris Net45xx line of main-boards, but is now capable of being targeted to any hardware/architecture FreeBSD itself supports. Although not feature complete, WifiBSD is expected to be ready for 5.2-RELEASE. The design goal is to meet, or exceed, the functionality of commercial/consumer 802.11 wireless gear. Features that need attention (to name just a few) are: http interface, consol menu interface, and installation. Volunters are welcome.
Work has restarted after a hiatus. Current focus is on getting loadable modules working, NEWBUSing the NetBSD dbdma code, and completing the BMAC ethernet driver.
There is a huge amount of work to do. Volunteers more than welcome!
The last known bug that prevented AMD64 machines completing a full release has been fixed - one single character error that caused ghostscript to crash during rendering diagrams. SMP work is nearing completion and should be committed within the next few days. The SMP code uses the ACPI MADT table based on John Baldwin's work-in-progress there for i386. We need to spend some time on low level optimization because there are several suboptimal places that have been ignored for simplicity, context switching in particular. MTRR support has been committed and XFree86 can use it. cvsup now works but the ezm3 port has not been updated yet. The default data segment size limit is 8GB instead of 512M, and the (primitive) i386 binary emulation support knows how to lower the rlimits for executing 32 bit binaries.
Notable things missing still: Hardware debug register support needs to be written; gdb is still being done as an external set of patches relative to the not-yet-released FSF gdb tree; DDB does not disassemble properly; DDB cannot do stack traces without -fno-omit-frame-pointer - a stack unwinder is needed; i386 and amd64 linux binary emulation is needed, and the i386 FreeBSD binary emulation still needs work - removing the stackgap code in particular.
The platform in general is very reliable although a couple of problems have been reported over the last week. One appears to be a stuck interrupt, but all that code has been redone for SMP support.
The FreeBSD Java community has started an effort to improve the current framework for Java-based ports. The main objective is the automation of JDK/JRE build and run dependency checking.
The original version was aimed to ease the life of porters. Although it has proved to be useful and reliable to a great extend, we are currently working on a new version. We intend to reach a high degree of flexibility to cope with the recent increase of available JDK/JRE flavors. Furthermore, the new version will be easier to maintain, which means improved reliability, and hopefully more frequent updates.
The BSD Java Porting Team has recently reached an exciting milestone with the release of the first "Diablo" JDK and JRE courtesy of the FreeBSD Foundation. The release of Diablo Caffe and Diablo Latte 1.3.1 was the first binary release of a native FreeBSD JDK since 1.1.8 and marks an important step forward in FreeBSD Java support.
The team is continuing development work, with a focus on achieving a compliant JDK 1.4 release in the near future.
With the introduction of ATAng, some users of ATAPI/CAM have experienced various problems. These have been mostly tracked down to issues in the new ATA code, as well as two long-standing problems in portions of the CAM layer that are rarely exercised with "real" SCSI SIMs. This has also been an occasion to cleanup ATAPI/CAM to make it more robust, and to enable DMA for devices accessed through it, resulting in improved performances.
We have released Japanese translation of 5.1-RELEASE online manual pages on June 10.
Several months ago, I took it upon myself to to try present the information contained on the bento build cluster to be presented in a more user-friendly fashion; that is, to be browsed by error type, by maintainer, and so forth. An early addition was code to attempt to classify ports PRs by either "existing port" (after assiging the most likely category and portname); "new port"; "framework" (e.g. bsd.port.mk changes); and "unknown". Various columns about the ports PRs were added to the reports.
The initial intent of this was to make life easier for ports maintainers; however, the "general" reports are also useful to anyone who just wants to, e.g., find out if a particular port is working on their particular architecture and OS combination before downloading it. Those with that general interest should start with the overview of one port.
A lot of work done since last report: site reworked completly (see new +
A lot of work done since last report: site reworked completely (see new URL), console design with console message in text or graphic modes implemented, implementation of a compatibility layer to compile Linux fbdev drivers with more or less changes in the original driver (experimental).
Except some memory allocation bugs, X (XGGI based on XFree 3.3.6) is now working with the same driver as the console. A basic terminal has now to be implemented.
-Volonteers are welcome to the project...
+Volunteers are welcome to the project...
A number of races have been identified in locking device_t. Most of the races have been identified in making device_t have to do with how drivers are written. Efforts are underway to identify all the races, and to contact the authors of subsystems that can help the drivers. Of special concern is the need for the driver to ensure that all threads are completely out of the driver code before detach() finishes. Of additional concern is making sure that all sleepers are woken up before certain routines are called so that other subsystems can ensure the last condition and leave no dangling references. Locking device_t is relatively straight forward apart from these issues. Towards the end of proper locking, sample strawmen drivers are being used to work out what, exactly proper is. Once these issues are all known and documented in the code, efforts will be made to update relevant documentation in the tree. There are many problems with driver locking that has been done to date, but until we nail down how to write a driver in current, it will be premature to contact specific driver writers with specific concerns.
Support for several new crypto devices was added. The SafeNet 1141 is a medium performance part that is not yet available on retail products. The Hifn 7955 and 7956 parts are starting to appear on retail products that should be available by the end of the year. Both devices support AES encryption. Support for public key operations for the SafeNet devices was recently done for OpenBSD and will be backported. Public key support for the Hifn parts is planned.
A paper about the performance work done on the cryptographic subsystem was presented at the Usenix BSDCon 2003 conference and received the best paper award.
NetBSD recently imported the cryptographic subsystem.
The release of 4.9 is just around the corner and offers Physical Address Extensions (PAE) for x86 along with the same world-class stability and performance that is expected from the 4-STABLE series. As always, don't forget to purchase a copy of the CD set from your favorite FreeBSD vendor.
FreeBSD 5.1 was released in June and offered vastly improved stability over 5.0 along with a working implementation of Kernel Scheduled Entities, allowing for true multithreading of applications across multiple CPUs. FreeBSD 5.2 will be released by the end of 2003 and will focus on improved network and overall performance.
Numerous bugs have been fixed since the last status report (and of course a few new ones added). Progress on improved security has been slowed by other work. But new features and fixes are coming in from other groups that are now sharing the code. In particular NetBSD recently imported the revised 802.11 layer and the Linux-based MADWIFI project is using it too (albeit in an older form). The MADWIFI users have already contributed features such as fragmentation reassembly of 802.11 frames and improved signal monitoring. Power save polling and an improved rate control algorothm are expected to come in from the NetBSD folks. WPA support is still in the plans; the best estimate is that work on that will start in January.
The purpose of this project is to improve performance of the network subsystem. A major part of this work is to complete the locking of the networking subsystem so that it no longer depends on the "Giant lock" for proper operation. Removing the use of Giant will improve performance and permit multiple instances of the network stack to operate concurrently on multiprocessor systems.
This project started in August. The emphasis has been on locking the "lower half" of the networking code so that packet forwarding through the IPv4 path can operate without the Giant lock as part of the 5.2 release. To this end locking was added to several network interface drivers and much of the "middleware" code in the network was locked (e.g. ipfw, dummynet, then routing table, multicast routing support, etc). Work towards this goal is still ongoing but should be ready for 5.2. A variety of test systems have been running for several months without the Giant lock in the network drivers and IP layer.
Past the 5.2 release Giant will be removed from the "upper half" of the network subsystem and the socket layer. Once this is done the plan is to measure and improve performance (though some work of this sort is always happening). The ultimate goal is a system that performs at least as well as 4.x for normal use on uniprocessor systems. On multiprocessor systems we expect to see significantly better performance than 4.x due to greater concurrency and reduced latency.
The FreeBSD status report is back again after another small break. The second half of 2004 was incredibly busy; FreeBSD 5.3 was released, the 6-CURRENT development branch started, and EuroBSDCon 2004 was a huge success, just to name a few events. This report is packed with an impressive 44 submissions, the most of any report ever!
It's also my pleasure to welcome Max Laier and Tom Rhodes to the status report team. They kindly volunteered to help keep the reports on time and help improve their quality. Max in particular is responsible for the reports being divided up into topics for easier browsing. Many thanks to both for their help!
Frenzy is a "portable system administrator toolkit," Live-CD based on FreeBSD. It generally contains software for hardware tests, file system check, security check and network setup and analysis. Current version 0.3, based on FreeBSD 5.2.1-RELEASE, contains almost 400 applications in 200MB ISO-image.
Tasks for next release: script for installation to HDD; unified system configuration tool; updating of software collection.
ALTQ is part of FreeBSD 5.3 release and can be used to do traffic shaping and classification with PF. In CURRENT IPFW gained the ability to do ALTQ classification as well. A steadily increasing number of NIC drivers has been converted to support ALTQ. For details see the ALTQ(4) man-page.
Currently TCP segment reassembly is implemented as a linked list of segments. With today's high bandwidth links and large bandwidth*delay products this doesn't scale and perform well.
The rewrite optimizes a large number of operational aspects of the segments reassembly process. For example it is very likely that the just arrived segment attaches to the end of the reassembly queue, so we check that first. Second we check if it is the missing segment or alternatively attaches to the start of the reassembly queue. Third consecutive segments are merged together (logically) and are skipped over in one jump for linear searches instead of each segment at a time.
Further optimizations prototyped merge consecutive segments on the mbuf level instead of only logically. This is expected to give another significant performance gain. The new reassembly queue is tracking all holes in the queue and it may be beneficial to integrate this with the scratch pad of SACK in the future.
Andrew Gallatin was able to get 3.7Gb/sec TCP performance on dual-2Gbit Myrinet cards with severe packet reordering (due to a firmware bug) with the new TCP reassembly code. See second link.
The old TTCP according to RFC1644 was insecure, intrusive, complicated and has been removed from FreeBSD >= 5.3. Although the idea and semantics behind it are still sound and valid.
The rewrite uses a much easier and more secure system with 24bit long client and server cookies which are transported in the TCP options. Client cookies protect against various kinds of blind injection attacks and can be used as well to generally secure TCP sessions (for BGP for example). Server cookies are only exchanged during the SYN-SYN/ACK phase and allow a server to ensure that it has communicated with this particular client before. The first connection is always performing a 3WHS and assigning a server cookie to a client. Subsequent connections can send the cookie back to the server and short-cut the 3WHS to SYN->OPEN on the server.
TTCPv2 is fully configurable per-socket via the setsockopt() system call. Clients and server not capable of TTCPv2 remain fully compatible and just continue using the normal 3WHS without any delay or other complications.
Work on implementing TTCPv2 is done to 90% and expected to be available by early February 2005. Writing the implementation specification (RFC Draft) has just started.
Modern CPU's can only perform to their maximum if their working code is in fast L1-3 cache memory instead of the bulk main memory. All of today's CPU's support certain L1-3 cache prefetching instructions which cause data to be retrieved from main memory to the cache ahead of the time that it is already in place when it is eventually accessed by the CPU.
CPU Cache Prefetching however is not a golden bullet and has to be used with extreme care and only in very specific places to be beneficial. Incorrect usage can lead to massive cache pollution and a drop in effective performance. Correct and very carefully usage on the other can lead to drastic performance increases in common operations.
In the linked patch CPU cache prefetching has been used to prefetch the packet header (OSI layer 2 to 4) into the CPU caches right after entering into the network stack. This avoids a complete CPU stall on the first access to the packet header because packets get DMA'd into main memory and thus never are already pre-cache in the CPU caches. A second use in the patch is in the TCP input code to prefetch the entire struct tcpcb which is very large and used with a very high probability. Use in both of these places show a very significant performance gain but not yet fully quantified.
The final patch will include documentation and a guide to evaluate and assess the use of CPU cache prefetch instructions in the kernel.
The TCP code in FreeBSD has evolved significantly since the fork from 4.4BSD-Lite2 in 1994 primarily due to new features and refinements of the TCP specifications.
The TCP code now needs a general overhaul, streamlining a cleanup to make it easily comprehensible, maintainable and extensible again. In addition there are many little optimizations that can be done during such an operation propelling FreeBSD back at the top of the best performing TCP/IP stacks again, a position it has held for the longest time in the 90's.
This overhaul is a very involved and delicate matter and needs extensive formal and actual testing to ensure no regressions compared to the current code. The effort needed for this work is about two man-month of fully focused and dedicated time. To get it done I need funding to take time off my day job and to dedicate me to FreeBSD work much the way PHK did with his buffer cache and vnode rework projects.
In February 2005 I will officially announce the funding request with a detailed description of the work and how the funding works. In general I can write invoices for companies wishing to sponsor this work on expenses. Tax exempt donations can probably be arranged through the FreeBSD foundation. Solicitations of money are already welcome, please contact me on the email address above.
The ARP IP address to MAC address mapping does not belong into the routing table (FIB) as it is currently done. This will move it to its own hash based structure which will be instantiated per each 802.1 broadcast domain. With this change it is possible to have more than one interface in the same IP subnet and layer 2 broadcast domain. The ARP handling and the routing table will be quite a bit simplified afterwards. As an additional benefit full MAC address based accounting will be provided.
Qing Li has become the driver and implementor of this project and is expected to post a first patch for comments shortly in February 2005.
IPFW2 has been converted to use PFIL_HOOKS for the IP[46] in/output path. (See link.) Not converted yet is the Layer 2 Etherfilter functionality of IPFW2. It is still directly called from the ether_input/output and bridging code.
Layer 2 PFIL_HOOKS provide a general abstraction for packet filters to hook into the Layer 2 packet path and filter or manipulate such packets. This makes it possible to use not only IPFW2 but also PF and others for Layer 2 filtering.
CARP is an alternative to VRRP. In contrast to VRRP it has full support for IPv6 and uses crypto to protect the advertisements. It was developed by OpenBSD due to concerns that the HSRP patent might cover VRRP and CISCO might defend its patent. CARP has, since then, improved a lot over VRRP.
CARP is implemented as an in-kernel multicast protocol and displays itself as a pseudo interface to the user. This makes configuration and administration very simple. CARP also incorporates MAC based load-balancing.
Patches for RELENG_5 and recent HEAD are available from the URL above. I plan to import these patches in the course of the next two to four month. RELENG_5 has all necessary ABI to support CARP and I might MFC it for release 5.4 or 5.5 - depending how well the HEAD import goes.
A public Subversion mirror of the FreeBSD repository is provided at svn://svn.clkao.org/freebsd/. This is intended for people who would like to try the svk distributed version control system.
svk allows you to mirror the whole repository and commit when offline. It also provides history-sensitive branching, merging, and patches. Non-committers can easily maintain their own branch and track upstream changes while their patches are being reviewed.
In my continuing quest to secure the mechanisms by which FreeBSD users keep their systems up to date, I've added a new tool: Portsnap. Available as sysutils/portsnap in the ports tree, this utility securely downloads and updates a compressed snapshot of the ports tree; this can then be used to extract or update an uncompressed ports tree. In addition to operating in an end-to-end secure manner thanks to RSA signatures, portsnap operates entirely over HTTP and can use under one tenth of the bandwidth of cvsup for users who update their ports tree more than once a week.
FreeBSD Update -- my utility for secure and efficient binary tracking of the Security/Errata branches -- continues to be widely used, with over 100 machines downloading security or errata updates daily.
At some point in the future I intend to bring both of these utilities into the FreeBSD base system, probably starting with portsnap.
Currently FreeBSD supports three family of Cronyx sync adapters: Tau-PCI - cp(4), Tau-ISA - ctau(4) and Sigma - cx(4). All these drivers were updated (in 6.current) and now they are Giant free. However, this is true only for sppp(4). If you are using Netgraph or async mode (for Sigma) you may need to turn mpsafenet off for that driver with appropriate kernel variable.
This year's EuroBSDCon will be held at the University of Basel, Switzerland from 25th through 27th November. The call for papers should happen shortly. Please consider attending or even presenting. Check the conference homepage for more information.
FreeSBIE is a Live-CD based on the FreeBSD Operating system, or even easier, a FreeBSD-based operating system that works directly from a CD, without touching your hard drive.
On December, 6th, 2004, FreeSBIE Staff released FreeSBIE 1.1, based on FreeBSD 5.3-RELEASE. Some of the innovations are: a renewed series of scripts to support power users in the use of FreeSBIE 1.1, an installer to let users install FreeSBIE 1.1 on their hard drives, thus having a powerful operating system such as FreeBSD, but with all the personalizations FreeSBIE 1.1 carries, the presence of the best open source software, chosen and personalized, such as X.Org 6.7, XFCE 4.2RC1, Firefox 1.0 and Thunderbird 0.9.2.
For a complete list of the included software, please consult: http://www.freesbie.org/doc/1.1/FreeSBIE-1.1-i386.pkg_info.txt
At EuroBSDCon 2004 in Karlsruhe, Germany, people from the FreeSBIE staff gave a talk, deeping into FreeSBIE scripts implementation and use.
A natively built 6.0-CURRENT miniinst ISO is available at the above link. It runs best on G4 Powermacs, but may run on other Newworld machines. See the release notes for full details.
As usual, lots of help is needed. This is a great project for those who want to delve deeply into FreeBSD kernel internals.
In the last month we set up the project page noted above and also created a p4 branch for those of us who use p4 to do work outside of CVS.
We haven't produced a status report in a while, but that's just because we've been busy. Since our last report in March 2004, we have added three new team members: Koop Mast (kwm), Jeremy Messenger (mezz), and Michael Johnson (ahze). Jeremy has been quite helpful in GNOME development porting while Michael and Koop have been focusing on improving GNOME multimedia, especially GStreamer. The stable release of GNOME is now up to 2.8.2, and we are actively working on the GNOME 2.9 development branch with is slated to become 2.10 on March 9 of this year.
The GNOME Tinderbox is still cranking away, and producing packages for both the stable and development releases of GNOME for all supported i386 versions of FreeBSD.
Thanks to Michael Johnson, the FreeBSD GNOME team has recently been given permission to use the Firefox and Thunderbird names , official icons, and to produce officially branded builds. Mozilla has also been very interested in merging our local patches back into the official source tree. This should greatly improve the quality of Firefox and Thunderbird on FreeBSD moving forward.
Finally, Adam Weinberger (adamw) has been pestering the team for photos so that we can finally show the community who we are. It is still unclear as to whether or not this will attract more FreeBSD GNOME users, or land us on the Homeland Security no-fly list.
Lots of changes happened inside the network stack that will hopefully be covered by a separate report. Outside of the network stack, several changes were made however including changes to proc locking, making the kernel thread scheduler preemptive, fixing several priority inversion bugs in the scheduler, and a few performance tweaks in the mutex implementation.
Locking work on struct proc and its various substructures continued with locking added where needed for struct uprof, struct rusage, and struct pstats. This also included reworking how the kernel stores process time statistics to store the raw struct bintime and tick counts internally and only compute the more user friendly values when requested via getrusage() or wait4().
Support for kernel thread preemption was added to the scheduler. Basically, when a thread makes another thread runnable, it may yield the current CPU to the new thread if the new thread has a more important priority. Previously, only interrupt threads preempted other threads and the implementation would occasionally trigger spurious context switches. This change exposed bugs in other parts of the kernel and was turned off by default in RELENG_5. Currently, only the i386, amd64, and alpha platforms support native preemption.
Several priority inversion bugs present in the scheduler due to various changes to the kernel from SMPng were also fixed. Most of the credit for these fixes belongs Stephan Uphoff who has recently been added as a new committer. Fixes include: closing a race in the turnstile wakeup code, changing the sleep queue code to store threads in FIFO order so that the sleep queue wakeup code properly handles having a thread's priority changes, and abstracting the concept of priority lending so that the thread scheduler is now able to properly track priority inheritance and handle priority changes for threads blocked on a turnstile.
Works in progress include separating critical sections from spin mutexes some so that bare critical sections become very cheap as well as continuing to change the various ABI compatibility layers to use in-kernel versions of system calls to reduce stackgap usage and make the system call wrappers MPSAFE.
The ACPI PCI link support code was reworked to work around some limitations in the previous implementation. The new version more closely matches the current non-ACPI $PIR link support. Enhancements include disabling unused link devices during boot and using a simpler and more reliable algorithm for choosing ISA IRQs for unrouted link devices.
Support for using the local APIC timer to drive the kernel clocks instead of the ISA timer and i8254 clock is currently being worked on in the jhb_clock perforce branch. It is mostly complete and will probably hit the tree in the near future. By letting each CPU use its own private timer to drive the kernel clocks, the kernel no longer has to IPI all the other CPUs in the system every time a clock interrupt occurs.
System-wide and process-virtual counting-mode performance monitoring counters are now supported for the AMD Athlon and Intel P4 CPUs. SMP works, but is prone to freezes. Immediate next steps include: (1) implementing the system-wide and process-virtual sampling modes, (2) debugging, (3) writing a test suite and (4) improving the project's documentation.
After experiencing spam attacks on the old wiki-engine caused by non-existent authentification mechanism, I had to replace it with a more advanced software. Instead of usemod, we now run moinmoin. As a consequence it's no longer just a 'browse & edit', but you have to sign up and let someone who is already in the ACL group 'developers' add you to the group. So it is a 'developers-only' resource now. The old wiki is found at http://wiki2.daemon.li
The project was very quiet (but still alive!) and mostly dedicated to testing by volunteers. New documentation at http://wiki.daemon.li/moin.cgi/KGI .
OpenOffice.org 2.0 status
FreeBSD 5.3 is the first release to include PF. It went out okay, but some bugs were discovered too late to make it on the CD. It is recommend to update `src/sys/contrib/pf' to RELENG_5. The specific issues addressed are:
Most of these issues were discovered by FreeBSD users and got fed back to OpenBSD. This is a prime example of open source at work.
The Handbook's Firewall section was modified to mention PF as an alternative to IPFW and IPF.
Following a number of mailing lists discussions on the topic, work has been progressing on the development of a new modular input device layer for FreeBSD. The purpose of this is twofold:
Currently, implementing support for new input devices is a painful process and there is great potential for code-duplication. The new input device layer will provide a simple API for developers to send events from their hardware on to the higher regions of the kernel in a consistent way, much like the 'input-core' driver in the Linux kernel.
Using multiple input devices at the moment is painful at best. With the new input device layer, events from different devices will be properly serialized before they are sent to other parts of the kernel. This will allow one to easily use, for instance, multiple USB keyboards in a virtual terminal.
The work on this is still in very rudimentary state. It is expected that the first visible changes will be committed to -CURRENT around late February or early March.
A longish status report for the 6 months of funded development was posted on announce, rather than repeat it here, you can find it at the link provided.
The FreeBSD Dutch Documentation Project is a ongoing project to translate the documentation into the Dutch language. Currently we are mainly focused on the Handbook, which is progressing pretty well. However, lots need to be translated and checked before we have a 'complete' translation ready. So if you are willing to help out, please checkout our website and/or contact me.
Since the last report on the Ports Collection, much has changed. Organizationally, the portmgr team saw the departure of some of the long-term members, and the addition of some newer members, Oliver Eikemeier, Kirill Ponomarew and Mark Linimon. Later on, portmgr also had to say goodbye to Will Andrews. In addition, we have gained quite a few new ports committers during this time period, and their contributions are quite welcome!
Most effort was devoted to two releases. The 5.3 release saw an especially long freeze period, but due to the good shape of the ports tree, the freeze for the 4.11 could be kept to a minimum. Several iterations of new infrastructure changes were tested on the cluster and committed. Also, the cluster now builds packages for 6-CURRENT, increasing the total number of different build environment to 10.
Additionally, several sweeps through the ports tree were made to bring more uniformity in variables used in the different ports and their values, e.g. BROKEN , IGNORE , DEPRECATED , USE_GCC - , and and others.
+ , and others.In technical terms, the largest change was moving to the X.org codebase as our default X11 implementation. At the same time, code was committed to be able to select either the X.org code or the XFree86 code, which also saw an update during that time. Due to some hard work by Eric Anholt, new committer Dejan Lesjak, and Joe Marcus Clarke, all of this happened more smoothly than could have reasonably been expected.
As well, GNOME and KDE saw updates during this time, as did Perl and the Java framework. Further, there were some updates to the Porter's Handbook, but more sections are still in need of updates to include recent changes in practices. Also, during this time, Bill Fenner was able to fix a bug in his distfile survey .
Shortly before the release for 4.11 our existing linux_base was marked forbidden due to security issues. A lot of effort was spent to upgrade the default version to 8 from 7 to ship 4.11 with a working linuxolator.
Due to stability problems in the April-May timeframe, the package builds for the Alpha were dropped. After Ken Smith and others put some work into the Alphas in the build cluster, package builds for 4.X were reenabled late in 2004.
Ports QA reminders -- portmgr team members are now sending out periodic email about problems in the Ports Collection. The current set includes:
Finally, it should be noted that we passed yet another milestone and the Ports Collection now contains over 12,000 ports.
The FreeBSD Hardware Notes have been (mostly) converted to being directly generated from the driver manual pages. This makes it much simpler to maintain the Hardware Notes, so they should be more accurate. The Hardware Notes for FreeBSD 5.3 use this new system.
The default linux_base port port was changed from the RedHat 7 based emulators/linux_base to the RedHat 8 based emulators/linux_base-8 just in time for FreeBSD 4.11-Release because of a security problem in emulators/linux_base. In the conversion process several problems where fixed in some Linux ports.
Both RedHat 7 and 8 are at their end of life, so expect an update to a more recent Linux distribution in the future. For QA reasons this update wasn't scheduled before FreeBSD 4.11-Release.
During 2004, there were several notable changes and events related to the FreeBSD Security Officer role and Security Team.
The charter for the Security Officer (SO) as approved by Core in 2002 was finally published on the web site. This document describes the mission, responsibilities, and authorities of the SO. (The current SO is Jacques Vidrine.)
The SO is supported by a Deputy SO and the Security Team. In April, Chris Faulhaber resigned as Deputy SO and Dag-Erling Smorgrav was appointed in his place. Also during the year, the following team members resigned: Julian Elischer, Bill Fumerola, Daniel Harris, Trevor Johnson, Kris Kennaway, Mark Murray, Wes Peters, Bruce Simpson, and Bill Swingle; while the following became new members: Josef El-Rayes, Simon L. Nielsen, Colin Percival, and Tom Rhodes. A huge thanks is due to all past and current members! The current Security Team membership is published on the web site.
With the release of FreeBSD 4.8, the SO began extended support for some FreeBSD releases and their corresponding security branches. "Early adopter" branches, such as FreeBSD 5.0 (RELENG_5_0), are supported for at least six months. "Normal" branches are supported for at least one year. "Extended" branches, such as FreeBSD 5.3 (RELENG_5_3), are supported for at least two years. The currently supported branches and their estimated "end of life" (EoL) dates are published on the FreeBSD Security Information web page. In 2004, four releases "expired": 4.7, 4.9, 5.1, and 5.2.
With the releases of FreeBSD 4.10 and 5.3, the SO and the Release Engineering team extended the scope of security branches to incorporate critical bug fixes unrelated to security issues. Currently, separate Errata Notices are published for such fixes. In the future, Security Advisories and Errata Notices will be merged and handled uniformly.
17 Security Advisories were published in 2004, covering 8 issues specific to FreeBSD and 9 general issues.
2004 also saw the introduction of the Vulnerabilities and Exposures Markup Language (VuXML). VuXML is a markup language designed for the documentation of security issues within a single package collection. Over 325 security issues in the Ports Collection have been documented already in the FreeBSD Project's VuXML document by the Security Team and other committers. This document is currently maintained in the ports repository, path ports/security/vuxml/vuln.xml. The contents of the document are made available in a human-readable form at the FreeBSD VuXML web site. The "portaudit" tool can be used to audit your local system against the listed issues. Starting in November, the popular FreshPorts.org web site also tracks issues documented in VuXML.
sppp(4) was updated (in 6.current) to be able to work in mpsafe mode. For compatibility if an interface is unable to work in mpsafe mode, sppp will not use mpsafe locks.
Support of FrameRelay AnnexD was added as a historical commit. Many of Cronyx users were expecting this commit for a long long time, and most of them still prefer sppp vs netgraph because of simplicity of its configuration (especially for ppp (vs mpd) and fr (vs a couple of netgraph modules). After MFCing this I'll finally close a PR 21771, from 2000/10/05
Support for multibyte characters has been added to many more base system utilities, including basename, col, colcrt, colrm, column, fmt, look, nl, od, rev, sed, tr, and ul. As a result of changes to the C library (see below), most utilities that perform regular expression matching or pathname globbing now support multibyte characters in these aspects.
The regular expression matching and pathname globbing routines in the C library have been improved and now recognize multibyte characters. Various performance improvements have been made to the wide character I/O functions. The obsolete 4.4BSD "rune" interface and UTF2 encoding have been removed from the 6-CURRENT branch.
Work is progressing on implementations of the POSIX iconv and localedef interfaces for potential inclusion into the FreeBSD 6.0 release.
FreeBSD/arm made some huge progress. It can boot multiuser, and run things like "make world" and perl on the IQ31244 board. It also now has support for various things, including DDB, KTR, ptrace and kernel modules. A patch is available for early gdb support, and the libpthread almost works.
The ATA driver is undergoing quite a few important changes, mainly it is being converted into modules so it can be loaded/unloaded at will, and just the pieces for wanted functionality need be present.
This calls for ata-raid to finally be rewritten. This is almost done for reading metadata so arrays defined in the BIOS can be used, and its grown quite a few new metadata formats. This also paves the way for ataraid to finally be able to take advantage of some of the newer controllers "RAID" abilities. However this needs more work to materialize but now its finally possible
There is also support coming for a few new chipsets as usual.
The work is just about finished enough that it can be released as patches to sort out eventual problems before hitting current. The changes are pretty massive as this touches all over the driver infrastructure, so lots of old bugs and has also been spotted and fixed during this journey
The ath driver was updated to support all the new features added to the net80211 layer. As part of this work a new version of the Hardware Access Layer (HAL) module was brought in; this version supports all available Atheros parts found in PCI and Cardbus products. Otherwise, adhoc mode should now be usable, antenna management has been significantly improved, and soft LED support now identifies traffic patterns.
The transmit rate control algorithm was split out of the driver into an independent module. Two different algorithms are available with other algorithms (hopefully) to be added.
Work is actively going on to add Atheros' SuperG capabilities.
The OpenBSD dhcp client program has been ported and enhanced to listen for 802.11-related events from the kernel. This enables immediate IP address acquisition when roaming (as opposed to the polling done by the old code). The main change from the previous client is that there is one dhclient process per interface as opposed to one for the entire system. This necessitates changes to the system startup scripts.
Incorporation into the base system is waiting on a volunteer who will shepherd the changes into the tree and deal with bugs.
Finally all of the papers and presentations are online for download from our conference website. Thanks again to all who helped make EuroBSDCon 2004 a success.
The ifconfig program used to configure network interfaces was overhauled. Over the years ifconfig has grown into a complex and often contorted piece of software that is hard to understand and difficult to maintain. The primary motivation for this work was to enable minimal configurations (for embedded use) without changing the code and to support future additions in a modular way. Functionality is now broken out into separate files and operations are registered with the central ifconfig code base. Features are configured simply by specifying which code is to be included when building the program.
In the future the plan is for ifconfig to auto-load functionality through dynamic libraries. This mechanism will allow, for example, third party software packages to provide kernel services and ifconfig add-on code without changing the base system.
The netperf project is working to enhance the performance of the FreeBSD network stack. This work grew out of the SMPng Project, which moved the FreeBSD kernel from a "Giant Lock" to more fine-grained locking and multi-threading. SMPng offered both performance improvement and degradation for the network stack, improving parallelism and preemption, but substantially increasing per-packet processing costs. The netperf project is primarily focused on further improving parallelism in network processing while reducing the SMP synchronization overhead. This in turn will lead to higher processing throughput and lower processing latency. Tasks include completing the locking work, optimizing locking strategies, amortizing locking costs, introducing new synchronization primitives, adopting non-locking synchronization strategies, and improving opportunities for parallelism through additional threading.
Between July, 2004, and December, 2004, the Netperf project did a great deal of work, for which there is room only to include limited information. Much more information is available by visiting the URLS above, including information on a variety of on-going activities. Accomplishments include:
July, 2004: A variety of improvements to PCB locking in the IPv6 implementation; locking for the if_xl driver; socket locking for the NFS client; cleanup of the soreceive() code path including structural improvements, assertions, and locking fixes; cleanup of the IPX/SPX code in preparation for locking; additional locking and locking assertions for the TCP implementation; bug fixes for locking and memory allocation in raw IP; netatalk cleanup and locking merged to FreeBSD CVS ; locking for many netgraph nodes merged to FreeBSD CVS ; SLIP structural improvements; experimental locking for netatalk ifaddrs; BPF locking optimizations (merged); Giant assertions for VFS to check VFS/network stack boundaries; UNIX domain socket locking optimizations; expansion of lock order documentation in WITNESS, additional NFS server code running MPSAFE; pipe locking optimizations to improve pipe allocation performance; Giant no longer required for fstat on sockets and pipes (merged); Giant no longer required for socket and pipe file descriptor closes (merged); IFF_NEEDSGIANT interface flag added to support compatibility operation for unlocked device drivers (merged) ; merged accept filter locking to FreeBSD CVS; documented uidinfo locking strategy (merged); Giant use reduced in fcntl().
August, 2004: UMA KTR tracing (merged); UDP broadcast receive locking optimizations (merged); TCP locking cleanup and documentation; IPv6 inpcb locking, cleanup, and structural improvements; IPv6 inpcb locking merged to FreeBSD CVS ; KTR for systems calls added to i386; substantial optimizations of entropy harvesting synchronization (merged) ; callout(9) sampling converted to KTR (merged); inpcb socket option locking (merged); GIANT_REQUIRED removed from netatalk in FreeBSD CVS; merged ADAPTIVE_GIANT to FreeBSD CVS, resulting in substantial performance improvements in many kernel IPC-intensive benchmarks ; prepend room for link layer headers to the UDP header mbuf to avoid one allocation per UDP send (merged); a variety of UDP bug fixes (merged); additional network interfaces marked MPSAFE; UNIX domain socket locking reformulated to protect so_pcb pointers; MP_WATCHDOG, a facility to dedicate additional HTT logical CPUs as watchdog CPUs developed (merged) ; annotation of UNIX domain socket locking merged to FreeBSD CVS; kqueue locking developed and merged by John-Mark Gurney ; task list for netinet6 locking created; conditional locking relating to kqueues and socket buffers eliminated (merged); NFS server locking bugfixes (merged); in6_prefix code removed from netinet6 by George Neville-Neil, lowering the work load for netinet6 (merged); unused random tick code in netinet6 removed (merged); ng_tty, IPX, KAME IPSEC now declare dependence on Giant using compile-time declaration NET_NEEDS_GIANT("component") permitting the kernel to detect unsafe components and automatically acquire the Giant lock over network stack operation if needed (merged) ; additional locking optimizations for entropy code (merged); Giant disabled by default in the netperf development branch (merged).
September, 2004: bugs fixed relating to Netgraph's use of the kernel linker while not holding Giant (merged); merged removal of Giant over the network stack by default to FreeBSD CVS ; races relating to netinet6 and if_afdata corrected (merged); annotation of possible races in the BPF code; BPF code converted to queue(3) (merged); race in sopoll() corrected (merged).
October, 2004: IPv6 netisr marked as MPSAFE; TCP timers locked, annotated, and asserted (merged); IP socket option locking and cleanup (merged); Netgraph ISR marked MPSAFE; netatalk ISR marked MPSAFE (merged); some interface list locking cleanup (merged); use after free bug relating to entropy harvesting and ethernet fixed (merged); soclose()/sofree() race fixed (merged); IFF_LOCKGIANT() and IFF_UNLOCKGIANT() added to acquire Giant as needed when entering the ioctls of non-MPSAFE network interfaces.
November, 2004: cleanup of UDPv6 static global variables (merged); FreeBSD 5.3 released! First release of FreeBSD with an MPSAFE and Giant-free network stack as the default configuration! ; additional TCP locking documentation and cleanup (merged); optimization to use file descriptor reference counts instead of socket reference counts for frequent operations results in substantial performance optimizations for high-volume send/receive (merged) ; an accept bug is fixed (merged) experimental network polling locking introduced; substantial measurement and optimization of mutex and locking primitives (merged) ; experimental modifications to UMA to use critical sections to protect per-CPU caches instead of mutexes yield substantial micro-benchmark benefits when combined with experimental critical section optimizations ; FreeBSD Project Netperf page launched; performance micro-benchmarks benchmarks reveal IP forwarding latency in 5.x is measurably better than 4.x on UP when combined with optional network stack direct dispatch; several NFS server locking bugfixes (merged); development of new mbufqueue primitives and substantial experimentation with them permits development of amortized cost locking APIs for handoff between the network stack and network device drivers (work in collaboration with Sandvine, Inc) ; Linux TCP_INFO API added to allow user-space monitoring of TCP state (merged); SMPng task list updated; UDP static/global fixes merged to RELENG_5.
December, 2004: UDP static/global fixes developed for multi-threaded in-bound UDP processing (merged); socket buffer locking fixes for urgent TCP input processing (merged); lockless read optimizations for IF_DEQUEUE() and IF_DRAIN(); Giant-free close for sockets/pipes/... merged to FreeBSD CVS; optimize mass-dequeues of mbuf chains in netisr processing; netrate tool merged to RELENG_5; TCP locking fixes merged to RELENG_5; "show alllocks" added to DDB (merged); IPX locking bugfixes (merged); IPX/SPX __packed fixes (merged); IPX/SPX moved to queue(9) (merged); TCP locking fixes and annotations merged to FreeBSD CVS; IPX/SPX globals and pcb locking (merged); IPX/SPX marked MPSAFE (merged) ; IP socket options locking merged to FreeBSD; SPPP locked by Roman Kurakin (merged); UNIX domain socket locking fixes by Alan Cox (merged).
On-going work continues with regard to locking down network stack components, including additional netinet6 locking, mbuf queue facilities and operations; benchmarking; moving to critical sections or per-CPU mutexes for UMA per-CPU caches; moving to critical sections or per-CPU mutexes for malloc(9) statistics; elimination of separate mbuf allocator statistics; additional interface locking; a broad variety of cleanups and documentation of locking; a broad range of optimizations.
FreeBSD profile.sh is targeted at laptops. It allows to define multiple network environments (eg, home, work), and will then detect in which environment the laptop is started and configure it accordingly. Almost everything from under /etc can be configured per environment, and only the overrides to the default /etc have to be defined. Suspending in one environment and resuming in a different one is also supported.
Proper integration into the acpi/apm and several small improvements are underway. More testing with different system configurations is needed.
At long last, FreeBSD 5.3 was released in November of 2004. This marked the start of the RELENG_5/5-STABLE branch and the beginning of the 6-CURRENT development branch. Many thanks to the tireless efforts of the FreeBSD developer and user community for making this release a success.
FreeBSD 4.11 release engineering is also now in progress. This will be the final release from the 4.x series and is mainly incremental bug fixes and a handful of feature additions. Of note is that the IBM ServeRAID 'IPS' driver is now supported on 4.x and will be included in this release, and the Linux emulation layer has been updated to support a RedHat 8.0 userland. The release is expected to be available on January 24.
Looking forward, there will be several FreeBSD 5.x releases in the coming year. FreeBSD 5.4 release engineering will start in March, and FreeBSD 5.5 release engineering will likely start in June. These releases are expected to be more conservative than previous 5.x releases and will follow the same philosophy as previous -STABLE branches of fixing bugs and adding incremental improvements while maintaining API stability.
For the 6-CURRENT development branch as well as all future development and stable branches, we are planning to move to a schedule with fixed timelines that move away from the uncertainty and wild schedule fluctuations of the previous 5.x releases. This means that major branches will happen at 18 month intervals, and releases from those branches will happen at 4 month intervals. There will also be a dedicated period of testing and bug fixing at the beginning of each branch before the first release is cut from that branch. With the shorter and more defined release schedules, we hope to lessen the problem of needed features not reaching users in a reasonable time, as happened too often with 5.x. This is a significant change in our strategy, and we look forward to realizing the benefits of it. This will kick off with the RELENG_6 branch happing in June of 2005, followed by the 6.0 release in August of 2005.
Also on the roadmap is a plan to combine the live-iso disk2 and the install distributions of disk1 into a single disk which can be used for both installation and for recovery. 3rd party packages that currently reside on disc1 will be moved to a disk2 that will be dedicated to these packages. This move will allow us to deal with the ever growing size of packages and also provide more flexibility to vendors that wish to add their own packages to the releases. It also opens the door to more advanced installers being put in place of sysinstall. Anyone interested in helping with this is encouraged to contact us.
The wireless networking layer was updated to support the 802.1x, WPA, and 802.11i security protocols, and the WME/WMM multi-media protocol. As part of this work extensible frameworks were added for cryptographic methods, authentication, and access control. Extensions are implemented as loadable kernel modules that hook into the net80211 layer. This mechanism is used, for example, to implement WEP, TKIP, and CCMP crypto protocols. The Atheros driver (ath) is currently the only driver that uses the full set of features. Adding support to other drivers is simple but waiting on volunteers. Ports of the wpa_supplicant and hostapd programs enable use of the new security protocols.
The support for tracking stations in a bss (managed or adhoc) and stations found when scanning was overhauled. Multiple tables are now used, each with different management policies, reference counting is now done consistently, and inactivity processing is done more intelligently (e.g. associated stations are probed before removal). This is the first step towards proper roaming support and other advanced features.
AP power save support was added. Associated stations may now operate in power save mode; frames sent to them will be buffered while they are sleeping and multicast traffic will be deferred until after the next beacon (per the 802.11 protocol). Power save support is required in a standards-compliant access point. Only the ath driver currently implements power save support.
Work is actively going on to add Atheros' SuperG capabilities, WDS, and for multi-bss support (ssid and/or bssid) on a single device.
FreeBSD 5.2.1 is stable on the stable branch of Xen as a guest. FreeBSD 5.3 runs on the stable branch of Xen as a guest, but a couple of bugs need to be tracked down.
The second quarter of 2005 has again been very exciting. The - BSDCan and MeetBSD conferences were both very interesting and and the + BSDCan and MeetBSD conferences were both very interesting and the sources of very good times. I highly recommend attending them again next year.
The Google Summer of Code project has also generated quite a bit of excitement. FreeBSD has been granted 19 funded mentorship spots, the fourth most of all of participating organizations. Projects being worked on range from UFS Journaling to porting the new BSD Installer to redesigning the venerable www.FreeBSD.org website. We are quite pleased to be working with so many talented students, and eagerly await the results of their work. More information and status can be found at the Wiki site at http://wiki.freebsd.org/moin.cgi/SummerOfCode2005 .
The FreeBSD 6.0 release cycle is also starting up. The purpose of quickly jumping from 5.x to 6.0 is to reduce the amount of transition pain that most users and developers felt when switching from 4-STABLE to 5.x. 6.0 will feature improved performance and stability over 5.x, experimental PowerPC support, and many new WiFi/802.11 features. The 5.x series will continue for at least one more release this fall, and will then be supported by the security team for at least 2 years after that. We encourage everyone to give the 6.0-BETA snapshots a try and help us make it ready for production. We hope to release FreeBSD 6.0 by the end of August.
Thanks again to everyone who submitted reports, and thanks to Max Laier for running the show and putting the reports together. Enjoy reading!
The second annual BSDCan conference was well presented, well attended, and everyone went away with good stories to tell. If you know anything that attended, get them to tell you what they did, who they met with, and talks they listened to.
We had 197 people from 15 different countries. That's a strong turnout by any definition.
We'll be adding more people to the program committee for BSDCan 2006. This job involves prodding and poking people from your respective projects. You get them to submit papers. There are a lot of very interesting projects out there and not all of them submit a paper.
If you know someone doing interesting work, please let me know and urge them to start thinking about BSDCan 2006.
Progress towards integrating the BSD Installer for Google's Summer of Code is coming along nicely. The installation CD will boot to multi-user mode and run both the front and back ends. It can then partition a hard drive, install the base distribution and make the disk bootable.
The following new features have been added to FreshPorts:
The TCP code in FreeBSD has evolved significantly since the fork from 4.4BSD-Lite2 in 1994 primarily due to new features and refinements of the TCP specifications.
The TCP code now needs a general overhaul, streamlining and cleanup to make it easily comprehensible, maintainable and extensible again. In addition there are many little optimizations that can be done during such an operation, propelling FreeBSD back at the top of the best performing TCP/IP stacks again, a position it has held for the longest time in the 90's.
This overhaul is a very involved and delicate matter and needs extensive formal and actual testing to ensure no regressions compared to the current code. The effort needed for this work is about three man-month of fully focused and dedicated time. To get it done I need funding to take time off my day job and to dedicate me to FreeBSD work much the way PHK did with his buffer cache and vnode rework projects.
I've got the opportunity to work up to three man-month exclusively full-time on FreeBSD during the second half of 2005. That means up to 720 hours of full-steam coding (at 60 hours/week)! I will work as much time as the fundraise provides.
I need to raise enough money for each month from donations from the FreeBSD community to cover my fixed cost of living, office and associated overhead. These fixed cost amount to US$6,300/month (EUR5,200 or CHF8,000). Yes, Switzerland is not the cheapest place to live. :)
A detailed description of the tasks involved and the code I will write is on my FreeBSD website; Follow the link above.
Modern CPU's can only perform to their maximum if their working code is in fast L1-3 cache memory instead of the bulk main memory. All of today's CPU's support certain L1-3 cache prefetching instructions which cause data to be retrieved from main memory to the cache ahead of the time that it is already in place when it is eventually accessed by the CPU.
CPU Cache Prefetching however is not a silver bullet and has to be used with extreme care and only in very specific places to be beneficial. Incorrect usage can lead to massive cache pollution and a drop in effective performance. Correct and very carefully usage on the other can lead to drastic performance increases in common operations.
In the linked patch CPU cache prefetching has been used to prefetch the packet header (OSI layer 2 to 4) into the CPU caches right after entering into the network stack. This avoids a complete CPU stall on the first access to the packet header because packets get DMA'd into main memory and thus never are already pre-cache in the CPU caches. A second use in the patch is in the TCP input code to prefetch the entire struct tcpcb which is very large and used with a very high probability. Use in both of these places show a very significant performance gain but not yet fully quantified.
The final patch will include documentation and a guide to evaluate and assess the use of CPU cache prefetch instructions in the kernel.
Currently TCP segment reassembly is implemented as a linked list of segments. With today's high bandwidth links and large bandwidth*delay products this doesn't scale and perform well.
The rewrite optimizes a large number of operational aspects of the segments reassembly process. For example it is very likely that the just arrived segment attaches to the end of the reassembly queue, so we check that first. Second we check if it is the missing segment or alternatively attaches to the start of the reassembly queue. Third consecutive segments are merged together (logically) and are skipped over in one jump for linear searches instead of each segment at a time.
Further optimizations prototyped merge consecutive segments on the mbuf level instead of only logically. This is expected to give another significant performance gain. The new reassembly queue is tracking all holes in the queue and it may be beneficial to integrate this with the scratch pad of SACK in the future.
Andrew Gallatin was able to get 3.7Gb/sec TCP performance on dual-2Gbit Myrinet cards with severe packet reordering (due to a firmware bug) with the new TCP reassembly code. See second link.
The old TTCP according to RFC1644 was insecure, intrusive, complicated and has been removed from FreeBSD >= 5.3. Although the idea and semantics behind it are still sound and valid.
The rewrite uses a much easier and more secure system with 24bit long client and server cookies which are transported in the TCP options. Client cookies protect against various kinds of blind injection attacks and can be used as well to generally secure TCP sessions (for BGP for example). Server cookies are only exchanged during the SYN-SYN/ACK phase and allow a server to ensure that it has communicated with this particular client before. The first connection is always performing a 3WHS and assigning a server cookie to a client. Subsequent connections can send the cookie back to the server and short-cut the 3WHS to SYN->OPEN on the server.
TTCPv2 is fully configurable per-socket via the setsockopt() system call. Clients and server not capable of TTCPv2 remain fully compatible and just continue using the normal 3WHS without any delay or other complications.
Work on implementing TTCPv2 is done to 90% and expected to be available by early February 2005. Writing the implementation specification (RFC Draft) has just started.
The goal of this project is to review the network interface API and try to remove references to kernel-only data structures by removing the use of libkvm and instead rely on other interfaces to provide information. If there are no adequate interfaces, they would be created.
Currently netstat is being reviewed and parts of it have been modified to use sysctl rather than libkvm to provide the information.
A big thank you to Brooks Davis for mentoring :-)
In May 2005, Remko Lodder joined the FreeBSD Security Team, followed by Christian S.J. Peron in July 2005. In the same time period, Gregory Shapiro and Josef El-Rayes resigned from the team in order to devote their time to other projects. The current Security Team membership is published on the web site.
In the time since the last FreeBSD status report, twelve security advisories have been issued concerning problems in the base system of FreeBSD; of these, six problems were in "contributed" code, while five problems were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and the Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 97 new entries have been added, bringing the total up to 519.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.10, FreeBSD 4.11, FreeBSD 5.3, and FreeBSD 5.4. Their respective End of Life dates are listed on the web site.
Currently trying to restart bits of the project. Cleaning up the p4 branch. Recently more people have volunteered to help as well. Brooks Davis has completed removing the ifnet from the softc.
The FreeBSD Dutch Documentation Project is a ongoing project in translating the english documentation to the Dutch language. Currently we are almost done with the FreeBSD Handbook. Finishing the Handbook is our first priority, and we could use your help. Please contact Siebrand or myself if you want to helpout. After the handbook we will focus on other documents as well, so feel free to help us there as well
We are currently working on an updated implementation of Juan Navarro's transparent support for superpages in FreeBSD.
The idea is to take advantage of the architectural support for big memory pages (superpages) by using a reservation mechanism allowing us to transparently promote groups of base pages into superpages and demote superpages into several smaller superpages or base pages.
The advantage of using superpages vs. base pages is to significantly improve the TLB coverage of the physical memory, thus improving the peformance by reducing the number of TLB misses.
The modification of the FreeBSD kernel that we are working on involves the replacement of the current list based page allocation mechanism with a system using a buddy allocator to reserve groups of pages for a memory object. The promotion and demotion of the pages occur directly within the pmap module.
The former implementation was supporting the alpha and IA64 architectures. We are adding the support for amd64. We currently have an almost complete implementation. Once completed we will make a performance study with a particular emphasis on TLB and cache misses.
A lot of bugs were fixed in preparation for the 6.0 release. 6.0 will be the first release to include full WPA support (both supplicant and authenticator).
A presentation on the forthcoming multi-bss support was given at BSDCan 2005. The slides from the talk are available at http://www.freebsd.org/~sam/BSDCan2005.pdf. The plan is to commit this work to HEAD after 6.0 is released which means the first release that will have it is 7.0.
My Summer of Code project is reengineering and rewrite of FreeSBIE toolkit, in order to include it in the source tree. Let's call it FreeSBIE 2
Before being accepted, I worked hard on the FreeSBIE 1 toolkit to make it more flexible. It now supports amd64 and PowerPC architecture. The built filesystem can now boot from almost every media, from DVD to compact flash or hard disk. Also on i386 it is now possible to include the BSD Installer on the livefs. We've received reports that our toolkit is successfully used for the install CD of pfSense and PC-BSD projects.
My future goals are to make the toolkit even more flexible, capable to build embedded images (like nanoBSD) or big Live-DVD systems, depending on user's choice, to support all the architectures supported by FreeBSD and to write a set of tools for making a netboot server with a FreeSBIE image.
Florent Thoumie has updated the massively out-of-date platform page. Work continues to creating a 6.0 release of the PowerPC port.
GGATE is a mechanism for exporting storage devices over the network. It was reimplemented to be much faster and to handle network failures better. The ggatec uses two threads now: sendtd, which takes I/O request from the kernel and sends it to ggated; recvtd, which receives finished requests and forwards them to the kernel. The ggated uses three threads: recvtd, which receives I/O requests from ggatec; disktd, which executes I/O requests (reads or writes data); sendtd, which sends finished requests to ggatec. The new ggate has been committed to 6.x.
The work was sponsored by Wheel Sp. z o.o.
The schedule (as stated on the wiki page) is honoured, which means that the development has started, but there's not enough code for testing. Many details have been thought-out and the development is ongoing.
Google has generously funded 19 students to spend the summer working on FreeBSD related projects. Each student is working with one or more mentors to learn about how open source software development is done with FreeBSD. This development work is happening in the Perforce repository as //depot/projects/soc2005. This tree will soon be exported via CVSup -- check the Wiki for more information.
With the releases of FreeBSD 5.3 and 5.4, FreeBSD has been moving away from "old-style" vinum towards GEOM-enabled gvinum for logical volume management. While gvinum is a mostly feature-complete replacement for vinum, it does not implement the 'move' or 'rename' verbs which are rather useful when reorganizing one's volume layout, the alternative being a tedious process of deleting and recreating subdisks, plexes, or volumes. Additionally, gvinum is nearly completely undocumented, which contributes to the perception of gvinum as an unfinished project.
I'm working on implementing 'move' (being able to move a subdisk from one drive to another) and 'rename' (being able to rename an subdisk, plex, volume, or drive), as well as on documentation for gvinum.
So far, I've come up with a plan of attack with le@ and phk@, and implemented the bulk of the userland code for gvinum 'move' and 'rename'. Still to come are the kernel-side code and documentation.
This was committed to current on 5 Jun 2005 and will first appear in the 6.0 release, thanks to everyone who tested. Recent improvements include:
There is ongoing work to bring in some of the advanced features from OpenBSD such as IPSec bridging. People are encouraged to use if_bridge and report any problems to the mailing lists.
At the developer summit before BSDCan it was decided to remove IP6FW from the tree as it has a couple of problems. The most pressing one is the lack of synchronization and thus the need for debug.mpsafenet=0. As a replacement Brooks Davis has imported patches to teach the existing and well-locked IPFW2 code about IPv6.
Since the initial import I have added some features required to manage IPv4 and IPv6 in a single ruleset. I have also extended existing opcodes to work with IPv6. There are, however, still some opcodes that do not work with IPv6 and most of the more exotic ones haven't been tested. As long as IPFW2+v6 does not provide enough functionality and stability to work as a drop-in replacement for IP6FW, we won't remove IP6FW.
In order to get the new code to that point we really need more testers with real world IPv6 deployment and interest in IPFW+v6. The lack thereof (I haven't received a single answer on my requests to various FreeBSD mailing lists) has made it hard to progress.
So far progress has been slow, the autoconf build system has been removed from all of the launchd(8) code, and launchctl(1) is building and semi-functional on FreeBSD-CURRENT (i.e. CoreFoundation hooks have been removed).
I'm currently working on porting "liblaunch" which is the core backend to both launchd(8) (the actual daemon) and launchctl(1), there are some mach/xnu specific hooks and calls that need to be remove and either reimplemented or worked around.
We're also waiting on a response from Apple on a possible BSD-licensed version of the code (it's currently under the APSL) Progress is slow, but steady.
This project is an attempt to clean up handling of network interfaces in order to allow interfaces to be removed reliably. Current problems include panics if Dummynet is delaying packets to an interface when it is removed.
I have removed struct ifnet's and layer two common structures from device driver structures. This will eventually allow them to be managed properly upon device removal. This code has been committed and will appear in 6.0. Popular drivers have generally been fixed, but more testing is needed.
The OpenBSD rewrite of dhclient has been imported, replacing the ISC dhclient. The OpenBSD client provides better support for roaming on wireless networks and a simpler model of operation. Instead of a single dhclient process per system, there is one per network interface. This instance automatically goes away in the even of link loss and is restarted via devd when link is reacquired. To support this change, many aspects of the network interface configuration process were overhauled.
The current code works well in most circumstances, but more testing and polishing is needed.
I've sent the patch to jinmei@isl.rdc.toshiba.co.jp @KAME for review. I'm still waiting for feedback from Andre. There hasn't been any major change since the last report. I've kept the code in sync with CURRENT. Gleb has created a separate P4 branch and has been helping out on the locking side. Gleb is also helping out on the testing front.
The nsswitch / caching daemon project is being developed within the Google's Summer Of Code program. The first goal of this project is to implement a set of patches to extend the use of nsswitch subsystem. The second goal is the development of the caching library and daemon to add the caching ability to the nsswitch.
Currently services, protocols, rpc and openssh patches are finished. Support for services, services_compat, rpc, protocols, and ssh_host_keys databases is added with 'files', 'nis' and 'compat' (for services) sources possible. The nsswitch-friendly openssh port is almost completed.
We will have pf as of OpenBSD 3.7 for RELENG_6. Import has been completed in early May and FreeBSD release 6.0 will ship with it.
A few serious issues with pfsync on SMP have been discovered since CARP is around and more and more people use it on big iron. Everything that has been discovered is fixed in HEAD and (if applicable) MFCed back to RELENG_5. Some functional changes are undergoing testing right now and will be MFCed in the coming days.
With the import of if_bridge from Net/OpenBSD we finally have a bridge implementation that allows for stateful filtering as well as IPv6 filtering. Please see the respective report.
Modern CPUs have on-chip performance monitoring counters (PMCs) that may be used to count low-level hardware events like instruction retirals, branch mispredictions, and cache misses. PMC architectures and capabilities vary between CPU vendors and between CPU generations from the same vendor, making the creation of portable applications difficult. This project implements a cross-platform PMC management API for applications, and implements the infrastructure to "virtualize" and manage these PMCs. The creation of performance analysis tools that use this infrastructure is also part of the project's goals.
Work since the last status report:
Future work:
My SoC project is about improving libalias and integrating it with ipfw2, adding nat support into the firewall. Till now I ported libalias (as a kld) and ng_nat to 4.x and 5.x branches, and I've already a first working patchset that adds 'nat' action into ipfw. Next step will be to add a complete syntax to ipfw that will let us manipulate libalias operations, much like we already do with queue and pipes for dummynet. In the end the entire work will compile and work out of the box for 4.x, 5.x and 6.x. More details about the project and its status are available on wiki page.
Since Google's "Summer of Code" resulted in a lot of interest in open projects, I'm in the process of compiling a list of nice projects for volunteers. Unlike Google's SoC those projects aren't backed with money (but this doesn't means nobody is allowed to sponsor one of those projects), so we can only guarantee the social aspects (some "Thank you!" and "That's great!" messages). So far the list has several entries where the difficulty ranges from "someone just has to sit down and spend some time on it" up to "we need a guru for this".
FreeBSD lacks a way to remove old/outdated files and directories in the basesystem. I have a patch which removes obsolete files in a safe way (interactively, since only the administrator really knows if there's a need to keep an old file or not; there's a switch for batch-processing). This feature may or may not be available for 6.0-RELEASE, depending on the decision from the Release Engineering team.
Intel released version 9 of its C/C++ compiler. Work to port the x86 version to FreeBSD is in progress as time permits. Porting the EM64T (amd64) version is on the TODO list too, but is subject to enough free time and access to appropriate hardware.
The cleanup/streamlining and the possibility of overriding the default Linux base as reported in the last report happened without major problems. Work on the open tasks hasn't started yet, but is scheduled to start "soon". If a volunteer wants to spend some hours on one of the open tasks, he should tell it on the emulation mailinglist.
The VM subsystem has code to reduce the amount of cache collisions of VM pages. Currently this code needs to be tuned with a kernel option. I have a patch which changes this to auto-tuning at boot time. The auto-tuning is MI, the cache size detection is MD. Cache size detection is currently available for x86/amd64 (on other systems it uses default values).
As part of the Google Summer of Code, I'm working on improvements to the FreeBSD website (including a proposed website redesign). My mentor for this project is Murray Stokely.
filesystem. Journaling helps ensure the filesystem's integrity should the system crash. Journaling eliminates the need for fsck'ing a filesystem, as the filesystem is never in an inconsistent state (barring hardware failure). This implementation is inspired by Darwin's HFS+ filesystem and the SGI XFS filesystem. This is a Summer of Code project, with Scott Long as the mentor and Brian Wilson as the developer/mentee. Currently this project is still in the early stages, but will be in a usable state by September 1 (the Google Summer of Code completion date).
Thanks Robert Watson and Scott Long for their kind help.
FreeBSD's VFS layer has been fine grain locked along with the FFS filesystem for the FreeBSD 6.0 release. The locking has been underway for several years, with the project really picking up over the last 6 months thanks largely to sponsorship provided by Isilon Systems, Inc. a leading vendor of clustered storage systems. The project has entered a stabilization phase, with a few bugs being reported in extreme circumstances while the majority of users have seen no problems. Tests on a 8 and 16 way machines yield reasonable parallelization, however, it will be beneficial to do lock contention analysis once things are fully stable.
For those interested in technical details, there have been a few relatively significant changes with vnode life-cycle management. Vnode reference counting and recycling is now no longer an ad-hoc process involving a variety of flags, a use count and the hold count. A single hold count is used to track all vnode references and a destroyed vnode is freed in the context of the caller when the last ref is lost. The old system would never reclaim memory used by vnodes and also had pathlogical behavior with unreferenced vnode caching under pressure. The new system is much simpler than the old one, however, callers are now required to vhold a vnode that they lock directly without going through vget to prevent it from being recycled while they are waiting on a lock. Relying on 'location stable storage', which is a more strict version of 'type stable storage' is no longer a valid approach.
Some other side effects include a much simpler and faster nullfs implementation, an improved buf daemon flushing algorithm which eliminated high latency that caused audio skipping, and a lots of minor cleanups and debugging aids.
The fourth European BSD conference in Basel, Switzerland is a great opportunity to present new ideas to the community and to meet some of the developers behind the different BSDs.
The two day conference program (Nov 26 and 27) will be - complemented by a tutorial day preceeding the conference (Nov + complemented by a tutorial day preceding the conference (Nov 25).
The program committee is looking for tutorial and paper submissions. For details, please see: The call for papers online.
Significant work has occurred over the last few months relating to the SMP network stack work. A few of the highlights are covered here at a high level:
In most cases, these changes will appear in FreeBSD 6.0-RELEASE; some have been, or will be, merged to FreeBSD 5.x.
On-going tasks include:
The TrustedBSD Project has released a new snapshot of "SEBSD", a port of NSA's SELinux FLASK and Type Enforcement implementation to FreeBSD based on a late 2005 FreeBSD 6.x snapshot. The SEBSD distribution has now been updated in Perforce to a recent 6.x snapshot, and a new distribution will be made available in the near future.
Work has been performed to merge additional dependencies for SEBSD back into the base FreeBSD tree, including most recently, changes to devfs, and System V and POSIX IPC.
In the past few months, significant work has been done relating to the TrustedBSD audit implementation, including preparatory work to merge audit into the FreeBSD CVS repository for FreeBSD 6.x. In particular:
The hope is still to provide audit as "experimental" in 6.0; the primary blocking factor is our awaiting relicensing of the last remaining audit files from Apple's APSL license to BSDL so that they can be included in the FreeBSD kernel. This is anticipated to complete in the near future. Once this is done, the changes can be merged to CVS, and then MFC'd to RELENG_6. If this is not complete by 6.0-RELEASE, the work will be merged shortly after the release, as all ABI-sensitive data structures have been updated as needed.
libmemstat(3) provides a user space library API to monitor kernel memory allocators, currently uma(9) and malloc(9), with the following benefits:
libmemstat(3) and the appropriate allocator changes for uma(9) and malloc(9) are currently in HEAD (7-CURRENT), and MFC has been approved to RELENG_6 for inclusion in 6.0-RELEASE. These changes may also be backported to 5.x.
Sample applications include memstat(8), an allocator-independent statistics viewing tool, memtop(8), which provides a top(1)-like interface for monitoring kernel memory use and active memory types. None of these are "pretty".
netstat -mb has also been updated to use libmemstat(3) to track network memory use using uma(9), rather than the less reliable mbuf allocator statistics interface. As a result, the statistics are now more reliable on SMP systems (this corrects the bug in which mbuf statistics sometimes "leaked", even though memory didn't), and more informative (cache information is now displayed, as well as mbuf tag information).
After a long, exhausting, yet very productive third quarter of 2005 FreeBSD 6.0 has been released. Many activities were put into the background in order to make this release the success it has become.
-Nontheless, we received a tremendous amount of reports covering +
Nonetheless, we received a tremendous amount of reports covering various projects that either found their way into FreeBSD 6.0 already or have started to develop in, what is now known as 7-CURRENT. The EuroBSDCon and the Developer Summit in Basel next week will be a good opportunity to help some of the ideas herein to take off.
Last round we had the pleasure to introduce our accepted Google Summer of Code projects. Now, that the summer is over, we are even more pleased to include reports about the outcome of these projects. Some already found their way into the tree or the general public otherwise - most ocularly the new webdesign.
Unfortunately, this publication has been delayed for various reasons - the before mentioned release of 6.0 being one of them. Thus, some of the reports might no longer be as up to date as they were when we received them and we encourage you - even more this time - to also visit the weblinks to get more recent information.
Thanks again to everyone who submitted reports, and our sincere apologies for running late this time.
A new netgraph-to-zaptel module that allows to use E1(ISDN PRI) WAN adapters as an interface card for open source PBX - Asterisk. All you need is an adapter that able to work in raw phone mode (like Cronyx Tau-PCI/2E1), eq. without HDLC-like framing and that has support of Netgraph.
We are in the process of recruiting new members for the program committee. If you would like to volunteer before you are recruited, please contact me.
The dates for 2006 have been announced: May 12-13, 2006. The venue will be the same as previous events: University of Ottawa. The prices will not increase from 2005.
Please start thinking about your papers. The call for papers will go out soon.
A great deal of work has gone into the Ports Collection since the last report in April, much of it behind-the-scenes.
As this report was being written, the 6.0 release was ongoing. Due to the amount of time that it has taken to get 6.0 through the beta process and into RC, we have been in ports freeze or slush for more than two months. Unfortunately this has held back needed work on the ports infrastructure.
The last major update to bsd.port.mk, in early May, was coordinated by Kirill Ponomarew added a number of new features and closed 15 PRs. Another similar set of changes has been tested and is ready for commit after release.
portmgr welcomed two new members to its team: Erwin Lansing (who had previously served as secretary, a role in which he is continuing) and Clement Laforet. Clement is interested in speeding up the adoption of new changes into the infrastructure, an item I'm sure that that everyone can support. He promises to bring some fresh ideas to bear on this, including the revitalization of devel/portmk as a testing ground for new changes to bsd.port.mk in which the larger community can help test changes.
The unfetchable distfile survey, which had been non-functional for quite some time, was revitalized by Bill Fenner, with many new pages of analysis added to it. Work is still ongoing. As a result of this analysis, Bill and Mark Linimon eliminated nearly 100 lines of bogus or outdated sites from bsd.sites.mk alone. They are continuing to work through many other sites and ports as successive iterations of the survey reveal more dimensions to the problem. We still need more help from the larger community (see below).
Edwin Groothius has instituted a similar but slightly different survey. His program attempts to visit each listed mastersite for each distfile and determine whether or not a newer version might be available. The results are stored in a database. This is helping to automate a function that had been left up to individual maintainers to look through numerous websites to try to find these updates. The survey has been hugely (if not universally) popular. Already, dozens of port updates have been committed as a direct result of this service.
In addition, portsmon, which had been down due to a machine change, was moved to portsmon.FreeBSD.org and updated during this time. Many thanks to Erwin Lansing for providing the loan of this machine, and Will Andrews for having provided the loan of the previous incarnation.
Both of the above surveys are now generating periodic email to ports maintainers advising them of problems. This is in addition to recurring email from portsmon. The surveys allow individual maintainers to ask to receive no further email. portsmon does not currently have this but it needs to be added. Although we have no doubt the mail can in some cases be annoying (especially given the fact that there will inevitably be some false positives), the fact is that these emails have had a direct impact on the quality on the ports. We ask for patience from the community while each of us continues to fine-tune the algorithms controlling what email is generated. (Because of the number of emails these systems generate, it is impossible to go over every one individually for a sanity check).
As a result of bounces from the above email, we have also been resetting maintainers who have become unreachable.
Pav Lucistnik has done a great deal of work on the Porter's Handbook, including some much needed reshuffling and cleanup. Expanded sections include Apache and PHP; Configure Scripts; Dealing With Shared Libraries; Dealing With User Configuration Files; Handling Empty Directories; Python; and Ruby. In addition, Edwin Groothius has contributed a section on OPTIONS, and numerous other sections have been improved by good suggestions from various other contributors.
A new article, "Maintaining and contributing to the FreeBSD Ports Collection", has been prepared by Sam Lawrance and has been reviewed and is ready for commit. This document attempts to codify the rights and responsibilities of ports maintainers, which until now had merely been "community lore" as discussed on various mailing lists.
We continue to add new committers regularly, 8 since the last report.
The ports collection now contains over 13,500 ports. This is an increase of over 750 since the last report in April.
The project met all the scheduled goals, and following are the new features implemented in libalias:
I'm in the process of adding personalized newsfeeds to the website. For each of your Watch Lists, you will also have a news feed just for that watch list. Any commit to any port in your watch list will turn up on your newsfeed. This fantastic new feature is available now for your RSS pleasure at the BETA site . I've also been doing some work in the area of supporting multiple platforms and architectures. This will allow FreshPorts to correctly report that a port is broken, for example, on i386, but not the other platforms. This feature will take note of BROKEN, FORBIDDEN, and IGNORE for the following architectures:
Fuse for FreeBSD is the outcome of my "ssh based networking filesystem for FreeBSD" SoC project.
The kernel interface for the comprehensive userspace filesystem API provided by the ( Fuse project ) has been implemented for FreeBSD (6.x and 7.x), under the BSD license. This has the benefit of opening up the possibility of porting the rich collection of Fuse based filesystems to FreeBSD.
Now it's ready for consumption by a broader audience. The sysutils/fusefs-kmod , sysutils/fusefs-libs , sysutils/fusefs-sshfs ports can be expected to be integrated into the FreeBSD ports tree in the next few days (the ports were created and are maintained by Anish Mistry, and Simon Barner's careful review also helps a lot).
Support for moving and renaming objects in gvinum was completed at the end of August 2005. All gvinum objects (drives, subdisks, plexes, and volumes) can be renamed, and subdisks can be moved from drive to drive. Also, a man page for gvinum was created.
This summer, we've had the pleasure of having two Google Summer of Code students hacking on our SNMP monitoring machinery. Victor worked on implementing the Host Resources, TCP and UDP MIBs in bsnmpd while Shteryana started on client-side SNMP tools.
With these modules and tools, a FreeBSD installation can be monitored without having to install any (heavy!) third-party tools.
While the modules and the tools currently in Perforce are generally functional, they still need some tidying up (style(9)) and testing before they can be committed to CVS.
At the time of this writing, the Hostres MIB is pretty much commit-ready in Perforce (//depot/user/philip/bsnmp/...), the other modules and tools live in //depot/projects/soc2005/bsnmp/... They'll be branched for tidying up and committing "Real Soon Now"[tm]
The dependencies to kernel-only datastructures in netstat (ifnet, etc.) have been removed almost completely (AppleTalk and IPX still needs work). In order to remove the dependencies, the debugging features of netstat had to be removed. However, a project to create a generic, modular 'data structure' examination tool is ongoing, and the debugging features factor out of netstat have been migrated to this tool.
Scott has been working on inserting journalling hooks into the ufs and ffs filesystem code. Brian has been balancing school and redesigning various things that were deemed necessary to update during the end of the actual SoC project.
pfSense is a m0n0wall derived operating system platform with radically different goals such as using Packet Filter, FreeBSD 6, ALTQ for excellent packet queueing and finally an integrated package management system for extending the environment with new features.
Work continues to stabilize pfSense in preparation for the FreeBSD 6 release. Once FreeBSD 6 is released pfSense will enter the final beta and release candidate phases in preparation for the 1.0 release.
In short, launchd can run perfectly fine on FreeBSD, and combined with launchctl, it can be used to manage daemons through the launchctl(1) interface. Jobs can be added and managed two ways as of yet from launchctl(1). Using zarzycki@'s experimental "submit" command within launchctl(1) or by using my lame/rudimentary/etc "launcher" format (launchd/launchers/*.launch) which uses property(3) to parse out three simple, and important details. The program label, path, and any program flags. Using the "load" command, one can load the data into launchctl(1) and then start the processes with the..."start" command. Jobs can be removed/stopped with the "remove" command. The "limit" command still throws launchctl(1) into an infinite loop, and yes, I plan on fixing this.
There are some things that need to be fixed, first off, some sort of boot time integration, whether as an init-replacement (i.e. PID 1, a la Mac OS X) or as the first thing started from init, that kicks all rcng things off. Along with, more importantly, a plist parser, so we can have full compatibility with Mac OS X's launchd via Core Foundation.
I'm also trying to get launchd(8) relicensed with the BSD license, as opposed to the APSL, anybody with tips, or methods for achieving this goal, contact me at tyler@tamu.edu
As of 26th July 2005, it is possible to run FreeBSD on your Xbox with minor patching effort. The framebuffer has initial support; The USB ports, IDE- and audio controllers are fully supported; the only part severely lacking now is the lack of support for the NForce Ethernet controller.
Currently, efforts are focussing on eliminating the XBOX kernel option and make the port self-detecting; this means the x86 and xbox kernels will be identical. The goal is to provide native xbox support in 7-CURRENT.
-Futhermore, a porting effort is planned from Linux' GPL-ed +
Furthermore, a porting effort is planned from Linux' GPL-ed forcedeth.c; not only the Xbox port will benefit from this but also all NForce motherboard owners. The resulting driver could be kldload-ed to keep the kernel GPL-free.
Ggtrace is "GEOM gate tracer", utility to track I/O requests on a storage device on FreeBSD. It uses the ggate facility of FreeBSD to attach to a file or device and produces a device that can be used for any I/O, including hosting filesystems.
I/O requests are presented in the form of a moving histogram that can be used to discern which parts of the storage device are used most often. One use of ggtrace is to analyze how filesystems arrange and access data on storage devices.
The project is working and usable only on the RELENG_6 branch.
Gjournal provides GEOM-level journaling and COW capabilities to storage devices. Unfortunately, it cannot be used as a substitute for filesystem journaling (fsck is still needed when gjournal device is used to host filesystems). Development has slowed down, and the existing code needs much more testing. If there is continued interest in it, I'll probably split the functionalities into two projects, one handling COW and one handling the journaling, in order to make the code cleaner.
The fundraise has been very successful and I want to thank everyone who has pledged their support and tipped the jar. The full amount plus a little bit more has been raised in a very short timeframe. More information on the exact amounts and their sponsors can be found at the first link.
Due to the extended (and unexpected long) code freeze for the release process of FreeBSD 6.0 (which is very high quality btw.) I've decided to push back on working full time until the freeze is lifted. So far I've done some work in the mbuf handling area and some other netinet cleanups in my local repository.
Once FreeBSD 6.0 is released I resume my work on this project and many changes and optimizations, as described in the first and second link, will go into into FreeBSD-current.
The TODO list for volunteers (see the last report for more) is now under review by some doc@ people.
As of September 27, the old bridge(4) implementation has been removed from HEAD and will not be part of FreeBSD 7 and later. FreeBSD 6 will serve as transition period. The full functional replacement if_bridge(4) is now available in FreeBSD 5 (not yet part of 5.4 however), FreeBSD 6 and -CURRENT. Any problems should be reported to Andrew Thompson, who is maintaining if_bridge in FreeBSD.
Due to some good recent commit and cleanup work by both Alexander Leidinger and Craig Rodrigues, the number of base-system PRs has stabilized somewhat. The number of incoming ports PRs continues to accelerate but except during freezes the ports committers have been battling to commit them as quickly as they come in. (The graphs very clearly show where the freezes are.) The net result is that we are holding our own but it takes a great deal of (mostly unheralded) effort to do so. Thanks are due to a large number of individuals who are doing this ongoing work.
There is ongoing work to ask committers who have had PRs assigned to them for a significant period of time, whether they are still interested in pursuing them or whether they should instead be reassigned to the pool. This is being done to try to get as many PRs 'unstuck' as possible to try to help improve our users' perceptions of the project.
As an experiment, Mark Linimon has been adding 'tags' to many of the kern and bin PRs, including such things as '[nfs]', '[if_em]', and so forth. The idea is to try allow searching and browsing based on these terms so that committers will find it easier to work with our current PR database. At the moment this is in the experimental stage, although it is possible for committers to work with them from the command line on systems with a database installed via query-pr(1).
This project is an attempt to clean up handling of network interfaces in order to allow interfaces to be removed reliably. Current problems include panics if Dummynet is delaying packets to an interface when it is removed.
I have removed struct ifnet's and layer two common structures from device driver structures. This will eventually allow them to be managed properly upon device removal. This code has been committed and will appear in 6.0. Popular drivers continue to be fixed. jhb's locking work has identified and corrected many issues. rwatson has also committed cleanups to the multicast code which fixed some issues in this area.
The OpenBSD rewrite of dhclient has been imported, replacing the ISC dhclient. The OpenBSD client provides better support for roaming on wireless networks and a simpler model of operation. Instead of a single dhclient process per system, there is one per network interface. This instance automatically goes away in the even of link loss and is restarted via devd when link is reacquired. To support this change, many aspects of the network interface configuration process were overhauled.
The current code works well in most circumstances, but more testing and polishing is needed. A few bugs are being tracked, but most of them are edge cases.
Work on further interface configuration enhancements is underway for FreeBSD 7.0.
The fourth European BSD conference in Basel, Switzerland is a great opportunity to present new ideas to the community and to meet some of the developers behind the different BSDs.
The two day conference program (Nov 26 and 27) will be - complemented by a tutorial day preceeding the conference (Nov + complemented by a tutorial day preceding the conference (Nov 25).
The FreeBSD developers will hold a DevSummit on Nov 24 and 25, so several developers will be at the conference.
The program is available for Saturday and Sunday providing very interesting FreeBSD talks and topics.
Today more than 160 people from 25 countries have registered for the conference.
Since our last status report, we have added a new member to the team: Jean-Yves Lefort (jylefort). We have even spiced up our contact page with pictures of ourselves and in some cases, a cute hippo. And our very own Adam Weinberger (adamw) has been made a GNOME Project committer heading up the Canadian English translation project.
We have finished the port GNOME 2.12 to FreeBSD. However, due to the ports slush in preparation for 6.0-RELEASE, the update has not been merged into the official ports tree. If people are eager to try out GNOME 2.12 while waiting for the ports tree to fully thaw, we have instructions on our website. GNOME 2.12 will be the first FreeBSD GNOME release not to include support for FreeBSD 4.X. While 4.X is still a very viable release for servers, it lacks many of the features needed for a Desktop Environment such as GNOME. We do plan to continue support of the GNOME development platform on 4.X, however. This includes Glib, GTK+, libgnome, etc. A new porting component will be introduced with GNOME 2.12 called, ``ltverhack''. This will help with future upgrades by keeping shared library versions from needlessly changing.
The FreeBSD GNOME Project is also committed to providing our users with a solid package experience. To that end, we have extended our Tinderbox to build amd64 packages for all supported versions of FreeBSD for both the production and development releases of the GNOME Desktop. The development packages are even built with debugging symbols to better help with reporting problems.
The project has been following the 6.0 release schedule by producing BETA-* builds and is now up to the RC1 build.
Dario Freni successfully built a FreeSBIE/ppc iso for his Summer-of-code project.
This iSCSI kernel module and its companion control program, are still under development, but the main parts seem to be working. A second round of public tests has started.
The current version of the test suite took form in the beginning of the year after discussions with Jeff Roberson, during a long period of testing Jeff's VFS SMP work.
At that time, Daniel Seuffert donated a Thunder 7500 motherboard complete with CPUs, RAM and coolers. This allowed me to do some serious SMP testing.
Mid July Murray Stokely suggested adding a link from the 6.0 todo web page to the Stress Test Status Page. At that time there were a few reoccurring panics that made it hard to test the kernel for other problems. Numerous people put a lot of hard work in fixing the panics and livelocks found during the next months. At the same time others stepped in and ran the test suite on their own hardware, thus increasing the focus on kernel stability.
As of 6.0, the kernel stress test suite cannot panic the kernel.
The nsswitch / caching daemon project was developed within the Google's Summer Of Code program. Almost all goals of the project were achieved. Thanks to Brooks Davis and Jacques Vidrine, who were my mentors and greatly helped me.
Nsswitch subsystem was extended to support new sources (services, protocols, rpc, openssh and GT4). The testing of the Globus Grid Toolkit 4 patch (which adds support for nsswitch to GT4) is still to be done. For nsswitch to support caching, the caching daemon was implemented on top of the caching library, which was also developed during the SoC. The current version of the daemon uses simple nscd-like configuration file and seems to be stable. To complete the SoC project, the experimental version of libc with in-process caching enabled was made. It's benchmarking will be done in the nearest future.
There were some requests for caching daemon to be able to act like NSCD (to perform the actual nsswitch lookups by itself), so it was modified to support this feature. But current implementation has some restrictions and requires a lot of testing. Right now the final polishing is being made to the project's sources, so that they could be added to the CURRENT
The new website has gone live! Thanks to Emily Boyd for all her hard work. We still have a lot of work to do to integrate suggestions that have been made by users since we went live. The new CSS design makes it much easier to rapidly change the look and feel of the site, so it is easy to experiment. We're still looking for more HTML/CSS designers to help us improve the site.
Due to an email snafu, the June report was not submitted, so this report covers since the last 6 months.
Summary: The 16-bit part of NEWCARD has been greatly enhanced. In addition, power control has become interrupt driven. Some drivers make use of the new functionality.
The pccard layer now exports the CIS for each device that is present, even if there's no driver for the card or parts of the card.
The power up and reset sequence is now interrupt driven. This has eliminated many of the long pauses that the system used to experience after a card insertion. We can not play glitch-free audio while inserting or removing a card.
A number of additional cards are recognized by PC Card. In addition, drivers now can read the CIS for more information about the card. Drivers have been enhanced to read the CIS for MAC addresses and the like where appropriate.
The ed driver now attaches the mii bus of the AX88190 and AX88790 fast ethernet PC Card chips. This allows better status reporting and increased functionality for PHY chips that need some help. The ed driver also supports the Tamarack TC5299J chipset (including attaching its MII bus) now, the only open source OS that does so (TC5299J cards will work with other open source OS, but they won't report their status or attach a mii bus).
A number of bugs have been fixed in the pccard or cardbus drivers. Most of these changes have been merged into the forthcoming 6.0. Others will be merged after the release.
Futher improvements have been made to pfsync to make it behave +
Further improvements have been made to pfsync to make it behave well in SMP scenarios. All bug fixes have been MFCed to RELENG_5 where applicable. A couple of bugfixes and feature improvements have been imported via OpenBSD (originally suggested by FreeBSD users).
As described in the last report, FreeBSD 6.0 and future RELENG_6 releases will be based on OpenBSD 3.7. Newer code will be imported as soon as 6.0 has settled down a bit.
By the end of August I managed to modify the release building process to build a live CD that loads the front and backends. It could install all the distfiles, install the ports tree and had minimal support to install and uninstall packages.
Since the end of the Summer of Code I have worked to integrate the new Lua backend. This has been successful, with it now past the point of the BSDINSTALLER-BETA-1 release. It can install the distfiles but not the ports tree or packages yet.
This projects implements a kernel module (hwpmc(4)), an application programming interface (pmc(3)) and a few simple applications (pmcstat(8) and pmccontrol(8)) for measuring system performance using event monitoring hardware in modern CPUs.
The last three months have been spent in bug fixing and in tweaking the code. A few more minor features and loose ends remain to be taken care of. Once these are done, I hope to get started on a graphical performance analyser.
The FreeBSD kernel is powerful, but it still lacks some realtime POSIX facilities, for example, sigqueue. Most of the code is ready, and I am testing it.
New MIBs are implmented for the BSNMP agent:
During the "Summer of Code" I worked on csup (a rewrite of CVSup in C). It already supported checkout-mode, so my task was to implement support for cvs-mode. The biggest part of the project was to implement support for rcs-files. As "byproducts" I also wrote the necessary code to create nodes/hardlinks and to update files using the rsync-algorithm. For what I know, the code works fine, but errorhandling is practically inexistent.
Recently a lot of fixes, specially in handling format / rate conversion and general stability was committed to -current. This include fixes for most LOR's and new features (software volume handling for soundcards without volume handling in hardware and the possibility to switch to spdif).
A lot of effort was expended by Ariff (and other people) to come up with those improvements. For this reason Ariff was "punished" with a commit bit, so he is able to commit further improvements on his own.
This work is not integrated into 6.0-RELEASE because of some lose ends (see 'sndctl' below).
You can help by looking at sound related PR's in GNATS and making follow-up's which tell us if a problem still persists or if a PR can be closed because the bug is fixed. Also feel free to submit patches for anything on the TODO list below.
The Ports Tinderbox is a packaged system for building FreeBSD ports in a clean environment. It can be used to test new ports, updates to existing ports, or simply as a package building engine. Tinderbox uses the same underlying code that the official FreeBSD package build cluster, pointyhat, uses. So if a port builds under Tinderbox, it is guaranteed to build on pointyhat.
More and more FreeBSD committers and ports maintainers are starting to use Tinderbox. We just released version 2.1.0 which added much-requested PostgreSQL support as well as fixed many bugs. We expect a 2.1.1 release soon with some additional bug fixes.
With the 2.1.0 release of Tinderbox, we have branched the code base so that we can focus on larger features in our HEAD branch while still producing stable releases on a more frequent basis. The biggest new feature planned for Tinderbox 3.0 is clustering support which is being spearheaded by Ade Lovett (ade).
This report is about the rather quiet last quarter of 2005, with the release of FreeBSD 6.0 and the holiday season things evolved in - the background. Nontheless, most exciting projects hit the tree (or + the background. Nonetheless, most exciting projects hit the tree (or are going to very soon).
Upcoming events, such as the release of FreeBSD 6.1/5.5 and the third BSDCan conference with a big developer summit promise to provide a busier start in 2006. The foundation for upcoming development, however, are the projects that are described herein.
We hope that you find interesting projects to look at or work on. The next status report collection will be April 7 2006. We are looking forward to your report then.
Thanks again to everyone who submitted reports, and thanks to Brad Davis who stepped up for an extensive spelling and grammar review. Enjoy reading!
libc's malloc implementation has been replaced with an implementation that is designed to scale well for multi-threaded applications running on multi-processor systems. This is accomplished by creating multiple allocation arenas that are independent of each other, and permanently assigning threads to these arenas. In the common case, threads do not access the same allocator arena at the same time, which reduces contention and cache sloshing.
Single-threaded application performance is approximately equivalent to what it was with phkmalloc, but for multi-threaded applications that make heavy use of malloc, the performance difference can be huge (orders of magnitude).
As with phkmalloc, the new malloc implementation supports runtime configuration via the MALLOC_OPTIONS environment variable. See the malloc(3) manpage for details on supported options, as well as more information about the allocator's architecture.
The OpenBSD rewrite of dhclient has been imported, replacing the ISC dhclient. The OpenBSD client provides better support for roaming on wireless networks and a simpler model of operation. Instead of a single dhclient process per system, there is one per network interface. This instance automatically goes away in the even of link loss and is restarted via devd when link is reacquired. To support this change, many aspects of the network interface configuration process were overhauled.
Support for adding aliases to DHCP configured interfaces has been committed to CURRENT and will be merged before 6.1-RELEASE. Soon work will begin to merge changes from OpenBSD that have taken place since the initial import.
Work on further interface configuration enhancements is underway for FreeBSD 7.0.
Currently splitting out the rest of the PF_KEY data-structures from the key database. This will mean the user level applications and the kernel will not share datastructures and that they can, hopefully, advance on their own without being in lockstep.
The "TODO list for volunteers" is now committed as the "FreeBSD list of projects and ideas for volunteers". So far the interest in the list is high and some volunteers already took the opportunity to start tackling some of the entries.
Unfortunately the FreeBSD project does not have enough human resources to provide a technical contact for every entry. Interested volunteers should not be afraid to try to come up with a solution for an entry without a technical contact. The people on the hackers and current mailing list are typically very helpful regarding answering specific questions (as long as they know the answer...).
We are looking forward to hear about new ideas, people willing to be technical contacts for generic topics (e.g. USB) or specific entries (already existing or newly created), suggestions for existing entries or completion reports for (parts of) an entry.
Most of the latest KAME code has been merged to 7-current and 6-stable, to prepare for the project conclusion in March 2006. For the same reason, we moved some ports applications (security/racoon, net/pim6sd, net/pim6dd, net/dhcp6) from KAME to sourceforge.net.
Some of the items (e.g. IGMPv3/MLDv2, Mobile-IPv6/NEMO, SCTP, DCCP, ISATAP) are not merged yet from the latest KAME code for several reasons. Other projects will continue to merge their work.
A lot of changes have taken place in the sound system since the - last status report. They range from less hickups and distortion by + last status report. They range from less hiccups and distortion by disk accesses and/or driver bugs to new and improved features (software volume control implemented for soundcards which do not have hardware volume control). Additionally a new driver (snd_atiixp) has seen the light and a lot of problem reports were fixed.
Most of those changes and the changes mentioned in the previous status report are already merged to RELENG_6 and will be part of 6.1-RELEASE.
The experiment to add 'tags' to many of the kern and related PRs, including such things as '[nfs]', '[fxp]', and so forth, continues. In addition, PRs with patches have been more consistently tagged with '[patch]'. Two new periodic reports based on both functional tags and PRs with patches have been added, with the goal of making these PRs more visible.
During this time, the number of ports PRs briefly dipped below 500 -- a number not seen since late 2000, when there were 4000 ports instead of our new total of over 14,000 ports. This is due to the hard work of a large number of individuals, including pav, edwin, mnag, garga, and many others. Congratulations folks! Some of this was due to more aggressively committing PRs where the maintainer had not responded within the timeout period. Although controversial, this new policy seems to be succeeding in its goal of improving the Ports Collection.
A new file, ports/KNOBS, was added by ahze to help bring some order in the chaos that had been the OPTIONS namespace.
dougb has changed the way that rc.d works in -HEAD to work more like the base rc.d scripts. We are hoping that this change will make ports maintenance easier in the future. However, in the meantime a few bugs have been introduced (which we intend to have fixed by the time 6.1 is released). While this regression is unfortunate, it was decided that now was the best time to try to make this change rather than waiting for 7.0. We hope our users can be patient with us in the interim.
Work continues to improve the marcuscom ports tinderbox, with new features added by marcus, aDe, and edwin in particular. Several ports committers are now running their own copies to test ports changes.
The www.FreeBSD.org/ports page, and the portmgr web pages, were reworked as well.
We have added 4 new committers since the last report.
Based on the Kame-Shisa Mobile IPv6 Software for FreeBSD 5.4, we implemented the performance optimization "Early Binding Updates" and "Credit-Based Authorization". The combined optimizations facilitate significant reductions in handoff delay without compromising protocol security [1][2].
Optimizations to reduce handoff delays inherent in Mobile IPv6 Route Optimization as well as IPv6 router discovery, address configuration, and movement detection have so far been mostly considered on an individual basis. This document evaluates three integrated solutions for improved handoff experience in surroundings with different preconditions: reactive handoffs with unmodified routers, reactive handoffs with router support, and movement anticipation and proactive handoff management.
This report covers the period July 2005 - January 2006, since the FreeBSD Security Team did not submit a status report for July - October 2005.
In August 2005, the long-time Security Officer, Jacques Vidrine, stepped down and was replaced by Colin Percival. Jacques remains with the team as Security Officer Emeritus, and the team thanks him for all his work over the past four years.
Also in August 2005, Dag-Erling C. Smørgrav was replaced by Simon L. Nielsen as Deputy Security Officer. In addition, Tom Rhodes and Guido van Rooij retired from the team in September 2005 and January 2006 respectively in order to devote their time to other parts of the FreeBSD project. The current Security Team membership is published on the web site.
In the time since the last status report, ten security advisories have been issued (five in 2005, five in 2006) concerning problems in the base system of FreeBSD; of these, four problems were in "contributed" code, while six were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and the Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 117 new entries have been added, bringing the total up to 636.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.10, FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, and FreeBSD 6.0. Their respective End of Life dates are listed on the web site.
Development is going on after the complete rewrite of the toolkit. There are many plugins available and we're testing a new implementation of unionfs for 6.x. Since it's a bit unstable, it won't be included in the release anyway. Developers hope to enter the BETA state on February 1st, to release an -RC image around February 15th and the RELEASE around March 1st. We need more people to test the images we provide. Torrents for them are available at torrent.freesbie.org .
The port of DragonFly's variant symlinks ( project ideas ) to FreeBSD. Variant symlinks is a dynamic symbolic link implementation. Source file of a variant symlink may contain one or more variable names. Each of these variable names is enclosed in braces and preceded by a dollar sign in the style of variable references in sh(1). Whenever a variant symlink is followed, each variable found in source file is replaced by its associated value. In this manner, a variant symlink may resolve to different paths based on context.
We are well into the process of selecting the talks for BSDCan 2006. Our new program committee has a hard selection task over the new few weeks. The deadline for the Call For Papers has passed, but it's not too late to submit a talk. Please see the above URL for details. After the success of the Work in Progress last year , we are going to do it again this year. If you are working on something you'd like to tell the world about, considering giving a 5 minute talk at BSDCan. The registration prices for BSDCan 2006 will be the same as they were for 2005 . We will be again in the SITE building at University of Ottawa and you'll have lots of opportunity to meet with people from all over the world. Be sure to make your travel plans now and don't miss out on the biggest BSD event this year: BSDCan 2006.
FreshPorts recently moved to a new webserver. This should speed things up considerably.
You can read all about the new hardware on the recently introduced FreshPorts Blog . This blog will include technical discussions about ports and the problems they present with respect to FreshPorts. Site announcements will be posted there. As bugs are found, they will be listed, as well as their fixes.
Supporting multiple platforms and architectures is still in the development stage. Lack of time is affecting progress.
A fix for virtual ports is in the works. I'm also going to implement more caching to speed things up. If interested in discussing the options there, please get involved in the blog.
The FreeBSD Dutch Documentation Project is an ongoing project, focussed on translating the English documentation and website to the Dutch language. Currently we are almost done with the FreeBSD Handbook and started the initial translation of the FreeBSD Website. We are always looking for people to help out, if you can help, please contact Siebrand or me so that we can divide the work amongst us.
Recent publications:
Recently the Printing and the Serial Communications chapters were
added to the FreeBSD Dutch Handbook.
Recently started items:
We started with the translation of the PPP and SLIP chapter and the
translation of the website.
FreeBSD/xbox support is nearing completion. Patches are available for nve(4) ethernet support, as well as a syscons(4)-capable console. I am working to integrate these in CURRENT, a backport to 6.x is planned too.
Work is under way to support X.Org as well; people with more detailed knowledge of X.Org are welcome to assist.
Major work has gone into improving both the performance of the - LSI MegaRAID (amr) driver, and in adding Linux compatiblity + LSI MegaRAID (amr) driver, and in adding Linux compatibility support. SMPng locking was added in Oct 2005 as well as a number of performance improvements. The result is 138% performance improvement in some local transaction tests.
Throughout 2005 a lot of work has gone into adding Linux compatibility to the driver. It is now possible to run many of the LSI-provided management apps for Linux under FreeBSD. Both this feature and the performance improvements are in the 7-CURRENT development branch of FreeBSD and are scheduled to be backported in time for the FreeBSD 6.1 release.
In an effort to solve the 'interrupt aliasing' problem that plagues many motherboards under FreeBSD, I modified the Intel e1000 network driver (if_em) to use a combination of fast interrupts and taskqueues. This technique avoids interrupt threads entirely, which in turn avoids triggering the aliasing problem in the Intel APIC. The result is that the driver now handles and masks interrupts immediately, and a private taskqueue is then scheduled to run to process the link events and rx/tx events. A side effect of this asynchronous processing is that it acts much as traditional polling does, in that the amount of work done in the taskqueue can be controlled, and the taskqueue rescheduled to process work at a later time. This leads to the driver having the low-latency benefits of interrupts and the workload segmentation of polling, all without complicated heuristics. Several users have reported that the driver can handle higher loads than traditional polling without deadlocks.
Along with this work, I modified the SMPng locking in the driver so that no lock is required for the RX path. Since this path is already implicitly serialized by the interrupt and/or taskqueue and/or polling handler (all of which are exclusive to each other), there was no need for extra synchronization. This has two benefits. The first is reduction in processing overhead to unlock and lock the driver for every RX packet, and significant reduction in contention of the driver lock when transmitting and receiving packets at the same time. I believe that it is further possible to run the TX-complete path without a lock, further reducing overhead and contention for high transmit loads. The reduced contention also - greatly benefitted the fast-forward bridging code in FreeBSD, with + greatly benefited the fast-forward bridging code in FreeBSD, with up to 25% performance improvement seen, as well as lower CPU utilization.
The work can be found in FreeBSD 7-CURRENT for now. There are still some rough edges relating to falling back to traditional ithread and polling behavior, and I do not intend to merge the changes back to FreeBSD 6.x until these are resolved. I also hope to extend the INTR_FAST+taskqueue model into a general framework for doing Mac OSX style filter interrupts. The work in the if_em driver can also be extended to other high-performance network drivers such as if_bge and if_ti. Any help with investigating these topics is welcomed.
Another very busy year for the FreeBSD Release Engineering Team. Recognizing the problems, both technical and emotional, surrounding the FreeBSD 5.x releases, our primary focus was in getting the bugs out of FreeBSD 6.0 and getting it released. We succeeded at that quite well, and the 6.0 release on Nov 18 was a huge success for the project. Many thanks to all of the developers who put in countless hours fixing bugs and improving performance, and to the users who helped find, fix, and verify bugs.
Moving forward to 2006, we plan on doing a joint release of FreeBSD 5.5 and 6.1 in late March. The 5.5 release will mark the end of active FreeBSD 5.x development and releases, and is intended to help users who have not yet switched to FreeBSD 6. It consists primarily of bug fixes and minor improvements. FreeBSD 6.1 will be an upgrade to 6.0 and will include new drivers, better performance in certain areas, as well as bug fixes. We expect to release FreeBSD 6.2 and 6.3 later in 2006.
FreeBSD 6 has evolved drastically in the development branch since FreeBSD 5.3 and especially so in the network area. The presentation and paper give an in-depth overview of all network stack related enhancements, changes and new code with a narrative on their rationale.
FreeBSD has gained fine grained locking in the network stack throughout the 5.x-RELEASE series cumulating in 6.0-RELEASE. Hardware architecture and performance characteristics have evolved significantly since various BSD networking subsystems have been designed and implemented. This paper gives a detailed look into the implementation and design changes in FreeBSD 7-CURRENT to extract the maximum network performance from the underlying hardware.
Sponsored by: TCP/IP Optimization Fundraiser 2005
The fundraiser has been very successful and I want to thank everyone who has pledged their support and tipped the jar. The full amount plus a little bit more has been raised in a very short timeframe. More information on the exact amounts and their sponsors can be found at the first link.
After the delays on this project caused by the FreeBSD 6.0 Release cycle code freeze work has picked up and a paper was written and a presentation held on "Optimizing the FreeBSD IP and TCP Stack" for EuroBSDCon 05 on November 27th. See related status report under that title.
From December 21st to January 11th I received access to a calibrated Agilent N2X gigabit tester and traffic generator. Stock FreeBSD 7-current was tested and profiled extensively in this timeframe. A first proof of concept optimization was developed in cooperation with Scott Long. It involved converting the Intel Gigabit ethernet em(4) driver to make use of fast interrupt handlers, taskqueues and lockless RX ring handling. This improved the performance from 570kpps to 750kpps, a 25% improvement, with IP fastforwarding enabled.
Basic audio capture is working. All of the parameters are set by userland, while the RISC program generation is by kernel. No real audio has been captured as there are no drivers for the tuner yet. Someone with a real Bt878 NTSC card that is supported by bktr(4) could use this to capture audio w/o using the sound card.
The real goal of this driver is to make HD capture possible with the DViCO FusionHDTV5 Lite card that I have. I have some of the documentation that I need, but I'm still missing two key docs. The docs for the LGDT3303 ATSC/8VSB/QAM demodulator chip and a block diagram of the board showing which GPIO lines go where and how the chips are interconnected. DViCO has been responsive in acknowledging my emails, but they have yet to produced any data besides pointing me to the Linux driver (which is difficult to figure out stuff by).
This project provides support for SysKonnect's SK-98xx, SK-95xx,SK-9Exx and SK-9Sxx PCI/PCI-Express Gigabit Ethernet adapters via the yk(4) driver, as well as Marvell's Yukon LOM Gigabit Ethernet controllers via the myk(4) driver. Driver source has been made available to selected members of the FreeBSD project.
Full domU support in p4 branch of -CURRENT, except suspend / restore. Dom0 work is in progress. Scott Long is working on xenbus integration with newbus. After newbus integration it will go into CVS. I hope to see it MFCed to RELENG_6 so it will be available for 6.1.
The highlights of this quarters report certainly include the availability of native Java binaries thanks to the FreeBSD Foundation , as well as progress has been made with Xen support and Sun's - Ultrasparc T1. Futhermore we are looking forward to FreeBSD 6.1 and + Ultrasparc T1. Furthermore we are looking forward to FreeBSD 6.1 and TrustedBSD audit support has been imported into FreeBSD 7-CURRENT. All in all, a very exiting start to 2006.
In just under a month the developers will be gathering at BSDCan 2006 for, FreeBSD Dev Summit, a two day meeting of FreeBSD developers. Once again the BSDCan schedule is filled with many interesting talks.
We hope you enjoy reading and look forward to hear from you for the next round. Consult the list of projects and ideas for ways to get involved. The submission date for the second quarter reports will be July, 7th 2006.
Thanks to everybody who submitted a report and to Brad Davis, who joined the Status Report team, for proof reading.
In March 2006, Marcus Alves Grando, George Neville-Neil, and Philip Paeps joined the FreeBSD Security Team. The current Security Team membership is published on the web site.
In the time since the last status report, eight security advisories have been issued concerning problems in the base system of FreeBSD; of these, three problems were in "contributed" code, while five were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and the Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 50 new entries have been added, bringing the total up to 686.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.10, FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, and FreeBSD 6.0. Upon their release, FreeBSD 5.5 and FreeBSD 6.1 will also be supported. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 4.10 and FreeBSD 5.4 will cease to be supported at the end of May 2006.
Support for NFS in FreeBSD received a boost this quarter as a kernel developer from Network Appliance has volunteered to help with the clients. Chuck Lever is now a src committer, mentored by Mike Silbersack. Mohan Srinivasan and Jim Rees have ended their apprenticeships and are now full committers. Mohan continues his effort to make the NFSv2/3 client SMP safe. He expects to make the changes available for review soon.
FreeBSD gained presence at the annual NFS interoperability event known as Connectathon. Rick Macklem's FreeBSD NFSv4 server is pretty stable now and available via anonymous ftp. NFSv4.1 features are not a part of it yet and are not likely to happen until at least the end of 2006. Contact rick@snowhite.cis.uoguelph.ca for details.
The schedule for BSDCan 2006 demonstrates just how strong and popular BSDCan has become in a very short time. Three concurrent streams of talks make sure that there is something for everyone. We provide high quality talks at very affordable prices .
BSDCan is the biggest BSD event of 2006. Ask others who attended in past years how much they enjoyed their time in Ottawa. Ask them who they met, who they talked to, the contacts they made, the information they learned.
Remember to bring your wife/husband/spouse/etc because we will have things for them to do while you are attending the conference. Ottawa is a fantastic tourist destination.
See you at BSDCan 2006!
During this time, the number of ports PRs rose dramatically from its impressive low number seen late last quarter. This was due to the holidays, the freeze for the 5.5/6.1 release cycle, and the aggressive work several submitters have been doing to correct long-standing problems with stale distfiles, stale WWW sites, port that only work on i386, and so forth. Over 200 new ports have also been added. The statistics do not truly reflect the state of the Ports Collection, which continues to improve despite the increased number of ports.
We now have 3 people who are qualified to run the 5-exp regression tests. Due to this, we were able to run several cycles, resulting in a series of commits that retired more than 3 dozen portmgr PRs. There were a few snags during one commit due to some unintended consequences, but the breakage was fixed in less than one day. Notable changes include the addition of physical category net-p2p and virtual categories hamradio and rubygems. Once 5.5 and 6.1 are released, portmgr hopes to be able to run regression tests more often.
We have added 5 new committers since the last report.
All dhclient changes in HEAD have been merged to 6-STABLE for 6.1-RELEASE. New patches currently in testing include startup script support for fully asynchronous starting of dhclient which eliminates the wait for link during startup and support for sending the system hostname to the server when non is specified.
Split out of PF_KEY code between the kernel and user space has been completed and committed to CVS.
The diff between Kame IPv4 based IPSec and FAST_IPSEC IPv4 did not show any glaring issues.
Moving on to making IPv6 work in FAST_IPSEC including being able to run the kernel with the following variations:
The project is alive and plans to release an ISO image of FreeSBIE 2.0 based on FreeBSD 6.1-RELEASE few day after the same has been release. FreeSBIE 2.0 will be available for i386 and amd64 archs. Tests images can be download via BitTorrent from torrent.freesbie.org .
A preliminary version of HP's hplip software for their printers and multi-function devices has been ported. This allows viewing of the status informantion from the printer. Such as ink levels, error messages, and queue information. If you have an Officejet you can also fax and scan. Photocard and Copies functionality is untested.
This projects implements a kernel module (hwpmc(4)), an application programming interface (pmc(3)) and a few simple applications (pmcstat(8) and pmccontrol(8)) for measuring system performance using event monitoring hardware in modern CPUs.
New features since the last status report:
The FreeBSD list of projects and ideas for volunteers is doing well. Several items were picked up by volunteers and have found their way into the tree. Others are under review or in progress.
We are looking forward to hear about new ideas, people willing to be technical contacts for generic topics (e.g. USB) or specific entries (already existing or newly created), suggestions for existing entries or completion reports for (parts of) an entry.
The FreeBSD Foundation released official certified JDK and JRE
1.5 binaries for the official FreeBSD 5.4 and FreeBSD 6.0 releases
on the i386 platform.
We were able to accomplish this by hiring a contractor to run the
Sun certification tests and fixing the problems found. This could
not have been completed without the support from the BSD Java
Team.
We provided financial support for Java development and funded the certification process. We spent a significant amount of time and money on legal issues from contract and NDA creation for our contractor to license agreements from Sun and creating our own for the binaries. We worked with OEMs who would like to use the binaries, but needed to understand what they need to do legally to be able to redistribute the binaries. This is an area we are still working on at our end. We are waiting for a letter from Sun to put on our website to OEMs. We are also in the process of updating our OEM license agreement. This should be available by mid-April.
We have received a positive response from the FreeBSD community regarding the release of the binaries. We received a few requests to support the FreeBSD 6.1/amd64 platform. We have decided to move forward and support this too. We currently are working with a contractor to provide Java support on 5.5/i386, 6.1/i386, and 6.1/amd64. Once 5.5 and 6.1 are released, we'll update the FreeBSD Foundation website with the Java status. Regular updates to the website will continue.
Work is underway to use the new linux_base-fc3 as the new default linux base. Since there's some infrastructure work to do before it can be made the new default, this will not happen before the release of FreeBSD 5.5 and 6.1. At the same time a new X.org based linux port will replace the outdated XFree86 based linux X11 port.
The use of fc3 instead of fc4 or fc5 is to make sure we have a smooth transition with as less as possible breakage. We already use several fc3 RPM's with the current default of linux_base-8, so there should be not much problems to solve.
The current mouse system is a mess with moused, psm, ums, and mse supporting, individually, multiple kinds of mice. This project aims to move all driver support into moused modules in userland. In addition, many features lacking in the existing mouse infrastructure are being added. It is my hope that this new system will make both using mice and writing drivers easier down the road.
The FreeBSD netperf project has recently focused on revising the socket and protocol control block reference counts to define and enforce reference and memory management invariants, allowing the removal of unnecessary checks, error handling, and locking. Use of global pcbinfo locks has now been eliminated from the socket send and receive paths into all network protocols, including netipx, netnatm, netatalk, netinet, netinet6, netgraph, and others. Checks have generally been replaced with assertions; so_pcb is now guaranteed to be non-NULL. This should improve performance by reducing lock contention and unnecessary checks, as well as facilitate future work to eliminate long holding of pcbinfo locks in the TCP input path through proper reference counting for pcbs. These changes have been committed to FreeBSD 7-CURRENT, and will be merged in a few months once they have stabilized.
pfSense continues to grow and fix bugs. Since the last report we have grown to 14 developers working part and full time on bringing pfSense to 1.0. Beta 3 is scheduled for release on 4/15/2006.
Symbol versioning libraries allows us to maintain binary compatibility without bumping library version numbers. Recently, symbol versioning for libc, libpthread, libthread_db, and libm was committed to -current. It is disabled by default, and can be enabled by adding "SYMVER_ENABLED=true" to/etc/make.conf. A final version bump for libc and other affected libraries (perhaps all) should be done before enabling this by default.
The last months has mostly been about stabilizing ATA for 6.1-RELEASE, and adding support for new chipsets. On that front JMicron has raised the bar for vendors as they have provided not only hardware but documentation on both their hardware and their software RAID implementation, making it a breeze to add support for their, by the way excellent, products. Other vendors can join in here. :) Otherwise I'm always in the need for any amount of time or means to get it if nothing else.
ATA has grown a USB backend so that fx. flash keys and external HD/CD/DVD drives can be used directly without atapicam/CAM etc. This is very handy on small (embedded) systems where resources are limited and kernel space at a premium. burncd(8) is in the process of being updated so it will support this along with SATA ATAPI devices, and if time permits adding DVD support.
The next months will be used to (hopefully) work on getting ATA to work properly on systems with > 4G of memory and utilize the 64bit addressing of controllers that support it. RAID5 support for ataraid is on the list together with hardening of the RAID subsystem to help keep data alive and well.
The BSDInstaller integration work has progressed since the previous report. The backend has been changed to the new Lua version. This is to ensure the version we use will be maintained. The release Makefile now uses the Lua package rather the local copy in Perforce. Ports are also being created for the required modules to remove the need to bring Lua into the base.
Libpkg is a package management library using libarchive to extract the package files. It is able to download, install and get a list of installed packages. Work has also been started on implementing the package tools from the base system. Most of pkg_info has been implemented and pkg_add has been started.
Work has been started to implement the Rapid Spanning Tree Protocol which supersedes STP. RSTP has a much faster link failover time of around one second compared to 30-60 seconds for STP, this is very important on modern networks. Some progress has been made but a RSTP capable switch will be needed soon to proceed, see http://www.freebsd.org/donations/wantlist.html .
Three betas have been released so far. The code is operational and seems to be stable but it is not MPSAFE yet.
The second and third betas used different mechanisms for data I/O. (sfbuf vs. kernel_map+vacache) and at present I am in the process on selecting one mechanism over the other. Your opinion is solicited.
A lot of fixes (bugs, LORs, panics) and improvements (performance, compatibility, a new driver, 24/32bit samples support, ...) have been merged to RELENG_6. FreeBSD 6.1 is the first release which ships with the much improved sound system. Additionally there's work underway:
Since 2003, I have introduced the (now quite widely used) FreeBSD Update and Portsnap tools, but rarely had time to make improvements or add requested features. Consequently, on March 30th, I sent email to the freebsd-hackers, freebsd-security, and freebsd-announce lists announcing that I was seeking funding to allow me to spend the summer working full-time on these and my role as FreeBSD Security Officer. Assuming that some cheques arrive as expected, I have reached my donation target and will start work at the beginning of May.
We had hoped to finish a prototype of Xen DomU and possible Dom0 in time for FreeBSD 6.1. The primary work was focused on bringing Xen into the FreeBSD 'newbus' framework. Unfortunately, an architectural problem in FreeBSD has stopped us. Xen relies on message passing between to child and parent domains to communicate device configuration, and this message passing requires that tsleep and wakeup work early in boot. That doesn't seem to be the case, and it's unclear what it would take to make it work. Without the newbus work, it's hard to complete the Dom0 code, and impossible to support Xen 3.0 features like domain suspension.
In the past three months, the TrustedBSD CAPP audit implementation has been merged to the FreeBSD 7-CURRENT development tree in CVS, and the groundwork has been laid for a merge to 6.X. OpenBSM, a BSD-licensed implementation of Sun's Basic Security Module (BSM) API and file format, as well as extensions to support intrusion detect applications. New features included support for audit pipes, a pseudo-device that provides a live audit record trail interface for intrusion detection applications, and an audit filter daemon that allows plug-in modules to monitor live events.
OpenBSM is a BSD-licensed implementation of Sun's Basic Security Module (BSM) API and file format, based on Apple's Darwin implementation. OpenBSM 1.0 alpha 5 is now available, and includes significant bugfixes, documentation, and feature enhancements over previous releases, including 64-bit token support, endian-independent operation, improved memory management, and bug fixes resulting from the static analysis tools provided by Coverity and FlexeLint. Recent versions are now built and configured using autoconf and automake, and have been built and tested with FreeBSD, Mac OS X, and Linux.
This is just an update to note that TS-7200 is building and running with a recent -current.
I have been working on getting FreeBSD/arm running on the TS-7200. So far the board boots, and has somewhat working ethernet (some unexplained packet loss). I can netboot from a FreeBSD/i386 machine, and I can also mount msdosfs's on CF.
FreeBSD has been ported the T1, Sun's newest processor. FreeBSD currently runs multi-user SMP. JMG is actively working on improving device support.
The port has taken several weeks longer than initially anticipated as the majority of the current sparc64 port could not be re-used.
Work towards importing the upcoming OpenBSD 3.9 version of pf is starting slowly. There are a couple of infrastructural changes (e.g. interface groups) that need to be imported beforehand. This work is in the final stage of progress.
-A couple of bugfixes have happend since the last report and will +
A couple of bugfixes have happened since the last report and will be available in FreeBSD 6.1/5.5. pf users are strongly encouraged to upgrade to RELENG_6 as the version present in RELENG_5 is collecting dust.
With the release of FreeBSD 5.5 and FreeBSD 6.1, the second quarter of 2006 has been productive. Google is sponsoring 14 students to work on FreeBSD as part of their Summer of Code Program (most of which already submitted a report for elaboration on their projects).
Sun's open-source software is starting to make its way into FreeBSD as a port of DTrace is nearing completion and a port to the UltraSparc T1 processor (which gives a great push to the ongoing SMP efforts). Having a powerful debugging tool combined with a CPU that can run up to 32 concurrent threads helps to identify scalability issues.
BSDCan 2006 was yet again a smashing success and much was covered in the 2-day developer summit. As a product of the conference, a new focus on FreeBSD for the embedded sector has started. Various ARM boards are targeted, a MIPS32 port is gearing up and people are looking for other interesting platforms to port FreeBSD to. Preparation for the EuroBSDCon (in Milan, Italy) on November has already issued a call for papers.
In addition, a lot of spring cleaning is taking place in the network stack. After conclusion of the KAME project, IPv6 code integration has been refocused and a fully locked port of SCTP is in the final stage of integration. Of course, all this goes without noting all the progress made with the other network projects.
Please read below for more detailed news on the projects that happened in FreeBSD during the last three months. If you are interested in helping, consider the "Open Tasks lists" provided with some reports. In addition we would like to point you at the list of projects and ideas for volunteers and hope to receive a status report from you next time.
Thanks to all reporters for your excellent work and timing! Enjoy reading.
BSDCan 2006 continues to impress. Again this year, we had a good collection of talks from a wide range of speakers. In all, we had over 200 people from 14 different countries.
Our sponsorship pool continues to grow. This year we had sponsorship from:
The 2007 planning has already begun and we look forward to another popular and successful event.
My thanks to the 2006 program committee, the speakers, the volunteers, the sponsors, and, of course, the attendees.
See you at BSDCan 2007.
The release engineering (RE) team announced the availability of FreeBSD 5.5 and 6.1, both in May 2006. FreeBSD 5.5 is the last planned release from the RELENG_5 branch in CVS. For the most part, its main features consist of bugfixes, security patches, and minor updates. We encourage users to move towards the 6.x series of releases whenever practical. FreeBSD 6.1 is the latest of the releases to come from the RELENG_6 branch in CVS. It includes (among many other things) improved support for WiFi devices, additional network and disk controller drivers, and a number of fixes for filesystem stability. The next release to be issued from this branch will be FreeBSD 6.2, which is currently scheduled for September 2006.
The RE team is currently in a ``between releases'' mode. Current activities include working with security-team@ on some errata fixes for the RELENG_6_1 branch and producing snapshots of HEAD and RELENG_6 at the start of each month.
Several personnel changes have taken place recently. Scott Long has stepped down from his position on the RE team; we thank him for his considerable efforts over the past four years. In his place, Ken Smith has taken over the role of lead release engineer. Bruce A. Mah has rejoined the RE team after a two-year sabbatical.
For some time now I have been working on converting the existing USB device drivers to my new and mutex enabled USB API. I have converted "ulpt", "ums", "uhid", "ukbd", "ugen", "uaudio", and a few others. Around 10 USB device drivers are left to convert. Most of these are network device drivers.
At the present moment I am working on getting scatter and gathering support working for all USB host controllers. Scatter and gathering means that one allocates PAGE_SIZE bytes of memory at a time, and then fills these memory blocks up as much as possible with USB host controller structures and buffers. This should solve problems allocating DMA-able memory when the system memory becomes fragmented.
Some FPU system and kernel memcpy/copyin/copyout changes have been performed. In particular, a per-CPU save area has been introduced (protected with an interlock) in order to assure a stable saving mechanism. copyout/copyin have changed in order to use vectorised version of memcpy and an xmm version of memcpy has been provided.
As part of my SoC 2006 project I am working on implementing a BRIDGE monitoring module for FreeBSD's BSNMP daemon. Initial prototyping is done and some kernel changes are coming to be able to access all needed data. In addition to IETF RFC 4188, which was designed for monitoring a single bridge, this snmp module will support monitoring of multiple bridge devices as supported by FreeBSD.
Anonymous enablings now work. There is a new option in the boot loader menu to load the DTrace modules and trace the kernel boot process.
Sun Microsystems has been very supportive of the FreeBSD port and has generously provided a Sun Fire T2000 server to allow Kip Macy's sun4v port to be merged into the DTrace project tree.
The DTrace project tree sources are now exported to cvsup10.freebsd.org
Refer to the project page for more details.
There are several projects moving forward in the embedded area. For now the main location for new information is www.embeddedfreebsd.org. We have also created a new mailing list, freebsd-embedded@freebsd.org , which is meant to eventually replace the freebsd-small. A call was put out on small for people to move to embedded.
This year's EuroBSDCon will be held in Milan, Italy, on November 10th through 12th.
Hosted in the foggy northern Italy, the fifth EuroBSDCon aims at being a new successful chapter in the itinerant series of European BSD conferences.
EuroBSDCon represents the biggest gathering for BSD developers from the old continent, as well as users and passionates from around the World. It is also a chance to share experiences, know-how, and cultures.
For the first time, parallel to the main event, an event for wives/girlfriends/friends will be organised. It will consist of guided tours of the city of Milan, a probable trip to Como and visits to various museums. We're also working towards offering a show at the Teatro alla Scala.
The FreeBSD developer summit will be also held on November 10th.
Continuing to add IPv6 support to FAST_IPSEC. Test environment is now stable. Can build and run kernels with FAST_IPSEC and INET6 enabled but IPSec in IPv6 is now broken and being worked on.
FreshPorts has seen several new features recently:
A 2U server was recently donated to the FreshPorts / FreshSource / FreeBSD Diary / BSDCan group. We have also received a RAID card. Now we're looking for some hard drives.
Over the past few weeks, work has concentrated on benchmarking the new server and getting it ready for production. Eventually it will need a new home as I don't really want it running in my basement all the time (it's really loud!).
Thanks to iXsystems and 3Ware for their contributions to this project.
GJournal is a GEOM class which provides journaling for GEOM providers. It can also be used to journal various file systems with just a minimal filesystem-specific portion of code. Currently only UFS journaling is implemented on top of gjournal. Being filesystem-independent and operating below the file system level, gjournal has no way to distinguish data from metadata, thus it journals both. One of the nice things about gjournal is that it works reliable even on disks with enabled write cache, which is often not the case for journalled file systems. And remember... fsck no more.
The purpose of gvirstor module is to provide the ability to create a virtual storage device of arbitrarily large size (typically several terabytes) which consists of an arbitrary number of physical storage devices (actually any lower-level GEOM providers, including RAID devices) of arbitrary size (typically 50 GB - 400 GB hard drives). Storage space from these components is carved into small chunks (for example 4 MB) and allocated (committed) to the virtual device on as-needed basis.
Development has started and is progressing as planned (though a little bit slow). Metadata format and virtual storage allocation formats have been defined and more serious coding is in progress.
The FreeBSD list of projects and ideas for volunteers is doing well. Several items were picked up by volunteers and have found their way into the tree. Others are under review or in progress. We are looking forward to hear about new ideas, people willing to act as technical contacts for generic topics such as USB or specific entries (already existing or newly created) and suggestions for existing entries or completion reports for (parts of) an entry.
Initial changes include:
Clement has been working both with libnet and gnn's Python based packet library (PCS) to produce code to test for vulnerabilities in IPv6. To Clement has found some issues, all of which have been reported to his mentor and to Security Officer at FreeBSD.org Vulnerabilities will not be reported here.
Project is in development with initial working software expected mid-July 2006. CPU limits will be implemented with a hierarchical scheduler: (initially) using a round-robin scheduler to select which jail to run a task in and then delegating which task in the jail to be run to a per-jail scheduler.
A simple lexer and parser have almost been completed. Also significant planing for future additions to K have been thought up.
FreeBSD linux emulation layer (linuxolator) currently implements most of the functionality necessary to emulate 2.4.2 linux kernel, but linux world has moved forward and current linux world requires 2.6.x features. The aim of this SoC task is to make Fedora Core 4 linux-base to be able to run with 2.6.x kernel. Currently this means extending clone() syscall and implement pthread related things. This involves TLS implementation (sys_set_thread_area syscall) and possibly tid manipulation (used for pthread_join etc.) and finally futexes (linux fast user-space mutexes implementation). This should enable pthread-linked programs to work. After this is done there may be other things necessary to implement however, only time will tell. I am funded by google.com in their SoC to do this work and I'll continue to work on this after the summer hopefully as a part of my MSc. thesis.
The improved support for the i386 binaries are ready for -exp run. It only allows installing such ports on amd64 and ia64 when there's a compatibility layer compiled into the kernel and the 32-bit libraries are installed under /usr/lib32.
The DESTDIR support are in progress. It works for the simplest ports without USE_* that don't have a [pre|do|post]-install target. There are more complicated issues with e.g. conflict checking in DESTDIR, deinstalling from DESTDIR, those have to be fixed as well.
The translated webpage is almost ready now. This Hungarian translation is a "lite" version of the original English webpages, since there are parts that are irrelevant for the Hungarian community, or has pieces of data that change quickly, so it's no use to translate these pages now, maybe later, if we have more Hungarian contributors, but this webpage would be a good starting point in translating the documentations, and we need a good place to put translated documentations anyway.
I'm going to be very busy with SoC this summer, but I'll try to find people that can help me out in this project. Any help appreciated.
As an intermediate step until FreeBSD will have full network stack virtualisation this work shall provide support for multi-IP IPv4/v6 jails.
These changes are based on Pawel Jakub Dawidek's work for multi-IPv4 jails and some initial work from Olivier Houchard for single-IPv6 jails.
The changes need some more testing but basically things work.
This is not considered to be the right thing todo so do
not
ask for official support or if this will be committed to the
FreeBSD source repository.
After some more cleanup of non-jail related IPv6 changes I will
publish a patch for HEAD and perhaps RELENG_6 for everyone who
wants to give it a try anyway.
Mohan Srinivasan committed his changes to make the NFSv2/3 client MP safe to HEAD this quarter. Changes may be back-ported to 6.x soon.
Robert Watson and Chuck Lever held a discussion about the future of the in-kernel NFSv4 client during BSDCan 2006. The current NFSv4 client is unmaintained. Chuck also pointed out the long series of unfixed PRs against the legacy client (NFSv2/3). These are at the top of his priority list. Robert is also interested in making NFSv4-style ACLs the lingua franca for FreeBSD file systems. There was some discussion about integrating Rick MacKlem's NFSv4 server into 7.x.
Chuck Lever became a full source committer during this quarter.
The basic goals of this SoC 2006 project are moving nsswitch-modules out of the libc, extending the caching daemon and importing nss_ldap into the base source tree. 2 milestones of the project are currently completed.
1. Nss-modules were successfully moved out of the libc into the separate dynamic libraries. In order for static binaries to work properly (they can't use dynamic nss-modules), nss-modules are linked statically into the libc.a. As the side-effect of nss-modules separation, getipnodeby***() functions were rewritten to use gethostby***() functions and not the nsdispatch(3) call. Caching daemon's "perform-actual-lookups" option was extended to support all implemented nsswitch databases.
2. A set of regressions tests was made to test nsswitch-related functions. These tests are also capable of testing the stability of these functions' behaviour after the system upgrade.
pfSense is rapidly approaching release. We are down to a handfull of bugs that should be fixed in the coming weeks. We should have a release around the time of our 2nd annual hackathon which is taking place on July 21st - July 28th. Many exciting sub-projects are taking place within pfSense and the project is gaining new developers monthly.
As an intermediate step towards implementing support for callgraphs and cross-architecture performance measurements, I am creating a BSD-licensed library for ELF parsing & manipulation. This library will implement the SysV/SVR4 (g)ELF[3] API.
Current status: Implementation of the library is in progress. A TET-based test suite for the API and manual pages documenting the library's interfaces are being concurrently created.
Work is being done in FreeBSD's Perforce repository. I hope to be ready for general review by the end of July '06.
During this time, a huge number of ports PRs were committed, bringing us back down below 800 for the first time since the 5.5/6.1 release cycle. This is due to a great deal of work, especially from some of our newest committers.
This is all the more notable given the fact that we have been adding new ports at a rapidly accelerating rate. We have now exceeded the 15,000 port mark!
Three sets of changes have been added to the infrastructure, including updates of default versions of MySQL, PHP, LDAP, and linux_base, and numerous bugfixes and improvements. About 2 dozen portmgr PRs were closed due to this.
In addition, a large-impact commit was made that attempts to move us to a single libtool that is as unmodified from 'stock' libtool as we can. Plans are also in place to do this for the autotools.
Several people are at work on implementing the modularised xorg ports. Most of the work is done but several key pieces remain. Once this is finished, an -exp regression test will be needed (most likely, more than one :-) ) It is possible that before this we will need to do a regression test that moves X11BASE back into LOCALBASE. This is still under study.
Gábor Kövesdán started a Google Summer of Code project on some highly needed improvements on the ports infrastructure (see elsewhere in this report). As this is a long term project, gtetlow kindly imported the most important ports infrastructure files into perforce to ease development. Other developers are encouraged to use perforce for ports development, especially as it can help keeping patches up-to-date while going stale in GNATS. Even though linimon has been pushing hard on running experimental builds on the test cluster, it will take some time to work through the backlog.
erwin added a ports section to the list of projects and ideas for volunteers at the FreeBSD website. Have a look if you want to work on the ports system. Don't hesitate to send additional ideas, and committers are encouraged to add themselves as technical contacts.
sem adopted portupgrade after it had been neglected for some time and has been very active on upgrades and bugfixing.
dougb has continued to enhance his portmaster script and people are finding success with it; although not designed to be as full-featured as portupgrade, it does seem to be easier to understand and use.
shaun has contributed portscout, a scanner for updated distfiles, to the ports collection.
marcus upgraded GNOME to 2.14.1.
As well, there have been new releases of the ports tinderbox code.
edwin has been hard at work on a PR-autoassigner for ports PRs, which has saved a lot of time and been well-received. It has now been installed on a freebsd.org machine (hub).
linimon has been more active in pursuing maintainer-timeouts, and has reset a number of inactive maintainers, with more in the pipeline. The intent is to try to reduce the number of PRs that sit around unanswered for two weeks. In almost all cases the resets are due to no response at all; maintainers who are merely "busy" are not the source of most of these problems, and deserve the benefit of the doubt. Some of the maintainers that have been reset haven't contributed in months or even years.
We have added 10 (!) new committers since the last report.
Since the last status report ports have been created for all parts of the BSDInstaller except the backend.
A snapshot of the BSDInstaller was released during this quarter. This has shown a number of bugs with the installation process. Most have now been fixed.
The patches to allow UFS operate with quotas in Giant-less mode are brewed for long now. Since recent huge pile of fixes into snapshots code, I think the problems you could encounter are caused solely by the patch.
Aside performance benefits, patch has another one, much more valuable. It makes UFS operating in one locking regime whatever options are compiled into kernel. I think, in long term, that would lead to better stability of the system.
We updated the default linux base port to Fedora Core 4 and the default linux X11 libs port to the X.org RPM in FC4.
An update to FC5 or FC6 has to wait until the kernel got support for syscalls of a newer linux kernel. See the corresponding SoC project report for more.
Since the last status report we fixed some more bugs, added basic support for envy24 chips and cleaned up the source for the emu10kx driver in the ports to make it ready for import into the base system.
We also got some patches with a little bit of infrastructure for Intel HDA support. It's not finished and also not usable by end users yet.
The XFS for FreeBSD project is an effort to port the publically +
The XFS for FreeBSD project is an effort to port the publicly available GPL'd sources to SGI's XFS filesystem to FreeBSD.
In December, we imported a version of XFS into FreeBSD-CURRENT which allows FreeBSD to mount an XFS filesystem as read-only.
As a side effort, we have been continuing on the work that PHK started to clean up the mount code in FreeBSD. We can use the existing FreeBSD mount(8) utility to mount an XFS partition, without introducing a new mount_xfs utility.
For the last several months Randall Stewart has been working in HEAD and STABLE to get us ready to integrate the SCTP protocol (Stream Transmission Control Protocol) into FreeBSD. He is currently working on a patch to share with a wider audience but needs to do some integration work first. Randall has a provisional commit bit and will be working with gnn on getting code committed to the HEAD of the tree.
In the time since the last status report, four security advisories have been issued concerning problems in the base system of FreeBSD; of these, one problem was "contributed" code, while three were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 71 new entries have been added, bringing the total up to 757.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, FreeBSD 5.5, FreeBSD 6.0, and FreeBSD 6.1. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 5.3 and FreeBSD 5.4 will cease to be supported at the end of October 2006, while FreeBSD 6.0 will cease to be supported at the end of November 2006.
I have been working on porting missing features in gvinum from vinum, as well as adding new features.
So far the resetconfig, detach, dumpconfig, setstate (on plexes and volumes) and stop commands have been implemented, as well as some other minor fixes. The attach command is currently being implemented, and started on disk-grouping. Currently most of this is in p4, but patches will be submitted as soon as possible.
The wireless support has been stable for a while so most work has focused on bug fixing and improving legacy drivers.
Max Laier and I worked on improving support for Intel wireless cards. The results of this work included significant improvements to the iwi(4) driver (for 2195/2200 parts) and the firmware(9) facility for managing loadable device firmware. There is also an updated ipw(4) that has improvements similar to those done for iwi that is in early test. Support for the latest Intel devices, the 3945 pci-express cards, is planned for later this summer.
Atheros support was updated with a new hal that fixes a few minor issues and provides known working builds for SPARC, PPC, and ARM platforms. There is also working MIPS support that will be used when the MIPS port is ready to test. Otherwise one useful bug was fixed that affected AP operation with associated stations operating in power save mode.
wpa_supplicant and hostapd were updated to the latest stable build releases from Jouni Malinen.
Experimental changes to support injection of raw 802.11 frames using bpf were posted for comment. This work was done in collaboration with Andrea Bittau.
With the help of Jim Thompson of Netgate ( http://www.netgate.com/ ) the FreeBSD Foundation arranged a purchase of xscale-based boards for folks interested in ARM support. Developers were able to purchase boards at a reduced cost. The goals were to accelerate and/or improve support for the ARM platform and to set forth at least one board as a reference platform for the ARM support. Netgate will be stocking lower-cost models of the board later in the year (a special order was made for boards with only 2 mini-pci slots).
With the introduction of fine grained locking in the SMPng project, the FreeBSD kernel went under a major redesign, and many subsystem changed significantly with it. In particular, device driver's interrupt context ("the bottom half") had the necessity to synchronise with process context ("the top half") and share data in a consistent manner without using spl*(). To overcome this problem, a new interrupt model based around interrupt threads was employed, together with a fast interrupt model dedicated to particular driver handlers that don't block on locks (i.e. serial port, clock, etcetc). Unfortunately, even if the interrupt thread model proved to be a reliable solution, its performance was not on par with the pre SMPng era (4.x), and thus others solutions were investigated, with interrupt filtering being one of that.
As part of my Summer of Code 2006 work, I'm implementing interrupt filtering for FreeBSD, and when the framework will be in place I'll compare the performance of filters, against all the previous models: pre-SMPng(4.x), ithread and polling.
The most important modifications to the src tree so far were:
The project is slowly starting to ramp up after a long move-induced hiatus.
Alan Cox has almost completed making the pmap module Giant-free.
TrustedBSD Audit provides fine-grained security event auditing in FreeBSD 7.x, with a planned merge to 6.x for FreeBSD 6.2. Work performed in the last three months:
This report covers FreeBSD related projects between June and October 2006. This includes the conclusion of this year's Google Summer of Code with 13 successful students. Some of last year's and the current SoC participants have meanwhile joined the committer ranks, kept working on their projects, and improving FreeBSD in general.
This year's EuroBSDCon in Milan, Italy has meanwhile published an exciting program. Many developers will be there to discuss these current and future projects at the Developer Summit prior the conference. Next year's conference calendar has a new entry - in addition to the now well established BSDCan in Ottawa - AsiaBSDCon - will take place in Tokyo at the begining of March.
+ will take place in Tokyo at the beginning of March.As we are closing in on FreeBSD 6.2 release many bugs are being fixed and new features have been MFCed. On the other hand a lot of the projects below already are focusing on FreeBSD 7.0 and promise a lot of exciting news and features to come.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
Most dhclient changes in HEAD have been merged to 6-STABLE for 6.2-RELEASE. The highlight of these changes is a fix for runaway dhclient processes when packets are not 4 byte aligned. Further changes including always sending client identifiers are scheduled for merge before the release. Work is ongoing to improve dhclient's interaction with alternate methods of setting interface addresses.
The FreeBSD/arm port has grown support for the Atmel AT91RM9200. Boards based on this machine are booting to multiuser off either NFS or an SD card. The onboard serial ports, PIO, ethernet and SD/MMC card controllers are well supported. Support for the SSC, IIC and SPI flash parts in the kernel will be forthcoming shortly.
In addition to normal kernel support, the port includes a boot loader that can initialize memory and boot off IIC eeprom, SPI DataFlash, BOOTP/TFTP and SD memory cards.
The port will be included in forthcoming commercial products.
We had another successful summer taking part in the Google Summer of Code. By all accounts, the FreeBSD participation in this program was an unqualified success. We received over 150 applications for student projects, amongst which 13 were selected for funding. All successful students received the full $4,500.
These student projects included security research, improved installation tools, new utilities, and more. Many of the students have continued working on their FreeBSD projects even after the official close of the program. At least 2 of our FreeBSD mentors will be meeting with Google organizers in Mountain View this month to discuss the program at the Mentor Summit.
The FreeBSD Release Engineering team is currently working on FreeBSD 6.2-RELEASE, which is scheduled for release in early November 2006. Some notable features of this release include the debut of security event auditing as an experimental feature, Xbox support, the FreeBSD Update binary updating utility, and of course many fixes and updates for existing programs. Pre-release images for all Tier-1 architectures are available for testing now; feedback on these builds is greatly appreciated. More information about release engineering activities can be found at the links above.
The focus of this project was to review past vulnerabilities, create vulnerability testing tools and to discover new vulnerabilities in the FreeBSD IPv6 stack which is derived from the KAME project code. During the summer Clement took two libraries, the popular libnet, and his mentor's Packet Construction Set (PCS) and created tools to find security problems in the IPv6 code. Several issues were found, bugs filed, and patches created. At the moment Clement and George are editing a 50 page paper that describes the project which will be submitted for conference publication.
All of the code from the project, including the tools, is online and is described in the paper.
By all measures, this was a successful project. Both student and mentor gained valuable insight into a previously externally maintained set of code. In addition to the new tools development in this effort, the FreeBSD Project has gained a new developer to help work on the code.
This project consisted in the improvement of the Interrupt Handling System in FreeBSD: while retaining backward compatibility with the previous models (FAST and ITHREAD), a new method called 'Interrupt filtering' was added. With interrupt filtering, the interrupt handler is divided into 2 parts: the filter (that checks if the actual interrupt belong to this device) and the ithread (that is scheduled in case some blocking work has to be done). The main benefits of interrupt filtering are:
Moreover, during the development of interrupt filtering, some MD dependent code was converted into MI code, PPC was fixed to support multiple FAST handlers per line and an interrupt stray storm detection logic was added. While the framework is done, there are still machine dependent bits to be written (the support for ppc, sparc64, arm and itanium has to be written/reviewed) and a serious analysis of the performance of this model against the previous one is a work-in-progress
We now have support for limiting CPU and memory use in jails. This allows fairer sharing of a systems' resources between divergent uses by preventing one jail from monopolizing the available memory and CPU time, if other users and jails have processes to run.
The code is currently available as patches against RELENG_6, and Chris is in the process of applying it to -CURRENT. More details can be found at JailResourceLimits on the wiki.
For me, the Google Summer of Code was a new and very exciting experience. I got actively involved in doing Open Source Software and giving something back to the community. Facing some challenges within the project forced me to look behind the scenery of FreeBSD. The result was a better understanding of the overall project. Working with a lot of developers directly also gave a very special spirit to the Google Summer of Code.
I really enjoyed the time and will continue to work on the project after the deadline. For me, it was a great chance to get involved in active development and not just some scripts and hacks at home. Getting paid for the work was just a small part of the overall feeling.
Thanks to the people at the FreeBSD Project and Google for the really, really great time!
The Project consisted of five parts:
Though none of the code was committed yet into the official FreeBSD tree, my experience from the previous year makes me think that this situation is normal. I hope, that the code will be reviewed and committed in the coming months.
Dongmei Liu spent the summer working on the basic footwork required to port the SEREF policy to SEBSD. This work has been submitted and can be viewed in the soc2006/dongmei_sebsd Perforce branch. This work was originated from the SEBSD branch: //depot/projects/trustedbsd/sebsd. Additionally setools-2.3 was ported from Linux and can be found in contrib/sebsd/setools directory. It is hoped that this work will be merged into the main SEBSD development branch.
There are currently patches available for testing. A planned integration to HEAD is set to happen in October.
Moved the HTML pages into the project CVS tree.
First working version of code. Does not pass all TAHI tests, but does pass packets correctly and does not panic.
During the last three months I have finished reworking nearly all USB device drivers found in FreeBSD-7-CURRENT. Only two USB drivers are left and that is ubser(4) and slhci. Some still use Giant, but most have been brought out of Giant. At the moment I am looking for testers that can test the various USB device drivers. Some have already been tested, and confirmed to work, while others have problems which need to be fixed. If you want to test, checkout the USB perforce tree or download the SVN version of the USB driver that is available on my homepage. At the moment the tarballs are a little out of date.
Ideas and comments with regard to the new USB API are welcome at: freebsd-usb@freebsd.org.
This iSCSI initiator kernel module and its companion control program are still under development, but the main parts are working.
GJournal seems to be finished. I fixed the last serious bug and it is now stable and reliable in our tests. I'm planning to commit it really soon now.
The work was sponsored by home.pl
Web site is up and we're soliciting papers and presentations. Some tutorials are already scheduled. Email secretary@asiabsdcon.org if you have questions or submissions.
In the previous quarter we primarily focused on overall quality of the translation rather than just increasing the number of translations, and we have strived to make sure that these translated stuff are up-to-date with their English revisions. Also, we have merged the translated website into the central repository.
In the next quarter we will focus on developing documentation that will help to attract more developers.
EuroBSDCon 2006 is taking place in Milan (Italy), from the 10th to the 12th of November.
EuroBSDCon represents the biggest gathering for BSD developers from the old continent, as well as users and passionates from around the World. It is also a chance to share experiences, know-how, and cultures.
The program is rich in talks about FreeBSD, with topics ranging from "How the FreeBSD ports collection works" to "Interrupt Filtering in FreeBSD". This means that both the novice and the hacker can enjoy the conference.
Registration is open. The EuroBSDCon Organizing Committee hopes to see you in Milan.
Since the last status report, there has been a lot of progress. I investigated a lot of charset issues and found out that HTML tidy breaks some entities when using iso-8859-2, so HTML tidy had to be disabled for Hungarian pages.
In the time since the last status report, six security advisories have been issued concerning problems in the base system of FreeBSD; of these, five problems were in "contributed" code, while one was in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 57 new entries have been added, bringing the total up to 814.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, FreeBSD 5.5, FreeBSD 6.0, and FreeBSD 6.1. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 5.3 and FreeBSD 5.4 will cease to be supported at the end of October 2006, while FreeBSD 6.0 will cease to be supported at the end of November 2006 (or possibly a short time thereafter in order to allow time for upgrades to the upcoming FreeBSD 6.2).
I spent the months of May through August working on improving Portsnap, FreeBSD Update, and devoting more time to my (continuing) role as Security Officer. FreeBSD Update is now part of the FreeBSD base system and is fully supported by the FreeBSD Security Team; updates are currently only being built for the i386 architecture, but AMD64 updates will become available soon.
In an attempt to reduce the number of people running out of date (and unsupported) FreeBSD releases, I wrote an automatic binary upgrade script for upgrading systems from FreeBSD 6.0 to FreeBSD 6.1; I will be releasing a new script for upgrading to FreeBSD 6.2-(RC*|RELEASE) soon (possibly before this status report is published).
Further improvements to Portsnap are still ongoing.
My work is moving slowly forward. ZVOL is, I believe, fully functional (I recently fixed snapshots and clones on zvols), which means you can put UFS on top of RAID-Z volume, take a snapshot of the volume, clone it if needed, etc. Very cool. The hardest part is the ZPL layer, I'm still working on it. Most file system methods work, but probably need detailed review and many fixes. Most of the time these days I'm spending on implementing mmap(2) correctly. It works more or less in simple tests but fails under fsx program. On the other hand, 'fsx -RW' works very stable and reliable. Other test programs (those that don't use mmap(2)) also work quite well. There is still a lot of work to do, mostly in ZPL area, many clean-ups, etc. Some functionality (like ACLs) I haven't even tried to touch yet.
TSO - TCP Segmentation Offload support has been committed to the network stack of FreeBSD-current in September 2006. With TSO, TCP can send data in the send socket buffer in bulk down to the network card which then does the splitting into MTU sized packets. On bulk high speed sending the performance is increased by 25% (normal writes) to 108% (sendfile). Jack Vogel and Prafulla Deuskar of Intel committed the driver changes for TSO hardware support of em(4) based network cards.
These changes are scheduled to be backported to FreeBSD 6-STABLE shortly after FreeBSD 6.2-RELEASE is published to appear in upcoming FreeBSD 6.3 early next year.
This work was sponsored by the TCP/IP Optimization Fundraiser 2005.
The addition of TSO (TCP Segmentation Offload) has highlighted some shortcomings in the sendfile(2) and sosend_*() kernel implementations.
The current sendfile(2) code simply loops over the file, turns each 4K page into an mbuf and sends it off. This has the effect that TSO can only generate 2 packets per send instead of up to 44 at its maximum of 64K. kern_sendfile() has been rewritten to work in two loops, the inner which turns as many pages into mbufs as it can -- up to the free send socket buffer space. The outer loop then drops the whole mbuf chain into the send socket buffer, calls tcp_output() on it and then waits until 50% of the socket buffer are free again to repeat the cycle. This way tcp_output() gets the full amount of data to work with and can issue up to 64K sends for TSO to chop up in the network adapter without using any CPU cycles. Thus it gets very efficient especially with the readahead the VM and I/O system do.
Looking at the benchmarks we see some very nice improvements: 181% faster with new sendfile vs. old sendfile (non-TSO), 570% faster with new sendfile vs. old sendfile (TSO).
The current sosend_*() code uses a sosend_copyin() function that loops over the supplied struct uio and does interleaved mbuf allocations and uiomove() calls. m_getm() has been rewritten to be simpler and to allocate PAGE_SIZE sized jumbo mbuf clusters (4k on most architectures). m_uiotombuf() has been rewritten to use the new m_getm() to obtain all mbuf space in one go. It then loops over it and copies the data into the mbufs by using uiomove(). sosend_dgram() and sosend_generic() have been changed to use m_uiotombuf() instead of sosend_copyin().
Looking at the benchmarks we see some very nice improvements: 290% faster with new sosend vs. old sosend (non-TSO), 280% faster with new sosend vs. old sosend (TSO).
Newly written is a specific soreceive_stream() function for stream protocols (primarily TCP) that does only one socket buffer lock per socket read instead of one per data mbuf copied to userland. When doing netperf tests with WITNESS (full lock tracking and validation enabled) the receive performance increases from ~360Mbit/s to ~520Mbit/s. Without WITNESS I could not measure any statistically significant improvement on a otherwise unloaded machine. The reason is two-fold: 1) per packet we do a wakeup and readv() is pretty much as many times as packets come it, thus the general overhead dominates; 2) the packet input path has a pretty high overhead too. On heavily loaded machines which do a lot of high speed receives a performance increase should be measureable.
The patches are scheduled to be committed to FreeBSD-current at end of October or early November 2006.
This work was sponsored by the TCP/IP Optimization Fundraiser 2005.
As a participant of Google's Summer of Code 2006, I am focusing on porting Xen to FreeBSD these months. The result of this summer's work include a domU kernel that could be used for installation, a guide for getting started with FreeBSD on Xen, and some other trivial improvements. But there are still a lot of work needing to be done in this area, e.g, the long-expeted dom0 support. So I will continue my work here and try to keep up with the update of Xen itself.
Gvirstor is a GEOM class providing virtual ("overcommit") storage devices larger than physical available storage, with possibility to add physical storage on-line when the need arises. Current status is that it's done and waiting commit to HEAD, scheduled for some time after 6.2 is released.
The ports PRs surged (especially due to a large number of new port submissions), but with some hard work we have been able to get back down to around 900. We are rapidly approaching 16,000 ports.
Due to this acceleration in adding new ports, portmgr is now very concerned that we are outstripping the capacity of both the build infrastructure and our volunteers to keep up with build errors and port updates. Accordingly, we've added a guideline (not a rule) that ports should be of more than just theoretical use to be added to the Ports Collection (e.g. we can't support all of CPAN + all of Sourceforge + everything else). Basically, use common sense as a guideline; certainly no one wants to see any kind of "gateway" procedure to get incoming ports approved.
Seven sets of changes have been added to the infrastructure, mostly refactoring and bugfixing.
As part of a Summer of Code project, we have also incorporated some of gabor@'s changes to incorporate better DESTDIR support. However, due to some unanticipated side-effects, more work is going to be needed in this area. gabor@ is continuing to work on the changes.
netchild@ and bsam@ have been doing a great deal of work to bring the linux emulator ports closer to sanity, including bringing up a regression-test suite.
The long-anticipated import of X.Org 7 has stalled due to developer time, mostly to deal with documentation and upgrade instructions. Hopefully this can get done in the early 6.3 development cycle. See the wiki for more information.
As a part of that work, the decision has been made to move away from using X11BASE and just put everything into LOCALBASE; /usr/X11R6 is simply an artifact at this point. A plan for a transition process is underway; a great deal of testing will need to be done, but in the end the ports tree will be much cleaner. The GNOME team has already done the work to move all of their ports over, and it will be incorporated after the 6.2 release is shipped.
tmclaugh@ is looking for someone to take over the C# ports. He has maintained them for over a year and wants more time to be able to work on other projects.
Some work has been done to get rid of FreeBSD 2.X cruft in ports. Further work is needed to get the 3.X cruft removed.
linimon@ did another pass through resetting inactive maintainers. Another list is waiting in the wings.
linimon@ is also working on adding the ability for portsmon to analyze successful packages (not just failed ones), so that queries such as "show me packages that build on i386 but not amd64" and "show me why dependent package foo was not built on bar". This is currently in alpha testing.
We have added 4 new committers since the last report.
CScout is a refactoring editor and source code browser for collections of C code. The aim of the project is to make it easy for FreeBSD developers to use CScout and to improve the FreeBSD source code quality through CScout-based queries and refactorings.
CScout was first applied to the FreeBSD kernel in 2003. Its application at that point involved substantial tinkering with the build system. The version released in October 2006 makes the running of CScout on the three Tier-1 architectures a fairly straightforward procedure. The current version can also draw a number of call graphs; this might help developers better understand foreign code.
Libelf is a BSD-licensed library for ELF parsing & manipulation implementing the SysV/SVR4 (g)ELF[3] API.
Current status: Implementation of the library is nearly complete. A TET-based test suite for the API is being worked on.
Progress this month has been limited due to my sea-change, moving house to the country.
Sun's OpenSolaris developers have followed through and released the DTrace test suite as part of the OpenSolaris distribution.
jkoshy@'s work on libbsdelf is nearing feature completion for DTrace and will make life easier in FreeBSD for DTrace, given that we have more architectures to support than Sun has.
The FreeBSD project has made available a dual processor AMD64 machine for DTrace porting.
I am currently working through the diffs between the DTrace project in P4 and -current, committing files to -current if they are ready.
The TrustedBSD audit implementation provides fine-grained security event logging throughout the FreeBSD operating system. The big news for the last quarter is that the TrustedBSD audit implementation has been merged into RELENG_6 branch, and appeared in 6.2-BETA2. Over the past few months, work has also occurred in the following areas:
Lots of testing as part of the 6.2-BETA cycle would be much appreciated. Audit support will be considered an experimental feature in FreeBSD 6.2-RELEASE, but we hope that it will be a production feature in 6.3-RELEASE.
The MMC/SD stack got a significant boost this quarter. Warner Losh and Bernd Walter have written a generic MMC/SD flash card stack for FreeBSD, and have implemented a host controller for the AT91RM9200 embedded ARM controller they are each using in separate projects.
The stack is presently experimental in quality. It is being used as the root file system for these embedded projects. There's been no work done to support hot insertion and removal of cards (neither board wires up the pins necessary, and besides, / disappearing is very bad). There are still many rough edges.
This is a freshly written stack. It has been written using the SD 1.0 (and recently 2.0) simplified specification, with the SanDisk MMC application notes supplementing. The Linux stack looks good, although not entirely standards conforming (there's work in progress that I've not seen that is supposed to fix this) and it is contaminated with the GPL. The OpenBSD stack also looks interesting, but Warner's experience porting NEWCARD over from NetBSD suggested that a fresh rewrite may be faster, at least for the bus and driver level. Since MMC is fairly simple, a port of the sdhci driver might be possible.
Please see the open tasks list.
Support for the UltraSparc T1 (Niagara) continues to improve. The code has recently been checked into public CVS under sys/sun4v.
It isn't clear whether or not I will have time to implement full logical domaining support before the APIs become publicly available. Testing indicates that substantial work will be needed before FreeBSD can take full advantage of all 32 threads.
Work on Xen support has slowly been continuing in perforce. The SOC student fixed several bugs and is continuing to work on it. Someone is needed who has the time to complete dom0 support and shepherd it production level stability.
Sufficient interest has been expressed in it that it probably makes sense to check it in to public CVS so that more people can try it out. Time permitting, I will bring it up to date and check it in the next month.
FreeSBIE is a FreeBSD based LiveCD.
On August 19th, Matteo Riondato, a member of the FreeSBIE staff, released an unofficial ISO, codename FreeSBIE GMV, based on FreeBSD -CURRENT (read the Announcement to download it). This is supposed to be the first in a series of four ISOs that will end up with the release of FreeSBIE 2.0. Matteo is now working on another ISO, codename FreeSBIE LVC, which is scheduled to be released October 12th.
FreeSBIE 2.0 will be based on FreeBSD 6.2-RELEASE and will hopefully be released at EuroBSDCon 2006 in Milan. It will be available for the i386 and AMD64 platforms.
Roman Divacky participated in the Google Summer of Code 2006 and implemented a major part of the syscall compatibility to the 2.6.16 Linux kernel. The work has been committed to -CURRENT (the default compatibility still being a 2.4.2 Linux kernel) and we are working on fixing the remaining bugs as time permits.
"Intron" submitted an implementation for the linux aio syscalls. His work has been committed to the Perforce repository.
We also started to consolidate a list of known bugs, open issues and helpful stuff (e.g. regression tests and their status) in -CURRENT on a page in the FreeBSD wiki (see the links-section). It also contains a link to a more or less up-to-date patch with stuff we have in the Perforce repository so that interested people can help with testing. Thanks to the help of Marcin Cieslak we already fixed some bugs (some of the fixes are already MFCed to -STABLE).
Thanks to the nice regression tests of the Linux Test Project (LTP) we have a list of small (and not so small) things which need to be looked at. This list makes up for a quick start into kernel hacking. So if you have a little bit of knowledge about C programming, and if you want to help us a little bit in improving FreeBSD, feel free to have a look at the list and to try to fix a problem or two. Sometimes it is as easy as "if (error condition) return Esomething;" (but you should coordinate with the emulation mailinglist, so that nobody does some work someone else just did too). Even if you do not know how to program, you can help. Have a look at the wiki page and tell us about things which should get mentioned there too. Or download the patch and test it.
Since the last status report we added basic support for envy24ht chips, imported the emu10kx driver into the base system and added support for High Definition Audio (HDA) compatible chips.
Additionally the work of Ryan Beasley as part of his Google Summer of Code 2006 participation is committed. It adds compatibility to the Open Sound System (OSS) v4 API as far as this was possible. This allows for more sophisticated programs to be written. For example it is now possible to synchronize the start of multiple sound channels. It is also possible for a driver to support more than the AC97 mixer devices, but so far no driver has been extended to support this yet. More about it can be found in the wiki and in the official OSS documentation.
The wiki page about the sound system was started to describe the current status of the sound system and to provide some information about where we are heading. But more work needs to be done to reach this goal. So far we collected some information about the status of the most recent work in the soundsystem. So if you have a look at it and you think that something important is missing, just tell us about it. While fully prepared content is very welcome, we are even happy about some ideas what we should list on the wiki page.
Work is almost finished to implement the Rapid Spanning Tree Protocol (RSTP) which supersedes Spanning Tree Protocol (STP). RSTP has a much faster link failover time of around one second compared to 30-60 seconds for STP, this is very important on modern networks. The code will be posted shortly for testing and feedback.
There were a number of OCaml ports in our tree, and each of them was doing the same work by maintaining OCaml ld.conf in the correct state, installing/removing their files/entries etc. To simplify the task of OCaml-language ports creation, the special framework (bsd.ocamk.mk) was developed and most of the ports were converted to use this framework. This allowed a lot of duplicate code to be removed. This new framework handles all the things required to install an OCaml-language library and properly register it. bsd.ocaml.mk also contains knobs to deal with findlib-powered libraries, modify ld.conf in the proper way, etc. Also, a lot of new Ocaml-related ports were added.
Integration of the new innovative e17 window manager into the ports tree is almost completed. A lot of new e17-related applications was ported, all old ports were updated to the latest stable cvs snapshot. The special framework (bsd.efl.mk) was created to support the whole thing and simplify the creation of dependent ports. I'll commit the changes in the days before the ports freeze.
Thanks to Sergey Matveychuk (sem@) for providing a machine to place CVS snapshots on. Without his help it will be impossible.
Last month I was working on a driver/module to update the microcode of Intel or AMD CPUs that support having their microcode updated. As you might know these processors are microcode-driven and this firmware can be updated. Intel(R) often releases microcode updates, and AMD(R) updates can be found in BIOS programs. The work is almost finished now, I just need to find a bit of time to test it on AMD64 systems and perform some code cleanup. The driver also provide a way for userland programs to access the Machine Specific Registers (MSR) and CPUID info for a certain cpu. This will allow some programs like x86info to provide more accurate information about cpus in SMP systems and make assumptions based on the contents of the MSR.
Thanks to John Baldwin, Kostik Belousov, John-Mark Gurney and Divacky Roman for helping during development.
During the Google Summer of Code 2006, Gábor worked on several ideas to improve the ports infrastructure:
The first three items have been completed and the next two items are being worked on. The DESTDIR support was more complicated than presumed and took more time than expected to complete. Gábor will continue working to finish these tasks and other ports related tasks. FreeBSD is happy to have interested him to keep working on ports and ports infrastructure.
I thought that since I sent a status report the last time, I might as well send one now.
Since the last status report I have done work on several of the remaining commands as attach, detach, and finally the concat command to be able to create concatenated volumes with one easy command. The mirror and stripe commands are the next step after this.
The most important thing I've been working on is maybe the implementation of drivegroups. I have posted a bit information on this mailinglists, but basically, it's a way to group drives with the same configuration. This way, you can make many commands operate on groups instead of drives, and the group-abstraction will handle how the underlying subdisks are created on the drives. In the future one will be able to move groups to different machines, etc.
I've created a patch of all my work that is not in HEAD yet here - (this is a snapshot of my developement branch, so how thing's are + (this is a snapshot of my development branch, so how thing's are done might be changed quite fast): http://folk.ntnu.no/lulf/patches/freebsd/gvinum/gvinum_all_current.diff
Be aware that a there will probably be bugs in the code, so don't use it in production yet!
Thanks to Greg Lehey for offering to help me on getting this into CVS, and all feedback on this has been good.
I have setup the FreeBSD Multimedia Resources List, a one-stop-shop for FreeBSD related podcasts, vodcasts and audio/video resources. Hopefully this list will make it easier for people to find and keep up to date with these recordings. The overview is available as a normal HTML page and as an XML/RSS feed.
The ultimate goal is to have this list to reside under the www.FreeBSD.org umbrella.
A BRIDGE monitoring module for FreeBSD's BSNMP daemon has been implemented. In addition to RFC 4188 single bridge support and extending the kernel to get access to all the information, a private MIB was designed in order to be able to monitor multiple bridges supported by FreeBSD. The kernel part has already been committed to -CURRENT (thanks to thompsa@), for -STABLE a patch is available (see the wiki), code has already been reviewed.
SoC 2005 work on SNMP client tools is now available too via port (net-mgmt/bsnmptools), thanks to Andrew Pantyukhin for the port.
The dates for BSDCan 2007 have been set: 11-12 May 2007. As is usual, BSDCan will be held at University of Ottawa, with two days of tutorials prior to the conference starting.
The call for papers will go out in mid December. Start thinking about your submissions now!
The new 2U server mentioned in the last report now has a collection of Raptor drives in a RAID-10 configuration. Thanks to very generous donations from the community, I purchased eight of these drives at very good prices. The server will be deployed in the next few weeks.
There has been quite a bit of work since the last report in June. Some highlights include:
For more detail, please review the FreshPorts Blog .
The FreeBSD Foundation continued to support the FreeBSD project and community through various activities. These activities include creating strategies for fund development and actively seeking funding for the FreeBSD community, coordinating a new IBM Bladeserver project, and protecting the image and integrity of FreeBSD by governing the use of the trademarks. We are pleased to be a sponsor of EuroBSDCon and will be sponsoring a few developers to attend the conference through our travel grant program. And finally, we have secured funds for a major project that will be announced later this month.
This report covers FreeBSD related projects between April and June 2007. Again an exciting quarter for FreeBSD. In May we saw one of the biggest developers summits to date at BSDCan , our 25 Google Summer of Code students started working on their projects - progress reports are available below, and finally the 7.0 release cycle was started three weeks ago.
If your are curious about what's new in FreeBSD 7.0 we suggest reading Ivan Voras' excellent summary at: http://ivoras.sharanet.org/freebsd/freebsd7.html and of course these reports.
The next gathering of the BSD community will be at EuroBSDCon in Copenhagen , September 14-15. More details about the conference and the developer summit are available in the respective reports below.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
Mpd-4.2 has been released. It includes many new features, performance improvements and fixes.
The most significant and unique new feature is a link repeater functionality. It allows mpd to accept incoming connection of any supported type and forward it out as same or different type outgoing connection. As example, this functionality allows mpd to implement real LAC with accepting incoming PPPoE connection from client and forwarding it using L2TP tunnel to LNS. All other software L2TP implementations I know is only a LAC emulators without real incoming calls forwarding abilities.
Also mpd-4.2 presents:
Replacing external ifconfig and route calls with their internal implementations and other optimizations in 4.2 gave significant performance boost in session management. Newly implemented overload protection mechanism partially drops incoming connection requests for periods of critical load by monitoring daemon's internal message queue. As result, simple 2GHz P4 system is now able to accept, authenticate and completely process spike of 1000 concurrent PPPoE connections in just a 30 seconds.
The basic idea behind this project is to implement secure and reliable log file shipping to remote hosts. While the implementation focuses on audit logs, the goal is to build tools that will make it possible to perform distributed logging for any application by using a simple API and linking with a shared library.
OpenBSD includes sysctl hw.sensors framework since 2003; since 2005 the frameworks supports raid drives and most known i2c sensors; since 2006 the framework is redesigned with a sensor device concept in mind to accommodate continued growth. Consists of kernel api, sysctl(3)/sysctl(8), sensorsd(8), ntpd(8), systat(1), ports/sysutils/symon and 51 drivers as of 2007-07-07.
This GSoC2007 project is to port the underpinnings of this unified hardware monitoring interface to FreeBSD. Whilst it won't be possible to port all of the drivers due to architecture differences, we aim at porting all other parts of the framework and accompanying userland utilities.
At this time, lm(4) at isa and some kernel api have already been ported. The next big step is to complete sysctl(3) glue code so that further work on porting userland utilities could be accomplished. Details about sysctl are being discussed on arch@.
The Linux kernel-based Virtual Machine (KVM) is a mechanism to exploit the virtualization extensions present in some modern CPUs (e.g., Intel VT and AMD-V). Virtualization extensions let ordinary processes execute a subset of privileged instructions in a controlled way at near-native speed. This in turn may improve the performance of system emulators such as qemu, xen, vmware, vkernel, User Mode Linux (UML), etc.
This project consists in porting to FreeBSD the Linux KVM, implemented as a loadable module, lkvm.ko. We use the approach in ports/devel/linux-kmod-compat to reuse the original Linux source code almost unmodified. We will also port a modified version of qemu which exploits the facilities made available by the Linux KVM to speed up emulation.
The URL above links to progress report detailing the exact project goals, milestones reached, and commit log details.
As of end of June 2007, we have mainly extended linux-kmod-compat to support the kernel API used by the Linux KVM code. The required functions have been implemented at various degrees, from simple stubs to fully functional ones. We have also imported the modified qemu and the libraries that are used to build the Linux KVM userspace client. In the second half of the SoC work we plan to complete the implementation of the kernel API and have a fully functional Linux KVM module, together with its client (qemu).
This project aims to create a multicast DNS daemon and service discovery utilities suitable for the base system. Multicast DNS is a part of Zero Configuration Networking (Zeroconf) and provides the ability to address hosts using DNS-like names without the need of an existing (unicast), managed DNS server. Work on the responder daemon is well underway and the only large missing piece of the puzzle is a way for local clients to do queries. The code can be found in the p4 branch projects/soc2007/fli-mdns_sd if anyone would like to give it a spin, even though it's incomplete. The project plan can be found on the wiki.
The project is split up with a front end to interact with the user and a back end to interact with freebsd-update. The back and front ends are able to communicate with each other using an XML protocol. The GUI is almost at the point it can take a command from the user and send it to the back end. The back end is able to detect when updates are ready.
The sixth EuroBSDCon will take place at Symbion in Copenhagen, Denmark on Friday the 14th and Saturday 15th of September 2007.
The programme is ready and online at the webpage. Registration
is open. Details about tutorials and Legoland trip are ready too.
The keynote will be John Hartman: Real men's pipes
If you share a room with friends at the hostel, then lodging is really inexpensive, and the lounge has high speed Internet access. Staying at the hostel is of course optional, and the area has several hotels.
KD85.com and O'Reilly will each have a booth at the conference.
We are still looking for more sponsors.
A public IRC channel #eurobsdcon on EFnet has been created for discussion and questions about the conference.
After the success of FreeSBIE-2.0.1-RELEASE, development slew down a bit, but we have a big task for the summer: enable unionfs again and trying the new efficient memory filesystem, tmpfs.
For all new ISO images we will be following RELENG_7, with the hope to release a stable image once 7.0-RELEASE have been released.
Gábor Kövesdán is working on some improvements for the Ports Collection infrastructure. This year, he aimed to work on long-standing issues, which are tracked in GNATS, but we have not had a volunteer for recently. With the mentorship of Andrew Pantyukhin, he is also reimplementing the DESTDIR support for Ports Collection in a more practical way. The complete description and status of this project is available on Gábor's SoC 2007 Wiki page.
We have added one translated article since the last status report about this project. The infrastructure is ready to support localized articles and books as well, we just lack of human resource. New volunteers are highly welcome! Please see the link below and contact Gábor if you are interested.
Tarfs is a simple tar file system implementation for FreeBSD.
The current goals are:
Here's the current state of things:
FAST_IPSEC has now replaced Kame IPsec as the IPsec stack in HEAD. This will be part of the 7.0 release. The merge happened in early July with George handling the kernel bits and Bjoern handling user space.
The kernel option IPSEC is now the ONLY option for IPsec support in the FreeBSD kernel.
During the last three months there has been several changes to the USB stack. Here is a quick list of the most important changes:
Markus Brueffer is still working on the USB HID parser and support. Nothing has been committed yet.
If you want to test the new USB stack, checkout the USB perforce tree or download the SVN version of the USB driver from my USB homepage. At the moment the tarballs are a little out of date.
Ideas and comments with regard to the new USB API are welcome at freebsd-usb@FreeBSD.org .
Gvirstor is a GEOM class which provides virtual storage capacity (something like virtual memory for storage devices). It's ready to be committed to HEAD (the plan is for it to get into 7.0-RELEASE).
Project "finstall" aims to create a next-generation FreeBSD installer that will make use of the newest features present in the system. The project should yield something usable for 7.0-RELEASE, but the intention is to keep it as a "second" installer system during 7.x, alongside sysinstall. In any case, sysinstall will be kept for architectures not supported by finstall (e.g. all except i386 and amd64).
This entry was previously the Bt878 Audio Driver (aka FusionHDTV 5 Lite driver) announcement, but as work expanded slightly, it's a bit more generic now.
A few bugs in bktrau has been fixed since January. If you have been running an earlier version, it is recommended to upgrade as the driver could panic. The driver works with multiple cards in the same machine (tested with two).
FusionHDTV 5 Lite -- Due to lack of documentation from DViCO and LG, I have copied magic values from the Linux driver to get ATSC capturing working.
ATI HDTV Wonder -- After years of trying to get into the ATI developer program, they have finally suspended it, so no support from ATI. I have started work on a driver, cxd, for the Conexant CX2388x based cards. The ATI HDTV Wonder uses ATI's own demodulator, and I was able to get it to tune, after cribbing from the Linux driver. When capturing, I get some valid data, but not all the data. Due to lack of support from ATI and linux-dvb the project has been put on indefinite hold.
If someone has another CX2388x based card, it shouldn't be too hard to take the driver and get it working with a different tuner.
A Python module is available for both drivers/cards, along w/ a sample capture application using it. The module is now known to work well with threads so that tuning (expensive due to i2c ioctl's) can happen in another thread without causing program slow down. The module is working well with a custom PVR backend.
Gavin Atkinson has joined the bugbuster team via getting a GNATS account on the FreeBSD cluster. He is following in the footsteps of Matteo Riondato, who later graduated to a full src commit bit. So far, he has helped close nearly 150 PRs, including many that had become stale. Welcome!
Our short-term goal is to try to identify bugs that we might be easily able to fix before the 6.3/7.0 simultaneous release. So far, great progress has been made on ata- and usb-related PRs.
The goal for the rest of this year is to generate more developer interest in fixing bugs. To do this, we are, first, trying to do more work on triaging PRs as they come in, to help flag ones that seem to be valid problems (especially if they include patches.) Secondly, we have started a new weekly periodic posting to the freebsd-bugbusters@FreeBSD.org mailing list, which is a short list of PRs that we feel are ready for committer action. This posting is automatically generated from a text-file list that we maintain.
We are continuing to try to manage our community's expectations of what we can do with the incoming PRs. In particular, we are trying to discourage submissions of the form "I cannot get the XYZ function to work". In practice, these PRs are not worked on. Instead, we are now encouraging these postings to go to one of the mailing lists such as freebsd-questions@, freebsd-x11@, and so forth. The idea is to emphasize GNATS as a "Problem Report" method, rather than a "general FreeBSD support" method. I feel that, otherwise, we were creating a false expectation.
The overall PR count has dropped to below 5000, despite the extra PRs still not cleared up from the ports freeze for the xorg7.2 import. Significant progress has been made on the i386, kern, and bin PRs, as well as PRs in the 'feedback' state. In addition, Warner Losh has made progress on closing many of the usb PRs.
The ports count is over 17,300. The PR count has been stable at around 800; we have not quite cleared up the backlog that showed up during the freeze to import xorg7.2.
There have been 4 experimental runs on the build cluster, most notably resulting in some speedups for package registration. A further experimental run to genericize autotools handling is in progress.
One of the most sweeping ports commits to happen in years was the upgrade of xorg from 6.9 to 7.2. This involved a complete rework of the internals of the port, as X.org itself has effectively pushed the responsibility for packaging to the OSes that incorporate it. The idea was to be able for them to update individual code (such as video drivers) without having to reroll the entire distribution. This commit caused us to have the longest period of preparation work, and actual tree lockdown, that I am aware of. The commit continues to be controversial, partly due to the fact that none of our port upgrade tools was up to the task of doing the upgrade without manual intervention.
At the same time that xorg was upgraded, we moved the installation directory from the obsolete /usr/X11R6 to our default /usr/local. This further complicated the upgrade.
There have been new releases of the ports tinderbox code, the portmaster update utility, and portupgrade.
GNOME was updated to 2.18.2.
We have added 7 new committers since the last report. We appreciate all the new help. However, a few committers have turned in their commit bits for safekeeping, due to lack of time.
Unfortunately, Clement Laforet has also had to step down from portmgr due to lack of time. We thank him for his help so far.
Erwin, Kris and Mark met up at BSDCan and reviewed all the portmgr-owned PRs. A large number were closed, or suspended pending more work from the submitter. After closing the PRs that were committed after the -exp builds, the number of portmgr owned PRs came down to an all time low of 48 from around 70. We hope to make further progress during the rest of the year.
The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This will allow for complete networking independence between jails on a system, including giving each jail its own firewall, virtual network interfaces, rate limiting, routing tables, and IPSEC configuration.
I believe that the prototype, which is kept in sync with FreeBSD -CURRENT, is now sufficiently stable for testing. It virtualizes the basic INET and INET6 kernel structures and subsystems, including IPFW and PF firewalls, and more. In the next month I plan to have the IPSEC code fully virtualized, and refine and document the management APIs. The short-term goal is to deliver production-grade kernel support for virtualized networking for FreeBSD 7.0-RELEASE (as a snap-in kernel replacement), while continuing to keep the code in sync with -CURRENT for possible merging at a later date.
IP can easily be tunneled over a plethora of network protocols at various layers, such as IP, ICMP, UDP, TCP, DNS, HTTP, SSH. While a direct connection may not always be possible due to a firewall, the IP packets could be encapsulated as payload in other protocols, which would get through. However, each such encapsulation requires the setup of a different program and the user has to manually probe different encapsulations to find out which of them works in a given environment.
mtund is a tunneling daemon using run-time loadable plugins for the different encapsulations. It automagically selects the best encapsulation in each environment and fails over to another encapsulation in case the environment changes. There already is running code available, capable of tunneling via TCP and UDP with a working failover mechanism. As this is a Summer of Code project, rapid changes and addition of new features can be expected during the summer. Please see the wiki page for more details and up-to-date information.
Note that the project originally started under the name of Super Tunnel Daemon, but was later renamed to mtund for Magic Tunnel Daemon.
Over the past 6 months several developers undertook an effort to replace the global scheduler lock with a finer-grain interface modeled on the Solaris container lock approach. This significantly reduces contention on higher-end multiprocessor machines.
This patch went into 7.0-CURRENT and has proven to be very stable. The last remaining bugs are in rusage and effect only process time accounting statistics.
SCHED_SMP is a fork of the ULE scheduler which makes use of the new fine grain scheduler locking in 7.0-CURRENT to significantly improve SMP performance on some workloads. It has improved and stronger affinity, smarter CPU load balancing, structural improvements and many sysctl tunables. This can be considered ULE 3.0. Discussions are ongoing as to whether this will go into 7.0 as SCHED_SMP or as SCHED_ULE in 7.0 or 7.1.
SCHED_ULE has had many bugfixes and performance improvements over the 7.0 development cycle and should no longer be considered unstable or experimental. On most workloads it significantly outperforms SCHED_4BSD on SMP and even slightly outperforms it on UP. There are some pathlogical workloads which exhibit as much as a 5% performance penalty. Many thanks to Kris Kennaway and current users for bug reports and performance testing.
FreeBSD developers have been using mysql as a testbed to find kernel contention hotspots in the kernel. As a result of this we have seen a 5x performance improvement over 6.0 on 8way machines. Recent changes include finer locking in fcntl(), removing Giant from flock and fcntl F_SETLK. These changes will be available in 7.0 and primarily improve write performance. Experimental changes to select() have also been discussed on arch@ that solve contention issues there however these will not be ready in the 7.0 timeframe.
The last major updates are currently being made to PC-BSD 1.4, which will include KDE 3.5.7, Beryl, Flash, Intel Wireless, Nvidia Drivers and more! This release will also include new utilities to make running PC-BSD on the desktop easier than ever, including:
Once any final major issues are resolved, we will be issuing a public beta of PC-BSD 1.4 to ensure compatibility across a variety of platforms.
The next developer summit will be different from the previous ones.
Very different.
Gone are the auditorium style seating, beamers, endless presentations and soggy sandwiches.
Instead we head out to an old village school in the beautiful Danish countryside, we hang around all over the place, sleep in the old science room, cook our own food and hack the living daylights out of anything we care for.
September 17th and 18th, right after EuroBSDcon2007 in Copenhagen. (Well, right after the optional trip to legoland...)
Be there!
PS: Yes, it's not uncivilized, there is a full speed ADSL and WLAN.
Main goal of project is to introduce code working in PXE preboot environment, able to download from web server via direct connection or http proxy and prepare booting of FreeBSD kernel.
Already implemented, but haven't thoroughly tested: PXE wrappers core code, ARP, ICMP echo request/reply, sockets code similar to common sockets (UDP and TCP modules). On base of sockets: simple DHCP client, DNS client.
Currently working on http client, TCP testing, kernel booting and documenting main concepts of project modules.
Code freeze in preparation for FreeBSD 7.0 began on June 18th. There are several large projects still being finished up as well as some issues that resulted as "fallout" from the work done just before the code freeze started (e.g. things resulting from the GCC 4.2 import). A schedule for the 7.0 release has not been set yet but the hope is that the first BETA build will be done near the end of July with a "fairly normal" release cycle (a few BETA builds followed by two or three RCs, each separated by around two weeks).
We are planning to release FreeBSD 6.3 around the same time as FreeBSD 7.0 is released so the release schedule for that will be set at the same point we set the release cycle for 7.0, hopefully late in July.
Support was added for two more 10gigabit network drivers and there were major advances on improving system performance over 10g media.
Kip Macy committed a new driver for the Chelsio adapters. The cxgb driver supports all current 10g adapters, as well as the new four-port gigabit model. The cxgb driver work was supported by Chelsio.
Drew Gallatin made significant improvements to the Myricom 10g driver mxge. With these updates the driver does line rate transfers with less system overhead.
Neterion contributed the nxge driver to support all their Xframe 10Gbe Server/Storage adapters. The initial driver import was done by Sam Leffler; a switch over to vendor support will happen soon.
Jack Vogel is preparing a driver to support the latest Intel 10g hardware devices. The new driver - ixgbe - will complement the existing ixgb driver that supports older Intel 10g cards.
Kip and Drew worked with other folks on performance analysis and tuning. This work improved cpu affinity and reduced overhead for managing network resources. Work is also underway to define a common Large Receive Offlaod (LRO) infrastructure. LRO is analogous to TSO on the receive side enabling drivers to receive at near line rate with normal sized frames. This common code base will help replace driver-specific code.
This project is due to provide a GUI audit log analysis tool for FreeBSD. Refer to ethereal/wireshark packet parsing engine and its framework to view and parse audit logs.
A basic implementation of ar(1) (include ranlib) was finished and available in the perforce repository. Currently it provides all the main functions an ar(1) should have and it is based on the libarchive and libelf library thus is expected to have a better and simpler structure than the GPL'ed version. The work left in this part of the project is to perform a elaborate test and add additional functions.
The kernel/hwpmc(4) bits of stack trace capture have been implemented and are available in Perforce under path '//depot/user/jkoshy/projects/pmc/...'. I'm currently enhancing pmcstat(8) to extract and summarize this information. Support by Google Inc. for this project is thankfully acknowledged.
Just like last year I got the opportunity to work on updating the Linuxulator to Linux version 2.6. This year I work on finishing futexes, *at syscalls and epoll/inotify.
I, cooperating with Konstantin Belousov, have managed to fix futexes to the state of passing the official futex testing program. The fix was committed and 7.0R will ship with correct futex implementation. Work is planned on removing Giant locking - from futexes. This only needs some carefull review and + from futexes. This only needs some careful review and testing.
These days I mostly focus on *at syscalls, the patch is almost finished for committing and I hope that it will make it into 7.0R. As a part of this work I implemented native FreeBSD syscalls as well. Watch arch mailing list as I post the patch there.
I also finished writing my master thesis describing how the Linuxulator works and Gábor Kövesdán is working on integrating it into official FreeBSD articles.
No work has happened in the epoll/inotify area but I hope to work on it right after I finish the *at syscalls.h
Security Regression Test is supported by the project of Google summer code 2007. The main objective of this stage is to test the correctness of FreeBSD Mandatory Access Control Framework including correctly passing the security label from userland to kernel and non-bypassibility of Mandatory Access Control Hooks.
Work performed in the last month:
General cleanups in preparation for 7.0.
Process audit state moved to the credential to allow it to be accessed lock-free in most cases, as well as allowing it to be used in asynchronous contexts.
OpenBSM 1.0a14 has been imported, which: fixes IPv6 endian issues, makes OpenBSM gcc41 warnings clean, teaches audit_submit(3) about getaudit_addr(), adds zonename tokens; other changes since the existing CVS 1.0a12 release previously imported include man page improvements, XML printing support, better audit.log.5 documentation, additional 64-bit token types, and new audit event identifiers.
MAC checks have been added so that MAC policies can control use of audit system calls.
Additional system call arguments are now audited.
Audit now provides a security.audit sysctl node in order to determine if audit support is compiled in; boot-time console printfs have been removed.
"options AUDIT" is now in the 7-CURRENT GENERIC kernel, so AUDIT support will be available out of the box in 7.0 without a kernel recompile. Manually enabling audit support in rc.conf will still be required. With FreeBSD 7.0, AUDIT will be a fully supported, rather than experimental, feature.
The FreeBSD Foundation ended Q2 raising over $116,000. We're almost half way to our goal of raising $250,000 this year! We continued our mission of supporting developer communication by helping FreeBSD developers attend BSDCan. We were also a sponsor of BSDCan and the developer summit. We are a sponsor of EuroBSDCon 2007 and are now accepting travel grant applications for this conference. Foundation board members met with representatives of companies that use or are thinking of using FreeBSD both in the bay area and Ottawa.
The Foundation has negotiated a joint development agreement with Google, Inc. to sponsor FreeBSD developer Joseph Koshy to improve FreeBSD's HWPMC implementation, including adding stacktrace support, and a donation of SMP hardware for future SMP scalability work. We greatly appreciate Google's support for this project, which will facilitate performance measurement and optimization of both the FreeBSD operating system and applications running on it.
To learn more about what we're doing, go to our website at http://www.FreeBSDFoundation.org/ . Our July newsletter will be published soon to update you on how we've been supporting the project and community worldwide.
Cleanup of MAC Framework API/KPI layers: mac.h is now just the user and user<->kernel API; mac_framework.h is the kernel<->MAC Framework KPI, and mac_policy.h is the MAC Framework<->MAC policy module KPI. Along similar lines, mac_label_get() and mac_label_set() accessor functions now allow policies to access label data without encoding struct label binary layout into policy modules, opening the door to more efficient layouts. struct label is now in mac_internal.h and used only inside the MAC Framework.
General MAC policy cleanup, including removing no-op entry points and sysctls for some sample policies. mac_test(4) has been cleaned up significantly, and counters for all entry points added.
A MAC check for UNIX domain socket connect has been added.
MAC checks have been added so that MAC policies can control use of audit system calls.
MAC checks that duplicate existing privileges but add no additional context have been removed (such as sysarch_ioperm, kld_unload, settime, and system_nfsd) -- checks aligned with privileges but that do provide additional context, such as additional arguments, have been kept.
The Biba and LOMAC policies now implement priv(9) checks, differentiating between privileges that may compromise system integrity models, and those that don't.
The essentially unused mnt_fslabel / mnt_label distinction has been eliminated by moving to a single mnt_label. No functional change to any policy.
Several MAC-related interfaces have been modified to synchronize with the naming conventions present in the version of the MAC Framework adopted in Mac OS X Leopard; significant further changes are in the pipeline to complete this synchronization. While it will not be possible to reuse a policy between the two platforms without careful thinking and modification, this makes porting much easier.
The custom file descriptor array lock has been replaced with an optimized sx lock, resulting in 2x-4x improvement in MySQL transaction rates on 8-core MySQL benchmarks. This improvement is due to moving to shared locking for frequent fd lookup operations, as well as significant optimization of the case where the filedesc lock is highly contended (as occurs in the threaded MySQL server performing constant socket I/O).
The custom socket buffer I/O serialization lock (sblock), previously created by interlocking SB_WANT and SB_LOCK flags with the socket buffer mutex, has been replaced with an optimized sx lock, leading to a 10% performance improvement in MySQL and PostgreSQL benchmarks on 8-core systems. As part of this change, sx locks now have interruptible sleep primitives to allow the SB_NOINTR flag to work properly.
These changes also correct a long-standing bug in socket buffer lock contention and SB_NOWAIT reported by Isilon; a simpler patch has been merged to 6.x to fix this bug without merging loocking changes.
TCP debugging is now properly synchronized using a new tcp_debug_mtx.
UMA allocation counters are now used for pipes rather than custom atomic counters, resulting in lowered overhead for pipe allocation and free.
Significant code cleanup, commenting, and in some cases MFC'ing, has taken place with respect to the network stack and synchronization. Additional DDB debugging commands for sockets of various sorts have been added, allowing listing of socket state from DDB without the use of GDB.
Certain non-MPSAFE subsystems have been removed or will be removed from FreeBSD 7.0, including IPX over IP tunneling (not general IPX/SPX support, just the tunneling over IP), KAME IPSEC (FAST_IPSEC is MPSAFE and now now supports IPv6), i4b, netatm (two other ATM stacks are still present), and ng_h4. Some of these features will be reintroduced in FreeBSD 7.1, but by removing them now, we are able to remove the NET_NEEDS_GIANT compatibility infrastructure that significant complicates and obfuscates the socket and network stack code.
Other measurement and optimization projects continue; however, the 7.0 locking/synchronization work for the network stack is essentially complete.
Further reduction of suser(9) consumers in order to attempt to remove the suser(9) KPI for 7.0. This includes resource limits, System V IPC, PPP, netinet port reuse, the NFS server, and netatalk. Remove unnecessary or redundant privilege checks were possible. UFS-privileges that apply to other file systems have been renamed to VFS privileges.
All suser_cred() flags and priv_check_cred() flags are no longer required, as SUSER_ALLOWJAIL and SUSER_RUID use are determined entirely inside kern_jail.c and kern_priv.c and selected based on the privilege number, not a calling context flag. All privileges are now consistently allowed or not allowed in jail, and consistently use the ruid or euid. We will leave the flags field there as it will likely be used for other things in the future.
Documentation in suser(9) and priv(9) has been updated.
Apple's MacBook computers are nicely designed and have neat features that other laptops don't. While Mac OS X is a nice operating system, UNIX folks (like me) would prefer to run other operating systems like FreeBSD. This project aims to bring bug fixes and new drivers to FreeBSD that would help running this OS on this platform.
In the time since the last status report, two security advisories have been issued concerning problems in the base system of FreeBSD; both of these problems were in "contributed" code maintained outside of FreeBSD. The FreeBSD Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated; since the last status report, 35 new entries have been added, bringing the total up to 925.
In order to improve handling of security issues in the FreeBSD Ports Collection a new "ports-security" team has been created to include ports committers who periodically help with fixing ports security issues and documenting them in the FreeBSD VuXML document. Committers who wish to help with this effort can contact simon@ for details.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 5.5, FreeBSD 6.1, and FreeBSD 6.2. The respective End of Life dates of supported releases are listed on the web site; it is expected that of the upcoming releases, FreeBSD 6.3 will be supported for two years after release, while FreeBSD 7.0 will be supported for one year after release.
The project consists in a rewriting of the lockmgr(9) interface on a lighter basis, using atomic instructions and direct usage of the sleepqueue interface. This should lead to a faster primitive, a saner interface and an higher maintainability of the code.
So far, 3 newly files called kern/kern_lockng.c, sys/_lockmgrng.h and sys/lockmgrng.h have been created for the new primitive and an initial implementation has been committed into the perforce branch: //depot/user/attilio/attilio_lockmgr/...
The implementation contains a good set of code intended to replace old lockmgr. Actually it only misses the support for lock draining that will be committed after an initial phase of testing and the inclusion of a better wake-up algorithm (which will simplify draining a lot and will improve performance on wakeup).
My previous status reports contained a lot of code that updated gvinum with the old vinum features.
This year gvinum has been significantly rewritten. Lukas Ertl began rewriting the way gvinum is organized from using a multi consumer/provider model, to use a single consumer and provider, and having an event-system that first handles user-requests, and then runs normal I/O operations (Much like other GEOM classes). This makes the code easier to read, and perhaps there will be less bugs :)
And of course, some time has gone to work out how things should be done, and to fix other bugs. I hope some of you are interested in trying this out (all the work has been in perforce so far), a patch can be found in the URL section. . This is a bit experimental, and although I've done much testing to hunt down bugs, there are most probably bugs left.
I have other goals this summer as well. However, since some parts of gvinum was rewritten, I might not be able to do all of these, but growing is already working for the concatenated volumes (and also mirrored). I'd also like to implement growing for Raid5 arrays as well. Logging plexes would also be cool to have, but this is not really needed, since we have g_journal. Both these features will be addressed after I've made sure gvinum does all old vinum does, and also perhaps better. As I might have some extra time on my hands this summer, so I gladly accept suggestions on what else I might fix or implement "while I'm at it".
Both libarchive 2 and bsdtar 2 are now in -CURRENT and will be in 7.0. Libarchive 1.9 and bsdtar 1.9 should be in 6-STABLE in time for 6.3.
libarchive 2 is much faster writing to disk than libarchive 1. It also supports new formats, has several minor API/ABI corrections, is more portable, and has many fewer bugs. Of special note is "libarchive_test", a new program that exercises much of the libarchive functionality; anyone interested in working on libarchive should become familiar with this test suite. bsdtar 2 is less ambitious, but does have a number of bug fixes and takes advantage of several new features in libarchive 2.
libarchive 1.9 is identical to libarchive 2 except it maintains the old API/ABI. Similarly, bsdtar 1.9 is nearly identical to bsdtar 2, lacking only a few features that would prevent it from being used with existing libarchive 1 libraries.
About 18 months ago, I started to remove the compatibility macros that we had in the USB stack. These macros made it very hard to read the code and to diagnose problems. They represented a barrier to entry for people reading and understanding the stack. In addition, many of them effectively hid bugs from all but the most intensive investigations of the code.
I've removed almost all of the macros in the client drivers, and all instances of the macros in the core FreeBSD USB stack. This makes the drivers more readable, and a little more robust. During this process, I fixed a lot of little bugs that people had been tripping over, and some that people hadn't reported. I've added a boatload of new vendor and product ids to the drivers from user PRs as well as from OpenBSD/NetBSD drivers.
I finished up this work so that the FreeBSD USB stack would be more maintainable during the RELENG_7 period of time. I plan on MFCing most of the changes I've made into RELENG_6 after they have been shaken out in current. There was only one API changes in this work, so this is doable, and makes sharing drivers between 6.x and 7.x much easier. At this stage, it is unclear how long RELENG_6 will be around, so I'm hoping this will make USB much better in 6.3 if that's the release people choose to run.
I've shied away from many of the more complicated changes to the stack. There's work being done outside of the tree by Hans Petter Selasky (hps) to make these sorts of changes. There is much in his stack that's ready to be merged, and I hope to integrate from that work useful bits that can be merged without disruption to improve the FreeBSD USB stack.
I'm also looking for other FreeBSD developers that can jump in and help. Nearly all of the improvements I've done by spending a few hours a week sorting through the PRs for extremely low hanging fruit. There's plenty of room for others to be involved as well in improving FreeBSD's USB stack, as well as chances for us to import the now-useful bits from the evolving hps USB stack, hopefully reducing the diffs between it and the present FreeBSD USB stack. In addition, I'm looking for someone to do similar device ID merges from DragonFlyBSD.
Finally, I've embarked on a mission to try to merge all the BSD's usbdevs files. There's no reason to have separate ones. I've started to modify usbdevs(1) to read the src/sys/dev/usb/usbdevs file and report more verbose information that way. A merged usbdevs would be larger, and take up more memory in a USBVERBOSE kernel, so to mitigate that effect, I'm making changes to usbdevs(1).
A major update of the 802.11 wireless support was committed. Changes include advanced station mode facilities such as background scanning and roaming, and support for 802.11n devices. In addition parts of the Atheros' SuperG protocol extensions were added so that wireless clients that communicate with Atheros-based access points can operate more effectively. The changes to the infrastructure are also important because they simplify future distribution of Virtual AP (VAP) support.
This work represents the effort of many people including Kip Macy, Andrew Thompson, Sepherosa Ziehau, Max Laier, and Kevin Lo. Getting these changes into the tree now ensures they will be present for the lifetime of the 7.x branch.
The scanning and SuperG work were supported by Atheros. The 802.11n-related work was supported by Marvell.
I have been working on making wake on lan (WOL) work with FreeBSD. Contrary to popular believe OS support is required for WOL to work properly. In particular network card drivers need to configure network cards for WOL during system shutdown, else the cards won't wake up. WOL is _not_ just a BIOS issue.
This is work in progress. Currently the following cards/chipsets are supported:
I would be glad to get more feedback on my patch. I can add support for more chipsets but I need testers for hardware I don't have. I would appreciate access to data sheets for any NIC chipsets that are supported by FreeBSD and have WOL support.
I would especially appreciate technical feedback on the patch, preferably by a committer who is willing to nitpick the patch to make it ready for inclusion in -CURRENT. I currently maintain the patch against RELENG_6_2 for my own use but I would port it to -CURRENT for inclusion.
Work is well under way to finish Kip Macy's FreeBSD/xen port, and get it into a shape which is suitable for inclusion in 7.0.
Generally, the port is stable and performs quite well. The major bottleneck is the inability to work with GCC 4.2, this is the last major TODO before the work can be committed.
pf in HEAD (soon to be FreeBSD 7.0) has been updated to OpenBSD 4.1 bringing in a couple of new features:
Some patches that went into OpenBSD after 4.1 and improve performance significantly will be merged later.
Work to support pf and netgraph interaction is underway and will be imported after 7.0. As all required ABI changes have been made during the update, we will be able to MFC this work for 7.1 later on.
FreeBSD's static analysis scans have been updated with a recent version of Coverity Prevent. Coverity is providing additional advice on configuration of the analysis to maximize the benefit from the tools.
At BSDCan2007, Coverity provided FreeBSD with a license for an additional analysis tool called Extend, which allows writing custom FreeBSD specific code checkers. David Maxwell presented training material for interested FreeBSD developers. Some applications of custom checkers have been considered, and more results will be forthcoming as they are implemented and tested.
This report covers FreeBSD related projects between July and October 2007. The sixth EuroBSDCon was held in Denmark in September. The Google Summer of Code project came to a close and lots of participants are working getting their code merged back into FreeBSD.
The bugs in the FreeBSD HEAD branch are being shaked out and it is being prepared for the FreeBSD 7 branching. If your are curious about what's new in FreeBSD 7.0 we suggest reading Ivan Voras' excellent summary here .
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
IP can easily be tunneled over a plethora of network protocols at various layers, such as IP, ICMP, UDP, TCP, DNS, HTTP, SSH. While a direct connection may not always be possible due to a firewall, the IP packets could be encapsulated as payload in other protocols, which would get through. However, each such encapsulation requires the setup of a different program and the user has to manually probe different encapsulations to find out which of them works in a given environment.
MTund is a tunneling daemon using run-time loadable plugins for the different encapsulations. It automagically selects the best encapsulation in each environment and can fail over to another encapsulation. Several plugins have been implemented and the daemon supports multiple concurrent clients.
Note that the project originally started under the name of Super Tunnel Daemon, but was later renamed to Magic Tunnel Daemon (MTund).
After a long break in this project, we started reviewing and refreshing our translations. We have to update the content to reflect the current state of the English version. There are a few parts written in a poor style, another task is to improve these a bit. Any kind of help is highly welcome.
We have a new volunteer, Gábor Páli, who provided us some high-quality contributions. As a result, we have been able to add 5 new articles since the last status report.
There is also an ongoing effort in the Perforce repository to translate the FreeBSD Handbook to Hungarian. Any kind of support is highly welcome.
The sixth EuroBSDCon went well. 215 people attended the conference. Feedback has been very positive.
At the conference we had a Best Talk contest. Steven Murdoch, Isaac Levy and Pawel Jakub "zfs-man" Dawidek each received a prize for their fantastic talks.
Also over 300 pictures from the conference has been uploaded to Flickr with the tag EuroBSDCon2007
Videos and slides from the talks are now online at the conference website.
We thank our speakers for graciously having permitted recording and publication of their talks
EuroBSDCon 2008 will take place in Strassbourg.
The "finstall" project is about the new graphical installer for FreeBSD. The basic frameworks (both client-side and server-side) are done during the SoC 2007 and it's ready for major new features to be implemented. This project should yield an usable installer for 7.0-RELEASE.
With the leaving of bsd@, we lost the GNATS statistics webpages. On this URL I generate a new set of graphs, right now a subset of what bsd@ had, hopefully a superset of that in the future.
We're happy to report the successful conclusion of our third consecutive Google Summer of Code. By all accounts, the FreeBSD participation in this program was an unqualified success. We narrowed down the many impressive applications to 25 that were selected for funding and 92% of these completed successfully and were awarded the full $4,500 stipend. The FreeBSD Foundation was also granted $500 per student from Google for a total of $12,500.
These student projects included security research, improved installation tools, new utilities, and more. Many of the students have continued working on their FreeBSD projects even after the official close of the program. Three students have already been granted full src/ commit access to CVS and more are expected. At least 2 of our FreeBSD mentors will be meeting with Google organizers in Mountain View this month to discuss the program at the Mentor Summit.
GEOM_VIRSTOR (virtual disk space / over-commit GEOM class) has been committed to 7-CURRENT and will ship in 7.0-RELEASE. Thanks to Pawel Jakub Dawidek and others who have made this possible.
During the last three months there has been a flush of changes - going into the FreeBSD USB P4 project. The changes mainly consern + going into the FreeBSD USB P4 project. The changes mainly concern the ability to support the USB device side and multi frame USB transfers. Up to date the FreeBSD USB stack has only supported the USB Host Side. Before Christmas 2007 the P4 USB project will offer USB device support and some simple USB device side implementations. Technically an USB device side driver will look very similar to an USB host side driver. Infact there will be very few differences. Support for multi frame USB transfers opens up the possibility to transfer multiple short-packet terminated USB frames to/from different memory locations resulting in only one interrupt on the USB Host Controller. More specific: I have implemented support for the "alt_next" pointer in the EHCI Transfer Descriptor. This should - give a noticable increase in the maximum number of short-packet + give a noticeable increase in the maximum number of short-packet terminated BULK frames that can be transferred per second.
I regularly get questions from people asking about when the USB P4 project will be merged into FreeBSD-current. The answer is not simple, but probably something like another year. The reason is not that the current code in the USB P4 project is not usable, but rather that the quality needs to be raised in means of making already good solutions more technically excellent, writing more documentation and styling the code.
Ideas and comments with regard to the new USB API are welcome at freebsd-usb@freebsd.org.
Linux KVM (Kernel-based Virtual Machine) is a software package that can be used to create virtual machines fully emulating x86 hardware on top of machines supporting Intel VT-x or AMD-V virtualization extensions, available on newer AMD and Intel processors, e.g., recent Athlon64, Core 2 Duo, Xeon and so on.
Linux KVM has been ported to FreeBSD as a loadable kernel module, using the linux-kmod-compat port (in /usr/ports/devel/) to reuse as much as possible of the original source code, plus an userspace client consisting in a modified version of qemu, that uses KVM for the execution of its guests.
The porting has been completed, many of the limitations present at the end of the Summer of Code have been removed and the known bugs have been fixed. Some configurations have been tested, FreeBSD-CURRENT i386 guests have been booted on Intel and AMD processors, both in i386 and amd64 (host) installations. Only one client at a time is supported by now and performance is not that exciting, but the project seems to be ready to receive wider testing.
The Summer of Code project went well and we reached interesting results. At least the Mac Mini should be fully supported by now. Regarding the other Apple systems, we still need to polish some edges.
The project (started out as a GSoC 2007 project) aims to provide a complete Multicast DNS and Service Discovery suite. Much progress have been made since the last status report and the project is slowly reaching a usable state. Most features are complete and the current focus is on fixing outstanding bugs, fine tuning and testing. However, there are still a few open tasks (see below). More information and snapshots can be found at the wiki page.
New mpd-5.x branch has been started and first public release is planned soon. The main goal of the new branch is to implement new operation principles based on dynamic on-demand links/bundles creation. There are several benefits received from new design:
The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This allows for networking independence between jail-like environmens, each maintaining its private network interface set, IPv4 and IPv6 network and port address space, routing tables, IPSec configuration, firewalls, and more.
The prototype, which is kept in sync with FreeBSD -CURRENT, should be sufficiently stable for testing and experimental use. The project's web page includes weekly code snapshots, as well as a virtualized FreeBSD system installed on a VMWare disk image available for download.
The short-term goal is to deliver production-grade kernel support for virtualized networking for FreeBSD 7.0-RELEASE (as a snap-in kernel replacement), while continuing to keep the code in sync with -CURRENT for possible merging at a later date.
The GSoC2007/cnst-sensors project was about porting the sysctl hw.sensors framework from OpenBSD to FreeBSD. The project was successfully completed, committed into DragonFly BSD, and is now pending final review and integration into the FreeBSD's CVS tree (subject to the tree being unfrozen).
The sensors framework provides a unified interface for storing, registering and accessing information about hardware monitoring sensors. Sensor types include, but are not limited to, temperature, voltage, fan RPM, time offset and logical drive status. In the OpenBSD base system, the framework spans sensor_attach(9), sysctl(3), sysctl(8), systat(1), sensorsd(8), ntpd(8) and more than 50 drivers, ranging from I2C temperature sensors and Super I/O hardware monitors to IPMI and RAID controllers. Several third-party tools are also available, for example, a plug-in for Nagios and ports/sysutils/symon.
As a part of this Google Summer of Code project, all core components of the framework were ported, including sysctl, systat and sensorsd. Some drivers for the most popular Super I/O Hardware Monitors were ported, too: it(4), supporting most contemporary ITE Tech Super I/O, and lm(4), supporting most contemporary Winbond Super I/O. Moreover, some existing FreeBSD drivers were converted to utilise the framework, for example, coretemp(4).
The PC-BSD derivative of FreeBSD is becoming increasingly popular for new users of BSD. Much of the content in the existing FreeBSD Handbook is directly applicable to PC-BSD. We are writing PC-BSD specific installation and port/packages chapters (PBI). These chapters will be checked into docs/en_US.ISO8859-1/books/pcbsd-handbook and will include some of the same chapters as the Handbook does, but with a different &os entity and possibly with some conditional changes in those chapter files.
The ports count is over 17,700. The PR count has decreased a bit to just over 700.
There have been 6 experimental runs on the build cluster. The resulting commits include the fixup of last year's DESTDIR changes, the refactoring of perl bits into bsd.perl.mk, the update of xorg from 7.2 to 7.3, the upgrade of all of the autoconf dependencies to the latest version (wherever possible), and the upgrade of Python to 2.5. This effort has resulted in the fewest number of 'open' portmgr PRs in quite some time. portmgr appreciates all the people who worked with us on these patches, and people's patience as we catch up.
As well, lofi@ committed the upgrade of QT to 4.3.1.
We have added 3 new committers since the last report.
The freebsd-update front end is able to wait for freebsd-update to download a new set of patches to apply. It can then install and rollback the patches on either the local computer or over a SSH tunnel.
Since the end of the Summer of Code work has moved to BerliOS. The focus has been on writing tests for the front end, back end and communication library. The library has had tests written for most of it while the front and back ends have none.
The two most important parts of this Summer of Code projects have been accomplished.
The DESTDIR support for the Ports Collection has been rewritten to use a chrooted install. Now it is much more lightweight and easier to understand, but it works well for the most common cases, where it is supposed to be useful.
The Perl parts of the Ports Collection infrastructure have been extracted into an own module. At the same time, a new version handling has been invented. You can find more info on the Wiki.
Over the last couple of months several FreeBSD.org systems have been experiencing hardware issues. This included the main web-server www.FreeBSD.org which had a bad fan. The bad fan has been replaced so it should hopefully be stable again. In general we are working on replacing older hardware with newer systems and consolidating machine functions in the process.
Since August most FreeBSD.org services have been available via IPv6 with connectivity provided from ISC using a tunnel.
To honor the "Eat your own dog-food" principle the first two FreeBSD.org infrastructure systems have been upgraded to FreeBSD 7 and more are being upgraded as time permit.
Due to heavy load on the project's Perforce and CVS server the - two services are being moved to seperate systems to improve + two services are being moved to separate systems to improve performance of both CVS and Perforce.
This report covers FreeBSD related projects between October and December 2007. AsiaBSDCon 2008 is approaching and will be held at the Tokyo University of Science in Tokyo, Japan on the 27th - 30th of March 2008. The FreeBSD Foundation has released a Newsletter detailing their activities over the past few months.
FreeBSD 7.0 is nearing release and the 2nd Release Candidate is ready for testing and is available for download now.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
As a result of a posting on freebsd-current@ complaining about a communication gap between users and developers, there has been a great deal of new interest in working on bugbusting -- in particular, we brainstormed on ideas on how non-committers can help. The two main ideas that are being discussed are incoming bug triage (classifying, rating, and so forth), and working with users (helping users to work through problems that aren't classical Problem Reports.
As a result of this, we held our first Bugathon in quite some time (on #freebsd-bugbusters on EFNet). Over 30 people participated. As a result of this, over 120 PRs were closed, and dozens more were put into the 'feedback' state. Most of these PRs were in the kern/ and bin/ categories, which are the two that need the most work. (The new arrival rate was over 40/day during this time, including ports, so there was a significant net decrease.)
Several new wiki pages were created to support this effort, and finally capture a lot of the previous discussions from both the mailing list and the IRC channel. There are even more good ideas which Mark Linimon has promised to work up and investigate, including:
Note: at this time we are not yet looking to replace GNATS. The idea right now is to see what we can learn about how our workflow does (and ought to) work, and experiment with some low-cost changes to get various people's reactions. Linimon's feeling is that any of these kinds of changes would carry over to a new system, if we were to change over.
rwatson also created a wiki page to put down some thoughts about how to work on the various kernel problems that are reported. Although preliminary, this captures some expertise and puts it into a place where prospective volunteers can more easily find it.
The overall PR count is back up to just under 5300. Although this is net increase from the previous report, there were long periods of src and ports freeze during this time, which creates a spike in the overall count. (src and ports both remain in slush during that time). The peak number was approaching 5500.
Overall, we seem to have some momentum and new volunteers interested in working on user-reported problems. bugmeister is hopeful that we can capitalize on this and make some good progress in the rest of 2008.
A large number of bugs have been fixed in the FreeBSD "coda" kernel module over the past six months, and a man page has been added to describe the module. Many of these bugs were the result of the coda module failing to keep up with the many enhancements to FreeBSD VFS over the last few years. As a result of these fixes, it is now possible to use Coda with FreeBSD 7.x and 8.x without immediate panics, and possibly for an extended period. The new man page does clarify that Coda is an experimental distributed file system and not yet appropriate for production use on FreeBSD, but things are looking a lot better than they were.
Since the last status report we have made a nice progress about the website translation. The structure of the translated sites is polished and we have brought a significant set of pages up-to-date. New pages with important content have also been translated. Apart from the good progress, there is a still a lot to do. Some pages are still seriously outdated and some important parts are missing.
At the same time, we have added one new article translation and one is still awaiting review before being committed.
We have added the translation of the FreeBSD Flyer and maintained the existing translations. A huge progress is being made to provide a Hungarian translation of the FreeBSD Handbook. Also, there is an ongoing effort to provide Hungarian release notes for the upcoming FreeBSD releases.
Thanks to support from Cisco Systems, Inc, the port of the DTrace dynamic tracing framework from OpenSolaris to FreeBSD is active again. A solution to the integration issues surrounding the CDDL and BSD licenses has been found. There is an entirely BSD licensed set of hooks/shims which are optionally compiled into the kernel. This option can be included in the GENERIC kernel and shipped without any CDDL patent encumberance. The CTF (Compact C Type Format) tools now work across all architectures enabled in a 'make universe'. A BSD licensed DWARF library has been developed. The kernel DTrace support is limited to amd64 and i386 at the moment. It currently passes 822 of the tests in the DTrace Test Suite. It is expected that the initial commit to FreeBSD-CURRENT will occur within the next month after review. Refer to the change summary page for details of the proposed changes.
The FreeBSD Installer project (FIN) is yet another attempt to replace the aging sysinstall(8). I am attempting to keep the best parts of sysinstall(8) and combine them with the framework provided by the BSDInstaller (bsdinstaller.org) to create an installation program for FreeBSD that is multi-lingual, supports multiple installation media, supports remote installation, and is easily extensible to other installation types (gui, cgi, etc). The current implementation will slice disks, install your choice of base distributions, and set hostname and root password.
glvm is a geom class which reads the metadata from a LVM2 (Linux volume manager) disk and creates a geom provider for each logical volume. An example is the logs lv on a volume group called vg0 appearing as /dev/lvm/vg0-logs, this can be mounted as a disk.
The code is working and will be posted for testing soon.
malloc(3) has been enhanced in several ways to reduce lock contention when multi-threaded programs concurrently use the malloc(3) functions. The primary enhancements are lazy deallocation and dynamic arena load balancing.
Lazy deallocation is designed to reduce contention for programs that use the producer-consumer model, where a thread produces (allocates) objects, and a pool of worker threads consumes (deallocates) those objects. As a side benefit, lazy deallocation also substantially reduces lock contention if multiple unrelated threads are using the same arena.
Allocation activity patterns can change throughout the lifetime of a program. Dynamic arena load balancing monitors arena lock contention and re-assigns threads to other arenas as necessary, thus smoothing out allocator performance.
In order to monitor lock contention in support of arena load balancing, I had to switch to using pthreads mutexes. This all by itself smoothed out allocator performance under high load, since the internal libc "spinlocks" aren't really spinlocks, whereas malloc now spins for a bit before blocking.
I plan to MFC these changes to RELENG_7, hopefully in time for the FreeBSD 7.1 release.
FreeBSD/mips boots to multiuser using gxemul on the MALTA board with a 4Kc based CPU. The port is targeting MIPS32 and MIPS64 release 1 and release 2 based systems. Work is underway to support multicore systems.
Preliminary ports to adm 5120, the IDT RC32434, the Sentry 5, and a few other targets have started. These ports are in various stages of stability.
Juniper Networks has donated a generic MIPS FreeBSD port. This port doesn't run on any real hardware, but contains the necessary parts to run on idealized MIPS hardware. The FreeBSD/mips workers have been merging the current base and the Juniper code into a unified base. In addition, Cavium Networks has donated code supporting their multicore mips64r2 platform. This code is also being merged into the tree and cleaned up as well. The merged code base presently is making it to the first (or maybe second) call to cpu_switch before dying. Active work is underway in this area.
There are several websites already with overview of the FreeBSD FTP mirrror sites, but they all seem to have one problem: They are not manually updated with the list of sites. For example, http://mirrorlist.freebsd.org/FBSDsites.php, despite being hosted by an Australia, doesn't have the Australian mirrors on it, while http://people.freebsd.org/~kuriyama/mirrors/ doesn't tell you which files are available from there. The data on my page shows the availability of the ISO images on all FTP mirror sites. The list of FTP mirror sites is obtained from DNS by either doing a zone-transfer or by just trying the standard names. The first data block shows a quick overview of the availability of the ISO image directories per server, architecture and mirror site. The second data block shows a verbose availability of the contents of the ISO image directories per server.
The multi-IPv4/v6 jails project was resumed in early January after previous work had been abandoned in 2006.
As an alternate solution to full network stack virtualization, this work shall provide a lightweight solution for multi-IP virtualization. The changes are even more important because of the emerging demand for IPv6.
The current status includes updated user space utilities. Kernel side has grown support for multiple IP addresses for both address families in jails, while the old kernel internal lookup/checking functions were kept and can be compiled in during the transition period limiting jails to one IP address. Additionally a show jails DDB command was added to ease debugging.
As an auxiliary project the last suser(9) checks were replaced in netinet6/ to support optional raw IPv6 sockets with jails. The new priv(9) checks were committed to HEAD.
The if_nxge driver, contributed by Neterion, has been merged into FreeBSD 8-CURRENT and FreeBSD 7-STABLE, and will appear in FreeBSD 7.0.
The nxge driver provides support for Neterion Xframe-I and Xframe-II adapters. The driver supports TCP Segmentation Offload (TSO/LSO), Large Receive Offload (LRO), Jumbo Frames (5 buffer mode), Header Separation (Rx 2 buffer mode), VLAN, and Promiscuous mode.
For general information and support, please visit the Neterion support page http://www.neterion.com/support/support.html.
The nxge driver supports Neterion Xframe 10 Gigabit Ethernet adapters listed in http://www.neterion.com/how/pricing.html.
As part of my thesis, I've been working on a framework to monitor the performance of CURRENT over time. The project is now in a state where a server and a slave are producing benchmark results and publishing the results to a web page for testing. Already, the setup has detected regressions. Lots of improvements can be made, but it is already quite useful. Over the next month I'll be adding a few features, fixing bugs and writing documentation.
Completed initial requirements gathering. Selection of development tools complete. General internal design complete.
Ports 2.0 goals are:
The ports count continues to accelerate and is now over 18,000. The PR count, which had dipped to around 750 before the 6.3/7.0 freeze, is now back up to about 1000, due to the fact that we remain in ports slush.
Because of the freeze/slush, no experimental ports runs have been committed since the last report. Although 2 more -exp runs have been completed, we are waiting for 7.0R to commit them.
Once 7.0R happens, a lot of chaos is going to happen in the Ports Collection. This has built up during the long release cycle. Get ready for the following changes, among others:
Most of the portmgr activity was related to the QA process for the releases. In addition, linimon spent quite some time trying to get the sparc64 ports into better shape, and sent out a request for more people to help test sparc64 ports. Some people have responded with offers for letting committers get accounts on their machines.
Unfortunately during this time period, we became unable to build packages for ia64-7. As a result, we are not currently building packages for ia64 any more. If any one wants to step up to work on this architecture, let portmgr know.
We are currently building packages for amd64-5, amd64-6, amd64-7, amd64-8, i386-5, i386-6, i386-7, i386-8, sparc64-6, and sparc64-7. Note, however, that RELENG_5 will reach the end of its supported life on May 31, and package builds for those 2 buildenvs will stop as of that date. (8 buildenvs * 18,000 ports should be enough to keep us busy.)
Other than that, the packages are in the best shape that they have been in for some time. linimon continues to work on package analysis tools for portsmon.
We have added 2 new committers since the last report.
A new command line tool, procstat(1), allows detailed inspection and printing of process properties, including file descriptors, threads, kernel thread stacks, credentials, and virtual memory mappings of processes. Several new sysctls have been added to the kernel in order to export this information cleanly, and the stack(9) facility has been enhanced to allow the capture of kernel stacks from threads other than curthread. None of these features depends on procfs, continuing the effort to remove a requirement for procfs in order to print process information, as well as adding new types of information not available with procfs. Kernel stack printing is particularly useful as it provides much more detailed information on why a thread is blocked in kernel beyond the useful but limited wmesg context provided to date. This is helpful in debugging both user process problems and kernel problems. procstat has been merged into FreeBSD 8-CURRENT, and will be merged to 7-STABLE after FreeBSD 7.0 is released.
The Greek doc translation team has grown significantly since we started the translations. Most of the Handbook has already been translated to Greek (and committed to the CVS tree), as a collaborative effort of Manolis Kiagias, Nikos Kokkalis, Panagiotis Kritikakos, Vaggelis Typaldos, Stylianos Sideridis and others. Manolis has started translating the FAQ too, and we also tagged most of the Greek documents with their original, English revision ID. There are also plans for a translation of www/en , but these may have to be deferred until we find the time to complete the Handbook, which is our primary target right now.
There are a variety of on-going projects relating to improving SMP scalability of the FreeBSD network stack post-7.0. These include:
Detailed profiling of application workloads such as BIND9, MySQL, PgSQL and Apache have been used to identify performance bottlenecks and to guide changes to the source code.
rwlock(9) use for pcbinfo and inpcb locking , allowing the acquisition of only read locks for pcbinfo and inpcb during UDP receive and transmit--this is highly desirable in order to improve BIND9 performance, which sends and receives from many threads at a time on a single UDP socket.
Breaking out pcbinfo into a series of parallel data structures , where the particular pcbinfo instance is selected using a hash of the connection tuple (and where ambiguous cases are present in all instances). This would allow greatly reducing pcbinfo contention for parallel input cases, which are increasingly likely with multiple input queue network devices, such as the Chelsio cxgb 10gbps driver.
Investigation of use opportunities for rmlock(9) -- rmlocks provide very lightweight acquisition for read, but expensive acquisition for write, and may be an appropriate replacement for rwlocks where significantly more reads than writes take place -- such as for firewall rule list protection, pf hook registration, address lists, etc.
Weak connection affinity , in which the effective affinity of a connection, determined by its hash/rss work assignment to a particular input queue by the network stack or network card, is tracked and exposed to user space so that work associated with that connection can be performed on or close to the CPU where the kernel will be processing input for the connection. Software work placement has been done using the netisr2 implementation, which creates per-CPU netisr threads and assigns work based on connection properties.
There are also many other pieces of related work going on, especially relating to 10gbps network drivers, and workloads of particular interest include BIND9, MySQL, pgsql, Apache, and general TCP parallelism.
In the time since the last status report, four security advisories have been issued concerning problems in the base system of FreeBSD; one of these problems was in "contributed" code maintained outside of FreeBSD. The FreeBSD Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated; since the last status report, 61 new entries have been added, bringing the total up to 1023. Many of these new VuXML entries were made by members of the "ports-security" team.
The "ports-security" team is still looking for more committers who can periodically help with fixing ports security issues and documenting them in the FreeBSD VuXML document. Committers who wish to help with this effort can contact simon@ for details.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 5.5, FreeBSD 6.1, FreeBSD 6.2, and FreeBSD 6.3. The respective End of Life dates of supported releases are listed on the web site; it is expected that the upcoming FreeBSD 7.0 release will be supported for one year after its release.
Like every year for the past few years is held what in France is mostly called "Solutions Linux" in Paris La Défense. The exhibition will take place the 29, 30 and 31st of January in the CNIT.
The interesting thing about this event is that 80% of the floor is taken by companies (IBM, Novell, Oracle), and the remaining 20% is given freely to associations and non-profit organizations, where you'll find many (if not most) french LUGs, *BSDs, most Linux distributions, Mozilla, OOo...
This year, FreeBSD will once again have a booth, and we'll be showing what FreeBSD is, why it's the damn best OS out there. We'll also be distributing flyers and CD's for the whole three days
Admission to the exhibitions is free, so if you ever happen to pass by, come and see us, we'll be at booth A39.
Completed and tested. Awaiting review from other committers.
The FreeBSD TCP reassembly queue system has reached its limits - with todays high speed links over long distances and large socket + with today's high speed links over long distances and large socket buffers. The old code is almost unchanged compared to 4.4BSD and gets quite inefficient with large mbuf chains.
The new code aggregates consecutive segments into blocks and inserts the blocks into a tail queue. The insertion points for a newly arrived segment are checked in order of their probability. This prevents full chain traversals and is very efficient.
To prevent easy resource exhaustion attacks the effective mbuf usage is accounted for and limited by the size of socket buffer. This way the reassembly queue can't be abused with many holes among small segments.
A further addition is the combination of received SACK block tracking with the reassembly queue. The reassembly queue now tracks all blocks of segments. This makes tracking it again for SACK unnecessary. Additionally the limitation to six SACK blocks is lifted and the size of the inpcb structure is reduced quite a bit.
The new code is stable and in testing correctly handles the download of a full set of FreeBSD CDROM images and 180 ports distfiles from widely distributed sites around the world at 2% packet loss.
The FreeBSD TCP code has evolved a lot over time and many new features were added. However over time it got crufty, complex and hard to read and track. In some places functionality was moved away but the corresponding code in the main TCP functions was not or not fully removed.
The main purpose of of the TCP code cleanup and rewrite is to make the code:
Quite a bit of code is already (re)written but a lot still remains to be done.
The kernel DDB facility has been enhanced to add several new features:
DDB scripting allows the user to define a set of simple scripts from within the debugger or userspace using the new ddb(8) tool to automate debugging steps. Scripts can be automatically executed when the debugger is entered ("kdb.enter.panic", "kdb.enter.break", ...) or manually using the DDB "run" command.
DDB output capture allows the user to request that the output of DDB be captured into a buffer for access from user space or to be written out in a textdump.
DDB textdumps, a new dump format that writes out a tarball of text-based debugging information, such as the kernel message buffer, panic message, kernel configuration, kernel version, and DDB capture buffer to the swap partition, to be extracted via savecore(8). This provides a compact, portable, and kernel compile independent debugging package.
Various interesting formulas for use are described in ddb(4) and textdump(4); the facilities are separable, so you can, for example, run a few DDB commands and capture their output, then write a regular dump and extract that output using kgdb, or you can do the same and write it out as a textdump. Likewise, scripts can be used to automate manual debugging, or implement textdumps by enabling output capture, running a series of commands, and forcing a textdump to be written before rebooting.
Support for these facilities has been merged into 8-CURRENT, and will be merged to 7-STABLE after the release of FreeBSD 7.0.
The TrustedBSD Project was proud to release OpenBSM 1.0, the first production release of OpenBSM, which is shipped with FreeBSD 6.3 and will ship with FreeBSD 7.0. This release represents largely polishing, bug fixing, and cleanup over the previous alpha release, but for FreeBSD 6.x introduced features such as XML audit trail printing, new token types, and new event identifiers.
A variety of development work continues on audit, including initial work on OpenBSM 1.1 alpha, work on improving the performance and semantics of audit pipes, and the experimental bsmtrace host intrusion detection package.
The patch to account the possibly required swap space and limit it by total amount of configured swap or per-uid limit is revived, ported to the 8-CURRENT. Now it is intensively tested by Peter Holm. Please, give it a run in the diverse workloads. Your comments are welcome!
The port will only run as a guest (i.e. domU) right now, on i386/PAE platforms. Status:
This Status Report covers FreeBSD related projects between January and March 2008. During this time FreeBSD 7.0 was released. BSDCan is upon us with the Developer Summit starting the 14th and the Conference starting the 16th.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
As one of the results of our January and February bugathons, we have granted Volker Werth (vwe@) direct access to GNATS. During the past few months he has been instrumental in working on several hundred PRs (mainly src-related), and either closing them or helping users work through issues they are having. There have been several commits to the src tree that directly resulted from this. Welcome Volker!
As well, several new people are assisting us in classifying incoming PRs, working with users, and reviewing patches. Among the most active are Bruce Cran, Dylan Cochran, and Harrison Grundy. We appreciate everyone's efforts.
As a direct result of the above, we have been able to hold the overall PR count down to around 5300 (the peak was around 5500). despite the facts that PR submissions have jumped recently, and the ports PR backlog is a little higher than recent trends (due to the long freeze/slush cycle). What is most encouraging, however, is not the absolute number, as much as that we are handling incoming PRs much more quickly and completely. While we are still not where we need to be, this trend is very encouraging.
As well, The Bugbusting Team has learned some lessons about how we can best involve new people in bugbusting, e.g., how to best leverage people who have varying levels of experience and areas of interest. Our old response of "just look through the bug reports and let us know if you see anything that needs doing" tends to discourage all but the most highly-motivated. Some of these ideas are being studied to figure how to change our process flow.
There are still a number of good technical suggestions from the two Bugathons that need to be written up and discussed. The first few have resulted in the following: there are a few new web pages that include: new PRs in the last 7 days; the web representation of the "recommended by bugbusting team" list; and "PRs in feedback with no change for 2 months". (See above). Many more need to be added.
Much of the work of the second Bugathon was in identifying and closing PRs for which fixes had already been committed. Others were identified and relabled as 'patched' to move them along.
This patch modifies the build infrastructure in order to use GCC's stack-smashing protection (SSP, aka ProPolice) when building world, kernel and ports. Don't forget to see the website and especially the FAQ for a list of ports that fail to build with ProPolice. The patch extends the meaning of src.conf(5) WITHOUT_SSP so as to prevent both building libssp and using ProPolice when compiling. An interesting thing to note is that libssp is GNU licensed (it is provided with GCC 4.2.1) but since libc includes the mandatory symbols, programs won't be linked against GNU libssp. A new knob USE_SSP has been also added for the ports infrastructure, you can set it to "yes" in make.conf(5) and use USE_SSP= on command-line to disable ProPolice for some ports. The patch has been reviewed and should hopefully be committed soon. The port part hasn't been reviewed yet, though.
"finstall" is a graphical installer project for FreeBSD, sponsored by Google during the 2007 Summer of Code. Its goal is to create a modern installer, usable by both novice users and experts. Because it is divided into front end and back end, it can potentially be used for advanced purposes as system configuration, remote and custom installs, etc. The project has resulted in a simple installer ISO image for i386 that can be used for new installations on empty hard drives. Development has continued post-SoC but somewhat slowly; recently implemented features include ZFS support and BSDStats support. To attract more potential developers (especially those without an account on FreeBSD's official development systems), the project has moved to SourceForge. Future development plans include support for headless / remote installs, partitioning, etc. Talks about finstall will be given at BSDCan 2008.
The foundation provided legal counsel for the project to understand the impact of GPLv3 on the project and to create a policy on software licenses. We approved a budget of $250,000 for 2008. We were a sponsor for AsiaBSDCon and provided travel grants to three people to attend the conference. We are a sponsor for BSDCan and the BSDCan Developer Summit. We have approved travel grants for 10 people to attend BSDCan. We are supporting projects that will provide Java 1.6 binaries for FreeBSD 6.3 and 7.0. Join - our mailing list to recieve monthly updates. See you at BSDCan!
+ our mailing list to receive monthly updates. See you at BSDCan!A prototype web application has been written for the http://www.FreeBSD.org website which allows authenticated users to add new development ideas or comment and vote on ideas added by others. This application is a proposed replacement for the static webpage that is currently maintained with project ideas for summer of code students and others looking to get involved with FreeBSD.
Some of the features currently available include:
The code is checked into perforce under //depot/user/murray/www/apps/django/ideas/... and I would eventually like to see this hosted on FreeBSD.org hardware, linked from the main website, and checked into www/apps/django/ideas.
The multi-IPv4/v6 jails project was resumed in early January after previous work had been abandoned in 2006.
As an alternate solution to full network stack virtualization, this work shall provide a lightweight solution for multi-IP virtualization. The changes are even more important because of the emerging demand for IPv6.
The current status includes updated user space utilities. Kernel side has grown support for multiple IP addresses for both address families in jails, as well as no IP addresses at all. 32bit and jail version 1 backward compatibility support were implemented.
The development was moved to perforce and patches for early adopters are available.
portmgr is pleased to announce that Florent Thoumie (flz) has joined us. We immediately put him to work on cleaning up the pkg_* tools.
After the extended freeze and then slush for 7.0R, we have finally been able to start catching up on the backlog that built up during that time.
The ports count is now over 18,200. The PR count has only dropped to around 1000. We are still turning around PRs fairly quickly, but are not making progress on the backlog.
We have only been able to do 2 -exp runs recently. Although a number of PRs have been closed, we are still at 57 portmgr PRs.
During this period, GNOME has been updated to 2.22.0. Also, a new port for linux emulation (emulators/linux_base-f8) has been introduced for general testing.
XFree86 has been removed. (It had been deprecated for quite some time; modern development seems to be happening in X.Org.) This simplifies the infrastructure. A few other stale ports have been reaped.
The following large changes are in the pipeline:
We are currently building packages for amd64-5, amd64-6, amd64-7, amd64-8, i386-5, i386-6, i386-7, i386-8, sparc64-6, and sparc64-7. Note, however, that RELENG_5 will reach end of its supported life May 31, 2008, and package builds for those 2 buildenvs will stop as of that date.
We have been able to use some new machines to speed up the package builds (in particular, amd64) -- in fact, to the point that we are now outrunning the capacity of some of the mirrors to stay current. A solution for the problem is being investigated.
We have added 4 new committers since the last report.
About 10 weeks ago I started rewriting the TTY layer. The existing TTY code is about 20-25 years old and has been extended over and over, without really improving its design.
The new TTY layer will allow us to remove usage of the Giant from drivers. It also includes an improved buffering mechanism, which has more constant-time operations and prevents copying data multiple times when moving data to userspace.
Right now the code should work quite well for most users. The code in Perforce includes a new pseudo-TTY driver, which is finally capable of destroying TTY's and their associated buffers when needed. The syscons, uart and ucom drivers have also been ported to the new TTY layer.
The code is quite complete, but it still misses driver interaction for carrier/connection detection and sending breaks. Many drivers still need to be ported.
The student application period for the Summer of Code is over and the mentors and administrators are carefully reviewing the applications, clarifying the project parameters, and deciding which students to recommend for funding from Google.
This year we received over 100 student applications from students in 26 different countries. We also have over 60 potential mentors that we are currently matching up with students. We will soon announce the winning students on the summer of code website and the process of bringing these students into our development community will begin.
Each student will again be given Perforce and wiki access and all developers are encouraged to contact any students working in related areas, as we don't want the students to have access to our community only through their formal assigned mentor.
Our implementation of UnionFS has been merged into HEAD, 7-stable and 6-stable already. Now we are working on UnionFS stability improvement. We have developed the following 5 patches. If you are interested, please try them and report your results.
During the last three months there has mostly been bugfix and documentation commits. The code is currently in a stable and full featured state. The FreeBSD P4 USB project now has a fully symmetric USB stack at API level and has been tested to work with AT91RM9200 ARM based boards and USS820 based devices. There are currently two USB device side drivers implemented, namely CDC Ethernet and Mass Storage (SCSI+BBB) so that you can now make your custom USB Flash Disk using FreeBSD. Don't confuse USB device side drivers with USB host side drivers.
Currently the USB P4 project is under review.
Ideas and comments with regard to the new USB API are welcome on the FreeBSD USB Mailing List .
We are progressing better these days again. We have made some updates to the website and to the Handbook, including the complete translation of the jails chapter. We have also added a new translation of an article and an another one is under review.
We are pleased to welcome Gábor Páli as a doc committer. He has successfully completed the translation of the FreeBSD Handbook. The final review of his work is pending now and we will import it soon to the repository. We consider the translation of the release notes the next important milestone of this translation project.
This Status Report covers FreeBSD related projects between April and June 2008. During this period The FreeBSD Foundation has released their July Newsletter.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
After the last couple of months of intensive development going on towards FreeBSD support for Marvell System-on-Chip devices, we have FreeBSD 8.0-CURRENT running on the following systems:
The above families of SOCs are built around CPU cores compliant with ARMv5TE instruction set architecture definition. They share a number of integrated peripherals, for most of which we already have operational and stable drivers:
High level functional summary:
The code is partially available (Orion in Perforce), other variants will also be integrated with Perforce/SVN soon.
This project aims to implement graphics support for FreeBSD's boot loader. It will replace the existing ASCII menu. (Note that the ASCII menu will still be available when graphics mode cannot be used, such as on serial console or on unsupported hardware.)
For a more detailed description and screen shots please refer to the project's Wiki URL above.
Progress is slow (due to lack of time) but steady. The code currently lives in the Perforce repository. I'll try to prepare a first public CFT as soon as possible.
We have granted Bruce Cran (bruce@) direct access to GNATS and Volker Werth (vwe@) has been released from mentorship. We appreciate their help!
We had a third bugathon in June, which resulted in the closing of a number of bugs and the investigation/classification of several others. We are still trying to find ways to get more committers helping us with closing PRs that the team has already analyzed.
We continue to make good progress in categorizing PRs as they
arrive with 'tags' that correspond to manpages. (Special thanks go
to Dylan Cochran for the help.) As a result, we now have created
some prototype reports that allow browsing the database
In addition, another new report, oriented towards PR submitters,
summarizes the
Mark Linimon summarized the good technical suggestions from the
bugathons so far this year to the wiki. As a part of this, he
rearranged the wiki pages, so if you have not seen them for a
while, please see
Jeremy Chadwick (koitsu@) is now maintaining a
The overall PR count has been holding at around 5300 since the last release.
For the past couple of months I have been working on generalizing the package build cluster to allow it to host other batch and interactive jobs. Currently we make an inefficient use of build machines because various projects have dedicated machines that are either underloaded or overloaded for their particular tasks. The goal is to provide a framework for combining all of these machine resources into a single cluster that can be shared by many users, reducing dead time and allowing distributed build tasks to take advantage of extra build resources when available. Developers will be able to obtain on-demand interactive access to a jail running on any of the available architectures, with root access. Similarly, batch jobs will specify their resource requirements and be dispatched to run on a suitable machine in the cluster. Current status: The job queue manager is working and is now being used to map package builds to machines. Various package build scripts have been rewritten to use it instead of the previous build scheduler. The generic job dispatcher is being prototyped and will be validated with several existing services such as INDEX builds. Various support services like ZFS snapshot replication have been written.
The FreeBSD Dutch Documentation Project is an ongoing project to translate the FreeBSD Documentation resources to the Dutch language.
The project is currently progressing very well in translating the FreeBSD Handbook to the Dutch language, the last chapter is being translated by the project members.
Recent achievements include the translation of the Jails chapter, and the Virtualization chapter, as well as progression on the Advanced Networking chapter. Rene Ladan is a keyplayer in that region.
We also started with the FAQ translation, which is another major target which we should be reaching at some point.
If you care to helpout with the translation(s) and/or want to know something about it, please do not hesitate to contact us, we are glad to help where possible.
An extensive work on renovating the FreeBSD FAQ has been started to support its Greek and Hungarian translations. Further improvements and content changes are still possible, we hope other committers will help us to keep the FAQ updated and tuned further.
We have launched a renewal proposal to collect and organize the ideas around a more interactive, accurate, open for comments, consistent across several views etc. FAQ document. We would like to experiment with methods to implement the goals mentioned before, and help is more than welcome.
Between the last report and this one, the project has yielded a LiveCD installer for i386 containing FreeBSD 7.0-RELEASE. The project was presented at BSDCan 2008. The development is progressing slowly due to the lack of free time. I'm looking for funding that will allow me more involvement in the project. The big item currently in development is documentation and description of the protocol used between the front-end and the back-end, which will result in more robustness in the implementation and could support third-party clients. This sub-project is near completion. The project is currently hosted at SourceForge to allow contribution from non-FreeBSD developers.
Hungarian translation of the FreeBSD Handbook has been finally committed to the doc repository. The translation of the FreeBSD FAQ has also been started, however, the original document needed to be brought up to date first. Two other article translations has been added, compiz-fusion and linux-users.
Our Perforce depot was reorganized for the better layout, giving newcomers more space to play. The checkupdate script written by Giorgos Keramidas, a new tool for checking translations has been adopted to help the project's work.
Qt4 has been updated to 4.4.1 in our test repository. We ran into some runtime problems with Qt 4.4.0, so it was never committed it to the ports tree. Most of the problems have been fixed in 4.4.1 and we plan to commit it in a few days.
At the moment, the KDE 4.1 ports are ready for testing before they are committed to the FreeBSD ports tree. We have already had the first Call for Public Testing on July 17th, 2008 with KDE 4.1 beta2. The feedback has been positive so far. If you want to help to test them to speed up the process, please visit the Wiki page and provide feedback.
We plan to have it all committed by the middle of August.
Project aims to improve layer2 filtering in ipfw and pf. So far following project goals are achieved: pfil framework is extended to handle ethernet packets, ipfw layer2 filtering is greatly simplified, added l2filter and l2tag per interface flags. Both ipfw and pf firewalls support filtering by ethernet addresses, support stateful filtering with ethernet addresses and firewall's lookup tables are extended to contain ethernet addresses.
The ports count has jumped to over 19,000. The PR count has been holding steady at around 900.
KDE has been updated to 4.1. Special thanks go to Martin Wilke for a great deal of pre-testing.
GNOME has been updated three times, first to 2.22.1 and then to 2.22.2 and 2.22.3.
Other notable updates are automake, gettext, libtool, and m4.
Florent Thoumie has been working on some updates to the pkg_* tools.
Ion-Mihai Tetcu has set up a tinderbox with several purposes: first, to quickly try to build packages as changes are committed; secondly, to build them with a non-standard set of environment variables; and thirdly, to build older packages with the non- standard set of environment variables. As a result of all this work, and work by various committers, we are much closer to building packages corrected in the NOPORTDOCS case.
Kris Kennaway has done a substantial rewrite of the package
building tools, including moving as a default to ZFS, which allows
quick cloning of src and ports directories. It is now much easier
to manage and monitor the builds. Work on this is continuing. See
the commits to
We have been able to do many -exp runs since the last report, including those for bsd.cmake.mk, autotools update, CC environment passing, the KDE 4.1 pre-integration and post-integration checks, lockmgr changes, tty changes, and others.
Although a number of PRs have been closed, we are still at 57 portmgr PRs, the same as the last report.
The following large changes are in the pipeline:
We are currently building packages for amd64-6, amd64-7, amd64-8, i386-6, i386-7, i386-8, sparc64-6, and sparc64-7. RELENG_5 has reached the end of its supported life.
We have added 4 new committers since the last report.
The grep utility is ready for a thorough test on the portbuild cluster. It is almost compatible with GNU grep, but there are differences in the regex handling at the level of the regex libraries of GNU and the base system one, thus a better compatibility is very hard to implement.
Some progress has been made on diff, but some important options are still missing. The sort utility seems to be very problematic in the aspect of the wide character support by design, thus it was given a lower priority.
We have not made any significant progress in this period. We definitely need more active translators to progress with the translation project.
During the last three months there has been a number of changes. Most notably all global USB symbols have been renamed to "usb2_" to allow for co-existence with the old USB stack. Also there is now a completely new and reworked UGEN driver which allows multiple drivers to hook onto the same USB device. No more need to unload any kernel drivers. For example it is now possible to have a userland Mouse driver stealing half of the mouse events at the same time "ums" is loaded. The only disadvantage is that your mouse cursor will move slower on the screen. This is maybe not the most common use-case, but it illustrates that kernel USB drivers are no longer locking out other USB userland drivers. A new userland libusb is in the works for FreeBSD. The USB stack now also has support for independent USB BUS, USB Device, and USB Interface permissions. That means you can more easily give USB permissions to USB device drivers at either USB BUS, USB Device or USB Interface level. All USB modules have now been grouped into functional categories: usb2_bluetooth, usb2_ndis, usb2_controller, usb2_quirk, usb2_core, usb2_serial, usb2_ethernet, usb2_sound, usb2_image, usb2_storage, usb2_input, usb2_template, usb2_misc, and usb2_wlan.
Ideas and comments with regard to the new USB API are welcome on the FreeBSD-USB Mailing List.
This report covers FreeBSD related projects between April and September 2009. During that time a lot of work has been done on wide variety of projects, including the Google Summer of Code projects. The BSDCan conference was held in Ottawa, CA, in May. The EuroBSDCon conference was held in Cambridge, UK, in September. Both events were very successful. A new major version of FreeBSD, 8.0 is to be released soon. If you are wondering what's new in this long-awaited release, read Ivan Voras' excellent summary.
Thanks to all the reporters for the excellent work! We hope you enjoy the reading.
Please note that the next deadline for submissions covering reports between October and December 2009 is January 15th, 2010.
The libnetstat(3) project provides a user-space library API to monitor networking functions with the following benefits:
The supported abstractions are as follows:
There is a sample application, called nettop(8), which provides a simple ncurses-based top(1)-like interface for monitoring active connections and network buffer allocations via the library. A modified version of netstat(1) has also been created to use libnetstat(3) as much as possible.
Pefs is a kernel level filesystem for transparently encrypting files on top of other filesystems (like zfs or ufs). It adds no extra information into files (unlike others), doesn't require cipher block sized io operations, supports per directory/file keys and key chaining, uses unique per file tweak for encryption. Supported algorithms: AES, Camellia, Salsa20. The code is ready for testing.
The BSD# Project is devoted to porting the Mono .NET framework and applications to the FreeBSD operating system.
During the past year, the BSD# Team continued to track the Mono development and the lang/mono port have almost always been up-to-date (we however had to skip mono-2.2 because of some regression issues in this release). Most of our patches have been merged in the mono trunk upstream, and should be included in the upcoming mono-2.6 release.
In the meantime, a few more .NET related ports have been updated or added to the FreeBSD ports tree. These ports include:
Some time ago I started writing a new driver for the FreeBSD kernel called vt(4), which is basically a replacement of syscons. There is still a lot of work that needs to be done but it is probably useful to mention what it does (and what does not).
Right now there are just two graphics drivers for vt(4), namely a VGA driver for i386 and amd64 and a Microsoft Xbox graphics driver (because it was so easy to implement). I still have to figure out what I am going to do with VESA, because maybe it is better to just ignore VESA and figure out how hard it is to extend DRM to interact with vt(4).
Some random features: it already supports both Unicode (UTF-8) input and output, it is MPSAFE and supports per-window graphical fonts of variable dimensions, containing an almost infinite amount of glyphs (both bold and regular).
The libprocstat project is an ongoing effort to develop a library that can be used to retrieve information about running processes and open files in the uniform and platform-independent way both from a running system or from core files. This will facilitate the implementation of file- or process-monitoring applications like lsof(1), fstat(1), fuser, etc. The libprocstat repository contains a preliminary version of the library. It also includes rewrites of the fstat and the fuser utilities ported to use this library instead of retrieving all the required information via the kvm(3) interface; one of the important advantages of the versions that use libprocstat is that these utilities are ABI independent.
I have been working recently on writing a new debugger, primarily for the FreeBSD platform. For various reasons, I have been writing it in a relatively obscure C-like language called D.
So far, I have a pretty useful (if a little raw at the edges) command line debugger which supports ELF, Dwarf debugging information and (currently) 32 bit FreeBSD and Linux. The engine includes parsing and evaluation of arbitrary C expressions along with the usual debugging tools such as breakpoints, source code listing, single-step etc. All the code is new and BSD licensed. Currently, the thing supports userland debugging of i386 targets via ptrace and post-mortem core file debugging of the same. I will be adding amd64 support real soon (TM) and maybe support for GDB's remote debugging protocol later.
The clang@FreeBSD team presents the status of clang/LLVM being able to compile FreeBSD system. The current status is:
All other platforms are untested.
A lot has happened over the spring/summer: amd64 got proper mcmodel=kernel support, compiler-rt has been introduced (paving the way for libgcc replacement), we have run two experimental port builds to see how clang does there. The C++ support is able to parse devd.cc without warnings. We have got the kernel working with -O2. FreeBSD has been promoted to be an officially supported plaform in LLVM. As a result of all this work, many parts of FreeBSD that did not compile before now build without problems.
We have ported libdispatch, Apple's Grand Central Dispatch event and concurrency framework to FreeBSD:
Jordan Hubbard has also prepared a blocks-aware clang compiler package for FreeBSD. When compiled with clang, libdispatch provides blocks-based, as well as function-based callbacks.
The port was presented at the FreeBSD Developer Summit in Cambridge, UK in September, and slides are online on the devsummit wiki page. A FreeBSD port is now available in the Ports Collection. After FreeBSD 8.0-RELEASE has shipped, the new kqueue primitives will be MFC'd so that libdispatch works out of the box on FreeBSD 8.1-RELEASE.
VirtualBox has been committed to the Ports tree and synchronized with the latest trunk version from Sun. Several known problems are already fixed and some new features have been added:
We would like to say thanks to all the people who helped us by reporting bugs and submitting fixes. We also thank the VirtualBox developers for their help with the ongoing effort to port VirtualBox on FreeBSD.
The current translations (Handbook and some articles) are kept
up to date with the English versions. Some parts of the website
have been
In May 2009, Benedict Reuschling received his commit bit to the www/de and doc/de_DE.ISO8859-1 trees under the mentorship of Johann Kois. Since then, he has been working primarily on the Handbook, updating existing chapters and translating new ones. Most notably, the filesystems and DTrace chapters have been recently translated. Bugs found in the original documents along the way were reported back so that the other translation teams could incorporate them, as well.
Christoph Sold has put his time in translating the wiki pages of the BSD Certification Group into the German language. This is very helpful for all German people who want to take the exam and like to read the information about it in their native language. Daniel Seuffert has sent valuable corrections and bugfixes. Thanks to both of them for their time and efforts!
The website is translated and updated constantly. Missing parts will be translated as time permits.
We appreciate any help from volunteers in proofreading documents, translating new ones and keeping them up to date. Even small error reports are of great help for us. You can find contact information at the above URL.
Kicking off our fall fund-raising campaign! Find out more at http://www.freebsdfoundation.org/donate/.
We were a sponsor for EuroBSDCon 2009, and provided travel grants to 8 FreeBSD developers and users. We sponsored Kyiv BSD 2009, in Kiev Ukraine. We were also a sponsor of BSDCan, and sponsored 7 developers. We funded three new projects, New Console Driver by Ed Schouten, AVR32 Support by Arnar Mar Sig, and Wireless Mesh Support by Rui Paulo, which has completed. We continued funding a project that is making improvements to the FreeBSD TCP Stack by Lawrence Stewart. The project that made removing disk devices with mounted filesystems on them safe, by Edward Napierala, is now complete.
We recognized the following FreeBSD developers at EuroBSDCon 2009: Poul-Henning Kamp, Bjoern Zeeb, and Simon Nielsen. These developers received limited edition FreeBSD Foundation vests.
Follow us on Twitter now!
We continue to classify PRs as they arrive, adding 'tags' to the subject lines corresponding to the kernel subsystem involved, or man page references for userland PRs. These tags, in turn, produce lists of PRs sorted both by tag and by manpage.
The list of PRs recommended for committer evaluation by the Bugbusting Team continues to receive new additions. This list contains PRs, mostly with patches, that the Bugbusting Team feel are probably ready to be committed as-is, or are probably trivially resolved in the hands of a committer with knowledge of the particular subsystem. All committers are invited to take a look at this list whenever they have a spare 5 minutes and wish to close a PR.
A full list of all the automatically generated reports is also available at one of the cited URLs. Any recommendations for reports which not currently exist but which would be beneficial are welcomed.
Gavin Atkinson gave a presentation on "The PR Collection Status" at the EuroBSDCon 2009 DevSummit, and discussed with other participants several other ideas to make the PR database more useful and usable. Several good ideas came from this, and will hopefully lead to more useful tools in the near future. Discussions also took place on how it may be possible to automatically classify non-ports PRs with a view towards notifying interested parties, although investigations into this have not yet begun.
Mark Linimon also continues attempting to define the general problem and investigating possible new workflow models, and presented work on this at BSDCan 2009.
Since the last status report, the number of open bugs has increased to around the 5900 mark, partially because of an increased focus on getting more information into the existing PRs, in an attempt to make sure all the information required is now available. As a result, although the number of open PRs has increased, they are hopefully of better quality.
As always, more help is appreciated, and committers and non-committers alike are always invited to join us on #freebsd-bugbusters on EFnet and help close stale PRs or commit patches from valid PRs.
The ports count has soared to over 20,700. The PR count had been driven below 800 by some extraordinary effort, but once again is back to its usual count of around 900.
We are currently building packages for amd64-6, amd64-7, amd64-8, i386-6, i386-7, i386-8, sparc64-7, and sparc64-8. There have been preliminary runs of i386-9; however, to be able to continue builds on -9, we will either need to find places to host a number of new machines, or drop package building for -6. The mailing list discussion of the latter proved quite controversial.
We have added some new i386 machines to help speed up the builds, but this only makes up for the disk failures on some of our older, slower, i386 nodes.
We also appreciate the loan of more package build machines from several committers, including pgollucci@, gahr@, erwin@, Boris Kochergin, and Craig Butler.
The portmgr@ team has also welcomed new members Ion-Mihai Tetcu (itetcu@) and Martin Wilke (miwi@). We also thank departing member Kirill Ponomarew (krion@) for his long service.
Ion-Mihai has spent much time working on a system that does automatic Quality Assurance on new commits, called QAT. A second tinderbox called QATty has helped us to fix many problems, especially those involving custom PREFIX and LOCALBASE settings, and documentation inclusion options. Ports conformance to documented features / non-default configuration will follow.
Between pav and miwi, over 2 dozen experimental ports runs have been completed and committed.
We have added 5 new committers since the last report, and 2 older ones have rejoined.
Since the spring, the FreeBSD KDE team has been busy upgrading KDE from 4.2.0 up through to 4.3.1. As part of the ongoing maintenance of KDE, the team also updated Qt4 from 4.4.3 through to 4.5.2
We added two new committers/maintainers to the team, Kris Moore (kmoore@) and Dima Panov (fluffy@). We also granted enhanced area51 access to contributors Alberto Villa and Raphael Kubo da Costa. Alberto has been our key contributor updating and testing Qt 4.6.0-tp1. Raphael is a KDE developer, who has become our Gitorious liaison, he has been responsible for getting FreeBSD Qt patches merged in upstream.
Markus Brüffer (markus@) spent a lot of time patching widgets and system plugins so they would work under FreeBSD. We would like to thank him for all his effort!
Around 70 FreeBSD developers and guests attended the FreeBSD developer summit prior to EuroBSDCon 2009 in Cambridge, UK. Hosted at the University of Cambridge Computer Laboratory, the workshop-style event consisted of prepared presentations, as well as group hacking and discussion sessions. Talks covered topics including 802.11 mesh networking, virtual network stacks and kernels, a new BSD-licensed debugger, benchmarking, bugbusting, NetFPGA, a port of Apple's GCD (Grand Central Dispatch) to FreeBSD, security policy work, cryptographic signatures, FreeBSD.org system administration, time geeks, a new console driver, and the FreeBSD subversion migration. Slides for many talks are now available on the wiki page. A good time was had by all, including a punting outing on the River Cam!
EuroBSDcon 2009 happened in Cambridge, with over 160 users, developers, friends and others. Slides, papers and audio are now up on the website for those who could not make it to Cambridge. Next year's event in 2010 will take place in Karlsruhe from 8 to 10 October 2010. If you are interested in what you missed in 2009, or to join the mailing list so you do not miss out next year, visit http://2009.eurobsdcon.org.
Since their public launch in November 2008, the FreeBSD Forums (the most recent addition to the user community and support channels for the FreeBSD Operating System) have witnessed a healthy and steady growth.
The user population is now at over 8,000 registered users, who have participated in over 6,000 topics, containing over 40,000 posts in total. The sign-up rate hovers between 50-100 each week. The total number of visitors (including 'guests') is hard to gauge but is likely to be a substantial multiple of the registered userbase.
New topics and posts are actively 'pushed out' to search engines. This in turn makes the Forums show up in search results more and more often, making it a valuable and very accessible source of information for the FreeBSD community.
One of the contributing factors to the Forums' success is their 'BSD-style' approach when it comes to administration and moderation. The Forums have a strong and unified identity, they are neatly divided into sub-forums (like 'Networking', 'Installing & Upgrading', etc.), very actively moderated, spam-free, and with a core group of very active and helpful members, dispensing many combined decades' worth of knowledge to starting, intermediate and professional users of FreeBSD.
We expect the Forums to be, and to remain, a central hub in FreeBSD's community and support efforts.
Problem: Over the years the FreeBSD locale database (share/colldef, share/monetdef, share/msgdef, share/numericdef, share/timedef) has accumulated a total of 165 definitions (language - country-code - character-set triplets). The contents of the files for Western European languages are often low-ASCII but for Eastern European and Asian languages partly or fully high-ASCII. Without knowing how to display or interpret the character-sets, it is difficult to make sure by the general audience that the local language (language - country-code) definitions are displayed properly in various character-sets.
Suggested approach: With the combination of the data in the Unicode project (whose goal is to define all the possible written characters and symbols on this planet) and the Common Locale Data Repository (whose goal is to document all the different data and definitions needed for the locale database), we can easily keep track of the data, without the need of being able to display the data in the required character sets or understand them fully when updates are submitted by third parties.
Current status: Conversion of share/monetdef, share/msgdef, share/numericdef, share/timedef to the new design is completed. The Makefile infrastructure is converted. Regression checks are done. Most of the tools are in place, waiting on the import of bsdiconv to the base system.
The code has been extracted from NetBSD and has been transformed into an independent shared library. The basic encodings are well supported. Almost all forward conversions (foo -> UTF-32) are compatible with GNU but the reverse ones are not so accurate because of GNU's advanced transliteration. Some extra encodings have also been added. There are two modules, which segfault; they need some debugging. I can keep working on this project as part of my BSc thesis, so I hope to be able to solve the remaining issues. Improved GNU compatibility is also very desired (extra command line options for iconv(1), iconvctl(), private interfaces, etc.).
FreeBSD's ext2fs had some parts under GPL. The aim of my project was to rewrite those parts and free ext2fs from GPL. I have been successful in rewriting the parts and NetBSD's ext2fs was a great help in this. Certain critical parts under GPL were also removed due to which the write performance suffered. I also implemented Orlov Block Allocator for ext2fs. Currently I am planning to make ext2fs Multiprocessor Safe (MPSAFE). My work resides in truncs_ext2fs branch of Perforce.
In the last months, we have not added new translations, although we have been working on the existing ones to have them updated. We need more translators and volunteers to keep the amount of the translated documentation growing, so feel free to contribute. Every line of submission or feedback is appreciated and highly welcome.
If you want to join our work, please read the introduction to the project as well as the FDP Primer (both of them are available in Hungarian).
Recently, we have added one new article translation. The existing translations have not been updated, though. We need more human resources to keep up with the work and keep the translations up-to-date.
This project was started as part of Google Summer of Code 2008 but there is still a bit of work to complete some missing parts. The BSD-licensed grep implementation is feature-complete and has a good level of GNU compatibility. Our only current concern about the BSD-licensed version is to improve its performance. The GNU variant is much more complex, has about 8 KSLOC, while BSD grep is tiny, has only 1.5 KSLOC. GNU uses some shortcuts and optimizations to speed-up calls to the regex library; that is why it is significantly faster. My point of view is that such optimizations must be implemented in the regex library, keeping the dependent utilities clean and easy to read. BSD grep is so tiny that there is hardly any optimization opportunity by simplifying the code, so the regex library is the next important TODO. There is another issue with the current regex library. It does not support some invalid regular expressions, which work in GNU. We need to maintain compatibility, so we cannot just drop this feature. Actually, BSD grep is linked to the GNU regex library to maintain this feature but due to the lack of the mentioned shortcuts, it is still slower than GNU. Anyway, if we can live with this little performance hit until we get a modern regex library, I think grep is ready to enter HEAD. As for the regex library, NetBSD's result of the last SoC is worth taking a look.
The sort utility has been rewritten from scratch. The existing BSD-licensed implementation could not deal with wide characters by design. The new implementation is still lacking some features but is quite complete. There is a performance issue, though. Sorting is a typical algorithmic subject but I am not an algorithmic expert, so my implementation is not completely optimal. Some help would be welcome with this part.
The bc/dc utilities have been ported from OpenBSD. They pass OpenBSD's and GNU's regression tests but they arrived too late to catch 8.X, so they will go to HEAD after the release.
The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This allows for networking independence between jail environment, each maintaining its private network interfaces, IPv4 and IPv6 network and port address space, routing tables, IPSec configuration, firewalls, and more.
During the last months the remaining pieces of the VIMAGE work were merged by Marko, Julian and Bjoern. Robert Watson developed a vnet allocator to overcome ABI issues. Jamie Gritton merged - his hierachical jail framework that now also is the management + his hierarchical jail framework that now also is the management interface for virtual network stacks.
During the FreeBSD Developer Summit that took place at EuroBSDCon 2009 in Cambridge, UK, people virtualized more code. As a result SCTP and another accept filter were virtualized and more people became familiar with the design of VImage and the underlying concepts. Finally getting more hands involved was a crucial first step for the long term success of kernel virtualization.
The next steps will be to finish the network stack virtualization, generalize the allocator framework before thinking of virtualizing further subsystems and to update the related documentation. Along with that a proper jail management framework will be worked on. Long term goals, amongst others, will be to virtualize more subsystems like SYS-V IPC, better privilege handling, and resource limits.
In the upcoming FreeBSD 8.0 Release, vnets are treated as an experimental feature. As a result, they are not yet recommended for use in production environments. There was lots of time spent to finalize the infrastructure for vnets though, so that further changes can be merged and we are aiming to have things production ready for 8.2.
In case you want to help to achieve this goal, feel free to contact us and support or help virtualizing outstanding parts like two firewalls, appletalk, netipx, ... as well as generating regression tests.
TCP appropriate byte counting (RFC 3465) support has been merged into the FreeBSD 8 branch and will ship in FreeBSD 8.0-RELEASE.
The reassembly queue auto-tuning and SIFTR work was not ready in time to safely integrate for 8.0-RELEASE. Padding has been added to necessary TCP structs to facilitate MFCing features back to the 8-STABLE branch after 8.0 is released.
Candidate patches against FreeBSD-CURRENT will be ready for wider testing in the coming weeks. The freebsd-net mailing list will be solicited for testing/feedback when everything is ready.
The patch has received some significant rototilling in the past few months to prepare it for merging to FreeBSD-CURRENT. Additionally, I completed an implementation of the CUBIC congestion control algorithm to complement the existing NewReno and H-TCP algorithm implementations already available.
I have one further intrusive change to make, which will allow congestion control modules to be shared between the TCP and SCTP stacks. Once this is complete, I will be soliciting for review/testing in the hope of committing the patch to FreeBSD-CURRENT in time to be able to backport it for 8.1-RELEASE.
SCTP continues to have minor fixes added to it as well as some new features. First and foremost, we now have VIMAGE and SCTP working and playing together. This goal was accomplished with the help of bz@, my new mentee tuexen@ and myself working together at the FreeBSD DevSummit in Cambridge, UK. Also the non-renegable SACK feature contributed by the university of Delaware was fixed so that now its safe to turn on (its sysctl). If you are using SCTP with CMT (Conncurrent Multipath Transfer) you will want to enable this option (CMT is also a sysctl). With CMT enabled you will be able to send data to all the destinations of an SCTP peer.
We welcomed a new mentee (soon to be a commiter) to FreeBSD. Michael Tuexen is now a mentee of rrs@. Michael has been contributing to the SCTP work for quite some time and also moonlights as a Professor at the University of Muenster in Germany (when not doing SCTP coding).
We believe that the ZFS file system is now production-ready in FreeBSD 8.0. Most (if not all) reported bugs were fixed and ZFS is no longer tagged as experimental. There is also ongoing work in Perforce to bring the latest ZFS version (v19) to FreeBSD.
Currently working on board bringup. I have looked over the docs for how MIPS provides performance counters and will begin adding code soon.
Andreas Tobler made the classic mistake of sending us a lot of powerpc and sparc64 related patches. The usual punishment, of giving him a commit bit to the Gecko repository, has been applied.
We currently have some old ports in the ports tree:
A patch that includes the following changes has been tested on pointyhat and is ready for commit:
We are also working on Firefox 3.6 (Alpha 2), Thunderbird 3.0 (Beta 4), new libxul 1.9.1.3 and Seamonkey 2.0 (Beta 2) ports. All of them are already committed to our Gecko repository.
A current status and todo list can be found at http://trillian.chruetertee.ch/freebsd-gecko/wiki/TODO.
I am currently seeking funding for further development work on portmaster. There are several features that are regularly requested by the community (such as support for installing packages) that I would very much like to implement but that will take more time than I can reasonably volunteer to implement correctly. There is information about the funding proposal available at the link above.
Meanwhile I have recently completed another round of bug fixes and feature enhancements. The often-requested ability to specify the -x (exclude) option more than once on the command line was added in version 2.12. Also in that version I added the --list-origins option to make it easier to reinstall ports after a major version upgrade, or install the same set of ports on another system.
The Valgrind suite in the FreeBSD ports collection has been updated to version 3.5.0 (the latest available version). Most of the issues of the previous version should be resolved now: we expect memcheck, callgrind and cachegrind to be fully functional on both i386 and amd64 platforms as well as for i386 binaries running on amd64 system. DRD/hellgrind should work too, though they generate a lot of false-positives for now, so their output is a bit messy.
Noteworthy developments regarding FreeBSD/sparc64 since the last Status Reports are:
During Google Summer of Code 2008, I have implemented native support for NFSv4 ACLs for both ZFS and UFS. Most of the code has already been merged to CURRENT. NFSv4 ACLs are unconditionally enabled in ZFS and the usual tools, like getfacl(1) and setfacl(1) can be used to view and change them. I plan to merge the remaining bits (UFS support) this month. It should be possible to MFC it in order to ship in FreeBSD 8.1-RELEASE.
2009 was The FreeBSD Project's fifth year of participation in the Google Summer of Code. We had a total of 17 successful projects. Some GSoC code will be shipping with FreeBSD 8.0-RELEASE and others will be integrated into future releases.
The FreeBSD GSoC admin team would like to thank Google and our students and mentors of another great year!
This work's purpose is a generic and flexible framework for systems equipped with Time Division Multiplexing (TDM) units, often found on embedded telecom chips. The framework is designed to support various controllers and many types of TDM channels e.g. voiceband, sound and miscellaneous data channels. Currently, voiceband infrastructure is being developed on Marvell RD-88F6281 reference board. It will serve as an example of how to use the TDM framework for other channel types. The direct objective of using TDM with voiceband channels is bringing a FreeBSD based VoIP system, capable of bridging analog telephone world with digital IP telephony. Together with third party VoIP software (e.g. Asterisk), the design can serve as VoIP Private Branch Exchange (PBX).
Current state highlights:
The Release Engineering Team continues to work on FreeBSD 8.0-RELEASE. Public testing has turned up quite a few problems, many related to the low-level network (routing/ARP table) changes and their interactions with IPv6.
Progress continues to be made on fixing up the issues that have been identified during the public testing. At this point in time we are shooting for two more public test builds (RC2 and RC3) followed by the release late October or early November.
This report covers &os; related projects between October and December 2009. This is the last of the four reports covering 2009, which has shown to be a very important year for the &os; Project. Besides other notable things, a new major version of &os;, 8.0-RELEASE, has been released, while the release process for 7.3-RELEASE is soon to begin.
Thanks to all the reporters for the excellent work! We hope you enjoy reading. Let us also take this opportunity to wish you all a happy and successful new year for 2010.
Please note that the deadline for submissions covering the period between January and March 2010 is April 15th, 2010.
A DAHDI support module for &os; has been created in the official Asterisk SVN repository.
The following drivers are currently ported:
The following HW drivers are currently ported and tested:
Existing ata(4) infrastructure, which has been around many years, has various problems and limitations when compared to modern controllers/device support. Although the CAM subsystem (used for SCSI) is almost as old as ata(4), it is more eligible to solve the current problems. To reduce code duplication and better support border cases such as ATAPI and SAS, we have started to develop a new CAM based ATA implementation.
As such, CAM infrastructure has been extended to support different transports. New transport has been implemented to support PATA/SATA buses. To support ATA disks, a new CAM driver (ada) has been written. ATAPI devices are supported by existing SCSI drivers cd, da, sa, etc. To support SATA port-multipliers another new CAM driver (pmp) has been written. To support most featured and widespread SATA controllers, new drivers ahci(4) and siis(4) have been developed.
To support legacy ATA controllers, a kernel option ATA_CAM has been added. When used, it makes all ata(4) controllers directly available to CAM, deprecating ata(4) peripheral drivers and external APIs. To make this possible, ata(4) code has been heavily refactored, making controller driver API stricter.
Command queuing support gives new ATA implementation up to double performance benefit on some workloads, with 20-30% improvement quite usual.
SATA Port Multiplier support makes it easy to build fast and cheap storage with huge capacities, by using dozens of SATA drives in one system or external enclosures,
Some of that code has been presented in the recently released &os; 8.0-RELEASE but 8-STABLE now includes a much improved version.
Chromium is a Webkit-based web browser that is largely BSD licensed. It has been ported from Linux to &os; in October and we have been posting patches and test builds periodically since then. Chromium works well on &os; — it is very fast and stable but there are a handful of rough edges that need to be polished up. Two remaining bugs should probably be fixed before releasing a chromium-devel port. We are looking for volunteers to test and maintain this port to make this BSD browser a viable option on &os; desktop systems.
I have been adding a small intent log to SoftUpdates to eliminate the requirement for fsck after an unclean shutdown. This work has been funded by Yahoo!, iXsystems, and Juniper. Kirk McKusick has been aiding me with design critiques and helping me better understand SoftUpdates.
Extensive testing by myself and Peter Holm has yielded a stable patch. Current users are encouraged to follow the instructions posted to the current@FreeBSD.org mailing list to verify stability in your own workloads. Updates are forthcoming and it is expected to be merged to 9.0-CURRENT before the end of January. Ports to older versions of &os; will be available in SVN under alternate branches. Official backports will be decided by re@ when 9.0-CURRENT is stable.
The changes are fully backwards and forwards compatible as there are very few metadata changes to the filesystem. The journal may be enabled or disabled on existing FFS filesystems using tunefs(8). The log consumes 64 MB of space at maximum and fsck time is bounded by the size of the log rather than the size of the filesystem. Other details are available in my technical journal.
Most of the recent activity has been dealing with the 8.0-RELEASE process. As an experiment, we have tried to decouple the ports QA timeline as much as possible from the src QA timeline. Although this meant that the impact on people actively maintaining and using ports has been much less than in previous releases, it still has not solved the problem of the release going out with a stale set of packages. We are still trying to come up with a better solution for the problem.
The ports count is over 21,000. The PR count jumped to over 1,000 but is now back to around 950.
We are currently building packages for amd64-6, amd64-7, amd64-8, i386-6, i386-7, i386-8, i386-9, ia64-8, sparc64-7, and sparc64-8. This represents the addition of i386-9 and ia64-8 since the last report.
There has been some discussion of when to drop regular package builds for 6.X but no decision has been made yet. The cluster and the port managers are struggling to keep up with so many branches being active all at the same time.
Mark Linimon continues to make progress on the cluster nodes. Almost every node that does not have a hard disk failure is now online. In addition, he continues to make progress debugging problems that occasionally take nodes offline.
The next task is to characterize the overall performance of the build cluster. The question has been asked of us, "what would it take to speed up package builds?" There is no one simple answer. It is not merely a matter of having a larger number of package building machines, so before asking for funding we first need to identify the current bottlenecks. While we are starting to understand the problems on the nodes, the problems on the dispatch machine itself are much harder. Complicating the matter is that there are several periodic processes (ZFS backup, ZFS expiration, and errorlog compression, among others) that can combine to slow that machine significantly. The simultaneous interaction of all these is proving difficult to quantify.
Between Pav Lucistnik and Martin Wilke, many more experimental ports runs have been completed and committed.
We have added 3 new committers since the last report, and 1 older one has rejoined us.
The main thing that has taken place since the last Status Report is that I have gotten to the bottom of the remaining PCI problems with Sun Fire V215/V245 and support for these has been completed and since r202023 now is part of 9.0-CURRENT. With some luck it will also be part of the upcoming 7.3-RELEASE.
Some other news:
Recently, a bunch of new device IDs have been added for the u3g(4) cellular wireless driver; the list should be comparable now with other operating systems around. A lot of these devices have a feature where the unit first attaches as a disk or CD-ROM that contains the Win/Mac drivers. This state should be detected by the u3g driver and the usb device is sent a command to switch to modem mode. This has been working for quite some time but as it is implemented differently for each vendor I am looking for feedback on any units where the auto switchover is not working (or the init is not recognized at all). Please ensure you are running an up to date kernel, like r201681 or later from 9.0-CURRENT, or 8-STABLE after the future merge of this revision.
We are happy to announce that Benedict Reuschling is now free from mentorship and can commit to the documentation tree on his own.
Since the last status report, the German Documentation Team has chased updates to various sections of the &os; Handbook, FAQ and the German website. Many handbook pages have been updated to the latest version, including chapters about configuration, disks, kernel configuration, printing, multimedia and virtualization.
We require help from volunteers that are willing to contribute bug fixes or translations. The following documents need active maintainership and are a good training ground for those willing to join the translation team:
There is one article translation pending review. Apart from this, - neither translations nor maintainance work have been done. We need + neither translations nor maintenance work have been done. We need more volunteers, mostly translators but we are glad to have more reviewers, as well. One can join by simply subscribing to the translators' mailing list where all the work is done.
In the last months, no new translation has been added. Lacking human resources, we can only manage to keep the existing documentation and web page translations up to date. If you are interested in helping us, please contact us via the email addresses noted above.
Since the last report we have seen a growth of 2,000 users on our forums resulting in approximately 10,000 registered users at this time. The posts count is about to reach 60,000 soon, which are contained in almost 9,000 threads.
The sign-up rate still hovers between 50-100 each week. The total number of visitors (including 'guests') is currently hard to gauge, but is likely to be a substantial multiple of the registered userbase.
New topics and posts are actively 'pushed out' to search engines. This in turn makes the forums show up in search results more and more often, making it a valuable and very accessible source of information for the &os; community.
One of the contributing factors to the forums' success is their 'BSD-style' approach when it comes to administration and moderation. The forums have a strong and unified identity and are very actively moderated, spam-free, and with a core group of very active and helpful members, dispensing many combined decades' worth of knowledge to starting, intermediate and professional users of &os;.
V4L video support in the Linux emulator is now available.
This work allows Linux applications using V4L video calls to work with existing &os; video drivers that provide V4L interfaces. It is tested and working with the net/skype port and also with browser-based Flash applications that access webcams. An early version has been committed to 9.0-CURRENT and work is in progress to commit the latest version and then MFC. It is also tested on &os;-8.0/amd64 and &os;-7.2/i386.
Note: to be clear, this does not add V4L support to all webcams. The &os; camera driver must already offer V4L support itself in order for a Linux application to be able to use that camera. The multimedia/pwcbsd port provides the pwc(4) driver that already has V4L support. If your camera is supported by a different driver, you will need to enhance that driver to add V4L support.
The webcamd daemon enables hundreds of different USB based webcam devices to be used under the &os;-8/9 operating system. The webcam daemon is basically an application, which is a port of Video4Linux USB webcam drivers into userspace on &os;. The daemon currently depends on libc, pthreads, libusb and the VIDEO4BSD kernel module.
Historically, &os; has limited the number of supplemental groups per process to 15 (NGROUPS_MAX was incorrectly declared to be 16). In &os; 8.0-RELEASE we raised the limit to 1023, which should be sufficient for most users and will be acceptably efficient for incorrectly written applications that statically allocate NGROUPS_MAX + 1 entries.
Because some systems such as Linux 2.6 support a larger group limit, we have further relaxed this restriction in 9.0-CURRENT and made kern.ngroups a tunable value, which supports values between 1023 and INT_MAX - 1. We plan to merge this to 8-STABLE before 8.1-RELEASE.
This import is based on OpenBSD 4.5 state of pf(4). It includes many improvements over the code currently present in &os;. The actual new feature present in pf45 repository is support for divert(4), which should allow tools like snort_inline to work with pf(4) too.
Currently, the pf(4) import is considered stable with normal kernel, as well as VIMAGE enabled kernels.
Native NFSv4 ACL support in ZFS and UFS has been committed into 9.0-CURRENT. It is expected to be MFCed in order to make it into &os; 8.1-RELEASE.
The run(4) driver brings support for Ralink RT2700U/2800U/3000U USB wireless devices. For detailed information and list of all the supported devices, please see the above mentioned URL. The source code has been imported to the USB P4 repository on January 10, 2010 (172906).
The base/projects/mips branch has been merged into 9.0-CURRENT. The merge is complete and the sanity tests have passed. The code has booted on both a Ubiquiti RouterStation (big endian) as well as in gxemul (little endian).
The branch lived for one year, minus a day, and accumulated much work:
The development branch had been updated incorrectly several times over the past year, and the damage was too much to repair. We have retired the branch and will do further mips development in 9.0-CURRENT for the time being. If you have a checked out tree, the suggested way to update the projects/mips tree you have is to do a "svn switch svn://svn.FreeBSD.org/base/head" in that tree.
I would like to thank everybody that has contributed time, code or hardware to make &os;/mips better.
As development proceeds, I will keep posting updates. In addition, I hope to have some mini "how-to" wiki pages done for people that want to try it out.
The purpose of this project is to provide &os; with support for the Flattened Device Tree (FDT) technology, the mechanism for describing computer hardware resources, which cannot be probed or self enumerated, in a uniform and portable way. The primary consumers of this technology are embedded &os; platforms (ARM, AVR32, MIPS, PowerPC), where a lot of designs are based on similar chips but have different assignment of pins, memory layout, addresses bindings, interrupts routing and other resources.
Current state highlights:
Work on this project has been sponsored by the &os; Foundation.
HAST software will provide synchronous replication of any GEOM provider (eg. disk, partition, mirror, etc.) or file from one &os; machine (primary node) to another one (secondary node).
Because data is replicated at the block level neither applications, nor file systems have to be modified to take advantage of this functionality.
The functionality that HAST software will provide is very similar to the functionality provided by the DRBD project for Linux.
The HAST project is sponsored by the &os; Foundation.
Work is progressing well; first milestone was reached in December 2009 and the expected project completion date is January 31, 2010.
Check out &os; mailing lists for patches to test in February and wish me good luck!
And by the way, do not forget to donate to the &os; Foundation, as your donations make projects like this possible.
Thank you!
Development of the &os; 802.11s stack continues. The code in &os; HEAD has been updated to comply with draft 4.0. Merge to &os; 8-STABLE will be done soon.
The developer is looking for funding to be able to implement mesh link security algorithms and/or coordinated channel access (performance improvement).
Important changes regarding &os; TDM Framework since the last status report:
On January 13, I removed the utmp user accounting database and replaced it with a new POSIX utmpx implementation. Unfortunately, the upgrade path is a bit complex, because the utmp interface provided almost no library interface to interact with the database files.
This change may have caused some regressions. Some ports may fail to build, while there could also be bugs in the library functions.
Good compatibility has been ensured and there are only few pending items that have to be reviewed/enhanced. Recently, an enhancement has been completed, which makes it possible to accomplish better transliteration, just like in the GNU version. An initial testing patch is expected at the beginning of February.
As 8.0-RELEASE is out, BSD bc/dc can be now committed to 9.0-CURRENT. We are only waiting for an experimental package building to make sure there are no regressions after this change. BSD grep is complete but it cannot be integrated yet because of some regex library issues. We need first a fast and modern regex library so that we can change to BSD grep. BSD sort has few incomplete features and needs some performance review.
NVIDIA has released the first BETA version of its graphics drivers for &os;/amd64. Note that this driver will work on &os; versions 7.3-RELEASE or 8.0-RELEASE and later. It also works on very recent versions of 7.2-STABLE. More details are provided in the official release announcement.
Bugmeister Gavin Atkinson has now been granted a src commit bit, and is now starting to work through some of our backlog.
The list of PRs recommended for committer evaluation by the Bugbusting Team continues to receive new additions; however, it has not yet achieved high visibility. (This list contains PRs, mostly with patches, that the Bugbusting Team consider potentially ready to be committed as-is, or are probably trivially resolved in the hands of a committer with knowledge of the particular subsystem.) One of the suggestions at the Cambridge devsummit was to create a way for people to be emailed the weekly summary that is posted to freebsd-bugs@, and this has now been implemented. Please email linimon@FreeBSD.org to ask to be added to the recommended_subscribers.txt file (see above).
We continue to classify PRs as they arrive, adding 'tags' to the subject lines corresponding to the kernel subsystem involved, or man page references for userland PRs. These tags, in turn, produce lists of PRs sorted both by tag and by manpage. At this point most of the PRs that refer to supported versions of &os; have been converted, and we are keeping up as new ones come in. We hope that this is making it easier to browse the PR database.
The overall PR count jumped to over 6,200 during the 8.0-RELEASE release cycle but seems to have stabilized at around 6,100. As in the past, we have a fairly good clearance rate for ports PRs but much less so for other PRs. (Partly this is due to the concept of individual ports having 'maintainers'.)
Work continues on our ia64 port. Many recent commits to help improve stability have been made to 9.0-CURRENT and MFCed to 8-STABLE.
Due to interest from a very motivated user, package builds have been restarted for ia64-8. This is primarily intended as a QA step to discover and fix bugs on ia64, rather than to create packages for upload.
Based on the above, Mark Linimon documented how to add more architectures to the package cluster scheduler. (This work will also be of use in an upcoming effort to start powerpc package builds.)
There are currently 3 ia64 machines online and building packages. The machines seem stable as long as multiple simultaneous package builds are not attempted, in which case they get machine checks. This is puzzling, since other heavy workloads seem stable on the same machines.
bwn(4) is replacing bwi(4) driver for to the following reasons:
Currently, it is tested on big/little endian machines and 32/64-bit DMA operation with STA mode. A major patch for siba(4) is being reviewed before committing into 9.0-CURRENT.
The Release Engineering Team announced &os; 8.0-RELEASE on November 26th, 2009. With 8.0-RELEASE completed planning has begun for 7.3-RELEASE. The schedule has been set with the release date planned for early March 2010.
The Release Engineering Team would like to thank George Neville-Neil (gnn@) for his service on the team. George continues to work with the &os; Project but has stepped down from the Release Engineering Team to focus on other activities.
One of the longer outstanding feature problems with the &os; IP security stack, broken IP Payload Compression Protocol (IPcomp) support, has been fixed.
While working on the fix, various problems had been identified:
Patches for all but the zlib support have been committed to 9.0-RELEASE and merged to all supported stable branches including 6-STABLE. Special thanks to Eugene Grosbein for helping with testing.
Despite a difficult economy, we more than doubled our number of donors, we raised $269K towards our goal of $300K, and with an improved economy hope to surpass that this year.
We have funded two new projects. One is the Flattened Device Tree by Rafal Jaworowski. And, the second one is Highly Available Storage by Pawel Jakub Dawidek. We continued supporting the New Console Driver by Ed Schouten and Improvements to the &os; TCP Stack by Lawrence Stewart. We also purchased equipment for several projects.
We have big plans for the new year! We are going to significantly increase our project development and equipment spending. Stay tuned for a project proposal submission announcement soon. We just announced that we are accepting travel grant applications for AsiaBSDCon and will be accepting them soon for BSDCan. And, we are working on infrastructure projects to beef up hardware for package-building, network-testing, etc.
Read more about how we supported the project and community by reading our end-of-year newsletter available at http://www.FreeBSDFoundation.org/press/2009Dec-newsletter.shtml.
We are fund-raising for 2010 now! Find out more at http://www.FreeBSDFoundation.org/donate/.
VirtualBox 3.1.2 has been committed to the ports tree.
Several changes to the port have been performed with this update including:
We would like to say thanks to all the people that helped us by reporting bugs and submitting fixes. We also thank the VirtualBox developers for their help with the ongoing effort to port VirtualBox to &os;
BSDCan, a BSD conference held in Ottawa, Canada, has quickly established itself as the technical conference for people working on and with 4.4BSD based operating systems and related projects. The organizers have found a fantastic formula that appeals to a wide range of people from extreme novices to advanced developers.
BSDCan 2010 will be held on 13-14 May 2010 at the University of Ottawa, and will be preceded by two days of Tutorials on 11-12 May 2010.
There will be related events (of a social nature, for the most part) on the day before and after the conference.
Please check the conference web site for more information.
AsiaBSDCon is a conference for users and developers on BSD based systems. AsiaBSDCon is a technical conference and aims to collect the best technical papers and presentations available to ensure that the latest developments in our open source community are shared with the widest possible audience. The conference is for anyone developing, deploying and using systems based on FreeBSD, NetBSD, OpenBSD, DragonFlyBSD, Darwin and MacOS X.
The next conference will be held at the Tokyo University of Science, Tokyo, Japan, on 11th to 14th March, 2010.
For more detailed information, please check the conference web site.
The meetBSD conference is an annual event gathering users and developers of the BSD operating system family, mostly &os;, NetBSD and OpenBSD. Afer the special California edition, meetBSD Wintercamp in Livigno, this year we are back to Krakow, Poland.
In 2010, meetBSD will be held on 2-3 July at the Jagiellonian University.
See the conference main web site for more details.
We are again able to build bootable i386/amd64 kernel. Nathan Whitehorn committed a fix to &os;, which enabled LLVM/clang to work mostly fine on PowerPC. There is some preliminary testing of LLVM/clang on ARM and MIPS being done. We have some ideas about sparc64 support which are currently being investigated. You are welcome to contact us if you want to help.
Since the last report, a lot has happened mostly in the area of C++; clang is currently able to build working groff, gperf and devd, i.e. all of the C++ apps we have in base. Unfortunately, it still cannot build any of the C++ libraries — two of them are missing builtins and libstdc++ is broken for other reasons.
Not much happened in the clangbsd branch as we cannot upgrade the clang/llvm there because we are blocked by a bug that requires using newer assembler than we can ship. This might be solved by either fixing this (short term) or using llvm-mc instead of GNU as for assembling (longer term).
Preliminary Hardware Performance Counter support for Intel XScale ARM processors was committed to &os; 9.0-CURRENT in December. This adds another supported architecture to hwpmc(9). The system works for basic performance counter usage but more advanced usage scenarios, namely callchain support, are not yet implemented.
This report covers &os; related projects between January and March 2010. Being the first of the four reports planned for 2010 with 46 entries, it shows a good progress of the &os; Project and proves that our committers are keeping up with the latest trends in the OS development. During this period, a new minor version of &os;, 7.3-RELEASE, has been released, while the release process for 8.1-RELEASE is soon to begin and is planned to be released later this summer.
Thanks to all the reporters for their excellent work! We hope you enjoy the reading.
Please note that the deadline for submissions covering the period between April and June 2010 is July 15th, 2010.
The ALQ(9) implementation and KPI has been rototilled and modified (one more patch needs to be committed) to support variable length messages. In addition, it can now be compiled and loaded as a kernel module.
With the ALQ changes in head, SIFTR can finally be imported.
Reassembly queue autotuning is in the project branch and needs to be extracted as a patch people can easily test.
I have just completed the last disruptive change to the KPI, which laid the groundwork to allow different congestion aware transports to share congestion control algorithms. The import into the head branch is a big job and my time is limited, so progress will be slow and I will not have it done and ready to MFC by 8.1 as I had hoped. I will aim to have it in 8.2 though.
In February work was done to address resource leaks in the (virtual) network stack, especially on teardown.
During that time also multiple general run-time problems and leaks were identified and fixed including leaked ipfw tables on module unload, routing entries leaked, in case of interfaces going away, as well as leaked link-layer entries in interaction with flowtable and timers.
For virtual network stacks resources are are no longer allocated
multiple times or freed upon teardown for eventhandlers, IP and upper
level layers, like TCP syncache and host cache, flowtable, and
especially radix/routing table memory.
In addition epair(4) was enhanced and debugging was improved.
This work was sponsored by ISPsystem.
We are once again participating in the Google Summer of Code. This is our 6th year of participation and we hope to once again see great results from our students. Currently applications have all been submitted and we are in the process of reviewing them. Accepted students will be announced April 26th and coding officially begins May 24th.
The Release Engineering Team announced &os;-7.3 on March 23rd, 2010. The schedule has been set for &os;-8.1 with the release date planned for mid July 2010.
mfsBSD is a set of scripts that generate a bootable image (e.g. an ISO file) that creates a working minimal installation of &os; that is completely loaded into memory (mfs).
The project has now reached a stable and well tested state. Images can be created from 8.0-RELEASE or 7.3-RELEASE ISO image files or from a custom makeworld.
A new feature is a script called "zfsinstall" that automates a ZFS-only install of &os; from a mfsbsd ISO (script works with 8-STABLE and 9-CURRENT, sample ISO images can be downloaded from the project web site).
We were proud to be a sponsor for AsiaBSDCon in March. We also committed to sponsoring BSDCan 2010 and NYCBSDCon 2010. We provided travel grants for AsiaBSDCon.
We funded a project by Murray Stokely to provide Closed Captioning of &os; Technical Videos in the BSD Conferences YouTube Channel. We were very pleased that the foundation funded HAST project completed.
We solicited project proposals and were very pleased with the number of proposals we received. With our project spending budget increase, we will be able to fund more projects this year.
We grew our board of directors by adding Erwin Lansing. This will expand our representation in Europe. Erwin brings ports knowledge and expertise to the board.
We continued our work on infrastructure projects to beef up hardware for package-building, network-testing, etc.
Follow us on Twitter now!
We are fund-raising for 2010 now! Find out more at http://www.FreeBSDFoundation.org/donate/.
'tbemd' stands for Target Big Endian Must Die. The current build systems requires that one define TARGET_BIG_ENDIAN for either big endian MIPS or big endian ARM processors. There are many problems with this approach. The resulting system will not create the proper binaries without TARGET_BIG_ENDIAN defined. There is no easy way to know what the endian is of the system you are running. There are many issues with ports, since they do not use bsd make, so do not pick up the extra flags that are added if TARGET_BIG_ENDIAN is defined.
The tbemd branch seeks to fix this. We will move from MACHINE_ARCH=mips for all mips platforms to MACHINE_ARCH=mipsel, mipseb, mips64eb and mips64el to match NetBSD's conventions. These represent 32-bit mips little endian, 32-bit mips big endian, 64-bit mips big endian and 64-bit mips little endian respectively. ARM will move to arm (little endian) and armeb (big endian), again following the standards set elsewhere. To facilitate a number of different MACHINE_ARCHs all built from the same source, a new MACHINE_CPUARCH is introduced and represents the sources needed to build CPU support for a given MACHINE_ARCH.
In addition, MACHINE_ARCH is overused in the build system today.
Many of its uses are gratuitous and can be simplified. Many of its
uses do not scale well and need to be refactored into a system that
will scale well. A per MACHINE/MACHINE_ARCH/MACHINE_CPUARCH selection
mechanism for makefile snippets will be introduced to move much of
the current if spaghetti into more controlled lists.
The branch can build everything we currently support with the new
names.
Work is underway to allow the &os; build system to use out of tree compilers and binary utililies (loaders, linkers, etc), especially in a cross compilation environment. While it is possible to swap out the compiler with a compatible compiler relatively easily, swapping out the toolchain is more involved. In addition, when using an external compiler to build the system, certain parts of buildworld can be omitted.
Portmaster version 2.22 is now in the ports tree and has full support for the following new features:
I am very excited about these new features, and owe a debt of gratitude to the companies and especially the individuals who stepped forward to support this work. I literally could not have done it without them.
DaVinci (TMS320DM644x) is an ARM9-based system-on-chip family from Texas Instruments with built-in DSP core and powerful multimedia/video features. This work is bringing support for &os; on these systems - it works in multiuser mode, using root filesystem mounted either via NFS or from SD/MMC card. The code is available in P4 at //depot/user/jceel/davinci/.
Current DaVinci support includes:
The purpose of this project is to provide embedded &os; with a generic and flexible scheme to support NAND Flash devices. The framework provides a set of KOBJ interfaces inside the kernel, which allow for uniform and flexible management of the NAND devices:
Part of the infrastructure is a full system simulator of ONFI-compliant devices (NANDsim), with a userland control application. This allows for exercising of the framework on platforms without real NAND chips.
Current state highlights:
Most of quarter one was spent dealing with the 7.3-RELEASE process. With apparent success enforcing Feature Safe ports commits during the 8.0-RELEASE, it was continued for the recent src/ freeze.
The ports count now exceeds 21,500 ports, and counting. The open PR count currently is over 1000. With the release of &os; 7.3, it is hoped this count will drop drastically.
Since the last report, we added four new committers, and had an old committer rejoin us.
With the donation of an Apple Xserve, powerpc builds have resumed. Renewed interest in ia64 has brought about new ports builds. A new sparc64 machine hosted by skreuser will help us with this build.
The Ports Management team have been running -exp runs on an ongoing basis, verifying how src code updates may affect the ports tree, as well as providing QA runs for major ports updates. Of note -exp runs were done for; gabor's BSD licensed bc/dc in src/, mva's OpenAL and SDL upgrades; brooks' removal of NGROUPS; ed's removal of libcompat and regexp.h; dinoex's jpeg update; a test run for m4 update; jilles' update for sh(1); johans' update for bison; and roam's curl update.
During 2009 work was done that allowed us to build a &os; kernel without INET and without INET6 (again). This work was the foundation for a prototype to get a kernel to compile and boot with only INET6 but no INET compiled in earlier this year.
The current focus is to identify general architectural problems and dependencies we do have between these two address families as well as with the upper layer protocols. This will at some point allow us to discuss the issues and seek solutions, preparing for a future where we can remove either INET or INET6 from the system.
Once we will have a stable, in-tree way to compile out either address family, optimizations wrt. size, as well as user space will need to be worked on. In addition to this, the work is believed to help should we further head in the direction of network stack modularization.
geom_sched is a GEOM module that supports pluggable schedulers for disk I/O requests. The main algorithm supported at the moment is an anticipatory Round Robin scheduler, which is especially effective in presence of workloads with highly random disk accesses. Other schedulers are available on the geom_sched page.
Developed in early 2009 and refined as a GSOC2009 project, geom_sched has been recently introduced in HEAD and is going to be soon merged to stable/8. A version for stable/7 also exists, with some restrictions.
To use the module, say on disk ad4, all you need to do is:
kldload geom_sched geom sched insert ad4
A number of sysctl variables under kern.geom.sched allow you to tune the parameters of the algorithm, or bypass the scheduler entirely so you can tell the difference of behaviour with and without the scheduler.
Although the bare bones of the NFS Version 4 support was released in &os; 8.0, the integration has been progressing slowly and support should be functional for &os; 8.1 for RFC3530 (NFS Version 4.0).
Post &os; 8.1, I believe the focus will be on code cleanup and, under a projects area of svn, some experimental work on aggressive whole file caching to client disk.
Based on the idea of Jeff Roberson, we reimplemented the path for read(2)/write(2) syscalls using page cache (in wide sense) to eliminate the issues with recursive vnode and buffer lock acquisitions. The usual reads and writes are no longer calls into VOP_READ/VOP_WRITE; the operation is done by copying user buffers to or from the pages of the vnode. This fixes known deadlocks when reads or writes are done over file-mmaped buffers.
The patch changes the performance characteristics of I/O, and we observed both better and worse behaviour. If filesystem implements VOP_GETPAGES and VOP_PUTPAGES without referencing buffer cache, buffers are completely eliminated from the i/o path (not true for UFS or NFS).
The alpha version of libunwind library port for &os; x86 and x86_64 is completed and imported into the official libunwind git repository. Libunwind is the library to perform dynamic unwinding of stacks, using dwarf call frame information. The library features remote unwinding using ptrace(2), very fast setjmp(3) implementation and more interesting features.
&os;/mips has been ported to D-Link DIR-320, wireless router based on BCM5354 SoC. Project aims to providing several working images tailored for different purposes (profiles). So far racoon based router-ipsec image is available.
We have recently completed a massive revision of ipfw and dummynet, and the result has been committed to HEAD and stable/8. The main features introduced with this work are:
Chromium is a Webkit-based web browser that is mostly BSD licensed. It works very well on &os; and even supports new features like HTML 5 video. I have started offering subscriptions to fund the porting effort to &os;, funding which has already paid to fix Chromium on BSD-i386. I am using a new funding model where subscriptions pay for development that is kept closed for at most 1 year, after which all patches used in a build are released to subscribers under the same BSD license as Chromium. Also, parts of the closed patches are continually pushed upstream, the BSD i386 fix has already been committed upstream. The goal is to fund Chromium development on BSD while continually pushing patches back to the BSD-licensed Chromium project. I will spin off a Chromium port for ports soon, for those who do not mind using an older, stable build that does not have all the paid features in the subscriber builds. You can read about the issues that a subscription would pay for, such as replacing the ALSA audio backend with OSS, and find out more about subscribing.
We restlessly keep the existing documentation and web page translations up to date. However, this will not last forever, and help is always welcome, so if you feel yourself Hungarian with some interests in translation, please contact our Documentation Project via the email addresses noted above.
Significant progress has been made in terms of stabilizing the uniprocessor Octeon port and adding support for MIPS ABIs other than o32 in the toolchain, rtld, libc and the kernel. Kernels built to the n32 ABI are currently supported with changes that will not be merged because they make invasive changes throughout the system with regard to db_expr_t and register_t, which are larger than a pointer in the n32 ABI. Once support for n64 kernels is completed (including the ability to run n32 worlds) and the n32 hacks are removed, the branch will be suitable for merging. Many nearby cleanups have occurred, particularly in the area of TLB and pmap code.
I wrote experimental code (please see my project page) and threw patch ( http://gist.github.com/350230 ) to freebsd-hackers. A lot of &os; hackers gave me precious advice, so I am going to reflect it as a next step.
Our last status report listed a number of documents that needed help. Thanks to the external contributions of Frank Boerner we were able to update a substantial amount of documents. This has resulted in a great reduction of our backlog. Subsequently, Benedict has agreed to take Frank under mentorship for the German doc project. We are looking forward to his future contributions and thank him for his past efforts.
Johann was busy keeping the German website in sync with updates to FreeBSD.org. However, there are still parts of the website that remain untranslated. We are looking for more support in maintaining the German website.
&os; users with German language skills are always welcome to join our efforts in translating the documentation and/or fixing bugs.
QAT has been running on a single server for about two years now and has proven very effective at catching problems with ports commits. Many of the problems it cannot catch are architecture or branch related. By moving QAT to a VMware box capable of running arbitrary versions of &os; on both amd64 and i386 this limitation will be removed.
Work on CAM-based ATA implementation continues. Since last report handling of heavy errors and timeouts was improved, Hot-plug now works for both Host and Port Multiplier ports. Series of changes were made to CAM to fix some old issues and honor some new ATA demands.
New drivers ahci(4) and siis(4) got some fixes and are quite stable now. "options ATA_CAM" kernel option shows good results in supporting other controllers using existing ata(4) drivers, so it is possible to start deprecating old ata(4) APIs now.
Started work on new Marvell SATA driver for both PCI-X/PCIe cards and ARM System-on-Chip SATA controllers. It is expected to support NCQ, Port Multipliers with FIS-based switching and other new features.
Most of the code is present in 8-STABLE.
snd_hda(4) audio driver got real multichannel playback support. It now supports 4.0 (quadro), 5.1 and 7.1 analog speaker setups. Digital multichannel AC3/DTS passthrough was already implemented earlier. Digital multichannel LPCM output via HDMI could also be possible now, but is not tested.
To use multichannel playback you should have fresh 8-STABLE kernel, instruct sound(4) vchans subsystem (if you are using it) about your speaker setup using dev.pcm.X.play.vchanformat sysctls and use your audio/video player application to play multichannel audio content without down-mixing it to stereo.
Bruce Cran (brucec) has graduated from GNATS-only access to having a src commit bit. He has been making commits to help us catch up with the PR backlog. Thanks!
We continue to classify PRs as they arrive, adding 'tags' to the subject lines corresponding to the kernel subsystem involved, or man page references for userland PRs. These tags, in turn, produce lists of PRs sorted both by tag and by manpage. The most recent use of these tags is the creating of a new report, Summary Chart of PRs With Tags, which sorts tagged PRs into logical groups such as filesystem, network drivers, libraries, and so forth. The slice labels are clickable. The chart is updated once a day. You can consider it as a prototype for browsing "sub-categories" of kernel PRs.
The "recommended list" has been split up into "non-trivial PRs which need committer evaluation" and the "easy list" of trivial PRs, to try to focus some attention on the latter.
New reports were added for "PRs which are from &os; vendors or OEMs", "PRs containing code for new device drivers", and "PRs referencing other BSDs". These will primarily be of interest to committers.
Some other bitrot on the "experimental PR reports" pages has been fixed.
It is now possible for interested parties to be emailed a weekly, customized, report along the lines of the above. If you are interested in setting one up, contact linimon@FreeBSD.org.
The overall PR count has recently jumped to around 6400. This may be due to increasing uptake of &os; 8.
Our clearance rate of PRs, especially in kern and bin, seems to be improving.
Mark Linimon polled various committers about their interest in specific PRs. As a result, the AssigningPRs page on the wiki and the src/MAINTAINERS file were updated based on feedback.
As always, anybody interested in helping out with the PR queue is welcome to join us in #freebsd-bugbusters on EFnet. We are always looking for additional help, whether your interests lie in triaging incoming PRs, generating patches to resolve existing problems, or simply helping with the database housekeeping (identifying duplicate PRs, ones that have already been resolved, etc). This is a great way of getting more involved with &os;!
The stability of the machines under package build has been improved by a number of recent commits. Some rework is underway to run with WITNESS. However, we are still limited in the number of simultaneous packages that can be built.
Based on this, we have completed the first full ia64-8 package build. 17187 were built (as compared to 19885 on a recent i386-8.) Mark Linimon has gone through the results to denote which packages do not build. A few fixes have already been committed based on this.
We currently have 3 available machines that are stable enough for package builds.
Support for the SGI Altix 350 has made its start. Porting is done on 2 SGI Altix 350 machines connected with NUMAFlex, giving a total of 4 CPUs and 24GB of DDR. The kernel boots with code on the projects/altix branch but since ACPI does not enumerate PCI busses, no hardware devices are found. SMP has been disabled because waking up the APs result in a machine check.
An Apple XServe G5 has been donated by Peter Grehan for package building. Based on the last two months' worth of testing, a large number of commits have been made to increase stability.
We have completed the first full powerpc-8 package build. Only 10918 were built (as compared to 19885 on a recent i386-8), primarily due to a few high-impact packages failing (such as lang/python25). Mark Linimon has gone through the results to denote which packages do not build. A few fixes have already been committed based on this; we have patches that are being tested in the next run.
Mark Linimon is working on getting us more XServes.
&os; is currently lacking support of LDAP based authentication and user identity.
We have integrated a stripped down OpenLDAP library (renamed to avoid conflict with ports OpenLDAP libraries), as well as some changes to OpenSSH as well as plugins for PAM, NSS and can support.
We have used several existing works and updated them to use new OpenLDAP API, fixed several bugs and integrated them together. All these works are under BSD or similar license and our new work would be under 2-clause BSD license. Currently, we support storing user identity, password and SSH public keys in LDAP tree.
Work on supporting EFI booting on &os;/i386 resumed. The boot loader can now read an ELF file from the EFI FAT partition. We are now working on trying to boot a kernel.
A full 64-bit PowerPC port of &os; is now complete, and should shortly be merged to HEAD, likely first appearing in &os; 9.0. This port supports SLB-based 64-bit server CPUs, such as the IBM POWER4-7, PowerPC 970 (G5), and Cell Broadband Engine. Current machine support is limited to Apple single and dual processor G5 systems, with future support planned for IBM Power Systems servers and the Sony PlayStation 3.
The net80211 (wireless) stack will support a modular rate control framework soon. The idea is to reduce some code in the drivers and add more rate control algorithms in the tree. All drivers that do rate control in software will automatically benefit from this project. On this stage, we are working on changing all the necessary drivers to cope with the new framework and making sure it all works as expected. Later this year we will bring the necessary changes to change the rate control algorithm with ifconfig(1).
If you are doing rate control algorithm or research on rate control algorithms for wireless networks, &os; is now an ideal candidate for testing your project!
802.11n support in the Atheros driver is being worked on. Right now it can do AMPDU RX in software and we are working on TX AMPDU. The code lives in a private Perforce branch, but some bits of it are already committed to HEAD.
This work is being sponsored by iXsystems, inc.
Atheros AR9285 support was added to &os; HEAD and 8-STABLE. There are still some issues but in general it works fine.
Webcamd is a userland daemon that enables use of hundreds of different USB based Linux device drivers under the &os;-8/9 operating system. Current focus has been on USB webcam and USB DVB-T/S/C devices. It is also possible to use the webcamd framework to make other Linux kernel USB devices work under the &os;-8/9 operating system, without violating the GPL license. The daemon currently depends on libc, pthreads, libusb and libcuse4bsd. Cuse4BSD is a new character device from userland implementation that fully supports open, read, write, ioctl, mmap and close file operations.
If you like this project or want me to spend more time on it, you can support it by transferring money to hselasky@c2i.net via paypal.
meetBSD is an annual event gathering users and developers of the BSD operating systems family, mostly &os;, NetBSD and OpenBSD. Afer the special California edition, meetBSD Wintercamp in Livigno, this year we are back to Krakow, Poland.
meetBSD 2010 will be held on 2-3 July at Jagiellonian University.
See the conference main web site for more details.
The ZFS file system has been updated to version 14 on both -HEAD and 8-STABLE. Ongoing work is undergoing to bring bug fixes and performance improvements from upstream svn -HEAD to approximately ZFS v15 in the near future, and a full upgrade of ZFS to version 24 including the de-duplication functionality, etc. The de-duplication functionality is currently partly supported, which is demonstrated below:
# uname -sr
FreeBSD 9.0-CURRENT
# zpool create tank ad{4,6,8,10}
# zpool get version tank
NAME PROPERTY VALUE SOURCE
tank version 24 default
# zfs set dedup=on tank
# dd if=/dev/random of=/tank/rand0 bs=1m count=1024
# zpool get allocated,dedupratio tank
NAME PROPERTY VALUE SOURCE
tank allocated 1.00G -
tank dedupratio 1.00x -
# dd if=/tank/rand0 of=/tank/rand1 bs=1m
# dd if=/tank/rand0 of=/tank/rand2 bs=1m
# dd if=/tank/rand0 of=/tank/rand3 bs=1m
# zpool get allocated,dedupratio tank
NAME PROPERTY VALUE SOURCE
tank allocated 1.01G -
tank dedupratio 4.00x -
Since the last status report we got to the state where we are able to build all of &os; (the C and C++ bits) on i386/amd64 with clang. The only exception is the bootloader which does not fit within the given size constraint. This is where the current efforts are going on. The C++ part got a big boost now being able to compile all C++ code in &os; and itself.
-We saw some movment on Mips and PowerPC. Mips got its driver +
We saw some movement on Mips and PowerPC. Mips got its driver definitions from Oleksander Tymoshenko and Nathan Whitehorn did the same for PowerPC and tested the kernel. Currently, the PPC kernel seems to boot but due to lack of va_arg implementation for PowerPC nothing is printed out. Nathan is working on that.
Overall ClangBSD is selfhosting on i386/amd64 and some progress has been made on PowerPC/PPC. We also saw some contribution to the Sparc64 but this seems to have stalled.
We need people to try out ClangBSD (see the wiki) and runtime test it. We also would appreciate help with other archs - namely ARM.
The soft-updates journaling project is nearing completion and will be available in head by the time this status report is released. Backports to other releases are maintained in SVN. SUJ is fully backwards compatible with non-journaled softupdates. Existing systems will not be affected. Journaling may be enabled and disabled by tunefs on unmounted filesystems. Journaling provides near-instant filesystem recovery after crash at the expense of some runtime performance and extra disk I/O.
We are currently doing a lot of code cleanup in the new System Installer backend for PC-BSD, pc-sysinstall, which can be used to install regular &os; as well. Some new features have already been implemented, such as:
This on-going project is to reduce tcbinfo/udbinfo lock and cache line contention; this global lock protects access to connection lists, and while it is a read-write lock, it is acquired for every in-bound packet (briefly) to look up the connection. This project adds a new connection group table, which assigns connections to groups, each of which has CPU affinity and aligns with RSS-selected queues in high-end 1gbps and most 10gbps implementations. The following tasks have been completed:
The following tasks remain:
This work is being performed in the &os; Perforce repository, and is sponsored by Juniper Networks. Connection groups and related features are slated for inclusion in &os; 9.0 (with possible backports to 8-STABLE of some features).
BSDCan, a BSD conference held in Ottawa, Canada, has quickly established itself as the technical conference for people working on and with 4.4BSD based operating systems and related projects. The organizers have found a fantastic formula that appeals to a wide range of people from extreme novices to advanced developers.
BSDCan 2010 will be held on 13-14 May 2010 at the University of Ottawa, and will be preceded by two days of Tutorials on 11-12 May 2010.
There will be related events (of a social nature, for the most part) on the day before and after the conference.
Please check the conference web site for more information.
This report covers &os;-related projects between April and June 2010. It is the second of the four reports planned for 2010, and contains 47 entries. During this period, a lot of work has gone into the development of new minor version of &os;, 8.1-RELEASE, which should be released within days.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
Please note that the deadline for submissions covering the period between July and September 2010 is October 15th, 2010.
A number of updates to the documentation were made since the last status report. We are especially grateful for the contributions from external people who sent the translations. People like Fabian Ruch, who updated the porters-handbook to the latest version (which had been on his to-do list for quite some time), and Benjamin Lukas, who did a great job with the from-scratch translation of the MAC chapter of the German handbook. We thank them both for their contributions and hope they will continue their efforts to enhance the German documentation.
Frank Börner was released from Benedicts mentorship and is now a full committer to the German Documentation Project. We are always looking for fresh blood that is willing to be mentored by us as a first step in becoming committers for the documentation project themselves.
Johann is keeping up the German website with the latest version. But we could use more translators for sections that are not fully translated yet.
The pkg_patch project is about creating a binary package patch infrastructure which would allow users to patch their live system's packages in an easy and efficient way. It is a C program written to interface with libpkg (for things which are common to all pkg utilities) meant to be included in the base system when it is done. It comes with built-in mass patch creation and application commands. It is funded by Google Summer of Code 2010.
For a while I have wanted to rework interrupt threads to address a few issues. The new design uses per-CPU queues of interrupt handlers. Interrupt threads are allocated by a CPU from a pool and bound to that CPU while draining that CPU's queue of handlers. Non-filter handlers can also reschedule themselves at the back of the current CPU's queue while executing. Filters with handlers are now always enabled and should provide a full replacement for the various uses of filters with "fast" taskqueues. A new class of "manual" handlers are also available which are not automatically scheduled, but are only explicitly scheduled from a filter. Thus, a filter can potentially schedule multiple handlers.
The code has been tested on amd64, but it needs wider review and testing. I hope to start soliciting review and feedback soon with the goal of getting the code into 9.0.
The work has been completed and the GNU compatibility levels seems to be quite high. One exception is the fallback support. It is difficult to implement that facility in this implementation because the design is somewhat different. Probably, it will not be a big problem because that functionality is not even documented in the GNU version so few applications might use it.
A portbuild test showed that grep is basically ready to enter HEAD, but there were a few failures that seem to be related. These have to be investigated and fixed before committing grep to 9-CURRENT.
The SGI IRIX operating system has a concept, called job, which is used to group processes together and then apply resource limits on them. The purpose of this project is to implement this facility on &os;.
I spent most of the time familiarizing myself with how things are done inside the kernel, how syscalls work, etc. So far, I have the basic understanding needed and I added the most important syscalls to group processes together into jobs and manipulate collective resource limits on them.
There is a bug, which I am tracking down at the moment, after this I can start to implement actual resource limit enforcement. For some of the limit types, it will be relatively easy but some others will take more effort and studies.
We need manpower. Existing documentation set has not been updated for quite some time because of lack of volunteers. Current members are busy with other projects and real life at the moment and we have not received anything from outside contributors. It is a shame because there are lots of users in Spain and Latin-America, as well. Besides, the world's first Free Software Street has been recently inaugurated in Spain. This obviously means that there is interest in free software but unfortunately, this translation project is not going very well nowadays.
Modern x86 systems include four different types of event timers: i8254, RTC, LAPIC, and HPET. First three are already supported by &os;. Depending on hardware and loader tunables, periodic interrupts from them are used to trigger all time-based events in kernel. That code has a long history, that made it tangled and at the same time limited and hard-coded.
New kernel event timers infrastructure was started to allow different event timer hardware to be operated in uniform way and to allow more features to be supported. Work consists of three main parts: writing machine-independent timer driver API and management code, updating existing drivers and improving HPET driver to support event timers.
The new driver API provides unified support for both per-CPU (independent for every CPU core) and global timers in periodic and one-shot modes. Management code at this moment uses only periodic mode, while one-shot mode use is planned by later tickless kernel work.
Different kinds of timers have different capabilities and could be present in hardware in different combinations. In every situation the infrastructure automatically chooses two best event timers to supply system with hardclock(), statclock(), and profclock() events. If some timer is not functioning — it will be replaced. If there is no second timer — it will be emulated. The administrator may affect that choice using loader tunables during boot and sysctl variables in run-time (kern.eventtimer.*, and so on).
Most of the code was recently committed to HEAD. Now it is used by i386 and amd64 architectures.
Thanks to Katalin Konkoly, the first few chapters of the &os; Handbook translation have been reviewed, therefore many typos and mistranslations were spotted and fixed. Apart from this, we are still keeping the existing documentation and web page translations up to date, currently without plans on further work. If you are interested in helping us, or you have any comments, or requests regarding the translations, do not hesitate to contact the project via the email addresses mentioned in the entry.
Our efforts on porting the generalized, general-purpose purely functional programming language, Haskell has rallied, since two new committers, Giuseppe Pilichi and Ashish Shukla joined recently, forming the &os; Haskell Team. Over the last months, &os;/i386 and &os;/amd64 have become Tier-1 platforms, featuring officially supported vanilla binary distributions for the Glasgow Haskell Compiler starting from version 6.12.1. We introduced a unified ports infrastructure for Haskell Cabal ports, which also makes possible the direct translation of Cabal package descriptions to &os; ports. The number of Haskell package ports increases steadily.
This project is about creating a wrapper library to support monitoring and management of networking with avoiding direct use of the &os; kvm(3) and sysctl(3) interfaces. This approach would allow the kernel implementation to change and monitoring applications to be extended without breaking applications and requiring them to be recompiled. We decided to merge the sources from the last year's Summer of Code project back to the &os; src/ repository piece by piece, and we have defined several phases of integration.
The first phase has been already posted for review. Note that we are looking for a sponsor with an src commit bit and enough time to represent the effort towards the Project.
The ZFS file system has been updated to version 15 on HEAD and it will be MFC'ed to 8-STABLE around September 13th, 2010. Work is in progress on porting the recent ZFS version 26 with deduplication functionality.
The purpose of this project was to provide &os; with support for the Flattened Device Tree (FDT) technology. A mechanism for describing computer hardware resources, which cannot be probed or self enumerated, in a uniform and portable way. The primary consumers of this technology are embedded &os; platforms (ARM, MIPS, PowerPC), where a lot of designs are based on similar chips, but have different assignment of pins, memory layout, addresses ranges, interrupts routing and other resources.
Current state highlights:
Work on this project was sponsored by the &os; Foundation.
This project focuses on updating the www/ja and doc/ja_JP.eucJP/ trees. Since last year www/ja tree has been mostly synchronized with the English counterpart and doc/ja_JP.eucJP has also been updated steadily. We are now working on &os; Handbook and Porter's Handbook.
We are once again participating in the Google Summer of Code. This is our 6th year of participation and we hope to once again see great results from our 18 students. Coding officially began May 24th, and we are in the middle of the mid-term evaluation period. You can see and comment on weekly status reports on the mailing list or on the wiki.
The project started with some cleanup on the network stack after all the import work and adjustments for virtualization to minimize changes to earlier branches. These made it into the tree already and to 8-STABLE, and it will be included in the upcoming 8.1 release.
The first major task was to generalize the virtualization framework, so that virtualization of further subsystems would be easier and could be achieved with less duplication.
In addition some documentation on the virtual network stack programming was written to help developers virtualizing their code. The interactive kernel debugger support was improved and libjail along with jls and netstat can work on core dumps now and query individual jails and attached virtual network stacks.
The second major task was network stack teardown, a concept introduced with the network stack virtualization. The primary goal was to prototype a shutdown of the (virtual) network stacks from top to bottom, which means letting interfaces go last rather than first. Work in this area is still in progress and will have to continue to allow long term stability and a leak and panic free shutdown.
The work on this project had been sponsored by the &os; Foundation and CK Software GmbH. Special thanks also to John Baldwin and Philip Paeps for helping with review and suggestions.
A significant part of quarter two was spent coordinating efforts for inclusion of Xorg 7.5, KDE 4, GNOME 2, plus preparation of ports for the 8.1 release process. Due to the success of enforcing Feature Safe ports commits during 7.3-RELEASE, it was continued for the recent src/ freeze.
The port count is approaching 22,000 ports. The open PR count currently floats at about 1200 entries.
Since the last report, we added four new committers, and had two old committers rejoin us.
The Ports Management Team is very grateful to the &os; Foundation for sponsoring two new head nodes for the ports building cluster, pointyhat. Each of the new head nodes has a larger capacity, both with regard to performance but also in amount of space available for the staging areas, allowing for faster, and thus more, build cycles. Additionally, having two head nodes will allow us to dedicate one of them for building production-ready binary packages, adding predicability for our users to when what types of packages are available for installation, and dedicate the other for regression testing of large port updates, ports infrastructure improvements, the cluster scheduling code, and &os; itself. Over the last few weeks, Mark Linimon has been working hard to get the first of the two new nodes online and has already completed its first package build. This has involved a substantial rework of our custom codebase.
The Ports Management team have been running -exp runs on an ongoing basis, verifying how base system updates may affect the ports tree, as well as providing QA runs for major ports updates. Of note, -exp runs were done for:
alepulver's Licensing Framework Summer of Code project has made it into the tree and the Port Management Team is currently assessing the fallout and it will come up with guidelines and documentation in due time.
On July 13, &os;/powerpc64 was integrated into HEAD. This provides support for fully 64-bit operation on 64-bit PowerPC machines conforming to the Book-S specification, including the PowerPC 970, Cell, and POWER4-7. Hardware support is currently limited to Apple machines, although this should expand in the near future.
Currently supported hardware:
Work has begun to port &os;/powerpc64 to the IBM Cell-based Sony Playstation 3, using the OtherOS feature present on some models of the console. As of July 14, the &os; boot loader is ported, and it is possible to netboot a kernel, which has support for the framebuffer, MMU, and device discovery. Once work on drivers for the network interface and interrupt controller is complete, it will be possible to boot the console multi-user.
AFS is a distributed network filesystem that originated from the Andrew Project at Carnegie-Mellon University; the OpenAFS client implementation has not been particularly useful on &os; since the 4.X releases. Recent work on the OpenAFS codebase has updated it to be consistent with current versions of &os;, and the client, though still considered experimental, is now relatively stable for light (single-threaded) use on 9-CURRENT. The auxiliary utilities for managing and examining the filesystem are functional, and reading and writing files works sufficiently well to copy /usr/src into and out of AFS. Compiling and running executables in AFS is unsuccessful, though, as mmap() is not always reliable.
There are several known outstanding issues that are being worked on, but detailed bug reports are welcome at port-freebsd@openafs.org.
The libpkg library will allow for fairly fine grained control over package management.
Presently libpkg has complete read functionality. Info and delete tools that have most of the current package tool features have already been implemented, and once they are completed they can be considered replacements for their counterparts.
Once the write and logging aspects of the library are more mature, add and create tools can be created quickly. A new set of more maintainable package tools that leverage libpkg will hopefully be available soon after.
This project purpose is adding support for general purpose DMA engines found in most embedded devices. GPDMA framework provides a unified KOBJ interface to DMA engine drivers and unified programming interface to use direct memory transfers in kernel and userspace applications.
This project is a part of Google Summer of Code 2010 and it is a work in progress. Current status can be observed on the wiki page.
First part of the project is mostly complete. I added support for new PORTS_CC variable which should be used in make.conf instead of CC to change ports compiler. This allows user to change ports compiler easily, while still respecting USE_GCC.
Some patches were written to get ports to work with Clang, and a lot of old patches written prior to the Google Summer of Code project were updated. There are still a lot of broken ports, and some that cannot be built because of Clang/LLVM bugs, but at this point, Clang can build most ports.
We were proud to be a sponsor for BSDCan in May. We also committed to sponsoring MeetBSD 2010 Poland and California. We provided 12 travel grants for BSDCan.
The Foundation and Core Team held a summit on BSD-licensed toolchains at BSDCan 2010.
We officially kicked off five new projects that we are funding. They are BSNMP Improvements by Shteryana Shopova, Userland DTrace by Rui Paulo, &os; jail-based virtualization by Bjoern Zeeb, DAHDI &os; driver port by Max Khon, and Resource Containers project by Edward Tomasz Napierała.
We continued our work on infrastructure projects to beef up hardware for package building, network testing, etc. This includes purchasing equipment as well as managing equipment donations.
We are half way through the year and we have raised around $48,000 towards our goal of $350,000. Find out how to make a donation at http://www.FreeBSDFoundation.org/donate/.
Our semi-annual newsletter will be published soon. Check out our website to find out more!
The old ata(4) driver is believed to be going away sometime in the future, to be replaced with ATA_CAM [1]. However, ATA pseudo-RAID support in &os;, ataraid(4), is implemented as part of said ata(4) driver, which means that it, too, will be going away. It was decided that pseudo-RAID support is desirable and that it should be reimplemented in GEOM [2] [3], which this project aims to do.
Currently, RAID-1 arrays can be used on VIA Tech V-RAID and Adaptec HostRAID controllers in a limited capacity. There is no support for writing metadata yet, so disks are not marked degraded, there is no rebuild support, etc. These features are planned, along with support for more hardware and RAID-0 and SPAN arrays.
A major setback for the current code is that it uses the device(9) family of functions to identify ATA pseudo-RAID controllers and constructs arrays based on that information. Unfortunately, ATA_CAM does not appear to add its devices to the device tree, so that tactic cannot be used with ATA_CAM. While this is fine for development of the actual RAID parts of the code, the project will be somewhat useless in the absence of the old ata(4) driver. There has been talk of exporting PCI information to GEOM [4] [5], but the work does not appear to have been completed yet.
The &os;/avr32 project was started by Arnar Mar Sing, and actively developed by him and Ulf Lilleengen. It successfully reached single-user stage but since then has not progressed much. At the moment I am trying to get it back into shape. So far some problems with toolchain on i386 host have been fixed, buildkernel succeeds, buildworld succeeds with some exceptions. Next step would be fixing pmap and bringing port back to single-user stage.
Implementation of General Purpose Input/Output interface for &os;. Current GPIO bus implementation allows user to control pins from userland and it could be expanded to support various type of peripheral devices. So far there are two drivers:
Framework is used in Alexandr Rybalko's port of &os; to D-Link DIR-320 and in Luis Otavio O Souza's work of bringing &os; to RouterBoard.
Chromium is a Webkit-based web browser that is largely BSD-licensed. It works very well on &os; and supports new features like HTML 5 video. This effort uses a new hybrid-source model, where the &os; patches are largely kept closed for a limited time. I submitted Chromium to ports a couple of months ago and recently updated the submission to the stable 5.0.375 branch. The port is ready to be committed pending final legal approval by the &os; Foundation. Further work remains to port Chromium to &os; completely, such as porting the task manager fully and making sure extensions work properly.
This project has two goals: pre-allocation algorithm and ext4 read-only mode.
The aim of pre-allocation algorithm is to implement a reservation window mechanism. Now this mechanism has been introduced. The performance comparison can be found on the wiki.
The aim of ext4 read-only mode is to make it possible to read ext4 file system in read-only mode when the hard disk is formatted with default features. Currently it only supports a few features, such as extents, huge_file. Others features will be added, such as dir_index, uninit_bg, dir_nlink, flex_bg and extra_isize. My work resides in extfs and ext4fs branch of Perforce.
90% of the functionality is working, the daemons sync two systems in a master-slave paradigm.
The aim of the project is to implement an inotify-compatible file system change notification mechanism for &os; and later, and add inotify support to linuxulator. The result, fsnotify is already functional but not yet compatible with inotify in some details.
As of now, &os; only offers very rudimentary resource controls — resource limits for many resources (e.g. SysV IPC) are missing, and there is no way to set resource limits for jails. As a result, users who want to run many different workloads on a single physical machine often have to replace jails with several &os; instances running in virtual machines.
The goal of this project is to implement resource containers and a simple per-jail resource limits mechanism. Resource containers are also a prerequisite for other resource management mechanisms, such as Hierarchical Resource Limits, for "Collective Limits on Set of Processes (aka. Jobs)" Google Summer of Code 2010 project, for implementing mechanism similar to Linux cgroups, and might be also used to e.g. provide precise resource usage accounting for administrative or billing purposes.
This project is being sponsored by The &os; Foundation.
I have been reimplementing VFS namecache to make it granularly locked and supporting reliable full-path lookup without calling underlying file system routines. I have successfully implemented directory cache that works in idealized environment with tmpfs. I am currently working on adding support for entries without associated vnodes and for "weak" entries and incomplete cached path.
&os; Services Control is a mix of binaries which integrate into the rc.d system and provide for service (daemon) monitoring. It knows about signals, pidfiles, and uses very few resources.
The fsc daemon (fscd) runs in the background once the system has started. Services are then added to this daemon via the fscadm control utility, and from there they will be monitored. When they die, depending on the reason, they will be restarted. Certain signals may be ignored (list not decided) and fscd will remove that service from monitoring. Every action is logged to the system logging daemon. Additionally, the fscadm utility may be used to inquire about what services are monitored, their pidfile location, and current process ID.
FSC provides several advantages over the third-party daemontools package. For example, fscd uses push notifications instead of polling; fscd is an internal, &os;-maintained software package accessible to all developers, where daemontools would have to be a port and require us to maintain patches; fscd could be easily integrated with the current rc.d infrastructure.
Partially based on the ideas of daemontools and Solaris Service Service Mangement Facility (SMF), this could be an extremely useful tool for &os; systems.
SIFTR was recently imported into HEAD and will be backported to 8-STABLE in time to be included in 8.2-RELEASE.
TCP reassembly queue autotuning will be ready for public testing within the next week and will be committed soon after. It too will be backported to 8-STABLE after an appropriate burn in period.
The ringmap stack is a complete &os; packet-capturing mplementation specialized for very high-speed networks. Similar to the "zero-copy BPF" implementation, the idea of ringmap is to eliminate packet copy operations by using shared memory buffers. However, unlike the "zero-copy BPF" model, ringmap eliminates ALL packet copies during capturing: the network adapter's DMA buffer is mapped directly into user-space. The ringmap stack also adapts libpcap accordingly to provide userspace applications with access to the captured packets without any additional overhead.
In the context of Google Summer of Code 2010:
Since the last status report some issues with cas(4) have been fixed, allowing it to work with Sun GigaSwift Ethernet 1.0 MMF cards (Cassini Kuheen, part no. 501-5524) as well as the on-board interfaces of Sun Fire B100s server blades (for the Sun Fire B1600 platform).
Support for Fujitsu (Siemens) PRIMEPOWER 250 based on SPARC64 V CPUs has been added. PRIMEPOWER 450, 650, and 850 likely also work but have not been tested. This also means that the building blocks for support of machines based on SPARC64 VI and VII CPUs like the Fujitsu/Sun SPARC Enterprise Mx000 series are now in place, but they need testing as well.
The problems with Schizo version 7 bridges (actually the firmware of these machines) triggering panics during boot finally should be solved.
The work on getting Sun Fire V1280 supported has been stalled due to access to such machines no longer being available.
The above mentioned improvements are/will be available in &os; 8.1-RELEASE and 7.4-RELEASE.
The new system installation backend, pc-sysinstall, was merged into HEAD recently and work is already underway to make it more functional and useful as a complete replacement to standard "sysinstall". It is written 100% in shell, not requiring any additional tools from what is standard to &os;. The backend already supports a number of exciting features such as:
In addition to the features above, pc-sysinstall is unique, in that every install ends up being a scripted install. Front-ends, be it GUI- or text-based, simply generate the appropriate system configuration file, and pc-sysinstall does the grunt work of the actual installation. This is important for a couple of reasons. First, it makes the task of front-end development much easier by not needing to worry about a backend-driven program flow. Second it means that any front-end can be used to generate the installation configuration file, which can then be copied or modified to perform automated installs.
While pc-sysinstall is still relatively new, it is already in use as the default backend for PC-BSD 8.0 and 8.1, and has been getting a very good reception and any bugs found are fixed quickly. A text-based front-end is already in the works which will allow installation media to be created without X11 support.
The purpose of DAHDI/&os; project is to make it possible to use &os; as a base system for software PBX solutions.
DAHDI (Digium/Asterisk Hardware Device Interface) is an open-source device driver framework and a set of hardware drivers for E1/T1, ISDN digital, and FXO/FXS analog cards [1]. Asterisk is one of the most popular open-source software PBX solutions [2].
The project includes porting DAHDI framework and hardware drivers for E1/T1, FXO/FXS analog, and ISDN digital cards to &os;. This also includes TDMoE support, software and HW echo cancellation (Octasic, VPMADT032), and hardware transcoding support (TC400B). The work is ongoing in the official DAHDI SVN repository with the close collaboration with DAHDI folks at Digium.
The project is nearing completion. The DAHDI framework and hardware drivers telephony cards have been ported and tested. There are a number of success stories from early adopters who have been using E1/T1 and FXO/FXS cards on &os; for several months.
Some bug fixes were applied, and the code was also tested and made to work with the cuse4bsd webcam driver, which supports a great many camera chipsets.
The code is still only in 9-CURRENT. We were going to MFC it to 8.x but ran into the code freeze for 8.1, so missed that. However, the code does work on 8-STABLE. We will try to get it MFC'd for 8.2.
In the past quarter we imported Clang into &os; and it is being - built by default on i386/amd64/powerpc. We have not yet commited + built by default on i386/amd64/powerpc. We have not yet committed the necessary changes to let world compile with Clang.
Some bugs and warnings were fixed in HEAD as a result of the Clang import and people are exploring more and more areas (DTrace, etc). There are some bug fixes in Clang/LLVM as well that stem from the import (unknown pragmas warnings, etc).
Roman Divacky and Matthew Fleming are working on ELF writer in LLVM. This is meant as a replacement for assembler (currently we use an outdated GNU as(1)). This work is progressing nice, currently it is able to produce working variants of hello world in C and C++, and some other small programs from "configure run".
After a long hiatus, we aim to hold a bugathon on the weekend of the 6th - 9th August. Everybody is welcome to help resolve or progress PRs from the database. We appreciate the help of committers and non-committers alike, please join us on IRC in #freebsd-bugbusters on EFnet if you are free at any time over that weekend and can help. Please see the "Bugathon" URL for more information.
Mark Linimon and Gavin Atkinson held a session on the State of Bugbusting at BSDCan, which was well attended and led to some interesting discussions. Time was also found to sit down with several committers to discuss long-standing PRs.
The bugbusting team continue work on trying to make the GNATS PR database more accessible and easier for committers to find and resolve PRs.
As a result, PRs continue to be classified as they arrive, by adding 'tags' to the subject lines corresponding to the kernel subsystem involved, or man page references for userland PRs. Reports are generated from these nightly, grouping related PRs in one place, sorted by tag or man page. Mark Linimon continues work on producing a new report, Summary Chart of PRs with Tags, which sorts tagged PRs into logical groups such as file system, network drivers, libraries, and so forth. The slice labels are clickable and may further subdivide the groups. The chart is updated once a day. You can consider it as a prototype for browsing "subcategories" of kernel PRs.
The "recommended list" has been split up into "non-trivial PRs which need committer evaluation" and the "easy list" of trivial PRs, to try to focus some attention on the latter. Various new reports exist, including "PRs containing code for new device drivers", "PRs which are from &os; vendors or OEMs", and "PRs referencing other BSDs".
It is now possible for interested parties to be emailed a weekly, customized, report similar in style to the above. If you are interested in setting one up, contact linimon@FreeBSD.org.
Our clearance rate of PRs, especially in kern and bin, seems to be improving. The number of non-ports PRs has stayed almost constant since the last status report.
As always, anybody interested in helping out with the PR queue is welcome to join us in #freebsd-bugbusters on EFnet. We are always looking for additional help, whether your interests lie in triaging incoming PRs, generating patches to resolve existing problems, or simply helping with the database housekeeping (identifying duplicate PRs, ones that have already been resolved, etc). This is a great way of getting more involved with &os;!
BSDCan 2010 was our 7th conference. As has become the custom, a &os; developer summit was held in the two days before the conference. Record numbers attended the Dev Summit which carried over into the conference proper. It was great to see representatives from so many more companies. I saw many great ideas take root and the start of cooperation on several projects.
The talks during the Dev Summit are beginning to attract a wider audience, and we have been talking about opening this up to the general audience by creating a fourth track at BSDCan 2011.
As impossible as it sounds, each year has seen an increase in the quality of talks and the number of proposals submitted.
meetBSD 2010 took place on July 2 - 3 in Krakow, Poland at the Faculty of Mathematics and Computer Science building of the Jagiellonian University.
The gathering was a much successful event which brought together developers, contributors, and users of the BSD systems from around the world. We had many interesting presentations, of various character and appeal for the diversified audience.
Attendees had a chance for taking the BSD Certification exam during the conference, as well as the advantage of face to face side conversations and discussions, which continued long during the social event on Friday night!
The conference presentation slides are already available for download. Video recordings edition is being finalized, and their publication is expected shortly.
We hope you enjoyed the event and had great time in Krakow. See you again soon!
The Release Engineering Team has been working on the &os; 8.1-RELEASE. At the time of this writing the final builds have been completed and uploaded to the master FTP site. The release announcement should be made within the next couple of days.
The 2010 &os; core team election was recently completed. The &os; core team acts as the project's "board of directors" and is responsible for approving new src committers, resolving disputes between developers, appointing sub-committees for specific purposes (security officer, release engineering, port managers, webmaster, et cetera), and making any other administrative or policy decisions as needed. The core team has been elected by &os; developers every 2 years since 2000, and this marks our 6th democratically elected core team.
The new core team would like to thank outgoing members Kris Kennaway, Giorgos Keramidas, George V. Neville-Neil, Murray Stokely, and Peter Wemm for their service over the past two (and in some cases, many more) years.
The core team would also especially like to thank Dag-Erling Smøgrav for running the election.
The newly elected core team members are:
The returning core team members are:
The purpose of this one-day event is to gather Central European developers of today's open-source BSD systems to popularize their work and their organization, and to provide an interface for real-life communication. There are no formalities, no papers, and no registration or participation fee. However the invited developers are encouraged to give a talk on their favorite BSD-related topic or join the live forum, then have a beer with the other folks around. The goal is to motivate potential future developers and users, especially undergraduate university students to work with BSD systems.
This year's BSD-Day will be held in Budapest, Hungary at Eötvös Loránd University, Faculty of Informatics on November 20, 2010.
Technical Contact: Warner Losh
The FreeBSD kernel has been optimized over the years for a server or workstation environment. Memory is plentiful in these environments, so little attention was given to the size of the kernel. There's a number items in the kernel that can be made optional without reducing affecting the functionality needed in an embedded environment. These include things like not compiling in strings into the kernel, less aggressively inlining code, making some non-optional features optional and investigating compile time flags. This task requires identifying potentially optional kernel content and building the infrastructure to make that content optional.
requirements:
Technical Contact: Warner Losh
There's about a dozen busses in the tree now that manage resources and activate children. They are far too hard to create. We need to abstract out the basics for these buses and provide a way to allow these buses to be a subclass of this new base class.
Requirements:
Technical Contact: Warner Losh
Adding a new board to the arm code is a lot harder than it needs to be. A lot of benefit could be had by creating tables for memory ranges, etc, and having more generic initialization code. Much of this can also be Machine Independent (MI).
Requirements:
Technical Contact: Warner Losh
There's a number of SoCs that are in consumer grade routers, etc that have chips that are supported by FreeBSD, or nearly supported by FreeBSD. Pick one and bring FreeBSD up on it. Integrate it into the tree.
Requirements:
Technical Contact: Warner Losh
Superpage support exists only for x86-based architectures at the moment. MIPS hardware supports a number of different page sizes on a per page basis. This could be used to implement superpages for MIPS(which just uses two sizes: 4k chunks and 4M chunks). The vm system supports it generically for all platforms, but the pmap for each platform must complete the implementation. Fortunately, superpages were MFC'd as one huge commit into FreeBSD 7.x, so finding the relevant changes that need to be made to the mips pmap won't be starting from scratch.
Requirements:
Technical Contact: Warner Losh
Right now the kernel is built twice: once for the static modules in the kernel, and once for the dynamically loaded. There's also inconsistent dependency tracking. We should fix this. Bikeshed included, along with three colors of paint.
Requirements:
A performance evaluation of the split cache (as is) and an unified cache (like e.g. NetBSD) would be interesting. More details in this mail to the hackers mailing list. Additional improvements are welcome too.
Requirements:
FreeBSD has an implementation of the ext2fs filesystems but it contains some files under the GPL which make it undesirable, among other things, to use it in the GENERIC kernel. Ext2fs is a rather simple but practical filesystem and NetBSD has had for a while an implementation based on UFS1 sources. The NetBSD implementation needs to be analyzed regarding features and performance. If it is on par or better with our GPLed implementation, it should be ported to FreeBSD.
Requirements:
Technical contact: Ulf Lilleengen
During Google Summer of Code, Aditya Sarawgi created a GPL-free implementation of ext2fs for FreeBSD, which has now been completed. However, there are several avenues for potential further work:
The Kernel part of NTFS implementation on Mac OS X is dual licensed under both a 3-clause BSD and GPL license, and supports writing.
Technical Contact: Andriy Gapon
Technical references: M. McKusick, G. Neville-Neil, The Design and Implementation of the FreeBSD Operating System (ISBN 0-201-70245-2), UDF 2.60 OSTA Specification , Volume and File Structure for Write-Once and Rewritable Media using Non-Sequential Recording for Information Interchange
FreeBSD has an implementation of the UDF filesystem but it is quite aged and doesn't support many features from newer UDF revisions (>=2.01), it's also readonly. NetBSD implementation by Reinoud Zandijk supports most of UDF features and has write support too. Initial port could be readonly so that at least newer pre-authored media (e.g. BD) is accessible. Porting write support is a bit trickier.
Requirements:
While FreeBSD's FFS includes many advanced features, such as snapshots and soft updates, Greg Granger has proposed other strategies that would be useful, especially when working with small files. Quoting Greg Ganger: "The key insight for why current file systems perform poorly is that locality is insufficient - exploiting disk bandwidth for small data objects requires that they be placed adjacently". Explicit grouping, in particular, seems to provide important performance improvements without less implementation complexity than embedded inodes. As this changes the on-disk structure, care needs to be taken that the implementation is backwards compatible.
Requirements:
Fix MDFS lockups when using async operation modes. Revision 1.115 of md.c has a discussion of the problem.
Requirements:
Technical contact: Konstantin Belousov
FreeBSD 6.0 introduced an multi-processor scalable Virtual File System (VFS) Framework, allowing file systems to operate without the Giant lock. Many of the most commonly used file systems have been adapted to be MPSAFE (i.e., not reliant on Giant), such as UFS, the NFS client and server, UDF, cd9660, ZFS, and most recently the msdosfs (FAT) file system. However, in order to remove the compatibility shims permitting a file system to declare itself non-MPSAFE, all file systems must be converted. The list of remaining file systems includes ntfs, hpfs, coda, smbfs, and ncpfs. This project idea is to take one or more file systems and convert them to use their own locking, rather than relying on the Giant lock. In some cases, this requires also adapting underlying infrastructure: for example, smbfs relies on the netsmb RPC framework, and ncpfs on the netcp RPC framework, both of which will also require conversion.
Technical contact: Kris Kennaway
Evaluate the possibility of merging the FIFO implementation with the pipe implementation for improved performance. Care would need to be taken to avoid regressions, so part of this project should be attention to previous and existing FIFO bug reports, and writing of conformance testing to verify correct behaviour. Possible extensions might include a re-evaluation of some of the performance tradeoffs made in the pipe code in light of modern CPUs.
Technical contact: Kris Kennaway
The gettimeofday syscall is a performance bottleneck in certain applications. An approach taken by other operating systems is to export the time counter to userland via a shared page, and to update it periodically (a prototype implementation is available). For some time consumers this is sufficient resolution. Other consumers need higher resolution. On the x86 architecture the TSC timecounter can be read from userland. However depending on the hardware there may be issues with synchronization between CPUs, as well as interaction with CPU frequency changes. With care it can be used as a delta against the timestamp updated by the kernel to provide improved resolution and avoid the need for the syscall.
Technical contact: Xin LI, Howard SU
In some recent operating systems, it is common that crashes are automatically reported to its vendor, which is very helpful for finding hidden problems that can not be easily triggered by usual test cases. Newer GNOME applications also have similar functionalities.
This project would consist two parts. One is some improvements over the current savecore rc.d script to teach it how to collect necessary information (of course, automatic reporting has to be explicitly enabled by individual system administrators, and should have at least three options: not to send out anything at all as a default, send out after administrator confirmation, and automatically send all necessary information). The FreeBSD kernel in 8-current has the textdump feature which may be interesting to use for this part
Another part after the first one is finished is the server side one, which will keep a database of backtraces where similar (call stack minus addresses) reports are kept together and be considered as a "vote", to make it possible for developers and release engineers to focus on the most commonly triggered issues.
Requirements:
Technical contact: Attilio Rao, Kris Kennaway
setproctitle() calls are a serious performance bottleneck in a default pgsql configuration (they are called at least once per query, which might be thousands of times per second - I measured a performance impact of about 33% on sysbench).
One idea for avoiding the syscall (and global sysctl lock) overhead for this kind of thing would be a memory page shared between kernel and userland which libc could read/write to access things like the process title. There are potentially many other data values that could be optimized by a similar method. This is presumably a well established technique in other OSes.
This project requires mentoring/review/planning with someone with significant VM experience to make sure this approach works properly. Done incorrectly, this could result in fairly massive security holes, performance issues (perhaps not visible in simple benchmarks), etc.
Work is currently in progress on this and is undergoing review. It is not complete, but interested parties should contact Attilio before beginning work.
Requirements:
Technical contact: John Baldwin, Attilio Rao
The project would need to extend the current CPU states of absent and present to include absent, offline, and online. A set of event handlers (probably using EVENTHANDLER(9)) will need to be created to allow subsystems to be notified of CPU state changes. At a minimum, the functionality of the current 'hlt_cpus_mask' should be reimplemented in both the 4BSD and ULE schedulers using this mechanism. Note that the ability to disable all but one thread within a core must still be supported easily. (This is currently done on x86 via the machdep.hyperthreading_allowed sysctl.) Each subsystem that uses CPU_ABSENT() will need to be evaluated to see if it needs updating to handle online vs offline CPUs. Further improvements may include teaching UMA to free per-CPU buckets when a CPU is taken offline.
Requirements:
Technical contact: Attilio Rao, John Baldwin.
FreeBSD SMP scalability has improved dramatically over the last ten years as a result of the now-complete SMPng project, which converted FreeBSD from a single Giant kernel lock to fine-grained locking and strong 8-core scalability. As commodity hardware is now reaching 16-24 cores, some low-level primitives in the SMP implementation must be revisited. In particular, 32 or 64-CPU limits are encoded in the CPU mask primitive (cpumask_t), and spin locks offer higher than desired cache line contention as well as unfairness. This project would combine these two related tasks by improving the expressibility of CPU masks and per-CPU objects and improving the scalability of the spinlock implementation. It is especially important that the latter portion of the work be driven by real-world measurement using workloads that trigger the specific problem, and experimentation with primitives across several types of hardware.
Note: a spinlock backoff patch has already been proposed, but not yet properly evaluated, and would make a good starting point for this work.
Requirements:
Technical contact: Attilio Rao
umtx is the underbelly of the FreeBSD pthreads synchronization primitives, allowing kernel-assisted sleeping, wakeups, priority propagation, and other features supporting efficient mutexes and reader-writer locks in userspace. As kernel features have evolved (such as the addition of a sleepqueue primitive), hardware scalability has grown, and application-writer expectations have changed, some weaknesses have come to light. In particular, there are concerns about overhead and scalability with highly-threaded applications. This project would profile and evaluate the existing system call service provided by umtex, and consider cleanups, optimizations, and the application of newer algorithms to improve functionality and performance. The effectiveness of the rewrite would be demonstrated through careful testing with sensitive workloads, such as MySQL on 16-core systems.
Requirements:
Technical contacts: Mathieu Arnold, Brad Davis
The sysctl(8) utility retrieves kernel states and allows processes with appropriate privilege to change kernel states. On request it is able to display description lines which document the kernel state. Unfortunately not every sysctl is documented. This task is possible to share with other volunteers. mat has done some development in Perforce, in the mat_sysctl_cleanup branch.
Requirements:
Technical contacts: Alexander Leidinger, Ariff Abdullah
Requirements:
Technical contact: Stacey Son, Robert Watson
URL: Perforce repository, DTrace for FreeBSD
DTrace is a dynamic tracing facility designed by Sun Microsystems and released in Solaris 10. They have since released the major part of Solaris under the banner of OpenSolaris and the Common Development and Distribution License (CDDL) 1.0. John Birrell has created an initial port provides strong kernel functionality, but significant work remains to be done. Some ideas for projects:
Requirements:
Technical contact: Konstantin Belousov
A debug kernel is not able to show stack traces with cross exceptions anymore. This is because we do not emit any dwarf2 call frame information for any assembler code, since gdb switched to the dwarf2 format. A volunteer should annotate every assembler file [*.[sS]] with dwarf2 call frame information.
Requirements:
Technical contact: Luigi Rizzo
In early 2007 we started a project to support building Linux device drivers on FreeBSD. This was done through an in-kernel emulation layer, which implements part of the linux kernel API on top of the FreeBSD kernel API. The initial implementation was good enough to support a few USB webcam drivers, and is documented here. The code is actually available as a port, devel/linux-kmod-compat, and a popular driver that uses this infrastructure is multimedia/linux-gspca-kmod.
We would like to use a similar approach to add support for DVB devices, which are widely supported in Linux but not in FreeBSD. In particular we expect the project to provide, within the FreeBSD kernel, enough of linux compatibility to build the core components of the drivers/media/ linux kernel, and then a few device drivers including one for a PCI DVB card (e.g. saa7134-based).
Before the start of the project, a Summer of Code applicant is expected to
i) become familiar with the approach used by linux-kmod-compat;
ii) set up a proper test environment, with a couple of DVB devices
supported by linux, and a working linux installation so that
one can compare results;
iii) become familiar with the architecture of the linux code in
drivers/media.
Probably the attention should be focused in PCI devices, because
at this stage the USB stack is in a transition phase and would
pose some additional difficulties.
Expected results are a working porting infrastructure, a working linux-dvb-kmod device driver, and a working application to demonstrate that the driver is working as expected. We suggest to look at "kaffeine" for which a FreeBSD port already exists.
Technical contact: Alexander Leidinger
The ktrace(1) facility allows to monitor what running processes do. It allows to determine if a process is stuck or if it still does useful work. The goal of this item is to look at the kernel interfaces, add missing "pieces" (e.g. syscall's) to the ktrace output and to extend the output with "decoded" (translating hex/dec values into human readable information, e.g. O_RDONLY in the case of open(2)) information. Some work has been completed and committed, but a few parts still remains. More information is available here.
Also, a related project would be to modify ktrace to write to pipes. Currently the ktrace infrastructure requires the dump output go to a file. It would be useful to be able to instead have it write to pipe, or in fact any type of file descriptor.
Requirements:
Technical contact: Attilio Rao, Jeff Roberson
The instruction pair sysenter and sysexit can contribute to certain performance improvements when a syscall is made on IA32. There is however no implementation of this available for FreeBSD, so a volunteer would have to add sysenter/sysexit support to the kernel. This needs to be properly evaluated and benchmarked though, so a complete implementation should therefore also contain informative benchmarks which shows a clear improvement in performance. It is also important to stress the fact that this project is of research quality and measures should be taken to ensure that no regressions are introduced. Another interesting extension to this project would be to investigate and evaluate the possibility to use mmx/xmm registers to gather syscalls arguments. David Xu has some work in progress in his sysenter branch in the perforce repository.
Requirements:
Technical contact: Philip Paeps
WIP: http://wiki.freebsd.org/GenericInputDeviceLayer
The kernel is lacking a generic input device layer analogous to the Linux 'input core' layer. Having such a layer would make it easy to write e.g. touchscreen support (Philip Paeps has some work-in-progress regarding pointer devices and touchscreen support, but not enough time to also cover keyboard support or other generic features). This project was worked on as part of Google Summer of Code 2007, and you can find more information on the FreeBSD.org wiki
Requirements:
Improve upon / replace the existing static VESA splash screen support in FreeBSD, with a script-driven back-end, which allows animation in the loading graphics. This would greatly improve the bootup experience for desktop users, while providing graphical feedback to the startup of the kernel / system services. Additionally this could be used to replace the beastie.4th menu, with a VESA driven graphical loader screen.
Technical contact: Bruce M. Simpson
Requirements:
Technical contact: Brooks Davis
In SGI's Irix operating system, there is a concept of a job which is a collection of processes. These processes share a set of resource limits similar to those accessible through the set/getrlimit and getrusage system calls. Schedulers such as Sun Grid Engine currently implement tracking the processes that make up a job and enforcing collective limits on them in an adhoc manner. Having first class kernel support would be useful.
It seems most likely that implementing something like the Irix interface would be the most useful approach since that would enable us to leverage existing code. Implementers will need to make sure that the job construct has no negative performance implications when not enabled (ideally, processes that are not part of jobs should be unaffected) and quantify the impact on performance when enabled.
References: Irix Man Pages mentioning jid_t
Requirements:
Technical contact: Luigi Rizzo
In a 2005 GSoC project, "Pluggable Disk Schedulers", Emiliano Mennucci explored the feasibility of pluggable disk schedulers for FreeBSD. The project was successful, but we could not explore certain approaches (e.g. "anticipation", where requests are delayed hoping that some future ones can served without a seek) due to architectural limitations the kernel had at the time.
Since then, the GEOM infrastructure has become available on FreeBSD for interacting with disk I/O requests. GEOM has enabled us to work on disk schedulers in a much more flexible way, allowing a much faster development of disk scheduling algorithms. With Fabio Checconi, we have developed a prototype implementation of some anticipatory schedulers, see GEOM_SCHED.
GEOM_SCHED works within the geom layer, i.e. above the device driver where queueing of requests may actually occur. The way GEOM_SCHED does scheduling is by limiting the number of outstanding requests to the device, and the performance implications of this approach need to be measured. An alternative approach is to push the scheduler (using the same algorithms developed in GEOM_SCHED, and most likely the same code) within the device drivers. This less general (as it needs to be replicated in all drivers) but it may be an interesting thing to do e.g. for some popular device drivers such as ATA.
The proposed SoC work can address one or more of the following aspects:
References: The Pluggable Disk Schedulers SoC project, Patches
Requirements:
Implement a suspend/resume from disk mechanism. Possibly use the dump functions to dump pages to disk, then use ACPI to put the system in S4 or power-off. Resume would require changes to the loader to load the memory image directly and then begin executing again.
Requirements:
Technical contact: John Baldwin
DragonFly invested a lot of time to clean up and document it. Additionally they fixed some bugs. Interesting files in the DragonFly CVS are sys/boot/i386/bootasm.h, sys/boot/i386/bootasmdef.c, sys/boot/boot0/*, sys/boot/boot2/*, sys/boot/i386/btx/*, sys/boot/i386/cdboot/*, sys/boot/i386/libi386/amd64_tramp.S, sys/boot/i386/libi386/biosdisk.c and sys/boot/i386/loader/main.c. An interested volunteer has to compare and evaluate both implementations and port interesting/good parts.
Requirements:
Technical contact: Kris Kennaway
Currently there is no way for e.g., a port makefile to tell whether things like FreeBSD 5.x compatibility are present on the system (just installing the compat5x port is not enough, you need a kernel built with COMPAT_FREEBSD5). All such optional kernel features need to register themselves with the FEATURE macro so that the userland can easily query whether a given feature is present. So far not all kernel features are using this infrastructure.
There needs also to be a way to spoof those values, e.g., when the ports build cluster is building for older FreeBSD versions in a jail. Suport for this is not available in the FEATURE macro.
Requirements:
Technical contact: Rui Paulo
Finish EFI support on the i386/amd64 ports. Final work should be able to boot a FreeBSD kernel into single user mode, at least.
Requirements:
Technical contact: Ulf Lilleengen
URL's: csup homepage, CVSweb
csup is a port of the cvsup high-speed CVS repository replication application from the original Modula-3 to the C language. It is now distributed with FreeBSD, but is missing some important features that would make useful projects to work on:
Requirements:
Technical contact: Robert Watson, George V. Neville-Neil
Design and implement a wire level regression test suite to exercise various states in the TCP/IP protocol suite. Ideally with both IPv4 and IPv6 support.
Requirements:
Technical contact: Andre Opperman.
Listens on an interface and tracks all TCP sessions it sees. In the normal case only general information is carried forward (seq#/ack#, negotiated SYN/ACK features, etc). Whenever an anomaly happens - that is a duplicate ACK, SACK response, out-of-order segment, retransmission or others; it captures those packets into a tcpdump file for later deep inspection with Wireshark or other tools. This tool is to be deployed on live hosts and passive monitors to collect reliable condensed data about real-world behavior of TCP on the global Internet. Currently no such quantitative data exist and contribution of such a tool that can be easily run is a significant step in helping further development of TCP algorithms.
Difficulty: Medium, good familiarity with the TCP RFCs is necessary and detection of many edge cases has to be implemented correctly.
Technical contact: Sam Leffler, Rui Paulo
WPA2 is the authentication protocol defined as part of the IEEE 802.11i specification. This protocol is now commonly used to authenticate wireless stations to access points. Part of this protocol is the ability to pre-authenticate a station with one or more access points so that roaming can happen quickly. FreeBSD lacks support for this aspect of the protocol in the hostapd program used to construct a WPA-enabled access point. This task would port the Linux code that exists to support pre-authentication in hostapd. This mostly involves rewriting some user-mode multicast code and testing the result.
Requirements:
Technical contact: Sam Leffler,Rui Paulo
Build a "packet fuzzer" tool that can be used to build test suites to improve reliability of the 802.11 code against garbage data. There are various tools out but we're not aware of any good ones that work with 802.11 and are generally available. The basic idea is to write a packet injector/playback tool that's driven by a scripting language. Then you need to build up a database of test cases. It's also possibly important to do time-based playback.
Requirements:
SCPS is a protocol suite designed to allow communication over challenging environments. Originally developed jointly by NASA and DoD's USSPACECOM, these protocols are used for commercial, educational, and military environments. A student project in this area would involve implementing various network protocols according to specification (SCPS File Protocol, similar to FTP; SCPS-Transport Protocol, based on TCP; and others.)
Technical contact: Rui Paulo
Implement TCP UTO (User Timeout Option) as defined by RFC 5482.
Requirements:
Technical contact: Robert Watson
The Berkeley Packet Filter (BPF) allows packet capture filters to be compiled into a bytecode that is either interpreted by a kernel virtual machine, or compiled into native machine code via a JIT and executed in in-kernel. Historically, the origin of packets has been the network interface, with each (synthetic) BPF device attached to exactly one NIC as requested by the application (for example, tcpdump). However, network interfaces have become significantly more complicated, and BPF has had to grow to support new features, such as Data Link Types (DLTs), in which BPF devices can tap network processing at different layers. This task would involve teaching BPF about a further dimension in network interface complexity: multiple input and output queues.
Modern 10gbps, and even 1gbps, network cards support multiple queues for various reasons: at first quality of service (QoS) differentiation in processing, but now especially to improve parallelism in network processing by distributing work to many CPUs on input. This same technica can also accelerate packet capture, but BPF currently doesn't know this. In this project, BPF would be enhanced to be aware of individual input and output queues on a NIC, which means providing network stack abstractions for these concepts, visible today only within device drivers. Userspace threads might then use enhanced ioctl(2)s to query the set of available queues and attach to one or more. Applications seeking maximum parallelism could open (n) devices, attaching each to a queue, and executing with appropriate CPU affinity. Ideally this would involve neither lock nor cache line contention throughout the entire stack, from device driver to userspace delivery.
Requirements:
Technical contact: Rui Paulo
The FreeBSD 802.11 layer, net80211, implements a broad range of advanced functionality: station, access point, virtual access point, mesh networking, authentication/encryption, and more. This task implements a net80211 driver simulating a wireless radio in order to allow protocol-layer testing without the hassle of real live hardware limitations: interference, signal propagation, and regulatory domains. The driver should let the user specify a number of radios to be simulated, with an eye towards future work to simulate physical properties.
A useful reference for this work would be the notionally similar mac80211_hwsim driver in Linux (http://linuxwireless.org/en/users/Drivers/mac80211_hwsim), which might provide a model in some areas of the work, and has similarly been used on that platform to test hostapd/wpa_supplication.
Requirements:
The pkg_* tools, which deal with the installation of pre-build binary package of ports, could do with a code cleanup or maybe even a rewrite from scratch. Some features of the ports tree are not supported by the pkg_* tools, e.g. versioned dependencies.
Requirements:
Technical contact: Brooks Davis
When bootstrapping systems it would be useful to be able to create a single package file that contains one or more packages and all the required dependent packages. This is conceptually similar to, but different from PC-BSD's PBI package format. PBI's contain a private copy of all dependencies, fat packages would contain each individual package and once installed it would be as though each package was individually installed in the usual manner.
This project would consist of additions to the pkg_tools to support creation and installation of a new package file format and to ports to build these packages.
Requirements:
A number of tasks of the Ports system can take quite a long time to perform. Some are well known for traversing each and every port in the tree, e.g. building INDEX, which does not scale well with the number of ports being added to the collection.
Detailed profiling of the most used commands in the ports framework needs to be done to analyze the bottlenecks and identify possible solutions. Refactoring of the main file bsd.ports.mk is also necessary to improve code readability, but not at the cost of performance.
Requirements:
Technical contact: Robert Watson
A number of kernel security subsystems, such as IPFW and pf, generate security log data. This task involves identifying potential sources of security event information in the kernel and modifying kernel subsystems to log that information using the kernel security event auditing system. User and programmer documentation of audit may be found on the TrustedBSD Documentation Page. There are also extensive manual pages relating to audit in FreeBSD. This project will require careful security analysis and kernel programming, and will likely need some re-working of the kernel audit framework (which is currently entirely focused on gathering user and kernel system call audit data).
Requirements:
Technical contact: Robert Watson, Simon Nielsen
WIP: http://wiki.freebsd.org/DistributedAuditDaemon
Create a tool to securely and reliably ship log files to remote hosts. The main focus is to manage per-machine audit records and submit them to a central site for processing and long-term archiving/management. Ideally with support for SSL (or the like) so they do not travel on the wire in the clear.
Requirements:
Technical contact: Robert Watson
FreeBSD 5.0 was the first FreeBSD release to ship with support for Mandatory Access Control (MAC), an access control technology allowing system administrators to implement multi-level security, integrity protection, and other "mandatory" policies. Policies may be compiled into the kernel, or loaded as loadable kernel modules. Later revisions of FreeBSD and the MAC Framework enhanced MAC support, and additional policy modules were made available, such as a port of the SELinux FLASK/TE framework available as a third party policy module. However, many of the sample MAC modules included with FreeBSD are considered experimental examples of what the technology can be used for, rather than production policies. For example, the Biba integrity policy can be deployed in production, but requires significant tuning to do so effectively.
This task involves a general review of the MAC Framework and Policy modules, with the goal of identifying improvement areas. It also involves specific cleanups, optimizations, and completeness work on specific policy modules -- most importantly, the Biba and MLS sample labeled policy modules. Work there includes improving memory overhead and efficiency; for example, moving from allocating complete labels for every labeled object to referencing common label storage where labels are identical, which occurs a great deal of the time in most systems. Other cleanups include moving towards a canonical/extensible on-disk label storage format, adding regression tests, investigating interactions with user applications, and writing documentation.
Requirements:
Technical contact: Robert Watson, TrustedBSD discuss mailing list
The TrustedBSD MAC Framework makes it easy to extend the FreeBSD kernel security model using pluggable modules, which has provided support for "traditional" mandatory access control, as well as allowed a number of companies to build local security policy extensions for workstation, server, and appliance products; one such example is Apple's "Seatbelt" policy for Mac OS X. However, the base system MAC policies are difficult to use, primarily because they either don't extend security meta-data (ugidfw) or they require extensive labeling (Biba, MLS). This project idea involves crafting a new OS security policy extension along the lines of ugidfw, the file system firewall, but using path names instead of existing file properties to manage protection. This requires careful thinking about what a file system path "is" in the UNIX environment, as well as regarding what sorts of policies would be useful.
Requirements:
Technical contact: Robert Watson
FreeBSD is undergoing constant and active improvement to all of its critical subsystems, from file systems to the network stack. With any change, there is a risk of introducing bugs or regressions. The goal of this task is to produce a security regression test suite, which encapsulates requirements regarding system security properties and tests that they (still) hold. Areas to test include file system access control, privilege, authentication, cryptography, process containment, and more. There are some current tests along these lines in the FreeBSD -regression test tree, but they are both incomplete and and inadequate. +regression test tree, but they are both incomplete and inadequate. New tests must be created; existing tests must be completed and updated.
Requirements:
Technical contact: Robert Watson, TrustedBSD audit mailing list
The current OpenBSM audit parsing API has a number of limitations, not least that it can't handle little endian BSM records that may come from Solaris x86 systems, in terms of ABI robustness in the presence of new record types, ability to process trails generated non-locally in terms of supporting uid/gid->name translation, and in terms of incrementally processing a byte stream from, for example, socket sources without using the C FILE API.
This task would consider existing audit parsing APIs in the industry, including POSIX.1e, relevant Open Group specs, and in-use APIs on other systems such as Solaris, Linux, Windows NT, and others, in order to first identify an existing candidate API or design a new candidate API, then implement the API and adapt existing audit applications to use it. The task would also document the API using man pages, create an audit parsing tutorial document, create a test suites, and require interaction with the OpenBSM and FreeBSD communities to identify audit parsing requirements.
If successful, the results of this work would be integrated into OpenBSM, the open source BSD-licensed audit framework shipped with FreeBSD and Mac OS X.
Requirements:
Technical contact: Robert Watson, TrustedBSD audit mailing list
The BSM (Basic Security Framework) audit trail format is the de facto industry standard for portable operating system audit trails, being supported on Solaris, FreeBSD, and Mac OS X. However, many other audit trail formats exist that are less portable, including audit trail formats local to Windows NT and Linux.
This task would create BSD-licensed conversion tools to import audit trails from other systems and convert them to BSM format so that they can be inspected and managed using the OpenBSM tool set. This would require the creation of BSD-licensed parsers for audit trail formats of interest, designing and documenting a semantic mapping to the BSM trail format, and writing conversion utilities using the new parsers, semantic mapping, and BSM generation routines in OpenBSM. A key part of this work would be to rigorously understand and document the mapping and its limitations (for example, perhaps some Windows NT concepts can't be represented in BSM in a natural way, such as larger SID than UID fields, which is important information for anyone using BSM trails from NT systems). A test suite is also required.
If successful, the results of this work would be integrated into OpenBSM, the open source BSD-licensed audit framework shipped with FreeBSD and Mac OS X.
Requirements:
Technical contact: Robert Watson, TrustedBSD audit mailing list
Currently, FreeBSD's audit trail support is a system-level facility, intended to track system-level security events. However, as the affordability of computers has gone up, they are increasingly single-application systems, and applications are increasingly playing a role in the Trusted Code Base (TCB) of larger integrated computing systems. As such, it is desirable to support reliable and secure auditing of application events that do not correspond to the traditional TCB components, such as events from databases, web servers, or virtual machines. However, these events need to be handled separately, as they have quite different security and selection requirements.
This task would extend the FreeBSD kernel's audit framework to allow multiple concurrent audit trails to be maintained, reflecting different "slices" of the system. The base slice would be the existing system audit trail, submitted to by the kernel itself, as well as using the audit(2) system call, and stored in /var/audit. New slices would be represented by named special devices in /dev, with ownership and permissions reflecting their desired application security properties; each slice would have its own audit worker and queue, with its own pipe reliability properties and target trail files. This would require extending OpenBSM's auditd and libbsm to support multiple trails as well. Finally, two or more sample applications would be modified to submit application-level auditing, such as the authentication routines in Apache and PostgreSQL. This design would have several interesting properties, not least that audit trails submitted by applications would be protected from application tampering after submission, as the kernel would mediate access using the TCB. Analysis of potential designs, security issues, documentation, and a test suite are required.
If successful, the results of this work would be integrated into the FreeBSD kernel and OpenBSM.
Requirements
Technical contact: Robert Watson
FreeBSD uses a largely standard UNIX privilege model, granting special OS rights to processes with an effective uid of 0. Exceptional behavior lies in securelevel (restricting root based on a global or per-jail level), Jail (limiting certain rights to ensure safe containment for a jailed root user), and the ability for the MAC Framework to augment privilege decisions. However, as of FreeBSD 7.0, privilege decisions internal to the kernel have been made with respect to a large set of named privileges (priv.h). This project idea is to create a new privilege model, similar perhaps to POSIX.1e or the Solaris privilege model, allowing masks of privileges to be associated with process credentials and managed with system calls. Of particular interest is supporting the assignment of privileges to users or roles -- such as the right to bind low port numbers, use DTrace, etc.
Technical contact: Joseph Koshy, Kai Wang
Create BSD-licensed versions of ELF processing tools (e.g., ld, dbx, as and others) using the ELF(3) and GELF(3) API set. Identify overlapping functions in those tools and create a library out of the common functions. Identify parts which can be generated by tools (e.g., machine code parser generators) to support our Tier-1 and Tier-2 architectures.
References:
Requirements:
Technical contact: Diomidis Spinellis, Gábor Kövesdán
Create/port BSD-licensed versions of one or more of the text processing tools that are currently missing from the FreeBSD distribution: sort, diff, groff/troff and the grep family. Licensed versions of some or all of these tools are already included in OpenBSD, so this task involves more porting and feature completion than development from scratch. Emphasis should be placed on performance, standards-compliance, and support for handling wide character sets.
Regarding groff/troff, there exist the OpenSolaris versions at SourceForge which at least do not come with a viral license like the current GNU versions we use. Additionally this implementation has support for common vector fonts and unicode. If those utilities are option-compatible or not has to be analyzed. A port of this is already available as textproc/heirloom-doctools. Alternatively, OpenBSD's mdocml can replace groff's mdoc functionality and may be sufficient for our purposes.
Requirements:
Technical contact: Alexander Leidinger, Gardner Bell
The new "delete-old" and "delete-old-libs" target in /usr/src for 6.1 and -CURRENT should be extended to support the WITHOUT_* knobs, e.g. WITHOUT_RESCUE or WITHOUT_CRYPT, and delete files which are covered by those knobs. Some switches have already been covered. You can view a list of all switches and what effect they have here.
Requirements:
Technical contact: Colin Percival
The freebsd-update(8) utility is used to fetch, install, and rollback binary updates to the FreeBSD base system. A nice project would be to develop a graphical front-end for freebsd-update(8), using the QT toolkit. A GTK frontend was developed as part of GSoC 2007 and exists at berlios; the QT frontend could maybe share common functions/classes and design ideas.
Requirements:
Technical contact: George V. Neville-Neil
Many userland network utilities do not work correctly with IPv6.
This project could also include a broader survey of other network services in /usr/bin and /usr/sbin to make sure they're all IPv6 clean.
OpenBSD has some improvements to lint(1) which may be beneficial to have.
Requirements:
Technical contact: Robert Watson
Create, similar to libmemstat, wrapper libraries to support monitoring and management applications to avoid direct use of kvm. Three parts to the project: for each of the above, add kernel support to export data in a less ABI-sensitive way using sysctl, write a library to present the information in an extensible way to applications, and update applications to use the library instead of reaching directly into kernel memory / consuming sysctls. The goal is to allow the kernel implementation to change without breaking applications and requiring them to be recompiled, and to allow monitoring functions to be extended without breaking applications. This should also facilitate writing new classes of monitoring and profiling tools.
Requirements:
URL: The NDMP Initiative
The NDMP initiative was launched to create an open standard protocol for network-based backup for network-attached storage. Major commercial storage systems come with a compliant service. This allows major commercial backup systems to backup such NAS devices. Including a NDMP disk server into FreeBSD would allow to play nice out of the box (modulo some configuring) regarding backups in a corporate environment.
Requirements:
The OpenBSD prebind is a secure implementation of prelinking that is compatible with address space randomization. Prelinking allows to speed up application startup when a lot of libraries are involved. This should show a noticeable effect with e.g. GNOME/KDE.
Requirements:
A proxy auto-config (PAC) file contains a JavaScript function "FindProxyForURL(url, host)" that determines which HTTP or SOCKS proxy, if any, to use to access a given URL. In most application the file may be specified manually or discovered using the Web Proxy Autodiscovery Protocol. Support for PAC files in libfetch would make fetch more versatile.
Supporting PAC files nominally requires a fairly complete JavaScript implementation. There appear to be no BSD Licensed JavaScript implementations so one will likely need to be written. A minimalist implementation of the language with commonly used constructs such as if/else, string comparison, and functions would be sufficient in many cases.
References:
Requirements:
It would be great to have a bundled PXE installer. This would allow one to boot an install server from a FreeSBIE live CD-ROM on one box, set the BIOS on subsequent boxes to PXE boot, and then have the rest happen by magic. This would be very helpful for installing cluster nodes, etc.
Markus Boelter is working on a bundled PXE installer as part of his BSDInstaller project within the Google Summer of Code 2006. The PXE Installer is working but some non-PXE related issues have to be solved before it can enter the tree.
Requirements:
Technical contact: Alexander Leidinger, Nik Clayton
Nik has written a regression test infrastructure using Perl. More of the regression tests should be made to work with libtap.
Porting LTP might also be a good idea.
Requirements:
Requirements:
Technical contact: Colin Percival, Tim Kientzle
Instead of installing using install, mkdir, mtree, etc, directly construct a tarball. This would allow creating install distributions without root access, as setuid etc would never hit the local disk. This would require some retrofitting of our installation mechanisms.
Bsdtar now (8-current, 20080101) has a feature that allows it to create a tar archive from a description provided in the form of an mtree file. This description can specify owner, permissions, and contents for each entry and does not require the files on disk to already have correct ownership. This should make it possible to build a FreeBSD distribution as a non-root user. Talk to Tim Kientzle for details of the new bsdtar features and look at NetBSD, which has a similar facility, for ideas about how to proceed.
Requirements:
Technical info:: on J.R. Oldroyd's Unicode Support on FreeBSD page
Many base system utilities grew multibyte support in 2004. It would be nice to continue this trend by teaching vi(1) to display and edit documents in UTF-8 encoding. The above referenced page contains info of what is needed to improve the Unicode support in vi(1).
Requirements:
Currently, cron(8) and atrun(8) are outdated in their implementation. Here are some directions for improvement:
Requirements:
Technical contact: Kris Moore
In 2008, Tijl Coosemans did excellent work getting wine to a very usable point on FreeBSD. However, there are some issues which still need to be addressed at http://wiki.freebsd.org/Wine. This would be a big improvement for our desktop users.
Technical contact: Shteryana Shopova
BSNMP is a portable SNMP framework consisting of a daemon, modules and tools. It includes libraries that ease the development of loadable modules for configuring and monitoring various subsystems. You can find more information about BSNMP on the wiki. Some project ideas, but not limited to that list, can be found here..
Requirements:
Technical contact: Jeff Roberson
This task would create an extension to the threading library that would allow application developers to measure and locate lock ordering and lock contention problems within their application. Such a tool is invaluable in debugging application deadlocks and creating high-performance multithreaded software. Existing lock ordering and profiling tools exist in the FreeBSD kernel, and could be used as the model for the userspace implementation. We would recommend beginning with profiling due to its immediate usefulness in optimizing performance, and to allow improvements in kernel scheduling to better manage user application lock contention.
Requirements:
Technical contact: Joe Marcus Clarke
This task is to port NetworkManager to FreeBSD. Porting NetworkManager will also require some core userland changes to FreeBSD, especially to ifconfig.
Requirements:
Technical contact: Joe Marcus Clarke
This task is to fix FreeBSD support in sysutils/system-tools-backends.
Requirements:
Technical contact: Joe Marcus Clarke
This task is to add hal support for some additional subsystems. In particular FreeBSD is lacking support for the ieee1394 (i.e. Firewire), bluetooth, and printer. Adding support for these subsystems will require changes to the FreeBSD kernel. Those interested should use the latest HAL Specification as a guide.
Requirements:
Technical contact: Konstantin Belousov
The FreeBSD binutils and gcc on amd64 are capable to build 32bit binaries. What we miss is the proper includes for i386 accessed when #include <machine/some.h> is used, for the -m32 invocation of the cc. Project would provide an installation of both amd64 and i386 includes in non-conflicting way, and changes to the gcc to select proper location for machine/ based on the compilation target architecture.
Requirements:
Technical contact: Bjoern Zeeb, Warner Losh, Robert Watson
FreeBSD's jail(8) facility provides lightweight operating system virtualization; with the addition of experimental virtual network stacks, ISPs are able to efficiently host thousands of FreeBSD instances on a single machine. This task improves FreeBSD's userspace management of jails by introducing jailinit(8), an enhancement to the init(8) utility responsible for starting up and shutting down the operating system. Currently, jails are started by executing their /etc/rc script directly using jail(8); in the proposed new world order, init(8) would have a jail start mode, and would continue running throughout the lifetime of the jail to manage services and eventually shut down the jail. This might require minor kernel changes in order to let jailinit perform some of the same functions for jails as it does for normal userspace. jailinit could also be responsible for setting up and managing console logging, launching tty-attached services such as getty within a jail, and other jail properties.
Requirements
The FreeBSD Project is looking forward to participating as a mentoring organization in Google Summer of Code 2011. This program funds students to contribute to an open source project over the summer break. We have had over 100 successful students working on FreeBSD as part of this program in 2005, 2006, 2007, 2008, 2009, and 2010.
This page will be updated throughout the GSoC 2011 application period to include new information, such as project ideas, proposal information, and potential mentor contact information. If you don't see an idea that interests you, visit again in a couple of days!
Google Summer of Code is an exciting opportunity for students to "intern" with an open source project for a summer. The FreeBSD Project, as one of the most successful and oldest open source projects, is an excellent place to do this internship. Founded in 1993, the project now consists of several hundred "committers" and tens of thousands of contributors. FreeBSD is the foundation for many commercial products, including Apple's Mac OS X, NetApp's OnTap/GX, Juniper's JunOS, as well countless other products, and is widely used in the Internet Service Provider and corporate IT worlds. Many of these sponsors participate daily in the FreeBSD community, and students have the opportunity to develop software ideas in an exciting environment with many real world applications, and under the mentorship of experienced developers.
After the summer ends, many of our students are sponsored by Google or the FreeBSD Foundation to attend operating systems and open source conferences to present on their work, and a significant number go on to become FreeBSD developers. It's also a great job networking opportunity!
For a complete list of student projects from previous years, visit:
See also our wiki pages for student projects [2008, 2007, 2006, and 2005].
The FreeBSD Project maintains an Ideas Page on our wiki. Projects specifically tagged as "GSoC" are believed to be sized for a useful summer hacking, and have technical contacts who can help answer questions as you write your proposal. Ideas not marked as GSoC-friendly are also fine to propose, but may be scoped larger or smaller than a summer, or might not have such a clear mentor -- we suggest e-mailing our soc-admins alias for help if you do decide to propose one of them. You are, of course, welcome to propose your own ideas, and if the proposal is strong, we'll try to match you with a mentor!
For additional ideas about upcoming development projects in FreeBSD, take a look at recent Developer Status Reports.
Students are responsible for writing a proposal and submitting it to Google before the application deadline. The following outline was adapted from the Perl Foundation. The objective of the proposal is to identify what is to be done, explain why this needs to be done, and convince us that:
A strong proposal will include (at least):
Name
Phone
IM/IRC
Availability
How many hours per week will you spend working on this? How many on other things? What other obligations (work, school, vacation, - weddings, etc) do you have this summer? Be as specific as possible: - when will the project begin and and? You should be ready to produce + weddings, etc.) do you have this summer? Be as specific as possible: + when will the project begin and end? You should be ready to produce a day by day schedule before the program starts.)
Please note: participating in Google Summer of Code is a significant time commitment, and you should not apply if you already have another full-time job planned for the summer.
Bio
Who are you? What skills do you bring to this project? What is your past involvement with The FreeBSD Project? (Past involvement is not required, but ideally you will have at least installed FreeBSD and perhaps fixed a bug or two) If your project includes programming in a particular language, such as C, or in a specific environment, such as the kernel or an embedded platform, what experience do you have working in that area? Are you familiar with or a user of revision control systems? Have you completed courses that will be relevant to your project idea? What do you think you will need to learn to complete this project?
Possible Mentor
Optional, but highly recommended. Do not put a name here if you have not contacted them.
Project Title
In forty characters or less, what you propose to do.
Project Description
A few paragraphs describing your project. Direct copies from the ideas page will be rejected - proposals should reveal that you have done some research into the problem and its solutions. Include both what you will be doing and why it is a good thing for The FreeBSD Project.
Deliverables
A list quantifiable results and related code milestones. We suggest at least two milestones before the mid-term evaluations and two after. Where appropriate, this schedule should include multiple committable or releasable points so people can benefit from and/or test your work as early as possible.
Test Plan
What parts of your code need testing and how do you plan to test them? This might include both functionality and performance tests.
Project Schedule
How long will the project take? When can you begin work?
A number of FreeBSD committers are willing to mentor students. A good place to start is the 'Technical contacts' listed with the example projects on the ideas page.
In previous years, the FreeBSD Project provided access to the FreeBSD Perforce revision control infrastructure in order to facilitate student collaboration, provide public access and archiving for the on-going student projects, and to help mentors and the community monitor on-going work. It is expected that students participating in future programs will be offered the same facilities. Students will also be asked to maintain wiki pages on their on-going projects. In the past, e-mail, IRC, and instant messaging have proven popular among students and mentors, and students participating in the FreeBSD summer program are encouraged to use these and other electronic communication mechanisms to become active in the community.
When are proposals due, and how do I submit mine?
At the time of writing, Google has announced the following dates of interest relating to the application process:
18 March - Google to announce what open source organizations will participate in Google Summer of Code 2011.
28 March - Student application period opens.
8 April - Student application period closes.
22 April - Organizations finish reviewing applications an mentors registered.
25 April - Accepted students announced.
Note that these dates may change, and the Google FAQ timeline is the authoritative source of detailed schedule information:
All students must register with, and submit applications via, the Google Summer of Code home page:
What advice do you have for a student who might want to submit a proposal?
Experience suggests that the strongest proposals come from students who contact FreeBSD developers and potential mentors well in advance of submitting their proposal, seek feedback on their proposal ideas, and write proposals that reflect time spent exploring and understanding the problem area to be addressed. Even if the FreeBSD developer(s) you contact aren't the eventual mentor of the project, their feedback can be invaluable.
Can I submit multiple project proposals to the FreeBSD Project?
Yes, but do make sure you invest adequate time in each proposal. We are not able to accept more than one project per student, so you may do better spending more time on one or two detailed proposals than by submitting lots of less-detailed ones.
Will the FreeBSD Project accept more than one student for the same idea?
In general, we will accept only one student for any given proposal idea, as most proposal ideas in our ideas list are sized with a single student summer project in mind. This is a good reason to consider coming up with your own idea, or at least, making sure that your proposal for one of our project ideas reflects your unique contribution and viewpoint. If you plan to submit multiple proposals, you might consider doing one with an idea from the list, and another with an original idea.
What if my proposal is not selected in the application process? Can I still participate?
We always have more good applications than student places, but that doesn't mean you can't do the project anyway. The FreeBSD Project always welcomes new volunteers to work on projects, and is generally happy to provide mentoring and support for students whose proposals could not be selected in order to allow them to work on their project anyway. You will need to work with the FreeBSD Project GSoC administrators to identify a possible mentor. However, Google will not fund that participation.
What projects were completed successfully by students in previous summers?
Please see the 2010 FreeBSD Summer of Code page, as well as older project pages from 2009, 2008, 2007, 2006, and 2005 for a list of the completed projects from previous years.
How can I learn more about FreeBSD?
The FreeBSD Project Home Page is the best way to learn more about the project -- from there you can reach the FreeBSD Handbook, FreeBSD Developer's Handbook, project mailing list archives, regular project status reports, and more. If you have questions about specific project ideas, e-mail the technical contacts for those ideas. If you have general GSoC questions relating to FreeBSD, such as if you are unable to reach a project technical contact, need help finding documentation, or want to know who might be a good person to talk to about your idea, send them to soc-admins@FreeBSD.org.
Is there an IRC channel I can join to talk about proposal ideas or get help finding out more?
You can join #freebsd-soc on the efnet IRC network to chat with FreeBSD developers interested in mentoring student proposals and projects, past FreeBSD/GSoC students, and other students applying to FreeBSD/GSoC this year.
Please help us advertise Google Summer of Code with FreeBSD at your local university or college campus! You can forward around our e-mail announcement to department and club mailing lists, and to department secretaries to distribute. You can also print out and post copies of the FreeBSD GSoC 2011 poster.