With the release of FreeBSD 5.5 and FreeBSD 6.1, the second quarter of 2006 has been productive. Google is sponsoring 14 students to work on FreeBSD as part of their Summer of Code Program (most of which already submitted a report for elaboration on their projects).
Sun's open-source software is starting to make its way into FreeBSD as a port of DTrace is nearing completion and a port to the UltraSparc T1 processor (which gives a great push to the ongoing SMP efforts). Having a powerful debugging tool combined with a CPU that can run up to 32 concurrent threads helps to identify scalability issues.
BSDCan 2006 was yet again a smashing success and much was covered in the 2-day developer summit. As a product of the conference, a new focus on FreeBSD for the embedded sector has started. Various ARM boards are targeted, a MIPS32 port is gearing up and people are looking for other interesting platforms to port FreeBSD to. Preparation for the EuroBSDCon (in Milan, Italy) on November has already issued a call for papers.
In addition, a lot of spring cleaning is taking place in the network stack. After conclusion of the KAME project, IPv6 code integration has been refocused and a fully locked port of SCTP is in the final stage of integration. Of course, all this goes without noting all the progress made with the other network projects.
Please read below for more detailed news on the projects that happened in FreeBSD during the last three months. If you are interested in helping, consider the "Open Tasks lists" provided with some reports. In addition we would like to point you at the list of projects and ideas for volunteers and hope to receive a status report from you next time.
Thanks to all reporters for your excellent work and timing! Enjoy reading.
BSDCan 2006 continues to impress. Again this year, we had a good collection of talks from a wide range of speakers. In all, we had over 200 people from 14 different countries.
Our sponsorship pool continues to grow. This year we had sponsorship from:
The 2007 planning has already begun and we look forward to another popular and successful event.
My thanks to the 2006 program committee, the speakers, the volunteers, the sponsors, and, of course, the attendees.
See you at BSDCan 2007.
The release engineering (RE) team announced the availability of FreeBSD 5.5 and 6.1, both in May 2006. FreeBSD 5.5 is the last planned release from the RELENG_5 branch in CVS. For the most part, its main features consist of bugfixes, security patches, and minor updates. We encourage users to move towards the 6.x series of releases whenever practical. FreeBSD 6.1 is the latest of the releases to come from the RELENG_6 branch in CVS. It includes (among many other things) improved support for WiFi devices, additional network and disk controller drivers, and a number of fixes for filesystem stability. The next release to be issued from this branch will be FreeBSD 6.2, which is currently scheduled for September 2006.
The RE team is currently in a ``between releases'' mode. Current activities include working with security-team@ on some errata fixes for the RELENG_6_1 branch and producing snapshots of HEAD and RELENG_6 at the start of each month.
Several personnel changes have taken place recently. Scott Long has stepped down from his position on the RE team; we thank him for his considerable efforts over the past four years. In his place, Ken Smith has taken over the role of lead release engineer. Bruce A. Mah has rejoined the RE team after a two-year sabbatical.
For some time now I have been working on converting the existing USB device drivers to my new and mutex enabled USB API. I have converted "ulpt", "ums", "uhid", "ukbd", "ugen", "uaudio", and a few others. Around 10 USB device drivers are left to convert. Most of these are network device drivers.
At the present moment I am working on getting scatter and gathering support working for all USB host controllers. Scatter and gathering means that one allocates PAGE_SIZE bytes of memory at a time, and then fills these memory blocks up as much as possible with USB host controller structures and buffers. This should solve problems allocating DMA-able memory when the system memory becomes fragmented.
Some FPU system and kernel memcpy/copyin/copyout changes have been performed. In particular, a per-CPU save area has been introduced (protected with an interlock) in order to assure a stable saving mechanism. copyout/copyin have changed in order to use vectorised version of memcpy and an xmm version of memcpy has been provided.
As part of my SoC 2006 project I am working on implementing a BRIDGE monitoring module for FreeBSD's BSNMP daemon. Initial prototyping is done and some kernel changes are coming to be able to access all needed data. In addition to IETF RFC 4188, which was designed for monitoring a single bridge, this snmp module will support monitoring of multiple bridge devices as supported by FreeBSD.
Anonymous enablings now work. There is a new option in the boot loader menu to load the DTrace modules and trace the kernel boot process.
Sun Microsystems has been very supportive of the FreeBSD port and has generously provided a Sun Fire T2000 server to allow Kip Macy's sun4v port to be merged into the DTrace project tree.
The DTrace project tree sources are now exported to cvsup10.freebsd.org
Refer to the project page for more details.
There are several projects moving forward in the embedded area. For now the main location for new information is www.embeddedfreebsd.org. We have also created a new mailing list, freebsd-embedded@freebsd.org , which is meant to eventually replace the freebsd-small. A call was put out on small for people to move to embedded.
This year's EuroBSDCon will be held in Milan, Italy, on November 10th through 12th.
Hosted in the foggy northern Italy, the fifth EuroBSDCon aims at being a new successful chapter in the itinerant series of European BSD conferences.
EuroBSDCon represents the biggest gathering for BSD developers from the old continent, as well as users and passionates from around the World. It is also a chance to share experiences, know-how, and cultures.
For the first time, parallel to the main event, an event for wives/girlfriends/friends will be organised. It will consist of guided tours of the city of Milan, a probable trip to Como and visits to various museums. We're also working towards offering a show at the Teatro alla Scala.
The FreeBSD developer summit will be also held on November 10th.
Continuing to add IPv6 support to FAST_IPSEC. Test environment is now stable. Can build and run kernels with FAST_IPSEC and INET6 enabled but IPSec in IPv6 is now broken and being worked on.
FreshPorts has seen several new features recently:
A 2U server was recently donated to the FreshPorts / FreshSource / FreeBSD Diary / BSDCan group. We have also received a RAID card. Now we're looking for some hard drives.
Over the past few weeks, work has concentrated on benchmarking the new server and getting it ready for production. Eventually it will need a new home as I don't really want it running in my basement all the time (it's really loud!).
Thanks to iXsystems and 3Ware for their contributions to this project.
GJournal is a GEOM class which provides journaling for GEOM providers. It can also be used to journal various file systems with just a minimal filesystem-specific portion of code. Currently only UFS journaling is implemented on top of gjournal. Being filesystem-independent and operating below the file system level, gjournal has no way to distinguish data from metadata, thus it journals both. One of the nice things about gjournal is that it works reliable even on disks with enabled write cache, which is often not the case for journalled file systems. And remember... fsck no more.
The purpose of gvirstor module is to provide the ability to create a virtual storage device of arbitrarily large size (typically several terabytes) which consists of an arbitrary number of physical storage devices (actually any lower-level GEOM providers, including RAID devices) of arbitrary size (typically 50 GB - 400 GB hard drives). Storage space from these components is carved into small chunks (for example 4 MB) and allocated (committed) to the virtual device on as-needed basis.
Development has started and is progressing as planned (though a little bit slow). Metadata format and virtual storage allocation formats have been defined and more serious coding is in progress.
The FreeBSD list of projects and ideas for volunteers is doing well. Several items were picked up by volunteers and have found their way into the tree. Others are under review or in progress. We are looking forward to hear about new ideas, people willing to act as technical contacts for generic topics such as USB or specific entries (already existing or newly created) and suggestions for existing entries or completion reports for (parts of) an entry.
Initial changes include:
Clement has been working both with libnet and gnn's Python based packet library (PCS) to produce code to test for vulnerabilities in IPv6. To Clement has found some issues, all of which have been reported to his mentor and to Security Officer at FreeBSD.org Vulnerabilities will not be reported here.
Project is in development with initial working software expected mid-July 2006. CPU limits will be implemented with a hierarchical scheduler: (initially) using a round-robin scheduler to select which jail to run a task in and then delegating which task in the jail to be run to a per-jail scheduler.
A simple lexer and parser have almost been completed. Also significant planing for future additions to K have been thought up.
FreeBSD linux emulation layer (linuxolator) currently implements most of the functionality necessary to emulate 2.4.2 linux kernel, but linux world has moved forward and current linux world requires 2.6.x features. The aim of this SoC task is to make Fedora Core 4 linux-base to be able to run with 2.6.x kernel. Currently this means extending clone() syscall and implement pthread related things. This involves TLS implementation (sys_set_thread_area syscall) and possibly tid manipulation (used for pthread_join etc.) and finally futexes (linux fast user-space mutexes implementation). This should enable pthread-linked programs to work. After this is done there may be other things necessary to implement however, only time will tell. I am funded by google.com in their SoC to do this work and I'll continue to work on this after the summer hopefully as a part of my MSc. thesis.
The improved support for the i386 binaries are ready for -exp run. It only allows installing such ports on amd64 and ia64 when there's a compatibility layer compiled into the kernel and the 32-bit libraries are installed under /usr/lib32.
The DESTDIR support are in progress. It works for the simplest ports without USE_* that don't have a [pre|do|post]-install target. There are more complicated issues with e.g. conflict checking in DESTDIR, deinstalling from DESTDIR, those have to be fixed as well.
The translated webpage is almost ready now. This Hungarian translation is a "lite" version of the original English webpages, since there are parts that are irrelevant for the Hungarian community, or has pieces of data that change quickly, so it's no use to translate these pages now, maybe later, if we have more Hungarian contributors, but this webpage would be a good starting point in translating the documentations, and we need a good place to put translated documentations anyway.
I'm going to be very busy with SoC this summer, but I'll try to find people that can help me out in this project. Any help appreciated.
As an intermediate step until FreeBSD will have full network stack virtualisation this work shall provide support for multi-IP IPv4/v6 jails.
These changes are based on Pawel Jakub Dawidek's work for multi-IPv4 jails and some initial work from Olivier Houchard for single-IPv6 jails.
The changes need some more testing but basically things work.
This is not considered to be the right thing todo so do
not
ask for official support or if this will be committed to the
FreeBSD source repository.
After some more cleanup of non-jail related IPv6 changes I will
publish a patch for HEAD and perhaps RELENG_6 for everyone who
wants to give it a try anyway.
Mohan Srinivasan committed his changes to make the NFSv2/3 client MP safe to HEAD this quarter. Changes may be back-ported to 6.x soon.
Robert Watson and Chuck Lever held a discussion about the future of the in-kernel NFSv4 client during BSDCan 2006. The current NFSv4 client is unmaintained. Chuck also pointed out the long series of unfixed PRs against the legacy client (NFSv2/3). These are at the top of his priority list. Robert is also interested in making NFSv4-style ACLs the lingua franca for FreeBSD file systems. There was some discussion about integrating Rick MacKlem's NFSv4 server into 7.x.
Chuck Lever became a full source committer during this quarter.
The basic goals of this SoC 2006 project are moving nsswitch-modules out of the libc, extending the caching daemon and importing nss_ldap into the base source tree. 2 milestones of the project are currently completed.
1. Nss-modules were successfully moved out of the libc into the separate dynamic libraries. In order for static binaries to work properly (they can't use dynamic nss-modules), nss-modules are linked statically into the libc.a. As the side-effect of nss-modules separation, getipnodeby***() functions were rewritten to use gethostby***() functions and not the nsdispatch(3) call. Caching daemon's "perform-actual-lookups" option was extended to support all implemented nsswitch databases.
2. A set of regressions tests was made to test nsswitch-related functions. These tests are also capable of testing the stability of these functions' behaviour after the system upgrade.
pfSense is rapidly approaching release. We are down to a handfull of bugs that should be fixed in the coming weeks. We should have a release around the time of our 2nd annual hackathon which is taking place on July 21st - July 28th. Many exciting sub-projects are taking place within pfSense and the project is gaining new developers monthly.
As an intermediate step towards implementing support for callgraphs and cross-architecture performance measurements, I am creating a BSD-licensed library for ELF parsing & manipulation. This library will implement the SysV/SVR4 (g)ELF[3] API.
Current status: Implementation of the library is in progress. A TET-based test suite for the API and manual pages documenting the library's interfaces are being concurrently created.
Work is being done in FreeBSD's Perforce repository. I hope to be ready for general review by the end of July '06.
During this time, a huge number of ports PRs were committed, bringing us back down below 800 for the first time since the 5.5/6.1 release cycle. This is due to a great deal of work, especially from some of our newest committers.
This is all the more notable given the fact that we have been adding new ports at a rapidly accelerating rate. We have now exceeded the 15,000 port mark!
Three sets of changes have been added to the infrastructure, including updates of default versions of MySQL, PHP, LDAP, and linux_base, and numerous bugfixes and improvements. About 2 dozen portmgr PRs were closed due to this.
In addition, a large-impact commit was made that attempts to move us to a single libtool that is as unmodified from 'stock' libtool as we can. Plans are also in place to do this for the autotools.
Several people are at work on implementing the modularised xorg ports. Most of the work is done but several key pieces remain. Once this is finished, an -exp regression test will be needed (most likely, more than one :-) ) It is possible that before this we will need to do a regression test that moves X11BASE back into LOCALBASE. This is still under study.
-Gabor Kovesdan started a Google Summer of Code project on some - highly needed improvements on the ports infrastructure (see - elsewhere in this report). As this is a long term project, gtetlow +
Gábor Kövesdán started a Google Summer of Code + project on some highly needed improvements on the ports infrastructure + (see elsewhere in this report). As this is a long term project, gtetlow kindly imported the most important ports infrastructure files into perforce to ease development. Other developers are encouraged to use perforce for ports development, especially as it can help keeping patches up-to-date while going stale in GNATS. Even though linimon has been pushing hard on running experimental builds on the test cluster, it will take some time to work through the backlog.
erwin added a ports section to the list of projects and ideas for volunteers at the FreeBSD website. Have a look if you want to work on the ports system. Don't hesitate to send additional ideas, and committers are encouraged to add themselves as technical contacts.
sem adopted portupgrade after it had been neglected for some time and has been very active on upgrades and bugfixing.
dougb has continued to enhance his portmaster script and people are finding success with it; although not designed to be as full-featured as portupgrade, it does seem to be easier to understand and use.
shaun has contributed portscout, a scanner for updated distfiles, to the ports collection.
marcus upgraded GNOME to 2.14.1.
As well, there have been new releases of the ports tinderbox code.
edwin has been hard at work on a PR-autoassigner for ports PRs, which has saved a lot of time and been well-received. It has now been installed on a freebsd.org machine (hub).
linimon has been more active in pursuing maintainer-timeouts, and has reset a number of inactive maintainers, with more in the pipeline. The intent is to try to reduce the number of PRs that sit around unanswered for two weeks. In almost all cases the resets are due to no response at all; maintainers who are merely "busy" are not the source of most of these problems, and deserve the benefit of the doubt. Some of the maintainers that have been reset haven't contributed in months or even years.
We have added 10 (!) new committers since the last report.
Since the last status report ports have been created for all parts of the BSDInstaller except the backend.
A snapshot of the BSDInstaller was released during this quarter. This has shown a number of bugs with the installation process. Most have now been fixed.
The patches to allow UFS operate with quotas in Giant-less mode are brewed for long now. Since recent huge pile of fixes into snapshots code, I think the problems you could encounter are caused solely by the patch.
Aside performance benefits, patch has another one, much more valuable. It makes UFS operating in one locking regime whatever options are compiled into kernel. I think, in long term, that would lead to better stability of the system.
We updated the default linux base port to Fedora Core 4 and the default linux X11 libs port to the X.org RPM in FC4.
An update to FC5 or FC6 has to wait until the kernel got support for syscalls of a newer linux kernel. See the corresponding SoC project report for more.
Since the last status report we fixed some more bugs, added basic support for envy24 chips and cleaned up the source for the emu10kx driver in the ports to make it ready for import into the base system.
We also got some patches with a little bit of infrastructure for Intel HDA support. It's not finished and also not usable by end users yet.
The XFS for FreeBSD project is an effort to port the publically available GPL'd sources to SGI's XFS filesystem to FreeBSD.
In December, we imported a version of XFS into FreeBSD-CURRENT which allows FreeBSD to mount an XFS filesystem as read-only.
As a side effort, we have been continuing on the work that PHK started to clean up the mount code in FreeBSD. We can use the existing FreeBSD mount(8) utility to mount an XFS partition, without introducing a new mount_xfs utility.
For the last several months Randall Stewart has been working in HEAD and STABLE to get us ready to integrate the SCTP protocol (Stream Transmission Control Protocol) into FreeBSD. He is currently working on a patch to share with a wider audience but needs to do some integration work first. Randall has a provisional commit bit and will be working with gnn on getting code committed to the HEAD of the tree.
In the time since the last status report, four security advisories have been issued concerning problems in the base system of FreeBSD; of these, one problem was "contributed" code, while three were in code maintained within FreeBSD. The Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated by the Security Team and Ports Committers documenting new vulnerabilities in the FreeBSD Ports Collection; since the last status report, 71 new entries have been added, bringing the total up to 757.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 4.11, FreeBSD 5.3, FreeBSD 5.4, FreeBSD 5.5, FreeBSD 6.0, and FreeBSD 6.1. The respective End of Life dates of supported releases are listed on the web site; of particular note, FreeBSD 5.3 and FreeBSD 5.4 will cease to be supported at the end of October 2006, while FreeBSD 6.0 will cease to be supported at the end of November 2006.
I have been working on porting missing features in gvinum from vinum, as well as adding new features.
So far the resetconfig, detach, dumpconfig, setstate (on plexes and volumes) and stop commands have been implemented, as well as some other minor fixes. The attach command is currently being implemented, and started on disk-grouping. Currently most of this is in p4, but patches will be submitted as soon as possible.
The wireless support has been stable for a while so most work has focused on bug fixing and improving legacy drivers.
Max Laier and I worked on improving support for Intel wireless cards. The results of this work included significant improvements to the iwi(4) driver (for 2195/2200 parts) and the firmware(9) facility for managing loadable device firmware. There is also an updated ipw(4) that has improvements similar to those done for iwi that is in early test. Support for the latest Intel devices, the 3945 pci-express cards, is planned for later this summer.
Atheros support was updated with a new hal that fixes a few minor issues and provides known working builds for SPARC, PPC, and ARM platforms. There is also working MIPS support that will be used when the MIPS port is ready to test. Otherwise one useful bug was fixed that affected AP operation with associated stations operating in power save mode.
wpa_supplicant and hostapd were updated to the latest stable build releases from Jouni Malinen.
Experimental changes to support injection of raw 802.11 frames using bpf were posted for comment. This work was done in collaboration with Andrea Bittau.
With the help of Jim Thompson of Netgate ( http://www.netgate.com/ ) the FreeBSD Foundation arranged a purchase of xscale-based boards for folks interested in ARM support. Developers were able to purchase boards at a reduced cost. The goals were to accelerate and/or improve support for the ARM platform and to set forth at least one board as a reference platform for the ARM support. Netgate will be stocking lower-cost models of the board later in the year (a special order was made for boards with only 2 mini-pci slots).
With the introduction of fine grained locking in the SMPng project, the FreeBSD kernel went under a major redesign, and many subsystem changed significantly with it. In particular, device driver's interrupt context ("the bottom half") had the necessity to synchronise with process context ("the top half") and share data in a consistent manner without using spl*(). To overcome this problem, a new interrupt model based around interrupt threads was employed, together with a fast interrupt model dedicated to particular driver handlers that don't block on locks (i.e. serial port, clock, etcetc). Unfortunately, even if the interrupt thread model proved to be a reliable solution, its performance was not on par with the pre SMPng era (4.x), and thus others solutions were investigated, with interrupt filtering being one of that.
As part of my Summer of Code 2006 work, I'm implementing interrupt filtering for FreeBSD, and when the framework will be in place I'll compare the performance of filters, against all the previous models: pre-SMPng(4.x), ithread and polling.
The most important modifications to the src tree so far were:
The project is slowly starting to ramp up after a long move-induced hiatus.
Alan Cox has almost completed making the pmap module Giant-free.
TrustedBSD Audit provides fine-grained security event auditing in FreeBSD 7.x, with a planned merge to 6.x for FreeBSD 6.2. Work performed in the last three months:
This report covers FreeBSD related projects between April and June 2007. Again an exciting quarter for FreeBSD. In May we saw one of the biggest developers summits to date at BSDCan , our 25 Google Summer of Code students started working on their projects - progress reports are available below, and finally the 7.0 release cycle was started three weeks ago.
If your are curious about what's new in FreeBSD 7.0 we suggest reading Ivan Voras' excellent summary at: http://ivoras.sharanet.org/freebsd/freebsd7.html and of course these reports.
The next gathering of the BSD community will be at EuroBSDCon in Copenhagen , September 14-15. More details about the conference and the developer summit are available in the respective reports below.
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
Mpd-4.2 has been released. It includes many new features, performance improvements and fixes.
The most significant and unique new feature is a link repeater functionality. It allows mpd to accept incoming connection of any supported type and forward it out as same or different type outgoing connection. As example, this functionality allows mpd to implement real LAC with accepting incoming PPPoE connection from client and forwarding it using L2TP tunnel to LNS. All other software L2TP implementations I know is only a LAC emulators without real incoming calls forwarding abilities.
Also mpd-4.2 presents:
Replacing external ifconfig and route calls with their internal implementations and other optimizations in 4.2 gave significant performance boost in session management. Newly implemented overload protection mechanism partially drops incoming connection requests for periods of critical load by monitoring daemon's internal message queue. As result, simple 2GHz P4 system is now able to accept, authenticate and completely process spike of 1000 concurrent PPPoE connections in just a 30 seconds.
The basic idea behind this project is to implement secure and reliable log file shipping to remote hosts. While the implementation focuses on audit logs, the goal is to build tools that will make it possible to perform distributed logging for any application by using a simple API and linking with a shared library.
OpenBSD includes sysctl hw.sensors framework since 2003; since 2005 the frameworks supports raid drives and most known i2c sensors; since 2006 the framework is redesigned with a sensor device concept in mind to accommodate continued growth. Consists of kernel api, sysctl(3)/sysctl(8), sensorsd(8), ntpd(8), systat(1), ports/sysutils/symon and 51 drivers as of 2007-07-07.
This GSoC2007 project is to port the underpinnings of this unified hardware monitoring interface to FreeBSD. Whilst it won't be possible to port all of the drivers due to architecture differences, we aim at porting all other parts of the framework and accompanying userland utilities.
At this time, lm(4) at isa and some kernel api have already been ported. The next big step is to complete sysctl(3) glue code so that further work on porting userland utilities could be accomplished. Details about sysctl are being discussed on arch@.
The Linux kernel-based Virtual Machine (KVM) is a mechanism to exploit the virtualization extensions present in some modern CPUs (e.g., Intel VT and AMD-V). Virtualization extensions let ordinary processes execute a subset of privileged instructions in a controlled way at near-native speed. This in turn may improve the performance of system emulators such as qemu, xen, vmware, vkernel, User Mode Linux (UML), etc.
This project consists in porting to FreeBSD the Linux KVM, implemented as a loadable module, lkvm.ko. We use the approach in ports/devel/linux-kmod-compat to reuse the original Linux source code almost unmodified. We will also port a modified version of qemu which exploits the facilities made available by the Linux KVM to speed up emulation.
The URL above links to progress report detailing the exact project goals, milestones reached, and commit log details.
As of end of June 2007, we have mainly extended linux-kmod-compat to support the kernel API used by the Linux KVM code. The required functions have been implemented at various degrees, from simple stubs to fully functional ones. We have also imported the modified qemu and the libraries that are used to build the Linux KVM userspace client. In the second half of the SoC work we plan to complete the implementation of the kernel API and have a fully functional Linux KVM module, together with its client (qemu).
This project aims to create a multicast DNS daemon and service discovery utilities suitable for the base system. Multicast DNS is a part of Zero Configuration Networking (Zeroconf) and provides the ability to address hosts using DNS-like names without the need of an existing (unicast), managed DNS server. Work on the responder daemon is well underway and the only large missing piece of the puzzle is a way for local clients to do queries. The code can be found in the p4 branch projects/soc2007/fli-mdns_sd if anyone would like to give it a spin, even though it's incomplete. The project plan can be found on the wiki.
The project is split up with a front end to interact with the user and a back end to interact with freebsd-update. The back and front ends are able to communicate with each other using an XML protocol. The GUI is almost at the point it can take a command from the user and send it to the back end. The back end is able to detect when updates are ready.
The sixth EuroBSDCon will take place at Symbion in Copenhagen, Denmark on Friday the 14th and Saturday 15th of September 2007.
The programme is ready and online at the webpage. Registration
is open. Details about tutorials and Legoland trip are ready too.
The keynote will be John Hartman: Real men's pipes
If you share a room with friends at the hostel, then lodging is really inexpensive, and the lounge has high speed Internet access. Staying at the hostel is of course optional, and the area has several hotels.
KD85.com and O'Reilly will each have a booth at the conference.
We are still looking for more sponsors.
A public IRC channel #eurobsdcon on EFnet has been created for discussion and questions about the conference.
After the success of FreeSBIE-2.0.1-RELEASE, development slew down a bit, but we have a big task for the summer: enable unionfs again and trying the new efficient memory filesystem, tmpfs.
For all new ISO images we will be following RELENG_7, with the hope to release a stable image once 7.0-RELEASE have been released.
Gábor Kövesdán is working on some improvements for the Ports Collection infrastructure. This year, he aimed to work on long-standing issues, which are tracked in GNATS, but we have not had a volunteer for recently. With the mentorship of Andrew Pantyukhin, he is also reimplementing the DESTDIR support for Ports Collection in a more practical way. The complete description and status of this project is available on Gábor's SoC 2007 Wiki page.
We have added one translated article since the last status report about this project. The infrastructure is ready to support localized articles and books as well, we just lack of human resource. New volunteers are highly welcome! Please see the link below and contact Gábor if you are interested.
Tarfs is a simple tar file system implementation for FreeBSD.
The current goals are:
Here's the current state of things:
FAST_IPSEC has now replaced Kame IPsec as the IPsec stack in HEAD. This will be part of the 7.0 release. The merge happened in early July with George handling the kernel bits and Bjoern handling user space.
The kernel option IPSEC is now the ONLY option for IPsec support in the FreeBSD kernel.
During the last three months there has been several changes to the USB stack. Here is a quick list of the most important changes:
Markus Brueffer is still working on the USB HID parser and support. Nothing has been committed yet.
If you want to test the new USB stack, checkout the USB perforce tree or download the SVN version of the USB driver from my USB homepage. At the moment the tarballs are a little out of date.
Ideas and comments with regard to the new USB API are welcome at freebsd-usb@FreeBSD.org .
Gvirstor is a GEOM class which provides virtual storage capacity (something like virtual memory for storage devices). It's ready to be committed to HEAD (the plan is for it to get into 7.0-RELEASE).
Project "finstall" aims to create a next-generation FreeBSD installer that will make use of the newest features present in the system. The project should yield something usable for 7.0-RELEASE, but the intention is to keep it as a "second" installer system during 7.x, alongside sysinstall. In any case, sysinstall will be kept for architectures not supported by finstall (e.g. all except i386 and amd64).
This entry was previously the Bt878 Audio Driver (aka FusionHDTV 5 Lite driver) announcement, but as work expanded slightly, it's a bit more generic now.
A few bugs in bktrau has been fixed since January. If you have been running an earlier version, it is recommended to upgrade as the driver could panic. The driver works with multiple cards in the same machine (tested with two).
FusionHDTV 5 Lite -- Due to lack of documentation from DViCO and LG, I have copied magic values from the Linux driver to get ATSC capturing working.
ATI HDTV Wonder -- After years of trying to get into the ATI developer program, they have finally suspended it, so no support from ATI. I have started work on a driver, cxd, for the Conexant CX2388x based cards. The ATI HDTV Wonder uses ATI's own demodulator, and I was able to get it to tune, after cribbing from the Linux driver. When capturing, I get some valid data, but not all the data. Due to lack of support from ATI and linux-dvb the project has been put on indefinite hold.
If someone has another CX2388x based card, it shouldn't be too hard to take the driver and get it working with a different tuner.
A Python module is available for both drivers/cards, along w/ a sample capture application using it. The module is now known to work well with threads so that tuning (expensive due to i2c ioctl's) can happen in another thread without causing program slow down. The module is working well with a custom PVR backend.
Gavin Atkinson has joined the bugbuster team via getting a GNATS account on the FreeBSD cluster. He is following in the footsteps of Matteo Riondato, who later graduated to a full src commit bit. So far, he has helped close nearly 150 PRs, including many that had become stale. Welcome!
Our short-term goal is to try to identify bugs that we might be easily able to fix before the 6.3/7.0 simultaneous release. So far, great progress has been made on ata- and usb-related PRs.
The goal for the rest of this year is to generate more developer interest in fixing bugs. To do this, we are, first, trying to do more work on triaging PRs as they come in, to help flag ones that seem to be valid problems (especially if they include patches.) Secondly, we have started a new weekly periodic posting to the freebsd-bugbusters@FreeBSD.org mailing list, which is a short list of PRs that we feel are ready for committer action. This posting is automatically generated from a text-file list that we maintain.
We are continuing to try to manage our community's expectations of what we can do with the incoming PRs. In particular, we are trying to discourage submissions of the form "I cannot get the XYZ function to work". In practice, these PRs are not worked on. Instead, we are now encouraging these postings to go to one of the mailing lists such as freebsd-questions@, freebsd-x11@, and so forth. The idea is to emphasize GNATS as a "Problem Report" method, rather than a "general FreeBSD support" method. I feel that, otherwise, we were creating a false expectation.
The overall PR count has dropped to below 5000, despite the extra PRs still not cleared up from the ports freeze for the xorg7.2 import. Significant progress has been made on the i386, kern, and bin PRs, as well as PRs in the 'feedback' state. In addition, Warner Losh has made progress on closing many of the usb PRs.
The ports count is over 17,300. The PR count has been stable at around 800; we have not quite cleared up the backlog that showed up during the freeze to import xorg7.2.
There have been 4 experimental runs on the build cluster, most notably resulting in some speedups for package registration. A further experimental run to genericize autotools handling is in progress.
One of the most sweeping ports commits to happen in years was the upgrade of xorg from 6.9 to 7.2. This involved a complete rework of the internals of the port, as X.org itself has effectively pushed the responsibility for packaging to the OSes that incorporate it. The idea was to be able for them to update individual code (such as video drivers) without having to reroll the entire distribution. This commit caused us to have the longest period of preparation work, and actual tree lockdown, that I am aware of. The commit continues to be controversial, partly due to the fact that none of our port upgrade tools was up to the task of doing the upgrade without manual intervention.
At the same time that xorg was upgraded, we moved the installation directory from the obsolete /usr/X11R6 to our default /usr/local. This further complicated the upgrade.
There have been new releases of the ports tinderbox code, the portmaster update utility, and portupgrade.
GNOME was updated to 2.18.2.
We have added 7 new committers since the last report. We appreciate all the new help. However, a few committers have turned in their commit bits for safekeeping, due to lack of time.
Unfortunately, Clement Laforet has also had to step down from portmgr due to lack of time. We thank him for his help so far.
Erwin, Kris and Mark met up at BSDCan and reviewed all the portmgr-owned PRs. A large number were closed, or suspended pending more work from the submitter. After closing the PRs that were committed after the -exp builds, the number of portmgr owned PRs came down to an all time low of 48 from around 70. We hope to make further progress during the rest of the year.
The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This will allow for complete networking independence between jails on a system, including giving each jail its own firewall, virtual network interfaces, rate limiting, routing tables, and IPSEC configuration.
I believe that the prototype, which is kept in sync with FreeBSD -CURRENT, is now sufficiently stable for testing. It virtualizes the basic INET and INET6 kernel structures and subsystems, including IPFW and PF firewalls, and more. In the next month I plan to have the IPSEC code fully virtualized, and refine and document the management APIs. The short-term goal is to deliver production-grade kernel support for virtualized networking for FreeBSD 7.0-RELEASE (as a snap-in kernel replacement), while continuing to keep the code in sync with -CURRENT for possible merging at a later date.
IP can easily be tunneled over a plethora of network protocols at various layers, such as IP, ICMP, UDP, TCP, DNS, HTTP, SSH. While a direct connection may not always be possible due to a firewall, the IP packets could be encapsulated as payload in other protocols, which would get through. However, each such encapsulation requires the setup of a different program and the user has to manually probe different encapsulations to find out which of them works in a given environment.
mtund is a tunneling daemon using run-time loadable plugins for the different encapsulations. It automagically selects the best encapsulation in each environment and fails over to another encapsulation in case the environment changes. There already is running code available, capable of tunneling via TCP and UDP with a working failover mechanism. As this is a Summer of Code project, rapid changes and addition of new features can be expected during the summer. Please see the wiki page for more details and up-to-date information.
Note that the project originally started under the name of Super Tunnel Daemon, but was later renamed to mtund for Magic Tunnel Daemon.
Over the past 6 months several developers undertook an effort to replace the global scheduler lock with a finer-grain interface modeled on the Solaris container lock approach. This significantly reduces contention on higher-end multiprocessor machines.
This patch went into 7.0-CURRENT and has proven to be very stable. The last remaining bugs are in rusage and effect only process time accounting statistics.
SCHED_SMP is a fork of the ULE scheduler which makes use of the new fine grain scheduler locking in 7.0-CURRENT to significantly improve SMP performance on some workloads. It has improved and stronger affinity, smarter CPU load balancing, structural improvements and many sysctl tunables. This can be considered ULE 3.0. Discussions are ongoing as to whether this will go into 7.0 as SCHED_SMP or as SCHED_ULE in 7.0 or 7.1.
SCHED_ULE has had many bugfixes and performance improvements over the 7.0 development cycle and should no longer be considered unstable or experimental. On most workloads it significantly outperforms SCHED_4BSD on SMP and even slightly outperforms it on UP. There are some pathlogical workloads which exhibit as much as a 5% performance penalty. Many thanks to Kris Kennaway and current users for bug reports and performance testing.
FreeBSD developers have been using mysql as a testbed to find kernel contention hotspots in the kernel. As a result of this we have seen a 5x performance improvement over 6.0 on 8way machines. Recent changes include finer locking in fcntl(), removing Giant from flock and fcntl F_SETLK. These changes will be available in 7.0 and primarily improve write performance. Experimental changes to select() have also been discussed on arch@ that solve contention issues there however these will not be ready in the 7.0 timeframe.
The last major updates are currently being made to PC-BSD 1.4, which will include KDE 3.5.7, Beryl, Flash, Intel Wireless, Nvidia Drivers and more! This release will also include new utilities to make running PC-BSD on the desktop easier than ever, including:
Once any final major issues are resolved, we will be issuing a public beta of PC-BSD 1.4 to ensure compatibility across a variety of platforms.
The next developer summit will be different from the previous ones.
Very different.
Gone are the auditorium style seating, beamers, endless presentations and soggy sandwiches.
Instead we head out to an old village school in the beautiful Danish countryside, we hang around all over the place, sleep in the old science room, cook our own food and hack the living daylights out of anything we care for.
September 17th and 18th, right after EuroBSDcon2007 in Copenhagen. (Well, right after the optional trip to legoland...)
Be there!
PS: Yes, it's not uncivilized, there is a full speed ADSL and WLAN.
Main goal of project is to introduce code working in PXE preboot environment, able to download from web server via direct connection or http proxy and prepare booting of FreeBSD kernel.
Already implemented, but haven't thoroughly tested: PXE wrappers core code, ARP, ICMP echo request/reply, sockets code similar to common sockets (UDP and TCP modules). On base of sockets: simple DHCP client, DNS client.
Currently working on http client, TCP testing, kernel booting and documenting main concepts of project modules.
Code freeze in preparation for FreeBSD 7.0 began on June 18th. There are several large projects still being finished up as well as some issues that resulted as "fallout" from the work done just before the code freeze started (e.g. things resulting from the GCC 4.2 import). A schedule for the 7.0 release has not been set yet but the hope is that the first BETA build will be done near the end of July with a "fairly normal" release cycle (a few BETA builds followed by two or three RCs, each separated by around two weeks).
We are planning to release FreeBSD 6.3 around the same time as FreeBSD 7.0 is released so the release schedule for that will be set at the same point we set the release cycle for 7.0, hopefully late in July.
Support was added for two more 10gigabit network drivers and there were major advances on improving system performance over 10g media.
Kip Macy committed a new driver for the Chelsio adapters. The cxgb driver supports all current 10g adapters, as well as the new four-port gigabit model. The cxgb driver work was supported by Chelsio.
Drew Gallatin made significant improvements to the Myricom 10g driver mxge. With these updates the driver does line rate transfers with less system overhead.
Neterion contributed the nxge driver to support all their Xframe 10Gbe Server/Storage adapters. The initial driver import was done by Sam Leffler; a switch over to vendor support will happen soon.
Jack Vogel is preparing a driver to support the latest Intel 10g hardware devices. The new driver - ixgbe - will complement the existing ixgb driver that supports older Intel 10g cards.
Kip and Drew worked with other folks on performance analysis and tuning. This work improved cpu affinity and reduced overhead for managing network resources. Work is also underway to define a common Large Receive Offlaod (LRO) infrastructure. LRO is analogous to TSO on the receive side enabling drivers to receive at near line rate with normal sized frames. This common code base will help replace driver-specific code.
This project is due to provide a GUI audit log analysis tool for FreeBSD. Refer to ethereal/wireshark packet parsing engine and its framework to view and parse audit logs.
A basic implementation of ar(1) (include ranlib) was finished and available in the perforce repository. Currently it provides all the main functions an ar(1) should have and it is based on the libarchive and libelf library thus is expected to have a better and simpler structure than the GPL'ed version. The work left in this part of the project is to perform a elaborate test and add additional functions.
The kernel/hwpmc(4) bits of stack trace capture have been implemented and are available in Perforce under path '//depot/user/jkoshy/projects/pmc/...'. I'm currently enhancing pmcstat(8) to extract and summarize this information. Support by Google Inc. for this project is thankfully acknowledged.
Just like last year I got the opportunity to work on updating the Linuxulator to Linux version 2.6. This year I work on finishing futexes, *at syscalls and epoll/inotify.
I, cooperating with Konstantin Belousov, have managed to fix futexes to the state of passing the official futex testing program. The fix was committed and 7.0R will ship with correct futex implementation. Work is planned on removing Giant locking from futexes. This only needs some carefull review and testing.
These days I mostly focus on *at syscalls, the patch is almost finished for committing and I hope that it will make it into 7.0R. As a part of this work I implemented native FreeBSD syscalls as well. Watch arch mailing list as I post the patch there.
I also finished writing my master thesis describing how the - Linuxulator works and Gabor Kovesdan is working on integrating it - into official FreeBSD articles.
+ Linuxulator works and Gábor Kövesdán is working + on integrating it into official FreeBSD articles.No work has happened in the epoll/inotify area but I hope to work on it right after I finish the *at syscalls.h
Security Regression Test is supported by the project of Google summer code 2007. The main objective of this stage is to test the correctness of FreeBSD Mandatory Access Control Framework including correctly passing the security label from userland to kernel and non-bypassibility of Mandatory Access Control Hooks.
Work performed in the last month:
General cleanups in preparation for 7.0.
Process audit state moved to the credential to allow it to be accessed lock-free in most cases, as well as allowing it to be used in asynchronous contexts.
OpenBSM 1.0a14 has been imported, which: fixes IPv6 endian issues, makes OpenBSM gcc41 warnings clean, teaches audit_submit(3) about getaudit_addr(), adds zonename tokens; other changes since the existing CVS 1.0a12 release previously imported include man page improvements, XML printing support, better audit.log.5 documentation, additional 64-bit token types, and new audit event identifiers.
MAC checks have been added so that MAC policies can control use of audit system calls.
Additional system call arguments are now audited.
Audit now provides a security.audit sysctl node in order to determine if audit support is compiled in; boot-time console printfs have been removed.
"options AUDIT" is now in the 7-CURRENT GENERIC kernel, so AUDIT support will be available out of the box in 7.0 without a kernel recompile. Manually enabling audit support in rc.conf will still be required. With FreeBSD 7.0, AUDIT will be a fully supported, rather than experimental, feature.
The FreeBSD Foundation ended Q2 raising over $116,000. We're almost half way to our goal of raising $250,000 this year! We continued our mission of supporting developer communication by helping FreeBSD developers attend BSDCan. We were also a sponsor of BSDCan and the developer summit. We are a sponsor of EuroBSDCon 2007 and are now accepting travel grant applications for this conference. Foundation board members met with representatives of companies that use or are thinking of using FreeBSD both in the bay area and Ottawa.
The Foundation has negotiated a joint development agreement with Google, Inc. to sponsor FreeBSD developer Joseph Koshy to improve FreeBSD's HWPMC implementation, including adding stacktrace support, and a donation of SMP hardware for future SMP scalability work. We greatly appreciate Google's support for this project, which will facilitate performance measurement and optimization of both the FreeBSD operating system and applications running on it.
To learn more about what we're doing, go to our website at http://www.FreeBSDFoundation.org/ . Our July newsletter will be published soon to update you on how we've been supporting the project and community worldwide.
Cleanup of MAC Framework API/KPI layers: mac.h is now just the user and user<->kernel API; mac_framework.h is the kernel<->MAC Framework KPI, and mac_policy.h is the MAC Framework<->MAC policy module KPI. Along similar lines, mac_label_get() and mac_label_set() accessor functions now allow policies to access label data without encoding struct label binary layout into policy modules, opening the door to more efficient layouts. struct label is now in mac_internal.h and used only inside the MAC Framework.
General MAC policy cleanup, including removing no-op entry points and sysctls for some sample policies. mac_test(4) has been cleaned up significantly, and counters for all entry points added.
A MAC check for UNIX domain socket connect has been added.
MAC checks have been added so that MAC policies can control use of audit system calls.
MAC checks that duplicate existing privileges but add no additional context have been removed (such as sysarch_ioperm, kld_unload, settime, and system_nfsd) -- checks aligned with privileges but that do provide additional context, such as additional arguments, have been kept.
The Biba and LOMAC policies now implement priv(9) checks, differentiating between privileges that may compromise system integrity models, and those that don't.
The essentially unused mnt_fslabel / mnt_label distinction has been eliminated by moving to a single mnt_label. No functional change to any policy.
Several MAC-related interfaces have been modified to synchronize with the naming conventions present in the version of the MAC Framework adopted in Mac OS X Leopard; significant further changes are in the pipeline to complete this synchronization. While it will not be possible to reuse a policy between the two platforms without careful thinking and modification, this makes porting much easier.
The custom file descriptor array lock has been replaced with an optimized sx lock, resulting in 2x-4x improvement in MySQL transaction rates on 8-core MySQL benchmarks. This improvement is due to moving to shared locking for frequent fd lookup operations, as well as significant optimization of the case where the filedesc lock is highly contended (as occurs in the threaded MySQL server performing constant socket I/O).
The custom socket buffer I/O serialization lock (sblock), previously created by interlocking SB_WANT and SB_LOCK flags with the socket buffer mutex, has been replaced with an optimized sx lock, leading to a 10% performance improvement in MySQL and PostgreSQL benchmarks on 8-core systems. As part of this change, sx locks now have interruptible sleep primitives to allow the SB_NOINTR flag to work properly.
These changes also correct a long-standing bug in socket buffer lock contention and SB_NOWAIT reported by Isilon; a simpler patch has been merged to 6.x to fix this bug without merging loocking changes.
TCP debugging is now properly synchronized using a new tcp_debug_mtx.
UMA allocation counters are now used for pipes rather than custom atomic counters, resulting in lowered overhead for pipe allocation and free.
Significant code cleanup, commenting, and in some cases MFC'ing, has taken place with respect to the network stack and synchronization. Additional DDB debugging commands for sockets of various sorts have been added, allowing listing of socket state from DDB without the use of GDB.
Certain non-MPSAFE subsystems have been removed or will be removed from FreeBSD 7.0, including IPX over IP tunneling (not general IPX/SPX support, just the tunneling over IP), KAME IPSEC (FAST_IPSEC is MPSAFE and now now supports IPv6), i4b, netatm (two other ATM stacks are still present), and ng_h4. Some of these features will be reintroduced in FreeBSD 7.1, but by removing them now, we are able to remove the NET_NEEDS_GIANT compatibility infrastructure that significant complicates and obfuscates the socket and network stack code.
Other measurement and optimization projects continue; however, the 7.0 locking/synchronization work for the network stack is essentially complete.
Further reduction of suser(9) consumers in order to attempt to remove the suser(9) KPI for 7.0. This includes resource limits, System V IPC, PPP, netinet port reuse, the NFS server, and netatalk. Remove unnecessary or redundant privilege checks were possible. UFS-privileges that apply to other file systems have been renamed to VFS privileges.
All suser_cred() flags and priv_check_cred() flags are no longer required, as SUSER_ALLOWJAIL and SUSER_RUID use are determined entirely inside kern_jail.c and kern_priv.c and selected based on the privilege number, not a calling context flag. All privileges are now consistently allowed or not allowed in jail, and consistently use the ruid or euid. We will leave the flags field there as it will likely be used for other things in the future.
Documentation in suser(9) and priv(9) has been updated.
Apple's MacBook computers are nicely designed and have neat features that other laptops don't. While Mac OS X is a nice operating system, UNIX folks (like me) would prefer to run other operating systems like FreeBSD. This project aims to bring bug fixes and new drivers to FreeBSD that would help running this OS on this platform.
In the time since the last status report, two security advisories have been issued concerning problems in the base system of FreeBSD; both of these problems were in "contributed" code maintained outside of FreeBSD. The FreeBSD Vulnerabilities and Exposures Markup Language (VuXML) document has continued to be updated; since the last status report, 35 new entries have been added, bringing the total up to 925.
In order to improve handling of security issues in the FreeBSD Ports Collection a new "ports-security" team has been created to include ports committers who periodically help with fixing ports security issues and documenting them in the FreeBSD VuXML document. Committers who wish to help with this effort can contact simon@ for details.
The following FreeBSD releases are supported by the FreeBSD Security Team: FreeBSD 5.5, FreeBSD 6.1, and FreeBSD 6.2. The respective End of Life dates of supported releases are listed on the web site; it is expected that of the upcoming releases, FreeBSD 6.3 will be supported for two years after release, while FreeBSD 7.0 will be supported for one year after release.
The project consists in a rewriting of the lockmgr(9) interface on a lighter basis, using atomic instructions and direct usage of the sleepqueue interface. This should lead to a faster primitive, a saner interface and an higher maintainability of the code.
So far, 3 newly files called kern/kern_lockng.c, sys/_lockmgrng.h and sys/lockmgrng.h have been created for the new primitive and an initial implementation has been committed into the perforce branch: //depot/user/attilio/attilio_lockmgr/...
The implementation contains a good set of code intended to replace old lockmgr. Actually it only misses the support for lock draining that will be committed after an initial phase of testing and the inclusion of a better wake-up algorithm (which will simplify draining a lot and will improve performance on wakeup).
My previous status reports contained a lot of code that updated gvinum with the old vinum features.
This year gvinum has been significantly rewritten. Lukas Ertl began rewriting the way gvinum is organized from using a multi consumer/provider model, to use a single consumer and provider, and having an event-system that first handles user-requests, and then runs normal I/O operations (Much like other GEOM classes). This makes the code easier to read, and perhaps there will be less bugs :)
And of course, some time has gone to work out how things should be done, and to fix other bugs. I hope some of you are interested in trying this out (all the work has been in perforce so far), a patch can be found in the URL section. . This is a bit experimental, and although I've done much testing to hunt down bugs, there are most probably bugs left.
I have other goals this summer as well. However, since some parts of gvinum was rewritten, I might not be able to do all of these, but growing is already working for the concatenated volumes (and also mirrored). I'd also like to implement growing for Raid5 arrays as well. Logging plexes would also be cool to have, but this is not really needed, since we have g_journal. Both these features will be addressed after I've made sure gvinum does all old vinum does, and also perhaps better. As I might have some extra time on my hands this summer, so I gladly accept suggestions on what else I might fix or implement "while I'm at it".
Both libarchive 2 and bsdtar 2 are now in -CURRENT and will be in 7.0. Libarchive 1.9 and bsdtar 1.9 should be in 6-STABLE in time for 6.3.
libarchive 2 is much faster writing to disk than libarchive 1. It also supports new formats, has several minor API/ABI corrections, is more portable, and has many fewer bugs. Of special note is "libarchive_test", a new program that exercises much of the libarchive functionality; anyone interested in working on libarchive should become familiar with this test suite. bsdtar 2 is less ambitious, but does have a number of bug fixes and takes advantage of several new features in libarchive 2.
libarchive 1.9 is identical to libarchive 2 except it maintains the old API/ABI. Similarly, bsdtar 1.9 is nearly identical to bsdtar 2, lacking only a few features that would prevent it from being used with existing libarchive 1 libraries.
About 18 months ago, I started to remove the compatibility macros that we had in the USB stack. These macros made it very hard to read the code and to diagnose problems. They represented a barrier to entry for people reading and understanding the stack. In addition, many of them effectively hid bugs from all but the most intensive investigations of the code.
I've removed almost all of the macros in the client drivers, and all instances of the macros in the core FreeBSD USB stack. This makes the drivers more readable, and a little more robust. During this process, I fixed a lot of little bugs that people had been tripping over, and some that people hadn't reported. I've added a boatload of new vendor and product ids to the drivers from user PRs as well as from OpenBSD/NetBSD drivers.
I finished up this work so that the FreeBSD USB stack would be more maintainable during the RELENG_7 period of time. I plan on MFCing most of the changes I've made into RELENG_6 after they have been shaken out in current. There was only one API changes in this work, so this is doable, and makes sharing drivers between 6.x and 7.x much easier. At this stage, it is unclear how long RELENG_6 will be around, so I'm hoping this will make USB much better in 6.3 if that's the release people choose to run.
I've shied away from many of the more complicated changes to the stack. There's work being done outside of the tree by Hans Petter Selasky (hps) to make these sorts of changes. There is much in his stack that's ready to be merged, and I hope to integrate from that work useful bits that can be merged without disruption to improve the FreeBSD USB stack.
I'm also looking for other FreeBSD developers that can jump in and help. Nearly all of the improvements I've done by spending a few hours a week sorting through the PRs for extremely low hanging fruit. There's plenty of room for others to be involved as well in improving FreeBSD's USB stack, as well as chances for us to import the now-useful bits from the evolving hps USB stack, hopefully reducing the diffs between it and the present FreeBSD USB stack. In addition, I'm looking for someone to do similar device ID merges from DragonFlyBSD.
Finally, I've embarked on a mission to try to merge all the BSD's usbdevs files. There's no reason to have separate ones. I've started to modify usbdevs(1) to read the src/sys/dev/usb/usbdevs file and report more verbose information that way. A merged usbdevs would be larger, and take up more memory in a USBVERBOSE kernel, so to mitigate that effect, I'm making changes to usbdevs(1).
A major update of the 802.11 wireless support was committed. Changes include advanced station mode facilities such as background scanning and roaming, and support for 802.11n devices. In addition parts of the Atheros' SuperG protocol extensions were added so that wireless clients that communicate with Atheros-based access points can operate more effectively. The changes to the infrastructure are also important because they simplify future distribution of Virtual AP (VAP) support.
This work represents the effort of many people including Kip Macy, Andrew Thompson, Sepherosa Ziehau, Max Laier, and Kevin Lo. Getting these changes into the tree now ensures they will be present for the lifetime of the 7.x branch.
The scanning and SuperG work were supported by Atheros. The 802.11n-related work was supported by Marvell.
I have been working on making wake on lan (WOL) work with FreeBSD. Contrary to popular believe OS support is required for WOL to work properly. In particular network card drivers need to configure network cards for WOL during system shutdown, else the cards won't wake up. WOL is _not_ just a BIOS issue.
This is work in progress. Currently the following cards/chipsets are supported:
I would be glad to get more feedback on my patch. I can add support for more chipsets but I need testers for hardware I don't have. I would appreciate access to data sheets for any NIC chipsets that are supported by FreeBSD and have WOL support.
I would especially appreciate technical feedback on the patch, preferably by a committer who is willing to nitpick the patch to make it ready for inclusion in -CURRENT. I currently maintain the patch against RELENG_6_2 for my own use but I would port it to -CURRENT for inclusion.
Work is well under way to finish Kip Macy's FreeBSD/xen port, and get it into a shape which is suitable for inclusion in 7.0.
Generally, the port is stable and performs quite well. The major bottleneck is the inability to work with GCC 4.2, this is the last major TODO before the work can be committed.
pf in HEAD (soon to be FreeBSD 7.0) has been updated to OpenBSD 4.1 bringing in a couple of new features:
Some patches that went into OpenBSD after 4.1 and improve performance significantly will be merged later.
Work to support pf and netgraph interaction is underway and will be imported after 7.0. As all required ABI changes have been made during the update, we will be able to MFC this work for 7.1 later on.
FreeBSD's static analysis scans have been updated with a recent version of Coverity Prevent. Coverity is providing additional advice on configuration of the analysis to maximize the benefit from the tools.
At BSDCan2007, Coverity provided FreeBSD with a license for an additional analysis tool called Extend, which allows writing custom FreeBSD specific code checkers. David Maxwell presented training material for interested FreeBSD developers. Some applications of custom checkers have been considered, and more results will be forthcoming as they are implemented and tested.
This report covers FreeBSD related projects between July and October 2007. The sixth EuroBSDCon was held in Denmark in September. The Google Summer of Code project came to a close and lots of participants are working getting their code merged back into FreeBSD.
The bugs in the FreeBSD HEAD branch are being shaked out and it is being prepared for the FreeBSD 7 branching. If your are curious about what's new in FreeBSD 7.0 we suggest reading Ivan Voras' excellent summary here .
Thanks to all the reporters for the excellent work! We hope you enjoy reading.
IP can easily be tunneled over a plethora of network protocols at various layers, such as IP, ICMP, UDP, TCP, DNS, HTTP, SSH. While a direct connection may not always be possible due to a firewall, the IP packets could be encapsulated as payload in other protocols, which would get through. However, each such encapsulation requires the setup of a different program and the user has to manually probe different encapsulations to find out which of them works in a given environment.
MTund is a tunneling daemon using run-time loadable plugins for the different encapsulations. It automagically selects the best encapsulation in each environment and can fail over to another encapsulation. Several plugins have been implemented and the daemon supports multiple concurrent clients.
Note that the project originally started under the name of Super Tunnel Daemon, but was later renamed to Magic Tunnel Daemon (MTund).
After a long break in this project, we started reviewing and refreshing our translations. We have to update the content to reflect the current state of the English version. There are a few parts written in a poor style, another task is to improve these a bit. Any kind of help is highly welcome.
We have a new volunteer, Gabor Pali, who provided us some - high-quality contributions. As a result, we have been able to add 5 +
We have a new volunteer, Gábor Páli, who provided us + some high-quality contributions. As a result, we have been able to add 5 new articles since the last status report.
There is also an ongoing effort in the Perforce repository to translate the FreeBSD Handbook to Hungarian. Any kind of support is highly welcome.
The sixth EuroBSDCon went well. 215 people attended the conference. Feedback has been very positive.
At the conference we had a Best Talk contest. Steven Murdoch, Isaac Levy and Pawel Jakub "zfs-man" Dawidek each received a prize for their fantastic talks.
Also over 300 pictures from the conference has been uploaded to Flickr with the tag EuroBSDCon2007
Videos and slides from the talks are now online at the conference website.
We thank our speakers for graciously having permitted recording and publication of their talks
EuroBSDCon 2008 will take place in Strassbourg.
The "finstall" project is about the new graphical installer for FreeBSD. The basic frameworks (both client-side and server-side) are done during the SoC 2007 and it's ready for major new features to be implemented. This project should yield an usable installer for 7.0-RELEASE.
With the leaving of bsd@, we lost the GNATS statistics webpages. On this URL I generate a new set of graphs, right now a subset of what bsd@ had, hopefully a superset of that in the future.
We're happy to report the successful conclusion of our third consecutive Google Summer of Code. By all accounts, the FreeBSD participation in this program was an unqualified success. We narrowed down the many impressive applications to 25 that were selected for funding and 92% of these completed successfully and were awarded the full $4,500 stipend. The FreeBSD Foundation was also granted $500 per student from Google for a total of $12,500.
These student projects included security research, improved installation tools, new utilities, and more. Many of the students have continued working on their FreeBSD projects even after the official close of the program. Three students have already been granted full src/ commit access to CVS and more are expected. At least 2 of our FreeBSD mentors will be meeting with Google organizers in Mountain View this month to discuss the program at the Mentor Summit.
GEOM_VIRSTOR (virtual disk space / over-commit GEOM class) has been committed to 7-CURRENT and will ship in 7.0-RELEASE. Thanks to Pawel Jakub Dawidek and others who have made this possible.
During the last three months there has been a flush of changes going into the FreeBSD USB P4 project. The changes mainly consern the ability to support the USB device side and multi frame USB transfers. Up to date the FreeBSD USB stack has only supported the USB Host Side. Before Christmas 2007 the P4 USB project will offer USB device support and some simple USB device side implementations. Technically an USB device side driver will look very similar to an USB host side driver. Infact there will be very few differences. Support for multi frame USB transfers opens up the possibility to transfer multiple short-packet terminated USB frames to/from different memory locations resulting in only one interrupt on the USB Host Controller. More specific: I have implemented support for the "alt_next" pointer in the EHCI Transfer Descriptor. This should give a noticable increase in the maximum number of short-packet terminated BULK frames that can be transferred per second.
I regularly get questions from people asking about when the USB P4 project will be merged into FreeBSD-current. The answer is not simple, but probably something like another year. The reason is not that the current code in the USB P4 project is not usable, but rather that the quality needs to be raised in means of making already good solutions more technically excellent, writing more documentation and styling the code.
Ideas and comments with regard to the new USB API are welcome at freebsd-usb@freebsd.org.
Linux KVM (Kernel-based Virtual Machine) is a software package that can be used to create virtual machines fully emulating x86 hardware on top of machines supporting Intel VT-x or AMD-V virtualization extensions, available on newer AMD and Intel processors, e.g., recent Athlon64, Core 2 Duo, Xeon and so on.
Linux KVM has been ported to FreeBSD as a loadable kernel module, using the linux-kmod-compat port (in /usr/ports/devel/) to reuse as much as possible of the original source code, plus an userspace client consisting in a modified version of qemu, that uses KVM for the execution of its guests.
The porting has been completed, many of the limitations present at the end of the Summer of Code have been removed and the known bugs have been fixed. Some configurations have been tested, FreeBSD-CURRENT i386 guests have been booted on Intel and AMD processors, both in i386 and amd64 (host) installations. Only one client at a time is supported by now and performance is not that exciting, but the project seems to be ready to receive wider testing.
The Summer of Code project went well and we reached interesting results. At least the Mac Mini should be fully supported by now. Regarding the other Apple systems, we still need to polish some edges.
The project (started out as a GSoC 2007 project) aims to provide a complete Multicast DNS and Service Discovery suite. Much progress have been made since the last status report and the project is slowly reaching a usable state. Most features are complete and the current focus is on fixing outstanding bugs, fine tuning and testing. However, there are still a few open tasks (see below). More information and snapshots can be found at the wiki page.
New mpd-5.x branch has been started and first public release is planned soon. The main goal of the new branch is to implement new operation principles based on dynamic on-demand links/bundles creation. There are several benefits received from new design:
The network stack virtualization project aims at extending the FreeBSD kernel to maintain multiple independent instances of networking state. This allows for networking independence between jail-like environmens, each maintaining its private network interface set, IPv4 and IPv6 network and port address space, routing tables, IPSec configuration, firewalls, and more.
The prototype, which is kept in sync with FreeBSD -CURRENT, should be sufficiently stable for testing and experimental use. The project's web page includes weekly code snapshots, as well as a virtualized FreeBSD system installed on a VMWare disk image available for download.
The short-term goal is to deliver production-grade kernel support for virtualized networking for FreeBSD 7.0-RELEASE (as a snap-in kernel replacement), while continuing to keep the code in sync with -CURRENT for possible merging at a later date.
The GSoC2007/cnst-sensors project was about porting the sysctl hw.sensors framework from OpenBSD to FreeBSD. The project was successfully completed, committed into DragonFly BSD, and is now pending final review and integration into the FreeBSD's CVS tree (subject to the tree being unfrozen).
The sensors framework provides a unified interface for storing, registering and accessing information about hardware monitoring sensors. Sensor types include, but are not limited to, temperature, voltage, fan RPM, time offset and logical drive status. In the OpenBSD base system, the framework spans sensor_attach(9), sysctl(3), sysctl(8), systat(1), sensorsd(8), ntpd(8) and more than 50 drivers, ranging from I2C temperature sensors and Super I/O hardware monitors to IPMI and RAID controllers. Several third-party tools are also available, for example, a plug-in for Nagios and ports/sysutils/symon.
As a part of this Google Summer of Code project, all core components of the framework were ported, including sysctl, systat and sensorsd. Some drivers for the most popular Super I/O Hardware Monitors were ported, too: it(4), supporting most contemporary ITE Tech Super I/O, and lm(4), supporting most contemporary Winbond Super I/O. Moreover, some existing FreeBSD drivers were converted to utilise the framework, for example, coretemp(4).
The PC-BSD derivative of FreeBSD is becoming increasingly popular for new users of BSD. Much of the content in the existing FreeBSD Handbook is directly applicable to PC-BSD. We are writing PC-BSD specific installation and port/packages chapters (PBI). These chapters will be checked into docs/en_US.ISO8859-1/books/pcbsd-handbook and will include some of the same chapters as the Handbook does, but with a different &os entity and possibly with some conditional changes in those chapter files.
The ports count is over 17,700. The PR count has decreased a bit to just over 700.
There have been 6 experimental runs on the build cluster. The resulting commits include the fixup of last year's DESTDIR changes, the refactoring of perl bits into bsd.perl.mk, the update of xorg from 7.2 to 7.3, the upgrade of all of the autoconf dependencies to the latest version (wherever possible), and the upgrade of Python to 2.5. This effort has resulted in the fewest number of 'open' portmgr PRs in quite some time. portmgr appreciates all the people who worked with us on these patches, and people's patience as we catch up.
As well, lofi@ committed the upgrade of QT to 4.3.1.
We have added 3 new committers since the last report.
The freebsd-update front end is able to wait for freebsd-update to download a new set of patches to apply. It can then install and rollback the patches on either the local computer or over a SSH tunnel.
Since the end of the Summer of Code work has moved to BerliOS. The focus has been on writing tests for the front end, back end and communication library. The library has had tests written for most of it while the front and back ends have none.
The two most important parts of this Summer of Code projects have been accomplished.
The DESTDIR support for the Ports Collection has been rewritten to use a chrooted install. Now it is much more lightweight and easier to understand, but it works well for the most common cases, where it is supposed to be useful.
The Perl parts of the Ports Collection infrastructure have been extracted into an own module. At the same time, a new version handling has been invented. You can find more info on the Wiki.
Over the last couple of months several FreeBSD.org systems have been experiencing hardware issues. This included the main web-server www.FreeBSD.org which had a bad fan. The bad fan has been replaced so it should hopefully be stable again. In general we are working on replacing older hardware with newer systems and consolidating machine functions in the process.
Since August most FreeBSD.org services have been available via IPv6 with connectivity provided from ISC using a tunnel.
To honor the "Eat your own dog-food" principle the first two FreeBSD.org infrastructure systems have been upgraded to FreeBSD 7 and more are being upgraded as time permit.
Due to heavy load on the project's Perforce and CVS server the two services are being moved to seperate systems to improve performance of both CVS and Perforce.
