diff --git a/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml b/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml
index 805774c3d5..be27ff3f28 100644
--- a/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml
+++ b/en_US.ISO8859-1/htdocs/news/status/report-2013-07-2013-09.xml
@@ -1,1688 +1,1702 @@
This report covers &os;-related projects between July and
September 2013. This is the third of four reports planned for
2013. Thanks to all the reporters for the excellent work! This report
contains 30 entries and we hope you enjoy reading it. The deadline for submissions covering between October and
December 2013 is January 14th, 2014. An enhancement to the AES-NI implementation for OpenCrypto has
- been committed that significantly improves AES-XTS and AES-CBC
- decryption performance. This gives geli(8) around a
- three times performance boost on gnop(8) using AES-XTS
- compared to the old code. An enhancement to the AES-NI implementation for OpenCrypto, the
+ kernel's cryptography framework, has been committed that
+ significantly improves AES-XTS and AES-CBC decryption
+ performance. This gives geli(8) around a three times
+ performance boost on gnop(8) using AES-XTS compared to
+ the old code. These improvements are available to users of the OpenCrypto
framework and crypto(4). With our own (old and unstable) instance of Coverity Prevent
gone, we have now fully transitioned to the Scan project run by
Coverity (see links), which Open Source projects can use to
learn about possible defects in their source code. We also continue to run our code base through the Static
Analyzer that is shipped with Clang/LLVM. It cannot track the
state of the code over time, but has the benefit that everyone
can use it without any special setup. See the home page at the
links section for more information on the Clang Static Analyzer
project in general, and head over to the &os; Clang Static
Analyzer Scan page (see links) to see those possible defects (no
signup required). We are looking for a co-admin for both of these projects to
increase the bus-factor and the chance of survival for these
services. Fame and fortune await! Last year's high-performance storage vendor summit reported a
performance bottleneck in the &os; block storage subsystem,
limiting peak performance to around 300-500K IOPS. While that
is still more than enough for average systems, detailed
investigation has shown a number of places that require radical
improvement. The unmapped I/O support implemented early this
year has already improved I/O performance by about 30% and moved
more focus toward GEOM and CAM subsystems scalability. Fixing
these issues was the goal of this project. The existing GEOM design assumed most I/O handling was to be
done by only two kernel threads (g_up() and
g_down()). That simplified locking in some cases, but
limited potential SMP scalability and created additional
scheduler overhead. This project introduces the concept of
direct I/O dispatch into GEOM for cases where it is known to be
safe and efficient. That implies marking some GEOM consumers
and providers with one or two new flags, declaring situations
when a direct function call can be used instead of normal
request queuing. That permits avoiding any context switches
inside GEOM for the most widely used topologies, simultaneously
processing multiple I/Os from multiple calling threads. Having GEOM pass through multiple concurrent calls down to the
underlying layers exposed major lock congestion in CAM. In the
existing CAM design, all devices connected to the same ATA/SCSI
controller share a single lock, which can be quite busy due to
multiple controller hardware accesses and/or code logic.
Experiments have shown that applying only the above GEOM direct
dispatch changes burns up to 60% of system CPU time or even more
in attempts to obtain these locks by multiple callers, killing
- any benefits of GEOM direct dispatch. To overcome that, a new
- fine-grained CAM locking design was implemented. It implies
- splitting the big per-SIM locks into several smaller ones:
- per-LUN locks, per-bus locks, queue locks, etc. After these
- changes, the remaining per-SIM lock protects only the controller
- driver internals, reducing lock congestion down to an acceptable
- level and keeping compatibility with existing drivers.
To overcome this scaling limitation, a new fine-grained CAM + locking design was implemented. It implies splitting the big + per-SIM locks into several smaller ones: per-LUN locks, per-bus + locks, queue locks, etc. After these changes, the remaining + per-SIM lock protects only the controller driver internals, + reducing lock congestion down to an acceptable level and keeping + compatibility with existing drivers.
Together, the GEOM and CAM changes double the peak I/O rate, reaching up to 1,000,000 IOPS on contemporary hardware.
The changes were tested by a number of people and will be committed into &os; head and merged to stable/10 after the end of the &os; 10.0 release cycle.
The project is sponsored by iXsystems, Inc.
A port of the OpenBSD vmx(4) ethernet driver for VMware virtual machines has been committed. The driver can be used in place of the VMware Tools vmxnet3 driver, which currently does not support 10.0-RELEASE (or anything past 9.0-RELEASE).
The VirtIO network driver, vtnet(4), recently gained - support for multiple queues, along with a significant cleanup - and support for a few additional features.
+The VirtIO network driver, vtnet(4), is used by &os; + systems running on hypervisors including bhyve(4) and + Linux's KVM. It recently gained support for multiple queues, + along with a significant cleanup and support for a few + additional features.
We are currently working on cleaning up the lang/python* ports to improve their compatibility with the original upstream build behaviour and to reduce the need for &os;-specific build patches. A first step was made in September by reducing the flags injected into the different Python interpreter versions.
The first tasks have been completed to support the installation of packages for different Python ports. A new metaport structure has replaced the original Python port behaviour, and will be enhanced over the next months to enable improved installation support of packages for different Python versions at the same time.
The Python ports framework was enhanced with automated packaging list creation and replacement macros, which improve the compatibility with multiple Python versions and reduce the packaging list sizes.
PyPy was heavily enhanced over the last couple of months. Major updates to the port solved integration issues and a new pypy-devel port for snapshots and previews was added. Since the PyPy 3 release, there is a new pypy3-devel port available to provide not only compatibility for Python 2.x specific scripts, but also for those using the 3.x language specification.
IronPython found its way into the &os; ports tree, providing an implementation of the Python language based on .NET and Mono.
The entities project branch has been successfully merged into the main documentation branch per revision 42226 of the doc repository (see link). The purpose of this branch was to remove the duplicated definitions of authors in both authors.ent and developers.ent. The latter file has been removed after migrating its contents to the former file. While most changes are not visible to end users, the Committer's Guide was changed to accomodate for changes related to adding a new committer. Translators were also informed of the update. The largest hurdle mentioned in the last report, processing the <email> element, was solved with the help of Gábor Kövesdán.
The &os; Release Engineering Team has completed the 9.2-RELEASE process. The release cycle changed with a last-minute addition of 9.2-RC4. The 9.2-RELEASE was announced September 30, four weeks behind the original schedule.
The &os; 10.0-RELEASE cycle has started, and testing is strongly encouraged. For testing purposes, both installation images and virtual machine images exist on the &os; Project FTP servers.
This is a Google Summer of Code 2013 project that aims to replace the fetch(1)-based method for getting distribution files, such as source tarballs, for the third-party applications (ports) with an intelligent Download Manager Service (see links for more information).
All the modules highlighted in the project wiki have been completed (see links). Specifically:
A few years ago, Ada-based ports almost completely disappeared from the Ports Collection. This was not surprising, as FSF GNAT, the only open-source Ada compiler, ceased to build correctly on any BSD flavor. Previously-built bootstrap compilers would not run on modern &os;, and certainly not on amd64. The first step, see the link for details, was to patch GCC in order to fix GNAT not only on &os;, but DragonFly, NetBSD, and OpenBSD as well. New bootstraps for both i386 and amd64 platforms were produced during this effort. Ada compilers on &os; now pass 100% of the ACATS and GCC testsuites.
With the introduction of the first new Ada compiler port, the GCC 4.6-based lang/gnat-aux, the GNAT Programming Studio (a multilanguage integrated development environment), XML/Ada, and GtkAda were among the first Ada ports resurrected.
With the latest compiler, lang/gcc-aux based on GCC 4.7, a cohesive Ada framework was created with the new USES= framework. Currently around 20 ports are part of this framework including Florist, ASIS, GPRbuild, QtAda, AdaControl, AdaBrowse, PolyOrb, and AWS (Ada Web Server).
The GNAT AUX compiler is also still in use to serve as a basis for the GNATDroid ports which are &os;-to-Android Ada+C cross-compilers. However, these will soon be integrated into the Ada Framework.
At this point, it looks like &os; (shared with DragonFly via DPorts) has taken the crown from Debian as the recognized best Ada development platform. The &os; versions of the software are more recent and the Ports Collection has ports not available on Debian, such as LibSparkCrypto, the Matreshka library, and the Ahven unit tester.
Future work potentially includes converting GCC AUX to GCC 4.8 to acquire better Ada 2012 support, importing Spark 2014 into ports when it arrives and to continue to add new Ada ports to the framework.
Initial support of Allwinner A20 SoC is committed to head. The A20 SoC on Cubieboard2 is pin-to-pin compatible with the A10 in Cubieboard1 and &os; supports the following peripherals:
&os; images are available for use in EC2 for 8.3-RELEASE, 8.4-RELEASE, 9.0-RELEASE, 9.1-RELEASE, and 9.2-RELEASE. In 9.2-RELEASE, &os; runs in EC2 using an unpatched source tree, but it needs the XENHVM kernel configuration.
Starting from &os; 10.0-ALPHA3, the GENERIC kernel configuration now contains all the XENHVM bits needed to allow &os; to run in EC2 natively. Consequently, &os; 10.0 will be the first release for which &os;/EC2 is purely "bits off the ISO". This also means that starting with 10.0 it will be possible to use freebsd-update(8) for all base system updates — in earlier releases it was necessary to recompile the XENHVM kernel manually.
Due to &os;'s use of HVM virtualization, running on "old" EC2 instance types (m1, m2, c1, t1) requires that &os; pretends to be Windows, which unfortunately results in paying the higher "windows" EC2 instance prices. On "new" EC2 instances (cc1, cc2, cg1, cr1, hi1, hs1, and m3) &os; can run as a "unix" image at the lower rate.
In the third quarter of 2013, the &os; Postmaster Team has implemented the following items that may be interest of the general public:
The ARM architecture is becoming more and more prevalent, with increasing usage beyond the mobile and embedded space. Among the more interesting industry trends emerging in the recent months, there has been the concept of "ARM server". Top-tier companies like Dell and HP have already started to develop such systems.
Key to the success of &os; in these new areas is dealing with the sophisticated features of the platform, for example adding support for superpages.
The objective of this project is to enable &os;/arm to utilize superpages, allowing efficient use of TLB translations (by enlarging TLB coverage), leading to improved performance in many applications and scalability. This is intended to work on ARMv7-based processors, however compatibility with ARMv6 will be preserved.
The following steps have been made since the last status report:
This project is jointly sponsored by The &os; Foundation and Semihalf.
Starting with &os; 10.0-ALPHA4, the projects/pseries branch has been merged into &os; head. This allows &os;/powerpc64 to run in an IBM POWER logical partition and on certain classes of older IBM-type PowerPC hardware.
Due to the quickly approaching time of 10.0-RELEASE, the priorities for the native iSCSI stack shifted somewhat, from performance optimizations to making sure the new stack is reliable, feature-complete, and is able to interoperate correctly with various implementations. Plenty of time was invested into testing and debugging, mostly on the initiator side, to make sure it works correctly with other targets, such as Solaris COMSTAR, and behaves properly in edge conditions like connection problems. Nevertheless, some fundamental optimizations, such as Immediate Data support, were implemented. The documentation has improved, and there will be a new section added to the &os; Handbook describing the use of the new stack.
The new stack was committed to head and will ship as part of 10.0-RELEASE. There is ongoing work on fixing issues reported by early adopters.
This project is being sponsored by The &os; Foundation.
SDIO is an interface designed as an extension of the existing SD card standard, to allow connecting different peripherals to the host with the standard SD controller. Peripherals currently sold at the general market include WLAN/BT modules, cameras, fingerprint readers, and barcode scanners. The driver is implemented as an extension to the existing MMC bus, adding a lot of new SDIO-specific bus methods. A prototype of the driver for the Marvell SDIO WLAN/BT (Avastar 88W8787) module is also being developed, using the existing Linux driver as the reference.
SDIO card detection and initialization already work, most needed bus methods are implemented and tested. There is an ongoing work to design a good locking model for the stack. The WiFi driver is able to load firmware onto the card and initialize it.
If threads or signal handlers call fork() and exec(), file descriptors may be passed undesirably to child processes, which may lead to hangs (if a pipe is not closed), exceeding the file descriptor limit, and security problems (if the child process has lower privilege). One solution is various new APIs that set the "close-on-exec" flag atomically with allocating a file descriptor. Some existing software will use the new features if present or will even refuse to compile without them.
With mkostemp(), dup3(), and a change to modes of fopen() and freopen(), everything proposed in Austin Group issue #411 has now been implemented. For all POSIX-specified functions that allocate file descriptors, it is possible to request that the new descriptor be set close-on-exec atomically.
Additionally, many file descriptors used internally by libc and libutil now have the close-on-exec bit set.
Random numbers require a lot more thought and preparation than would naively appear to be the case. For simulations, number sequences that are repeatable but sufficiently disordered are often needed to achieve required experimental duplication ability, and many programmers are familiar with these. For cryptography, it is essential that an attacker not be able to predict or guess the output sequence, thus giving a source of security-critical secret material for uses such as passwords or "key material".
&os;'s random number generator, available as the pseudo-file /dev/random produces unpredictable numbers intended for cryptographic use, and is thus a Cryptograpically-Secured Pseudo-Random Number Generator, or CSPRNG. The security is given by careful design of the output generator (based on a block cipher) and input entropy accumulation queues. The latter uses hashes to accumulate stochastic information harvested from various places in the kernel to provide highly unpredictable input to the generator. The algorithm for doing this, Yarrow, by Schneier et al, may be found by web search.
&os;'s CSPRNG also allowed for certain stochastic sources, deemed to be "high-quality", to directly supply the random(4) device without going through Yarrow. With recent revelations over possible government surveillance and involvement in the selection of these "high-quality" sources, it is felt that they can no longer be trusted, and must therefore also be processed though Yarrow.
The matter was discussed at various levels of formality at the Cambridge Developer Summit in August, and at EuroBSDcon 2013 in September.
This work is now done, and the random(4) CSPRNG is now brought to a more paranoid, modern standard of distrust with regard to its entropy sources. Infrastructure work was also done to facilitate certain entropy-source choices for the convenience of the system administrators.
Future work is now going ahead with the implementation of the Fortuna algorithm by Ferguson and Schneier as an upgrade or alternative to Yarrow. Initially a choice will be presented, and decisions on the future of the CSPRNG processing algorithms in use will be made in the future as needs arise.
GNUstep is the open source implementation of the Objective-C APIs based on the OpenStep specification that Apple brands as Cocoa. The similarities between the &os; and OS X libc make &os; an attractive target platform for porting OS X applications, with the addition of GNUstep.
The GNUstep ports in &os; have now been updated to the latest releases and now build with the GNUstep Objective-C runtime and Clang 3.3, with the non-fragile ABI by default. This means that all of the modern features of Objective-C are supported, including Automatic Reference Counting (ARC) and recent syntax improvements.
The devel/gnustep meta-port will install all of the core GNUstep libraries, ready for development. The x11/gnustep-app meta-port will install all of the GNUstep-based applications and libraries currently in the ports tree. Many of these are old and not well-tested with later GNUstep release, so consider them experimental at present. We are currently working on updating them, including moving from some abandoned upstream locations to the GNUstep Applications Project (GAP), which has taken over maintenance of a number of older GNUstep programs.
LLDB is the debugger project in the LLVM family. It supports the Mac OS X, Linux, and &os; platforms.
A number of improvements have been made to the port since the previous status update. Unit test failures have been triaged and have defects entered in LLDB's bug tracker. In combination with the lldb buildbot this allows for the quick identification of new failures introduced by other ongoing development. Core file support has also been added.
An LLDB snapshot has been imported into the &os; base system and is available as of SVN revision 255722. It is not yet built by default but may be enabled by adding WITH_LLDB= to src.conf(5).
This project is sponsored by DARPA/AFRL in collaboration with SRI International and the University of Cambridge.
The &os; Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the &os; Project and community worldwide. Most of our funding is used to support &os; development projects, conferences and developer summits, purchase equipment to grow and improve the &os; infrastructure, and provide legal support for the Project.
We listened to our donors who asked us to have more fundraising efforts throughout the year. This quarter we had the second of three fundraising campaigns planned for 2013. We started the quarter having raised $365,291. By the end of the quarter, we raised $410,000 for the year. These early donations have made a significant impact on our fundraising efforts this year.
Some of the highlights from this past quarter include:
Capsicum is the &os; sandboxing subsystem, which presents + programmers with a capability module allowing fine-grained + delegation of rights to less-privileged processes. Casper is a + friendly daemon that provides services to sandboxed processes, + allowing policy-based access to privileged services such as DNS + resolution.
+The work on Capsicum and related projects (such as Casper, libnv, etc.) is progressing nicely. An overhaul of the cap_rights_t was committed to &os; head and will be included in 10.0. This allows us to have more capability rights on file descriptors than the previous limit of 64 rights, which was almost reached. This change is not backward compatible, so it was very important to get it into 10.0.
libnv, used for communication between Casper services and consumers, but which will hopefully be used more widely, is finalized and comes with a nice set of regression tests.
The number of applications sandboxed using the Capsicum framework is increasing. We have around 10 of them already in base and more that are not yet committed.
This project is being sponsored by the &os; Foundation.
Glib 2.36 and Gtk 3.8 were imported into the ports tree. The GNOME Team is currently working on improving the quality of GNOME 3.6. The version of multimedia/cheese shipped with GNOME 3 is now able to use devd(8) to find the camera through multimedia/webcamd. Several build improvements have been made to the www/webkit-gtk3 port, however it still is rather fragile.
-MATE is about ready to go in.
+MATE, a desktop environment forked from the now-unmaintained + codebase of GNOME 2, is about ready to go in.
GNOME 2 will be removed at some point in the near future. How or when this will happen is not yet clear.
The &os; Documentation Project Primer had not changed at the same rate as the documents themselves. Some sections were - outdated and others were wordy and confusing, while information on - new changes to the documentation were not described at all. In - July, Warren gave the entire FDP Primer a fairly intense edit - for simplicity and clarity. Chapters and sections were moved - into a more logical order, and information was updated to be a - better guide to the current state. Markup examples were added - and revised. Style guidelines were also extended and updated. - The Primer is now far more consistent and usable. As always, - there is still room for improvement, and additions or - corrections are encouraged.
+ outdated and others were verbose and confusing, while + information on new changes to the documentation were not + described at all. In July, Warren gave the entire FDP Primer a + fairly intense edit for simplicity and clarity. Chapters and + sections were moved into a more logical order, and information + was updated to be a better guide to the current state. Markup + examples were added and revised. Style guidelines were also + extended and updated. The Primer is now far more consistent and + usable. As always, there is still room for improvement, and + additions or corrections are encouraged.There are some things going on with the &os;/sparc64 port - behind the scenes.
- -For one, after having fixed all remaining problems and starting - with 9.2-RELEASE, release bits for this architecture are - cross-built on the &os; Project cluster. As one might already - have noticed, this means that from now on, sparc64 install sets - and images including those for ALPHA, BETA, and RC builds, are - already available alongside those for the other platforms - supported by &os;. It also means that since August 2013, - automatically cross-built monthly &os;/sparc64 snapshots are - distributed via the official project mirrors. Hopefully, this - can soon be extended further with freebsd-update(8) - support for sparc64.
- -Second, the X.Org ports have been fixed to work on sparc64 when - built with the WITH_NEW_XORG knob. However, it still - needs to be evaluated whether the recently committed update to - Mesa 9.1.6 has introduced any breakage.
+There are several things going on with the &os;/sparc64 + port.
+ +After having fixed all remaining problems and starting with + 9.2-RELEASE, releases for this architecture are cross-built on + the &os; Project cluster. As one might already have noticed, + this means that from now on, sparc64 install sets and images + including those for ALPHA, BETA, and RC builds, are available + alongside those for the other platforms supported by &os;. + Since August 2013, automatically cross-built monthly + &os;/sparc64 snapshots are distributed via the official project + mirrors. Hopefully, this can soon be extended further with + freebsd-update(8) support for sparc64.
+ +The X.Org ports have been fixed to work on sparc64 when built + with the WITH_NEW_XORG knob. However, it still needs + to be evaluated whether the recently committed update to Mesa + 9.1.6 has introduced any breakage.
The ports tree contains approximately 24,400 ports, while the PR count exceeds 1,900. In the third quarter, we added four new committers and took in six commit bits for safekeeping.
A significant amount of effort has gone into tweaking and manipulating the infrastructure to modernize and update it, in preperation for pkg(8) replacing the old pkg_add(1) infrastructure, as well as preparing for - &os; with Clang as default compiler.
+ &os; 10.0 with Clang as default compiler, libc++ + as the default C++ standard library, and iconv(1) + integrated into libc. -Automated procedures for Quality Assurance have been +
Automated procedures for quality assurance have been implemented, notably pkg-fallout. All porters are encouraged to subscribe to the associated mailing list (see links), and do their part to fix ports for pkg(8) and Clang readiness.
Many iterations of tests were run to ensure that as many packages as possible would be available for the 9.2 release.
In the third quarter of 2013, the Core Team focused on officially launching pkg.freebsd.org, the Project's official pkg(8) repository, in cooperation with the Port Management Team, the Security Team, and the Cluster Administration Team. At the same time, there are plans to gradually deprecate the use of the old pkg_add(1), allowing pkg(8) to be the default binary package management solution for &os;, arriving with 10.0-RELEASE. Thomas Abthorpe has been appointed to the role of liaison between the Core Team and the Ports Management Team, in order to make the collaboration more effective.
David Chisnall has joined the group that publishes the Quarterly Status reports and compiled a special status report on the results of the BSDCan 2013 Developer Summit. David also took the lead role on the organization of an off-season developer summit in Cambridge, UK, which was finally held at the end of August. For the items discussed in Cambridge, preparation of a detailed report is still in progress.
There were src commit bits issued for 5 new developers and most of the src commits being idle more than 12 months have been taken into safekeeping as result of a major cleanup to the repository access file in July, performed by Gavin Atkinson.
Mesa 9.1 (libGL and dri) was updated in ports. This includes experimental ports for libEGL and libgles2: they are dependencies of the experimental ports for Wayland and Weston.
The radeonkms driver was committed to &os; head in the end of August and will be part of 10.0-RELEASE. It received several fixes since the initial commit and now seems quite stable. However, one missing major feature is support for suspend/resume: the GPU almost always locks up during resume on the test computer.
Thanks to the update of Mesa and the update of x11-drivers/xf86-video-ati to 7.2.0 in the ports tree, every pieces are in place to allow users to use recent AMD video cards (up to HD7000, maybe some HD8000).
The driver will now only receive bug fixes and focus will move on the update of the DRM generic code and the i915 driver.
The generic DRM code, shared by the i915kms and radeonkms video drivers is quite old now. Work has started to update and sync it with that of Linux 3.8. This code is available on GitHub.
The expected benefits are:
François Tigeot from DragonFly is also working on updates to their DRM code, and the X11 team is planning to share the effort.
An experimental devd(8) backend was added to the x11-servers/xorg-server port. This allows X.Org to use devd(8) to detect and configure input devices (for example, keyboards and mices) dynamically.
Our current wiki articles are used to describe projects and report status. However, they lack some consistency and links between them. We started to think about reorganizing them to:
Nothing is visible yet on the wiki.
The Newcons project is aimed to replace the old syscons(4)-based virtual terminals. The main objectives are: support Unicode characters, and move away from the dependency on fixed VGA and VESA graphics modes and built-in BIOS services.
This project was originally started by Ed Schouten, and it already featured the following features (among many others) in 2013:
And these have been extended by the following items recently:
Supported startup modes for KMS:
This project is being sponsored by The &os; Foundation. Many thanks to Ed Schouten, who started the Newcons project and did most of the work.