diff --git a/en_US.ISO8859-1/articles/wp-toolbox/Makefile b/en_US.ISO8859-1/articles/wp-toolbox/Makefile new file mode 100644 index 0000000000..d19a1dc514 --- /dev/null +++ b/en_US.ISO8859-1/articles/wp-toolbox/Makefile @@ -0,0 +1,19 @@ +# +# $FreeBSD$ +# +# Article: Casestudy about FreeBSD as a Software Testing Toolbox + +DOC?= article + +FORMATS?= html +WITH_ARTICLE_TOC?= YES + +INSTALL_COMPRESSED?= gz +INSTALL_ONLY_COMPRESSED?= + +SRCS= article.sgml + +URL_RELPREFIX?= ../../../.. +DOC_PREFIX?= ${.CURDIR}/../../.. + +.include "${DOC_PREFIX}/share/mk/doc.project.mk" diff --git a/en_US.ISO8859-1/articles/wp-toolbox/article.sgml b/en_US.ISO8859-1/articles/wp-toolbox/article.sgml new file mode 100644 index 0000000000..124b10ca09 --- /dev/null +++ b/en_US.ISO8859-1/articles/wp-toolbox/article.sgml @@ -0,0 +1,456 @@ + + + +%articles.ent; +]> + +
+ Creating a Software Testing Environment Using FreeBSD + + + + + Chris + McMahon + +
christopher.mcmahon@gmail.com
+ + + + + $FreeBSD$ + + + &tm-attrib.freebsd; + &tm-attrib.cvsup; + &tm-attrib.intel; + &tm-attrib.xfree86; + &tm-attrib.general; + + + A slightly altered version of this paper was first + published in the Pacific Northwest Software Quality Conference + Proceedings, 2005. + + + + + + Overview + + From late 2003 until early 2005, I was a tester in an + all-Windows environment. Although unlikely on the face of it, + FreeBSD became a valuable test tool platform in that context. + FreeBSD contains useful and powerful applications for any tester + in any environment. + + Unlike Linux, FreeBSD is a single monolithic project rather + than a collection of disparate parts assembled into a + distribution. And the most attractive part of FreeBSD for a + software tester is the FreeBSD ports collection--a very large, + managed set of software applications with a single simple and + uniform installation procedure. + + This paper describes several software test tools from the + FreeBSD ports collection that I used to test software and + systems in an all-Windows environment. + + + + + The Challenge + + Software testing environments are radically more complex + than software development environments. Interconnected systems + to test, network entities, databases, and filesystems present + challenges to testers that developers can for the most part mock + out and essentially ignore. Software testers need more tools, + and more complex tools, than do software developers. + + On the other hand, software development tools are much more + highly evolved than software testing tools. There is no Eclipse + or IntelliJ or even Visual Studio aimed at testing. Testers + struggle and scratch to find tools appropriate to their test + environments and appropriate to their Systems Under Test (SUTs). + + Every test environment is different. Finding appropriate + test tools is challenging. And when a tester has identified a + set of useful tools, introducing them to the test environment + and integrating them with the test environment is a challenge. + + + + + The FreeBSD Solution + + + Introduction + + The set of tools available with the FreeBSD Operating + System is amazing. The FreeBSD ports collection + contains more than thirteen thousand separate applications, + all of which have a standard installation procedure and + conform to a set of guidelines that make them reliable without + the need to manage dependencies, appropriate versions, and all + of the other problems that affect even the most well-managed + Linux distribution or the various versions of Microsoft + Windows. The monolithic nature of FreeBSD and the FreeBSD + ports collection removes much of the trouble of integrating + tools with the test environment, regardless of the OS under + which the SUT runs. FreeBSD is a highly evolved server + environment, and contains so many reliable applications, that + every tester should consider adding a FreeBSD machine (or + several) to their test environment. + + Of course, all of the applications available in the + FreeBSD ports collection won't be appropriate for any single + test environment. Some of the obvious choices for software and + systems testing are the six hundred or so system utilities, + the more than one thousand network tools, and the fifty-odd + benchmarking tools. Whether your test environment is Windows, + UNIX, Linux, Mac OS, FreeBSD itself, or some combination of + any of them, FreeBSD and the FreeBSD ports collection is a + great place to look first. + + + + + How To Use The Ports System + + Installing an application from the FreeBSD ports + collection is a simple matter of: + + &prompt.root; cd /usr/ports/foo +&prompt.root; make install + + and the system does the rest. It reports build status and + test status, and installs all the relevant documentation as + well. This aspect of FreeBSD is very attractive to a tester, + who typically is pressed for time! + + + + FreeBSD For Testing + + The test environment should be more stable than the SUT. + Once the tester decides to use the tools available on FreeBSD, + FreeBSD's long record of reliability makes it an easy choice + for a test tools platform. + + My own introduction to FreeBSD occurred when I was hired + by a major vendor of large-scale network security video + services to be their network testing person in an all Windows + environment. My first assignment was to replace the obsolete, + buggy, disk imaging system. I chose to do that with an Open + Source disk imaging system called Frisbee + which was implemented originally on FreeBSD. I built the + system--a feature-for-feature replacement for an expensive + proprietary system--but we never actually used it in our + production system. + + In the meantime, I had discovered the FreeBSD ports + collection and started to use some of those tools for testing; + and I had discovered the power of disk imaging with Frisbee, + especially for smoke testing and installation testing; and + FreeBSD became a permanent part of my test lab. The test lab + I built, and the FreeBSD systems I created still exist, and + still provide value to the testers there. + + + + + FreeBSD For Collaboration: Twiki + + A wiki is a simple set of web pages to allow many users to + share information and collaborate on any sort of documents. + Twiki is a wiki fine-tuned for + document management. It has built-in version control and + security implemented at the user/password level. I used it + for requirements management and traceability. + + The company I worked for managed their requirements in a + proprietary tool with a truly byzantine hierarchical + structure. Although the tool was fine for generating + requirements, it was horrible for testing. Also + unfortunately, the tool supported very few export mechanisms. + From the proprietary tool, I had to: + + + Export to Access + Export from Access to Excel + Export from Excel to a delimited file + Import the delimited file to the wiki by means of a Perl script + + + And in Twiki I was able to implement version control, + traceability, and test coverage, all features missing from the + proprietary tool. This gave me a remarkable ability to trace + requirements coverage with my tests. Having imported a + complex hierarchy of sets of requirements, I could use Twiki's + simple markup features to show each requirement as tested, + passed, or failed. I could attach test documentation to wiki + pages, and since I had already scripted the process, it was + easy to update the requirements as they changed. And Twiki + itself handled version control for such updates. + + As with all of the examples in this paper, installing + Twiki on FreeBSD is fairly simple. It takes just a few + minutes on a FreeBSD system. However, if you want to use Twiki + on a Microsoft Windows platform, I strongly suggest you read + the Twiki documentation extremely carefully. I know someone + who installed Twiki on Windows, and it took him several days. + Twiki on Windows requires not only knowledge of Windows, but + also deep knowledge of Cygwin and Perl. + + Furthermore, at one point in the project I had to migrate + my wiki from a machine running FreeBSD 4.8 to one running + FreeBSD 5.3. The migration consisted merely of installing + Twiki on FreeBSD 5.3; using tar on the FreeBSD 4.8 + machine to gather all of the Twiki data files specific to my + testing; FTPing the gathered files to the new FreeBSD 5.3 + machine; and untarring the file. The complete set of Twiki + documents migrated with no issues or problems at all. That's + the power of a unified system like FreeBSD. + + + + FreeBSD For Disk Imaging: Frisbee + + A disk imaging system is a mechanism for saving and + restoring all of the data on a physical disk. The most + popular commercial system for doing this is probably the + product Ghost from Symantec. + + The Frisbee enterprise disk imaging system mentioned above + had a lot of features I never implemented in the test lab. + Using Frisbee and an Open Source tool called PXELINUX, I was + able to: + + + Boot the Windows client machines from the network + Make an image of the client + Update the BIOS on the client + Lay down an existing image on the client machine + Make a set of restore CDs for the client + + + In the test lab, I only needed to boot from the Frisbee + CD, make an image, or lay down an image on the client machine. + Both Frisbee and proprietary imaging systems allow the user to + image individual drives on the client, but I never had a need + to do this. + + Installation testing was a large part of my duties at the + company where I used FreeBSD. To do this testing, I would + typically use Frisbee to make an image of a machine containing + only a Windows OS, install the SUT, and run a smoke test. The + smoke test typically left the test machine in a very bad + state. But instead of having to painstakingly clean up the + mess left by the failed installation, I simply re-imaged the + machine in question with the bare OS image and started over. + A typical re-image containing only the Windows OS and a few + test tools took less than three minutes. Using Frisbee, we + could run smoke tests on about six builds per day; before + Frisbee, we could run smoke tests on about three builds per + week. + + Of course, Ghost or other proprietary tools also image + machines quickly under these circumstances: once you buy the + tool, license the software, install it on an appropriate + server, and configure it properly. I prefer Frisbee to Ghost + because: Frisbee is marginally faster; Frisbee is very easy to + install on FreeBSD; and Frisbee is very efficient. Adding a + couple of small Perl scripts to the normal Frisbee + distribution gave me an imaging environment tailored for the + test lab. + + I also used Frisbee to preserve the state of a machine + after I had uncovered particularly complex defects. That is, + if it took a large effort (many steps and/or a long duration + of time) to demonstrate a defect, I could make an image of the + machine at the point at which the defect was visible, and + restore the image at will to demonstrate the defect to + developers or managers. + + + + FreeBSD Security Testing: Nessus + + Whenever you have more than one entity on a network, and + whenever you expose a server to the wider Internet, security + of the machine itself is always a concern. Nessus is an Open Source + remote vulnerability scanner for security and penetration + testing that consistently is rated among the top products of + its type throughout the security industry. + + Nessus probes a remote machine over the network for + security vulnerabilities. It does a port scan, finds which + ports are open, and investigates the software that has those + ports open for a huge number of security risks, for all major + OSs. It generates detailed reports in a number of formats + that anyone can understand. The number of security probes + available in the default installation of Nessus is very large, + but sophisticated security and penetration testers take + advantage of NASL, the Nessus Attack Scripting Language, to + craft their own attacks using Nessus' available features. + + Of interest is that, while Nessus is a free download for + UNIX-like systems (and is available in the ports collection of + FreeBSD), it is available on Windows only as a commercial + product from a company called Tenable. The Tenable product is + NeWT, Nessus on Windows Technology. + + + + FreeBSD Network Tools + + FreeBSD is most widely used as a robust server platform. + It follows, then, that tools related to network analysis and + performance will be highly evolved on FreeBSD. Here is a + brief description of network diagnostic tools that I found + invaluable in testing in a networked environment. + + From the name, one would assume that ntop emulates the functions of + the UNIX top command, but for the network + rather than for the local machine. Perhaps the first version + did; currently, ntop is capable of providing detailed + information about a huge number of hosts and their status and + activities on the network. + + For testing, two features I found very powerful: at a high + level, ntop shows the amount of network + traffic on the entire network segment minute-by-minute, + hour-by-hour, and day-by-day in a graphical format. Also, + ntop shows information about recent connections between + individual hosts on the network. + + It is easy to see traffic trends on the network as they + are occurring; also, if something anomalous appears, ntop + records detailed information about network connections between + hosts, including the ports over which the connection happened. + This was critically important when analyzing software + issues. If ntop showed a period of time for which traffic was + particularly high, I would find out which host was generating + the traffic. I would examine the software running on that + host, over that port. Often it was a new build with a + bug. + + Ettercap is + a tool for ARP poisoning which can also decipher passwords on + the fly and corrupt IP traffic by means of a Man In The Middle + (MITM) attack. However, I used ettercap as a performance tool. + In my test labs, all of my FreeBSD machines ran on discarded + hardware, Pentium II processors. I found that when I used + ettercap to sniff traffic between two hosts, the lack of + processing power caused ettercap on the slow MITM machine to + start dropping packets, making it look to the client machine + in the SUT as if there was interference or other trouble on + the network. And by varying the load on the FreeBSD machine, + I could in fact control the number of packets being dropped: + running ettercap and the UNIX yes utility + caused 100% packet loss. + + This was my most creative use of a FreeBSD tool for + testing. In a more straightforward application, any time a + tester needs to eavesdrop on traffic between two hosts on a + network, ettercap is an excellent choice because of its power + and ease of use. + + Perl gets a special mention here because Perl's network + utilities are outstandingly good compared to other + languages. Perl Net::* modules and + IO::Socket::* modules are robust and + powerful-- but they often fail to compile on Windows. It is + the ease of use of Perl's network utilities on FreeBSD that + gets Perl the mention in this section. + + I use Perl's network utilities to impersonate network + clients and servers for test purposes. On one occasion, I was + required to test software that was a client to an interface on + the New York Stock Exchange. Unfortunately, the NYSE test + server was down about nine days out of ten. I wrote a little + network server in Perl to emulate simple functions of the NYSE + server in order to test the client software. + + On another occasion, I had to test some functions on a web + server. It was not difficult to write a simple Perl HTTP + client to validate that the server was functioning properly. + + I have also used Perl to validate the output from a server + sending to a multicast address. I wrote a simple Perl + multicast client on FreeBSD to monitor the traffic on a + multicast address. Lincoln Stein's excellent + IO::Socket::Multicast module made it easy. + (Note: I never got IO::Socket::Multicast to + compile on Windows. I tried it on Windows 2000 and on Windows + XP.) + + + + + Conclusion + + I used tools from the FreeBSD ports collection in four + areas: in the network, where the operating system has very + little impact on how software behaves; for remote security + testing and performance testing in order to manipulate remote + machines over the network regardless of the operating system; + for disk imaging of Windows, Linux, and FreeBSD machines; and on + the webserver, where Twiki was my collaboration tool of choice. + + Because the installation procedure for all of these tools is + standard, I spent relatively little time installing and + configuring my tools. Because all the tools were hosted on a + single platform, I had all of my configuration and diagnostic + information located in just a few places. I kept all of my + potentially dangerous security tools on a single machine, which + made my presence on the network tolerable to the company's + network management staff. And the compatibility between FreeBSD + versions made it fairly simple to upgrade and to manage multiple + FreeBSD machines. And of course, I could rely on the + correctness of my test results, because the system itself is so + reliable. + + I have tried using Linux in a similar way, but my experience + is that package management quickly becomes tedious if not + overwhelming. The FreeBSD ports collection handled that for me. + And many of these tools are simply not available on Microsoft + Windows. And when they (or their equivalents) are available, + their cost, both financial and in terms of overhead, was simply + too high. + + FreeBSD's simple installation procedures and robust ports + collection makes it easy to experiment with the huge number of + tools available. I often find myself browsing the ports + collection looking for interesting applications to install, just + to see how they work. (I found ettercap by browsing the ports + collection, a tool that became very useful very quickly.) It + became clear that the more tools I used on FreeBSD, the more + economical became the management of those tools. + + The next time you need to reach into your toolbox for some + sophisticated, reliable, and powerful testing tools, I hope you + find them in FreeBSD. + + + + +
\ No newline at end of file