         Creating a Software Testing Environment Using FreeBSD

  Chris McMahon

   <christopher.mcmahon@gmail.com>

   $FreeBSD: doc/en_US.ISO8859-1/articles/wp-toolbox/article.sgml,v
   1.7 2012/01/11 01:21:58 eadler Exp $

   FreeBSD is a registered trademark of the FreeBSD Foundation.

   CVSup is a registered trademark of John D. Polstra.

   Intel, Celeron, EtherExpress, i386, i486, Itanium, Pentium, and
   Xeon are trademarks or registered trademarks of Intel Corporation
   or its subsidiaries in the United States and other countries.

   Microsoft, IntelliMouse, MS-DOS, Outlook, Windows, Windows Media
   and Windows NT are either registered trademarks or trademarks of
   Microsoft Corporation in the United States and/or other countries.

   Symantec and Ghost are registered trademarks of Symantec
   Corporation in the United States and other countries.

   Many of the designations used by manufacturers and sellers to
   distinguish their products are claimed as trademarks. Where those
   designations appear in this document, and the FreeBSD Project was
   aware of the trademark claim, the designations have been followed
   by the "(TM)" or the "(R)" symbol.

   A slightly altered version of this paper was first published in
   the Pacific Northwest Software Quality Conference Proceedings,
   2005.

     --------------------------------------------------------------

1 Overview

   From late 2003 until early 2005, I was a tester in an
   all-Windows(R) environment. Although unlikely on the face of it,
   FreeBSD became a valuable test tool platform in that context.
   FreeBSD contains useful and powerful applications for any tester
   in any environment.

   Unlike Linux, FreeBSD is a single monolithic project, rather than
   a collection of disparate parts assembled into a distribution. And
   the most attractive part of FreeBSD for a software tester is the
   FreeBSD Ports Collection--a very large, managed set of software
   applications with a single, simple, and uniform installation
   procedure.

   This paper describes several software test tools from the FreeBSD
   Ports Collection that I used to test software and systems in an
   all-Windows environment.

     --------------------------------------------------------------

2 The Challenge

   Software testing environments are radically more complex than
   software development environments. Interconnected systems to test,
   network entities, databases, and filesystems present challenges to
   testers that developers can, for the most part, mock out and
   essentially ignore. Software testers need more tools, and more
   complex tools, than do software developers.

   On the other hand, software development tools are much more highly
   evolved than software testing tools. There is no Eclipse, or
   IntelliJ, or even Visual Studio aimed at testing. Testers struggle
   and scratch to find tools appropriate to their test environments
   and appropriate to their Systems Under Test (SUTs).

   Every test environment is different. Finding appropriate test
   tools is challenging. And when a tester has identified a set of
   useful tools, introducing them to the test environment and
   integrating them with the test environment is a challenge.

     --------------------------------------------------------------

3 The FreeBSD Solution

  3.1 Introduction

   The set of tools available with the FreeBSD Operating System is
   amazing. The FreeBSD Ports Collection contains more than thirteen
   thousand separate applications, all of which have a standard
   installation procedure and conform to a set of guidelines that
   make them reliable without the need to manage dependencies,
   appropriate versions, and all of the other problems that affect
   even the most well-managed Linux distribution or the various
   versions of Microsoft Windows. The monolithic nature of FreeBSD
   and the FreeBSD Ports Collection removes much of the trouble of
   integrating tools with the test environment, regardless of the OS
   under which the SUT runs. FreeBSD is a highly evolved server
   environment, and contains so many reliable applications, that
   every tester should consider adding a FreeBSD machine (or several)
   to their test environment.

   Of course, all of the applications available in the FreeBSD Ports
   Collection will not be appropriate for any single test
   environment. Some of the obvious choices for software and systems
   testing are the six hundred or so system utilities, the more than
   one thousand network tools, and the fifty-odd benchmarking tools.
   Whether your test environment is Windows, UNIX, Linux, Mac OS,
   FreeBSD itself, or some combination of any of them, FreeBSD and
   the FreeBSD Ports Collection is a great place to look first.

     --------------------------------------------------------------

  3.2 How To Use The Ports System

   Installing an application from the FreeBSD Ports Collection is a
   simple matter of:

 # cd /usr/ports/foo
 # make install

   and the system does the rest. It reports build status and test
   status, and installs all the relevant documentation as well. This
   aspect of FreeBSD is very attractive to a tester, who typically is
   pressed for time!

     --------------------------------------------------------------

  3.3 FreeBSD For Testing

   The test environment should be more stable than the SUT. Once the
   tester decides to use the tools available on FreeBSD, FreeBSD's
   long record of reliability makes it an easy choice for a test
   tools platform.

   My own introduction to FreeBSD occurred when I was hired by a
   major vendor of large-scale network security video services to be
   their network-testing person in an all-Windows environment. My
   first assignment was to replace the obsolete, buggy, disk imaging
   system. I chose to do that with an Open Source disk imaging system
   called Frisbee which was implemented originally on FreeBSD. I
   built the system--a feature-for-feature replacement for an
   expensive proprietary system--but we never actually used it in our
   production system.

   In the meantime, I had discovered the FreeBSD Ports Collection and
   started to use some of those tools for testing; and I had
   discovered the power of disk imaging with Frisbee, especially for
   smoke testing and installation testing; and FreeBSD became a
   permanent part of my test lab. The test lab I built, and the
   FreeBSD systems I created still exist, and still provide value to
   the testers there.

     --------------------------------------------------------------

  3.4 FreeBSD For Collaboration: Twiki

   A wiki is a simple set of web pages to allow many users to share
   information and collaborate on any sort of documents. Twiki is a
   wiki fine-tuned for document management. It has built-in version
   control and security implemented at the user/password level. I
   used it for requirements management and traceability.

   The company I worked for managed their requirements in a
   proprietary tool with a truly byzantine hierarchical structure.
   Although the tool was fine for generating requirements, it was
   horrible for testing. Also unfortunately, the tool supported very
   few export mechanisms. From the proprietary tool, I had to:

     * Export to Access

     * Export from Access to Excel

     * Export from Excel to a delimited file

     * Import the delimited file to the wiki by means of a Perl
       script

   And in Twiki I was able to implement version control,
   traceability, and test coverage, all features missing from the
   proprietary tool. This gave me a remarkable ability to trace
   requirements coverage with my tests. Having imported a complex
   hierarchy of sets of requirements, I could use Twiki's simple
   markup features to show each requirement as tested, passed, or
   failed. I could attach test documentation to wiki pages, and since
   I had already scripted the process, it was easy to update the
   requirements as they changed. And Twiki itself handled version
   control for such updates.

   As with all of the examples in this paper, installing Twiki on
   FreeBSD is fairly simple. It takes just a few minutes on a FreeBSD
   system. However, if you want to use Twiki on a Microsoft Windows
   platform, I strongly suggest you read the Twiki documentation
   extremely carefully. I know someone who installed Twiki on
   Windows, and it took him several days. Twiki on Windows requires
   not only knowledge of Windows, but also deep knowledge of Cygwin
   and Perl.

   Furthermore, at one point in the project I had to migrate my wiki
   from a machine running FreeBSD 4.8 to one running FreeBSD 5.3. The
   migration consisted merely of installing Twiki on FreeBSD 5.3;
   using tar on the FreeBSD 4.8 machine to gather all of the Twiki
   data files specific to my testing; FTPing the gathered files to
   the new FreeBSD 5.3 machine; and untarring the file. The complete
   set of Twiki documents migrated with no issues or problems at all.
   That is the power of a unified system like FreeBSD.

     --------------------------------------------------------------

  3.5 FreeBSD For Disk Imaging: Frisbee

   A disk imaging system is a mechanism for saving and restoring all
   of the data on a physical disk. The most popular commercial system
   for doing this is probably the product Ghost(TM) from Symantec.

   The Frisbee enterprise disk imaging system mentioned above had a
   lot of features I never implemented in the test lab. Using Frisbee
   and an Open Source tool called PXELINUX, I was able to:

     * Boot the Windows client machines from the network

     * Make an image of the client

     * Update the BIOS on the client

     * Lay down an existing image on the client machine

     * Make a set of restore CDs for the client

   In the test lab, I only needed to boot from the Frisbee CD, make
   an image, or lay down an image on the client machine. Both Frisbee
   and proprietary imaging systems allow the user to image individual
   drives on the client, but I never had a need to do this.

   Installation testing was a large part of my duties at the company
   where I used FreeBSD. To do this testing, I would typically use
   Frisbee to make an image of a machine containing only a Windows
   OS, install the SUT, and run a smoke test. The smoke test
   typically left the test machine in a very bad state. But instead
   of having to painstakingly clean up the mess left by the failed
   installation, I simply re-imaged the machine in question with the
   bare OS image and started over. A typical re-image containing only
   the Windows OS and a few test tools took less than three minutes.
   Using Frisbee, we could run smoke tests on about six builds per
   day; before Frisbee, we could run smoke tests on about three
   builds per week.

   Of course, Ghost or other proprietary tools also image machines
   quickly under these circumstances: once you buy the tool, license
   the software, install it on an appropriate server, and configure
   it properly. I prefer Frisbee to Ghost because: Frisbee is
   marginally faster; Frisbee is very easy to install on FreeBSD; and
   Frisbee is very efficient. Adding a couple of small Perl scripts
   to the normal Frisbee distribution gave me an imaging environment
   tailored for the test lab.

   I also used Frisbee to preserve the state of a machine after I had
   uncovered particularly complex defects. That is, if it took a
   large effort (many steps and/or a long duration of time) to
   demonstrate a defect, I could make an image of the machine at the
   point at which the defect was visible, and restore the image at
   will to demonstrate the defect to developers or managers.

     --------------------------------------------------------------

  3.6 FreeBSD Security Testing: Nessus

   Whenever you have more than one entity on a network, and whenever
   you expose a server to the wider Internet, security of the machine
   itself is always a concern. Nessus is an Open Source remote
   vulnerability scanner for security and penetration testing that
   consistently is rated among the top products of its type
   throughout the security industry.

   Nessus probes a remote machine over the network for security
   vulnerabilities. It does a port scan, finds which ports are open,
   and investigates the software that has those ports open for a huge
   number of security risks, for all major OSs. It generates detailed
   reports in a number of formats that anyone can understand. The
   number of security probes available in the default installation of
   Nessus is very large, but sophisticated security and penetration
   testers take advantage of NASL, the Nessus Attack Scripting
   Language, to craft their own attacks using Nessus' available
   features.

   Of interest is that, while Nessus is a free download for UNIX-like
   systems (and is available in the Ports Collection of FreeBSD), it
   is available on Windows only as a commercial product from a
   company called Tenable. The Tenable product is NeWT, "Nessus on
   Windows Technology".

     --------------------------------------------------------------

  3.7 FreeBSD Network Tools

   FreeBSD is most widely used as a robust server platform. It
   follows, then, that tools related to network analysis and
   performance will be highly evolved on FreeBSD. Here is a brief
   description of network diagnostic tools that I found invaluable in
   testing in a networked environment.

   From the name, one would assume that ntop emulates the functions
   of the UNIX top(1) command, but for the network rather than for
   the local machine. Perhaps the first version did; currently, ntop
   is capable of providing detailed information about a huge number
   of hosts and their status and activities on the network.

   For testing, two features I found very powerful: at a high level,
   ntop shows the amount of network traffic on the entire network
   segment minute-by-minute, hour-by-hour, and day-by-day in a
   graphical format. Also, ntop shows information about recent
   connections between individual hosts on the network.

   It is easy to see traffic trends on the network as they are
   occurring; also, if something anomalous appears, ntop records
   detailed information about network connections between hosts,
   including the ports over which the connection happened. This was
   critically important when analyzing software issues. If ntop
   showed a period of time for which traffic was particularly high, I
   would find out which host was generating the traffic. I would
   examine the software running on that host, over that port. Often
   it was a new build with a bug.

   Ettercap is a tool for ARP poisoning which can also decipher
   passwords on the fly and corrupt IP traffic by means of a Man In
   The Middle (MITM) attack. However, I used Ettercap as a
   performance tool. In my test labs, all of my FreeBSD machines ran
   on discarded hardware, Pentium II processors. I found that when I
   used Ettercap to sniff traffic between two hosts, the lack of
   processing power caused Ettercap on the slow MITM machine to start
   dropping packets, making it look to the client machine in the SUT
   as if there was interference or other trouble on the network. And
   by varying the load on the FreeBSD machine, I could in fact
   control the number of packets being dropped: running Ettercap and
   the UNIX yes utility caused 100% packet loss.

   This was my most creative use of a FreeBSD tool for testing. In a
   more straightforward application, any time a tester needs to
   eavesdrop on traffic between two hosts on a network, Ettercap is
   an excellent choice because of its power and ease of use.

   Perl gets a special mention here because Perl's network utilities
   are outstandingly good compared to other languages. Perl Net::*
   modules and IO::Socket::* modules are robust and powerful--but
   they often fail to compile on Windows. It is the ease of use of
   Perl's network utilities on FreeBSD that gets Perl the mention in
   this section.

   I use Perl's network utilities to impersonate network clients and
   servers for test purposes. On one occasion, I was required to test
   software that was a client to an interface on the New York Stock
   Exchange. Unfortunately, the NYSE test server was down about nine
   days out of ten. I wrote a little network server in Perl to
   emulate simple functions of the NYSE server in order to test the
   client software.

   On another occasion, I had to test some functions on a web server.
   It was not difficult to write a simple Perl HTTP client to
   validate that the server was functioning properly.

   I have also used Perl to validate the output from a server sending
   to a multicast address. I wrote a simple Perl multicast client on
   FreeBSD to monitor the traffic on a multicast address. Lincoln
   Stein's excellent IO::Socket::Multicast module made it easy.
   (Note: I never got IO::Socket::Multicast to compile on Windows. I
   tried it on Windows 2000 and on Windows XP.)

     --------------------------------------------------------------

4 Conclusion

   I used tools from the FreeBSD Ports Collection in four areas: in
   the network, where the operating system has very little impact on
   how software behaves; for remote security testing and performance
   testing in order to manipulate remote machines over the network,
   regardless of the operating system; for disk imaging of Windows,
   Linux, and FreeBSD machines; and on the webserver, where Twiki was
   my collaboration tool of choice.

   Because the installation procedure for all of these tools is
   standard, I spent relatively little time installing and
   configuring my tools. Because all the tools were hosted on a
   single platform, I had all of my configuration and diagnostic
   information located in just a few places. I kept all of my
   potentially dangerous security tools on a single machine, which
   made my presence on the network tolerable to the company's network
   management staff. And the compatibility between FreeBSD versions
   made it fairly simple to upgrade and to manage multiple FreeBSD
   machines. And of course, I could rely on the correctness of my
   test results, because the system itself is so reliable.

   I have tried using Linux in a similar way, but my experience is
   that package management quickly becomes tedious if not
   overwhelming. The FreeBSD Ports Collection handled that for me.
   And many of these tools are simply not available on Microsoft
   Windows. And when they (or their equivalents) are available, their
   cost, both financial and in terms of overhead, was simply too
   high.

   FreeBSD's simple installation procedures and robust Ports
   Collection makes it easy to experiment with the huge number of
   tools available. I often find myself browsing the Ports Collection
   looking for interesting applications to install, just to see how
   they work. (I found Ettercap by browsing the Ports Collection, a
   tool that became very useful very quickly.) It became clear that
   the more tools I used on FreeBSD, the more economical became the
   management of those tools.

   The next time you need to reach into your toolbox for some
   sophisticated, reliable, and powerful testing tools, I hope you
   find them in FreeBSD.

     --------------------------------------------------------------
