automated bench tests script of FreeBSD -current

# Purpose #
There is a firewall performance regression on -current (source: http://pastebin.com/R7mjEwxr).
How to find the culprit commit ?
=> Need to generate lot's of FreeBSD -current regarding a list of svn revision (bisection method), and benchmarks each of them...

# First step: Generating lot's of FreeBSD -current #
This script will generate BSDRP (nanobsd based) images regarding a list of FreeBSD svn revision number.
=> http://sourceforge.net/p/bsdrp/code/HEAD/tree/trunk/BSDRP/tools/bisection-gen.sh

    BSDRP/tools/bisection-gen.sh
    Building image matching revsion 244900...
    Building image matching revsion 245423...
    Building image matching revsion 246146...
    Building image matching revsion 246792...
    Building image matching revsion 247463...
    Building image matching revsion 247916...
    Building image matching revsion 248267...
    Building image matching revsion 248584...
    Building image matching revsion 248830...
    Building image matching revsion 248944...
    etc...

# Second step: Automatically bench each previously generated images #
This script will do the job.
=> http://sourceforge.net/p/bsdrp/code/HEAD/tree/trunk/BSDRP/tools/bench-lab.sh

During this script development, we need a virtual BSDRP lab, the BSDRP vbox-lab script (http://sourceforge.net/p/bsdrp/code/HEAD/tree/trunk/BSDRP/tools/BSDRP-lab-vbox.sh).
  And just start it like that:
    BSDRP-lab-vbox.sh -i BSDRP-1.4-full-amd64-serial.img.xz -n 3 -c

It will produce a lab with 3 routers full-meshed, and with a network between the host and all 3 BSDRP VMs.
This will permit to reproduce the real lab:

 +----------+     +-------------------------+     +----------+
 | Tester_1 |<--->| Device Under Test (DUT) |<--->| Tester_2 |
 +----------+     +-------------------------+     +----------+
     |                            |                     |
 --------------admin network --------------------------------

Now we can run all bench tests with one command line: bench-lab.sh
I'm using (netblast/netreceive and/or netmap pkt-gen), here is an output example:

    BSDRP automatized upgrade/configuration-sets/benchs script
    Testing ICMP connectivity to each devices:
      192.168.56.11...OK
      192.168.56.13...OK
      192.168.56.12...OK
    Testing SSH connectivity with key to each devices:
      192.168.56.11...OK
      192.168.56.13...OK
      192.168.56.12...OK
    Starting the benchs
    Testing image serie: /tmp/BSDRP-244900-upgrade-amd64-serial.img
    Upgrading...done
    Rebooting DUT and waiting device return online...done
    Start bench serie /tmp/benchs/bench.1
    Waiting for end of bench 1/6...done
    Rebooting DUT and waiting device return online...done
    Waiting for end of bench 2/6...done
    Rebooting DUT and waiting device return online...done
    Testing image serie: /tmp/BSDRP-247463-upgrade-amd64-serial.img
    Upgrading...done
    Rebooting DUT and waiting device return online...done
    Start bench serie /tmp/benchs/bench.2
    Waiting for end of bench 3/6...done
    Rebooting DUT and waiting device return online...done
    Waiting for end of bench 4/6...done
    Rebooting DUT and waiting device return online...done
    Testing image serie: /tmp/BSDRP-249330-upgrade-amd64-serial.img
    Upgrading...done
    Rebooting DUT and waiting device return online...done
    Start bench serie /tmp/benchs/bench.3
    Waiting for end of bench 5/6...done
    Rebooting DUT and waiting device return online...done
    Waiting for end of bench 6/6...done
    All bench tests were done, results in /tmp/benchs

# Example of first results #

On my real lab with only 3 svn revision, here is the results, value in pps:

    ministat pf.244900 pf.247463 pf.249330
    x pf.244900
    + pf.247463
    * pf.249330
    +---------------------------------------------------------------------------------------------+
    |*   **          *x   *      x   x           x x                       +   +         +       +|
    ||____M___A________|  |__________MA___________|                      |_____M___A_________|    |
    +---------------------------------------------------------------------------------------------+
        N           Min           Max        Median           Avg        Stddev
    x   5        238517        241763        240221      240364.6      1334.621
    +   5        244467        246948        244907      245382.6      1095.694
    Difference at 95.0% confidence
            5018 +/- 1780.78
            2.08766% +/- 0.740867%
            (Student's t, pooled s = 1221.02)
    *   5        236646        239048        237209      237688.2      1003.186
    Difference at 95.0% confidence
            -2676.4 +/- 1721.83
        -1.11348% +/- 0.716339%
        (Student's t, pooled s = 1180.59)