The Official Unofficial Bigfoot Challenge: Throwing Down The Gauntlet of Science at Marketing BS.

I decided to move this from my blog entry She Blinded Me WIth Science: Tuning Up Your BS Detectors to make it more easily findable by Bigfoot Networks.

Bigfoot: All ribbing aside, I'm quite serious. Let's put your outlandish claims under the harsh light of scientific scrutiny. Win or lose, a school gets a nice PC, so think of it as an educational opportunity combined with worthwhile charity. You can comment here or e-mail me for the contact information for my counsel, where the contract for the challenge can be fine tuned based on the outline in the challenge below and then finalized.

I look forward to your response.

The Official Unofficial Bigfoot Challenge

(1)

Bigfoot networks ("Bigfoot") will supply a trained network performance expert player (at their expense) of their choice from their pool of network performance experts shills.

(2)
I will provide a newly built, high performance gaming PC, configured with components typically used by gamers with budgets, not to exceed $2000.00 in total costs. Certainly a machine far below what the highly compensated and sponsored professionals would have. You know, the pros making all the glowing statements on the Bigfoot web site.

Such a machine would of course be handicapped in performance compared to the machines of a pro gamer, so any benefit from the Bigfoot technologies would be expected to be magnified. The machine will have a fresh, unmolested installation of Windows 7 64 Bit operating system, patched to the current patch and update state at the time of the challenge.

The tester (me) will install the game to be used for the challenge on the day of the challenge. The game will be patched to the patch level available at that time. Any customization of the game other than that available via normal in-game interfaces will be provided to the tester for review and approval, and will be applied by the tester, subject to verification by the network performance expert player.

(3)
The network performance expert player will play a series of games, not less than 20 games or twenty hours, whichever is greater time. At the start of gaming, and between games or once every hour, the network interface will be switched between the NIC interface provided by the test PC, and the Bigfoot "Killer 2100" NIC. This will be done by an appropriate random selection, known only to the tester (me.)

(4)
 At each game or session start, the identity of the correct and current NIC in use will be noted and placed in a time stamped envelope by the tester.

(5)
The network performance expert player will at the end of each game or session write their guess as to which NIC was in use. In addition, the appropriate score metrics will be noted to provide data on network performance expert player performance during the game. This will be placed into the same envelope, which will then be sealed and be placed into a locked slotted box to prevent tampering.

Alternately, utilizing accepted data security techniques as outlined by Carnac the Magnificent and in light of Bigofoot's likely concerns of this data being 'leaked', the envelopes can be hermetically sealed, placed in a mayonnaise jar, and at a time no later than noon the following day the aforesaid jar shall be placed on Funk & Wagnalls' porch.

(6)
At no time will any employee of Bigfoot be allowed contact or interaction with the test PC, other than access supervised and approved by the tester to allow Bigfoot to validate the correctness and specifications of the test PC build. The network performance expert player will similarly not be allowed access to the machine other than through the mouse and keyboard(s) provided by the network performance expert player.

Specifically, the network performance expert player will not have access to the network connections, which will be switched via a switching unit utilizing a two-into-one topology, so that both NICs on the PC will have cables attached for the duration of the test. The switch box will be viewable only to the tester, and accessible only to the tester.

Any indicators or other applications that provide notification of NIC or network status, performance, or related information will be disabled or made otherwise unavailable.

(6a)
Any and all non-standard or non-default utilization of the network "stack" or operating system by the Bigfoot "Killer 2100" NIC and associated software will be specified in writing and certified as encompassing any and all such utilization by Bigfoot. Such documentation will be provided to the tester at least three days before the challenge test to allow configuration of the alternate NIC and OS as appropriate. This includes, but is not limited to, changes in use of "Nagling", packet fragmentation or coalescing behaviors, any other interaction with the underlying OS environment and network facilities, any interaction with the OS that affects process priorities or other characteristics that may affect system, network, game or process performance that are advantageous to the Bigfoot NIC and that can also be utilized by the alternate NIC but may not be in the default settings for Windows. Such documentation must include any and all system changes made by the installer(s) for Bigfoot related software components, and any and all system changes made by the utilization of Bigfoot related software components.

(6b)
The tester reserves the right to install and utilize monitoring applications to determine system changes made by the installation of any and all Bigfoot related software during the installations process and during the game play testing. This includes but is not limited to Windows registry, process control, and network object monitoring. In addition, the test reserves the right to utilize packet analysis tools during game play testing to ensure that the comparison between NICs uses packets of similar nature.

(6a) and (6b) ensure that the installation and use of the Bigfoot NIC and related software do not cause covert or overt changes to normal Windows OS, Network, or process behavior that would have performance impact that could also apply to the alternate NIC but may not be used by default in Windows. In other words, they ensure that it is the card and associated driver and only the card and associated driver that is affecting performance, if any such effect exists.

Since it would obviously be trivial for an installer, application, or driver to make such changes covertly (e.g., my installer could manipulate the on-board NIC parameters unfavorably, and/or manipulate the parameters for my "super nic" favorably to produce favorable results) and no prior "test" of the Bigfoot products has to my knowledge eliminated this possibility, we will ensure this for this challenge.

The tester reserves the right to install and utilize network performance monitoring tools to record quantitative data to be included with the primary score and NIC identification data. This data, if gathered, will be statistically analyzed to determine if any statistically significant differences exist between the Bigfoot NIC and the alternate NIC. Only open source monitoring tools will be utilized, such that Bigfoot can assess the correctness of the monitoring tool(s). By the same token, any tools that the tester chooses to utilize that are supplied by Bigfoot must be provided with complete source code and build environments, allowing independent validation and compilation of the tool(s). This ensures that no manipulation of measurement or output data is done by the Bigfoot supplied tools.

(7)
At no time when at the PC will the network performance expert player be allowed to utilize any application other than the game played, or directly access any programs, commands, or other services or applications intrinsic to Windows. No performance measurement device or application may be used, nor will the use of any in-game measurement related to network performance be allowed. This is to avoid the network performance expert player from attempting to discern which NIC is in use by means other than game play performance.

The only interactions allowed with the test PC for the network performance expert player will be for activities directly related to the playing of the test game, game configuration, and score data recording.

(8)
I and Bigfoot may agree to a 'disinterested' third party to conduct the actual testing, to include but not limited to the switching of the NICs and recording of data.

(9)
The test will be held at a mutually agreed on location with sufficient network bandwidth. The servers for the games will also utilize providers mutually agreed on by Bigfoot and me. For any game, at least ten providers will be specified, half by the tester and half by Bigfoot. The provider to be used for the tests will be selected by the appropriate random selection process by the tester at the time of the test. If it is determined there will be a LAN component to the challenge tests, the PC used for the server must use a standard retail NIC other than Bigfoot products. Any LAN server PC if used will only have a fresh, default Windows 7 installation and the default installation files and configuration for the game. This is to reduce the possibility of collusion with game server providers and to eliminate the possibility of any covert or overt use of communication techniques dependent on any form of endpoint cooperation and only available when Bigfoot products are at both endpoints of the network.

I humbly suggest the Google campus for the test. Sergey and Larry will surely accommodate us, and they both enjoy a good scientific venture. Heck, they might even help us televise it!

(10)
The Bigfoot "Killer 2100" NIC will be a retail model, purchased by the tester, as part of the test PC build.

No information, such as the MAC of the NIC will be made available to Bigfoot until the termination of the test. Both NICs will use a 'spoofed' MAC during game play, provided and controlled by the tester, to ensure that no identification of which NIC is in use can be attempted by packet analysis.

(11)
The data will be independently analyzed by an agency approved by both parties as to statistical significance of the data gathered.

(12)
The results will be published on the home / front page of the Bigfoot Networks web site. They will remain there for a period not less than one hundred eighty (180) days.

Any changes to the domain name or other infrastructure that would prevent access to this information by visitors to the site will provide for the needed changes to the published results to ensure they remain visible.

(13)
This item is not part of the challenge requirements. It's just a place holder for a "Hello!" to the apparently underpaid lawyers of Bigfoot that will surely be asked to review this and figure out a way to have some kind of "take down" issued so Bigfoot doesn't have to explain why they didn't take up the challenge. Hello!

(14)
If there is a statistically significant network performance expert player performance benefit shown when using the "Bigfoot Killer 2100" NIC, and the network performance expert player shows a statistically significant ability to correctly identify the NIC that was used for each session, I will donate the complete PC to the school or other educational facility of Bigfoot's choice.

(15)
If no statistically significant network performance expert player performance benefit is seen when utilizing the Bigfoot "Killer 2100" NIC or the network performance expert player is unable to identify which NIC was in use for each session to a statistically significant level,  Bigfoot agrees to reimburse the author (me) for all costs incurred for the test PC, which I will then donate to the school of my choice.

(16)
Any violation of the rules, specifications and stipulations agreed upon will constitute a forfeiture by the violator.