She Blinded Me WIth Science: Tuning Up Your BS Detectors.

PCs, operating systems, specialized hardware, 'tweaks'. Just some of the pieces in the FPS gamer's world. Like any field with such esoterica, there's plenty of snake oil to go around.

Vendors of software tempt the player with promises of improved game performance, while hardware is presented as being able to improve your pings. Mice are made with DPI ratings that need to use scientific notation. The list goes on and on.

What is the intelligent FPS gamer to make of these, and how can they be sure that their money is well spent on things that really can make a difference in their gaming acumen?

We'll tear into a couple of examples in this blog entry, and show how a healthy dose of skepticism toward many of the claims bantered about the Internet by producers of these products and posters in forums could save the gamer time and money.

I've chosen two areas that have in particular pegged my BS detector.

• Game Booster, and other software that alleges to 'speed up' your PC for gaming.
• Specialized 'gaming' Network Cards, that make claims of reduced pings and latency.

When deciding whether something might be a worthwhile addition to their gaming arsenal, the educated gamer should ask themselves the following:

• Is there a valid reason this should improve my gaming experience?
• Is there a repeatable, scientific test and measurement to validate the claims, or are they largely anecdotal and subjective?
• Even if there is a measurable difference, does it make a difference for gaming?

Game Booster is a piece of software that claims to improve your gaming experience by "... defragmenting game directories, temporarily shutting down background processes, cleaning RAM, and intensifying processor performance...".

What? How exactly does this piece of free software know how to manage these things better than Microsoft's own OS? Each of the techniques used have been shown to have little if any merit.

It has been shown repeatedly that file fragmentation has little bearing on the performance of Windows and applications for any modern version of Windows. In the days of the FAT file system, defragmentation could make a noticeable difference, but that was because of a file system design that had outgrown its usefulness.

With modern file systems on Windows (NTFS), defragmentation is frankly a waste of time, and its biggest effect is likely physical wear and tear on your hard drive. Defragmentation is strictly a no-no for Solid State drives, and garners absolutely no benefit.

I am not saying defragmentation of the file system under certain corner cases cannot show some measurable change. It can. I am saying such a change will have no material impact in the performance of a game. I invite any reader that can show otherwise with a repeatable test that will pass muster for scientific validity to comment with references to such a test. I know of none.

The 'shutting down of background processes', whether done manually or with some program has long been a 'tweak' recommended in enthusiast forums. In particular, recommendations are made to disable Windows intrinsic services and processes. Utter BS! There is again no test I've ever seen that shows this to have any material effect on game performance. Messing with Windows facilities can have very deleterious effects on the performance and reliability of the OS, and should be considered off limits by any informed gamer.

Now doing this can certainly affect boot times: these Windows components are typically loaded during the boot processes, so this will take some time obviously. But once booted, Windows manages these components quite efficiently, moving them aside if an application such as a game needs resources. There is no material penalty leaving these alone for a gamer.

Of course, if the gamer has tons of junkware or other software installed on their gaming machine, these may cause issues. In these cases, shutting down or disabling these could benefit game play. But seriously, what sane gamer that cares about game performance plays on a machine burdened with junk?

The solution to problems like this is to have a proper gaming environment, and not install crud in the first place. See Swimming in the Septic Tank with my Gaming Buddies for how I set up my gaming machines, a model that I believe is ideal for the serious gamer.

I'm not even sure how to comment on the claim of  'cleaning RAM, and intensifying processor performance'. Windows manages memory. The way it should be managed. 'Memory Cleaners' and 'Extenders' have long been known to belong to the snake oil camp of PC 'tweaks'. Enough said.

Let's answer our three questions for this product.

• There's no valid reason this product should have any material affect for a properly configured gaming environment. If you have junk software baggage getting in the way of your gaming pleasure, uninstall it or just build a proper environment in the first place.
• I've seen no repeatable, scientifically valid test to validate the claims for this product when installed in a proper gaming environment. Only anecdotal claims, about as valid as the miracle cures claimed for colon cleansing.
• Since there does not appear to be a measurable difference on a properly setup PC, I'd expect the product to have no material effect. So it doesn't matter to the gamer.

Next, let's look at the specialized 'gaming specific' NIC hardware offered by Bigfoot Networks. This company burst into the PC gaming scene in 2006, offering a product called the "KillerNIC" with claims of improved pings and game performance. I mentioned them in my earlier blog entry Perplexed Execs Dissect PhysX.

The marketing materials and hype that surrounded this initial product were mind numbing. Oral Roberts, Tony Robbins, and Billy Mays combined. There was (and still is) however at the same time a suspicious lack of any tests with reasonable protocols and environments showing any material improvement. This kind of hype should always raise an eyebrow. Their current campaign is filled with outlandish testimonials (how many of these players paid for their cards, you might ask) that they remind me of a revival meeting. No scientifically testable numbers to be found anywhere. Big surprise.

The tests I've seen, both in-house and out, seemed to all have been done on hardware that was borderline at best for a serious gamer. While offloading the network processing from an overloaded OS and hardware platform with a mediocre on-board NIC chip may demonstrate measurable performance benefits in network response and even frame rates due to reduced CPU utilization, no gamer with the $300.00 to spend at the time was likely to be running games on such hobbled hardware. And if they were, the money could have been spent far more effectively elsewhere to improve hardware performance (RAM, CPU, add-in NIC, etc.)

I am not aware of any rigorous, scientifically valid test of these products on a modern gaming machine that show any material effect on performance in areas that matter to the gamer. Reviews by magazines and tech sites reflect this view.

In their most recent incarnation, the company has moved the performance metric to a measurement of their own creation, using tools defined and built by them. "Trust me. See the difference it makes in the flow rate of the Flux Capacitors?''  This is starting to smell like some of the marketing done in other hobbies with kook esoterica, like high-end audio products.

Let's look at our BS detector questions for this item:

• Pings are important to gamers. They're also largely out of the gamer's control. Once the gamer's packets are on the WAN, there's nothing they can do to improve pings. Probably the reason the company dropped this as their marketing silver bullet, and moved to new nonsense. There's no valid reason improvements in the new metric should result in a material improvement of the gaming experience.
• The only test with any sort of measurement for the current product ("Killer 2100") extant at this time is the in-house test of the in-house created metric using the in-house created tool. There have been no scientifically valid tests showing any material benefit to a gamer. Suspiciously, there are absolutely no details in the marketing diarrhea found on the company site describing the hardware configuration used for their comparison. Lots of "testimonials" though. Sound like a late night TV commercial to you?
• Even if the results of the in-house test are valid, that this would make any material difference to game play strains credulity. Like audio cable makers that charge tens of thousands of dollars for a pair of speaker cables, claiming the incredible improvements wrought by the 10 MHz bandwidth of their cable. Problem is, human hearing drops out orders of magnitude lower in the spectrum, and there has never been a scientifically valid test to show any benefit of such cables. The numbers look good in their own tests for the NIC, as they do for the speaker cables. They just don't matter.

The answers to all three of these should raise red flags for the gamer considering laying out their hard earned cash for a product such as this.

If the gamer applies these three simple questions when considering a new piece of hardware, software, or the application of some 'twaek' seen in a forum, I think they will save themselves time and money, and avoid going down the rabbit hole of nonsense.

I try to look at the PC and gaming world, and the world in general, through glasses colored by logic and rationality. Take a look at my thoughts on other areas where I think the manure is deep at I Can See CXLVI Frames Per Second!, I Read it in a National Enquirer Survey, it Must Be True!, Mutation on the Bounty, and Port Forwarding: Slaying the Mythical Dragon of Online PC Gaming.

For a great overview of how to think using logic, rationality, and healthy skepticism, check out the materials at The James Randi Educational Foundation and their Million Dollar Challenge, most recently applied to ultra-expensive high-end speaker cables (well, almost applied: the cable vendor chickened out!)

There's also an amusing snippet How to be a Skeptic on WikiHow, check it out!

Update:

After reviewing some of the grotesque marketing tactics at the pages of Bigfoot Networks, I have made a public challenge.

I invite readers to e-mail them with a link (I wanted to put a cool "mailto" link here, but strangely the contact information for the company headquarters doesn't have an e-mail. For that matter, it doesn't have a phone number. The "Texas Office" has a phone number, but I'm not clear if the address has a suite number or a self-storage shed number. In any case, no e-mail there either. If a reader finds one, let me know! I want to donate a PC to a worthy school.)

Their lawyers (must be pretty cheap, if "Grubby" the master Warcraft player they use as a reference "earns more in a year than your average lawyer.") can contact me and my lawyers for details.

In the spirit of my earlier "Bounty" blog entries Mutation on the Bounty and I Can See CXLVI Frames Per Second!, I will open this up to a public challenge at some future date. I first want to give the Bigfoots at Bigfoot a chance to accept it.

"Grubby". That's a good one. I wonder how much pings really matter for flashing pink hooves on the royal mount, or whatever they call them in those kind of games like Warcraft where cat-like reactions are required. Not!

The challenge has beeen moved to its own entry here.

As an aside, as per the earlier bounties, I'll not be posting any anecdotal comments from readers claiming they've "seen" this particular bigfoot. If you think you can meet this challenge and are able to demonstrate the ability and willingness to suffer the penalty of a loss, or if you have an idea for a challenge related to this that you want to propose, do so via a comment or email. Again, I'll not be posting any of the "Well, I can tell the difference 'cause my cat says so" genre of comments.