She Blinded Me WIth Science: Tuning Up Your BS Detectors.

PCs, operating systems, specialized hardware, 'tweaks'. Just some of the pieces in the FPS gamer's world. Like any field with such esoterica, there's plenty of snake oil to go around.

Vendors of software tempt the player with promises of improved game performance, while hardware is presented as being able to improve your pings. Mice are made with DPI ratings that need to use scientific notation. The list goes on and on.

What is the intelligent FPS gamer to make of these, and how can they be sure that their money is well spent on things that really can make a difference in their gaming acumen?

We'll tear into a couple of examples in this blog entry, and show how a healthy dose of skepticism toward many of the claims bantered about the Internet by producers of these products and posters in forums could save the gamer time and money.

I've chosen two areas that have in particular pegged my BS detector.

• Game Booster, and other software that alleges to 'speed up' your PC for gaming.
• Specialized 'gaming' Network Cards, that make claims of reduced pings and latency.

When deciding whether something might be a worthwhile addition to their gaming arsenal, the educated gamer should ask themselves the following:

• Is there a valid reason this should improve my gaming experience?
• Is there a repeatable, scientific test and measurement to validate the claims, or are they largely anecdotal and subjective?
• Even if there is a measurable difference, does it make a difference for gaming?

Game Booster is a piece of software that claims to improve your gaming experience by "... defragmenting game directories, temporarily shutting down background processes, cleaning RAM, and intensifying processor performance...".

What? How exactly does this piece of free software know how to manage these things better than Microsoft's own OS? Each of the techniques used have been shown to have little if any merit.

It has been shown repeatedly that file fragmentation has little bearing on the performance of Windows and applications for any modern version of Windows. In the days of the FAT file system, defragmentation could make a noticeable difference, but that was because of a file system design that had outgrown its usefulness.

With modern file systems on Windows (NTFS), defragmentation is frankly a waste of time, and its biggest effect is likely physical wear and tear on your hard drive. Defragmentation is strictly a no-no for Solid State drives, and garners absolutely no benefit.

I am not saying defragmentation of the file system under certain corner cases cannot show some measurable change. It can. I am saying such a change will have no material impact in the performance of a game. I invite any reader that can show otherwise with a repeatable test that will pass muster for scientific validity to comment with references to such a test. I know of none.

The 'shutting down of background processes', whether done manually or with some program has long been a 'tweak' recommended in enthusiast forums. In particular, recommendations are made to disable Windows intrinsic services and processes. Utter BS! There is again no test I've ever seen that shows this to have any material effect on game performance. Messing with Windows facilities can have very deleterious effects on the performance and reliability of the OS, and should be considered off limits by any informed gamer.

Now doing this can certainly affect boot times: these Windows components are typically loaded during the boot processes, so this will take some time obviously. But once booted, Windows manages these components quite efficiently, moving them aside if an application such as a game needs resources. There is no material penalty leaving these alone for a gamer.

Of course, if the gamer has tons of junkware or other software installed on their gaming machine, these may cause issues. In these cases, shutting down or disabling these could benefit game play. But seriously, what sane gamer that cares about game performance plays on a machine burdened with junk?

The solution to problems like this is to have a proper gaming environment, and not install crud in the first place. See Swimming in the Septic Tank with my Gaming Buddies for how I set up my gaming machines, a model that I believe is ideal for the serious gamer.

I'm not even sure how to comment on the claim of  'cleaning RAM, and intensifying processor performance'. Windows manages memory. The way it should be managed. 'Memory Cleaners' and 'Extenders' have long been known to belong to the snake oil camp of PC 'tweaks'. Enough said.

Let's answer our three questions for this product.

• There's no valid reason this product should have any material affect for a properly configured gaming environment. If you have junk software baggage getting in the way of your gaming pleasure, uninstall it or just build a proper environment in the first place.
• I've seen no repeatable, scientifically valid test to validate the claims for this product when installed in a proper gaming environment. Only anecdotal claims, about as valid as the miracle cures claimed for colon cleansing.
• Since there does not appear to be a measurable difference on a properly setup PC, I'd expect the product to have no material effect. So it doesn't matter to the gamer.

Next, let's look at the specialized 'gaming specific' NIC hardware offered by Bigfoot Networks. This company burst into the PC gaming scene in 2006, offering a product called the "KillerNIC" with claims of improved pings and game performance. I mentioned them in my earlier blog entry Perplexed Execs Dissect PhysX.

The marketing materials and hype that surrounded this initial product were mind numbing. Oral Roberts, Tony Robbins, and Billy Mays combined. There was (and still is) however at the same time a suspicious lack of any tests with reasonable protocols and environments showing any material improvement. This kind of hype should always raise an eyebrow. Their current campaign is filled with outlandish testimonials (how many of these players paid for their cards, you might ask) that they remind me of a revival meeting. No scientifically testable numbers to be found anywhere. Big surprise.

The tests I've seen, both in-house and out, seemed to all have been done on hardware that was borderline at best for a serious gamer. While offloading the network processing from an overloaded OS and hardware platform with a mediocre on-board NIC chip may demonstrate measurable performance benefits in network response and even frame rates due to reduced CPU utilization, no gamer with the $300.00 to spend at the time was likely to be running games on such hobbled hardware. And if they were, the money could have been spent far more effectively elsewhere to improve hardware performance (RAM, CPU, add-in NIC, etc.)

I am not aware of any rigorous, scientifically valid test of these products on a modern gaming machine that show any material effect on performance in areas that matter to the gamer. Reviews by magazines and tech sites reflect this view.

In their most recent incarnation, the company has moved the performance metric to a measurement of their own creation, using tools defined and built by them. "Trust me. See the difference it makes in the flow rate of the Flux Capacitors?''  This is starting to smell like some of the marketing done in other hobbies with kook esoterica, like high-end audio products.

Let's look at our BS detector questions for this item:

• Pings are important to gamers. They're also largely out of the gamer's control. Once the gamer's packets are on the WAN, there's nothing they can do to improve pings. Probably the reason the company dropped this as their marketing silver bullet, and moved to new nonsense. There's no valid reason improvements in the new metric should result in a material improvement of the gaming experience.
• The only test with any sort of measurement for the current product ("Killer 2100") extant at this time is the in-house test of the in-house created metric using the in-house created tool. There have been no scientifically valid tests showing any material benefit to a gamer. Suspiciously, there are absolutely no details in the marketing diarrhea found on the company site describing the hardware configuration used for their comparison. Lots of "testimonials" though. Sound like a late night TV commercial to you?
• Even if the results of the in-house test are valid, that this would make any material difference to game play strains credulity. Like audio cable makers that charge tens of thousands of dollars for a pair of speaker cables, claiming the incredible improvements wrought by the 10 MHz bandwidth of their cable. Problem is, human hearing drops out orders of magnitude lower in the spectrum, and there has never been a scientifically valid test to show any benefit of such cables. The numbers look good in their own tests for the NIC, as they do for the speaker cables. They just don't matter.

The answers to all three of these should raise red flags for the gamer considering laying out their hard earned cash for a product such as this.

If the gamer applies these three simple questions when considering a new piece of hardware, software, or the application of some 'twaek' seen in a forum, I think they will save themselves time and money, and avoid going down the rabbit hole of nonsense.

I try to look at the PC and gaming world, and the world in general, through glasses colored by logic and rationality. Take a look at my thoughts on other areas where I think the manure is deep at I Can See CXLVI Frames Per Second!, I Read it in a National Enquirer Survey, it Must Be True!, Mutation on the Bounty, and Port Forwarding: Slaying the Mythical Dragon of Online PC Gaming.

For a great overview of how to think using logic, rationality, and healthy skepticism, check out the materials at The James Randi Educational Foundation and their Million Dollar Challenge, most recently applied to ultra-expensive high-end speaker cables (well, almost applied: the cable vendor chickened out!)

There's also an amusing snippet How to be a Skeptic on WikiHow, check it out!

Update:

After reviewing some of the grotesque marketing tactics at the pages of Bigfoot Networks, I have made a public challenge.

I invite readers to e-mail them with a link (I wanted to put a cool "mailto" link here, but strangely the contact information for the company headquarters doesn't have an e-mail. For that matter, it doesn't have a phone number. The "Texas Office" has a phone number, but I'm not clear if the address has a suite number or a self-storage shed number. In any case, no e-mail there either. If a reader finds one, let me know! I want to donate a PC to a worthy school.)

Their lawyers (must be pretty cheap, if "Grubby" the master Warcraft player they use as a reference "earns more in a year than your average lawyer.") can contact me and my lawyers for details.

In the spirit of my earlier "Bounty" blog entries Mutation on the Bounty and I Can See CXLVI Frames Per Second!, I will open this up to a public challenge at some future date. I first want to give the Bigfoots at Bigfoot a chance to accept it.

"Grubby". That's a good one. I wonder how much pings really matter for flashing pink hooves on the royal mount, or whatever they call them in those kind of games like Warcraft where cat-like reactions are required. Not!

The challenge has beeen moved to its own entry here.

As an aside, as per the earlier bounties, I'll not be posting any anecdotal comments from readers claiming they've "seen" this particular bigfoot. If you think you can meet this challenge and are able to demonstrate the ability and willingness to suffer the penalty of a loss, or if you have an idea for a challenge related to this that you want to propose, do so via a comment or email. Again, I'll not be posting any of the "Well, I can tell the difference 'cause my cat says so" genre of comments.

Mutation on the Bounty.

What James 'The Amazing Randi' Randi has done in debunking nutjob psychics, fringe audiophiles and other forms of pseudoscience goes mutatis mutandis for much of the 'expert advice', 'facts', and 'tweaks' often seen in gaming enthusiast forums.

Some of my posts have dealt with just such things (see I Read it in a National Enquirer Survey, it Must Be True!, I Can See CXLVI Frames Per Second!, Port Forwarding: Slaying the Mythical Dragon of Online PC Gaming. for some examples). I do this for a few reasons. I hate bullshit. I hate even more bullshit from the clueless (the truly ignorant can be forgiven - they're too clueless to know they don't know). There's enough BS in enthusiast forums to fertilize the entire planet. (As a side note, if you enjoy seeing this kind of thing blown out of the water 'boom headshot!' style, catch some episodes of the hilarious Penn & Teller: Bullshit! series).

I laid out a challenge bounty in my frames per second entry, which I'm quite sure will remain standing until we as humans are genetically altered or evolve into some kind of über gaming race. I've got another one in mind, prompted by recent posts in a myriad of Battlefield Bad Company 2 forums regarding a tweak to the MCSS (Multimedia Class Scheduler Service) for windows Vista/7, alleging greatly improved game performance. The fact that this mechanism only comes into play with respect to network throttling at rates over 10,000 packets per second seems to be utterly lost on the Jim Jones like posters of this 'miracle tweak'. This is far beyond the actual rate used by the game.

BS! I say! I firmly believe this is a classic placebo combined with a sort of mass delusion often seen with 'tweaks' like this. This facility of Windows does not even come into play in the typical gaming environment, and any gamer worth their salt is not even trying to watch streaming videos and listen to streaming music while gaming (not that even that would necessarily invoke this feature's use).

So here's the challenge: I publicly state that the first person to demonstrate a measurable and statistically significant difference in game performance (including but not limited to ping and FPS rates, measured using metrics and tools acceptable to both parties) and a statistically significant change in player score (frags per minute or some other metric agreed to by both parties) or the ability to even recognize whether the feature is enabled or not to a statistically significant level while playing Battlefield Bad Company 2, playing only the game and optionally using some VOIP communications such as Ventrilo or Teamspeak on a fresh installation of Windows 7 with no other applications installed or running other than those present in the default Windows installation, will garner a bounty of a $100.00 U.S. gift card from the author.

For a twist on the last challenge, where a challenger has nothing to lose and everything to gain by showing fault in my statements, any challenger agrees that once the detailed conditions outlined above are agreed upon by both parties, and the challenge is accepted, failure to produce supporting evidence for the hypothesis that this 'tweak' has any material affect will require the challenger to pay $100.00 U.S. within 30 days of conclusion of the challenge to the charity of the author's choice.

As before, my Shekels are safe, perhaps I can however enrich the coffers of some charities!

As an aside, I wil NOT post any comments of the 'Well, I can see the difference!' genre for this blog entry. You think you can beat the challenge? Then post your public acceptance of the challenge. Put up, or shut up!

I Can See CXLVI Frames Per Second!

You can find a myriad of posts in gaming enthusiast forums debating the need, or lack thereof, of high frame rates in PC gaming. Inevitably, someone posts a claim that at least some 'X' frames per second are needed before it doesn't matter, or even that no such upper limit of  'X' exists, to which I usually reply with the title of this entry.

And just as inevitably, the 'experts' and 'pro gamers' chime in with claims that ridiculous frame rates are needed, and how they can see and notice a difference with hundreds of frames per second.

Invariably, these forum arguments boil down to someone referencing a Wikipedia article, or some other forum post, all of which seem lead back to the same 'authoritative' reference: a couple of 'articles'  (one a dedicated web site) by an 'expert' making ludicrous claims based on research done by the Air Force many years ago regarding the 'persistence' of vision to a rapidly displayed single image.

The problem here (aside from the fact that the Air Force study has no real relevance with respect to our ability to utilize rapidly changing scene information) is that these two 'articles' are written by someone with no background of note, nor any expertise of any sort in the subject matter that I could discern.

In my opinion, the 'author' may also be a 'kook', having published other 'articles' of such scholarly note as 'We Made Contact', an expose on their 'scientific' analysis of newly discovered crop circles, with the conclusion that these are a response from a message sent to a distant globular cluster as part of the SETI experiments. Complete with a 'decoding' of the alien message. This 'article' demonstrates such an incredibly naive understanding of the physics involved in the transmission, or a purposeful use of misinformation in an attempt to bolster the credibility and 'wow' factor of the 'article', or both, that I'm really not sure what to make of it! Read it at "We Made Contact" for a serious laugh.

The 'gotcha' there: even though the target in question would require a round-trip message time of tens of thousands of years, the 'reply' took only a handful according to the 'expert' author of the 'scientific' article, while a trivial analysis shows that the probability of some mystery planet hosting aliens being close enough and in the needed direction yet unknown to us is nonexistent.

All this is aside from the fact you would need to believe the conclusion of the 'scientific' research of the 'author' that these Aliens exist, have somehow violated Einstein's Theory of Relativity, and even bothered to respond to us by carving these circles (and the 'Face on Mars', according to the 'study').

If you need to use this kind of thing as your authoritative reference, you might as well stop playing games, and just check with Sylvia Browne to have her tell you what your scores and stats will be.

There is no academically, scientifically sound study that supports the outlandish claims of this 'author', or the logical leaps made by many based on the 'facts' from these articles, or the claims by gamers that wildly overestimate their own physiological capabilities.

I actually heavily researched this, and consulted with real experts in the field during a venture funding analysis.

I want to be clear: I am not saying higher frame rates are not beneficial.

I am saying that the value where the ability of even gifted humans to take advantage of high frame rates is lower by a large margin than the claims of this oft referenced 'expert' and that of poseurs in forums.

Think you know better than the experts? Think you have supernatural abilities?

I publicly state here that the first person to either:

(1) Show a peer-reviewed academic/scientific study that supports the hypothesis "Humans can effectively utilize frame rates over 300 frames per second and can demonstrate a statistically significant increase in target acquisition and game score (test score) compared to 120 frames per  second"

or

(2) Demonstrate such an ability under controlled and accepted scientifically valid testing protocols,

will be awarded a $100.00US prize by me.

Interested parties can post their acceptance of this challenge here or email the author. The precise terms and conditions will then be determined by both parties.

(As an aside: there is at least one game I am aware of where the physics algorithms of the game appear to be inexplicably tied to the frame rate limit settings. In this game, one can raise the default rate cap, and this allows slightly higher jumps, etc. to be accomplished. Obviously, such coding horrors do not mean the higher frame rates are the cause of better target acquisition by the player...)

My shekels are safe.

Dedicated to James "The Amazing Randi" Randi.
Debunker extraordinaire. A hero of mine.
See Google to find his $1,000,000.00 challenge to 'audiophile' kooks that think $25,000 cables sound better. Hilarity ensues with the funambulistc help of  Pear Cable, maker of goof-ball speaker cables.