Wolfram|Alpha: Systematic knowledge, immediately computable.

Friday, April 30, 2010

My Orthographer Sayz Mah Spellun iz Getten Bettur!

I spell horribly. I despised English classes - I always wanted to be goofing off with math / physics / chemistry / computer stuff. Add to that growing up between Europe and the U.S., and a German Mom, yeah, things got scrambled. I even once received an 'F' on a paper (later remedied by my explanation), accused of plagiarism. The forensic clues to professor used? I spelled color as 'color' and 'colour' in the paper. To him, I must have copied some passages.

Sadly, this handicap has followed me into adulthood. So I'm always cutting and pasting text into spell checkers to ensure correctness. I don't know what took me so long, but I recently 'discovered' the Google Toolbar.

If you aren't using it, check it out! It has tons of cool features. The ones I find invaluable include a spell-checker that can check spelling on any form, portable bookmarks (all of mine, anywhere, any PC, any browser), search on site, search my google docs, and a unified login for those that use Google ID for login.

Pretty slick, pretty stable and bug free. I like it!

Book 'Em, Danno!

My Daughter likes to talk  about her schoolwork. Often. Pretty cool, but it can be a bit hard to do when remote: Try having someone describe something like a physics problem over the phone. So I asked her to get me copies of her books, a request easier said than done. Eventually, the school administration asked for a personal letter from me justifying the request. I ran across it going through some old e-mails. It got the job done, the books showed up quickly after its receipt.

Dear Ms. L$$$$ H$$$$:

Per your request, communicated through my daughter Juliana Cook, a junior at $$$$$$$$$, I am requesting one copy of each of the textbooks she uses in her classes, to facilitate tutoring and other assistance from me.

Because I am away from the school area more often than not, having a copy of the books locally will allow me to much more easily respond to questions from Juliana, and to be proactively prepared for tutoring her in her advanced studies.

I understand from Juliana that this email from me is needed to facilitate this request. I trust and hope that our wishes can be accommodated. Barring this, I will have no choice but to lock Juliana in the basement, properly converted to a medieval scriptorium of course and supplied with parchment and pen, to have her copy all of the books manually for my use. While such monastic activity will likely aid her in her understanding of ancient history, I'm sure you'll agree that getting the books from you will be far more time effective.

Thank you for your attention in this matter, should you have any questions, do feel free to contact me. If time is critical, feel free to have Juliana deliver the message personally. I have trained her in the traditions of Marathon (we only buy the Wheaties boxes with Pheidippides on them), and she will gladly run to deliver the message before dying from exhaustion.

Otherwise, please email me or phone me at (###) ###-####.

Kind Regards,
Rob Cook

Thursday, April 29, 2010

I Scream, You Scream, We All Scream for DICE Scream!

Doing some mathematical work with Maple 13 tonight and needed a break. Thinking of the recent fiasco with DICE and their new game Battlefield Bad Company 2 (see Pings? We Don't Need No Stinkin' Pings!), I decided to see how hard it would be to connect to EA's central server (where the information for the game server browser comes from - the one that doesn't work properly in the game) using Maple. I'd already done some work reverse engineering the traffic for this and the patcher, looking to ease the pain of users with the  game browser and the patch process (yay for Steam - patches there just work, no worries of overloaded, flaky EA servers!)

Maple, for those not familiar with it, is an extremely sophisticated application for doing all things mathematical. The product has an amazing list of capabilities for mathematical analysis, graphing, and programming. It is however primarily a mathematical tool, competing with the likes of Mathematica and Matlab. I use all three, Mathematica being my personal favorite. For quick and dirty, however, I find the 'Document Mode' in Maple to be ideal for rapid exploration. I often do proof of concepts there, and when the ideas are fleshed out, move them to Matlab or Mathematica.

So how hard would it be to get to the Electronic Arts centralized server, using a tool completely out of its domain (kind of like using a champagne bottle for a baseball bat), without any of the raw socket nonsense that the game developers used? See for yourself - seven lines of Maple gets you the initial connection and response. A handful more lines would get you a complete server browser. Without the hassles the game introduces by using raw sockets. Pretty powerful tool, doing things out of its real domain. It makes me wonder even more: what were the game developers thinking when they chose to use raw sockets?

Maple Code:

with(Sockets);
sid := Open("159.153.235.12", 18395);
reply := Array(1 .. 65, datatype = integer[1]);

WriteBinary(sid, Array([67, 79, 78, 78, 64, 0, 0, 0, 0, 0, 0, 91, 80, 82, 79, 84, 61, 50, 10, 80, 82, 79, 68, 61, 98, 102, 98, 99, 50, 45, 112, 99, 10, 86, 69, 82, 83, 61, 49, 46, 48, 10, 80, 76, 65, 84, 61, 80, 67, 10, 76, 79, 67, 65, 76, 69, 61, 101, 110, 95, 85, 83, 10, 83, 68, 75, 86, 69, 82, 83, 73, 79, 78, 61, 53, 46, 49, 46, 50, 46, 48, 46, 48, 10, 84, 73, 68, 61, 49, 10, 0], datatype = integer[1]));

ReadBinary(reply, sid);
Close(sid);
convert(subs(0 = 32, 10 = 32, convert(reply, list)), bytes);

EA Server Response:

"CONN ATIME=1272627041 TID=1 activityTimeoutSecs=240 PROT=2"

Hello, EA! All your base are belong to us!

I Can See CXLVI Frames Per Second!

You can find a myriad of posts in gaming enthusiast forums debating the need, or lack thereof, of high frame rates in PC gaming. Inevitably, someone posts a claim that at least some 'X' frames per second are needed before it doesn't matter, or even that no such upper limit of  'X' exists, to which I usually reply with the title of this entry.

And just as inevitably, the 'experts' and 'pro gamers' chime in with claims that ridiculous frame rates are needed, and how they can see and notice a difference with hundreds of frames per second.

Invariably, these forum arguments boil down to someone referencing a Wikipedia article, or some other forum post, all of which seem lead back to the same 'authoritative' reference: a couple of 'articles'  (one a dedicated web site) by an 'expert' making ludicrous claims based on research done by the Air Force many years ago regarding the 'persistence' of vision to a rapidly displayed single image.

The problem here (aside from the fact that the Air Force study has no real relevance with respect to our ability to utilize rapidly changing scene information) is that these two 'articles' are written by someone with no background of note, nor any expertise of any sort in the subject matter that I could discern.

In my opinion, the 'author' may also be a 'kook', having published other 'articles' of such scholarly note as 'We Made Contact', an expose on their 'scientific' analysis of newly discovered crop circles, with the conclusion that these are a response from a message sent to a distant globular cluster as part of the SETI experiments. Complete with a 'decoding' of the alien message. This 'article' demonstrates such an incredibly naive understanding of the physics involved in the transmission, or a purposeful use of misinformation in an attempt to bolster the credibility and 'wow' factor of the 'article', or both, that I'm really not sure what to make of it! Read it at "We Made Contact" for a serious laugh.

The 'gotcha' there: even though the target in question would require a round-trip message time of tens of thousands of years, the 'reply' took only a handful according to the 'expert' author of the 'scientific' article, while a trivial analysis shows that the probability of some mystery planet hosting aliens being close enough and in the needed direction yet unknown to us is nonexistent.

All this is aside from the fact you would need to believe the conclusion of the 'scientific' research of the 'author' that these Aliens exist, have somehow violated Einstein's Theory of Relativity, and even bothered to respond to us by carving these circles (and the 'Face on Mars', according to the 'study').

If you need to use this kind of thing as your authoritative reference, you might as well stop playing games, and just check with Sylvia Browne to have her tell you what your scores and stats will be.

There is no academically, scientifically sound study that supports the outlandish claims of this 'author', or the logical leaps made by many based on the 'facts' from these articles, or the claims by gamers that wildly overestimate their own physiological capabilities.

I actually heavily researched this, and consulted with real experts in the field during a venture funding analysis.

I want to be clear: I am not saying higher frame rates are not beneficial.

I am saying that the value where the ability of even gifted humans to take advantage of high frame rates is lower by a large margin than the claims of this oft referenced 'expert' and that of poseurs in forums.

Think you know better than the experts? Think you have supernatural abilities?

I publicly state here that the first person to either:

(1) Show a peer-reviewed academic/scientific study that supports the hypothesis "Humans can effectively utilize frame rates over 300 frames per second and can demonstrate a statistically significant increase in target acquisition and game score (test score) compared to 120 frames per  second"

or

(2) Demonstrate such an ability under controlled and accepted scientifically valid testing protocols,

will be awarded a $100.00US prize by me.

Interested parties can post their acceptance of this challenge here or email the author. The precise terms and conditions will then be determined by both parties.

(As an aside: there is at least one game I am aware of where the physics algorithms of the game appear to be inexplicably tied to the frame rate limit settings. In this game, one can raise the default rate cap, and this allows slightly higher jumps, etc. to be accomplished. Obviously, such coding horrors do not mean the higher frame rates are the cause of better target acquisition by the player...)

My shekels are safe.

Dedicated to James "The Amazing Randi" Randi.
Debunker extraordinaire. A hero of mine.
See Google to find his $1,000,000.00 challenge to 'audiophile' kooks that think $25,000 cables sound better. Hilarity ensues with the funambulistc help of  Pear Cable, maker of goof-ball speaker cables.

Wednesday, April 28, 2010

Flipping You the Bird, With a Flipping Bird That's Flipping Out.

Oh man, I want one of these! This bird borders on unbelievable. It not only can mimic dozens of calls from other birds perfectly, but when exposed to man made sounds, it mimics them too. I'd make a pair of little tiny bird headphones, and have it listen to BFBC2, so I could have it mimic the crazy accented Russian/English and weapon sounds in the game.

This video shows a bird from a zoo where construction was in progress. Among other things, it mimics:

1. Hammer
2. Chainsaw
3. Jack hammer
4. Lawn mower hitting sticks
5. Leaf blower starting
6. Power drill
7. Wood saw
8. Human voices
9. Two-way radio
10.Worker whistling

Video here: Lyre Bird in Action

Swimming in the Septic Tank with my Gaming Buddies.

C'mon in, water's warm! And there's flotation devices all over the place!

Seriously, you wouldn't go swimming at your local water treatment facility, so why on earth would a serious gamer try to run their games in an environment that is even more unhygienic?

I helped a few posters recently with a puzzling 'lag' issue. They would enter the game, but each spawn was met with a many seconds delay, and in some cases, the same puzzling delay would happen when accessing the game menus.

Turned out to be their anti-virus software getting in the way, and either disabling it, or adding all the game directories to the exceptions list, fixed the issue. I see this all the time, where some software turd causes a problem with a game, slowing things down or otherwise interfering with the game. Anti-virus, anti-malware, peer-protection, printer drivers, iTunes, Quicktime, peripheral drivers, etc., the list of cesspool floaters is endless.

Which got me thinking about something I ponder about periodically: Why on earth would a serious gamer have anything but the leanest, meanest OS environment for playing their games? What on earth is the reason to have an anti-virus running with a game that is from a trusted source (if you're stealing games, that's another story)?

I myself have always had separate Windows installations, one hardened for day-to-day activity and any uncontrolled network access, the others having only the drivers and software needed to play my games. Gives me the protection I want for regular activities and maximum performance, minimum hassle for gaming.

Many use things like Hardware Profiles (unfortunately deprecated in Vista and beyond) to 'minimize' unnecessary system load, but that can be cumbersome. Others use snake oil programs that purport to improve game performance by shutting down system processes and optimizing memory. I'll not argue the myth of messing with MS system processes, and what effect (none) it has on game performance. This is just another messy and questionable 'tweak'.

It is trivially simple to start with a clean Windows install and clone it to a separate partition to provide a multi-boot environment where one copy is hardened, the other is for gaming. The web is full of helpful tutorials that can guide even the most novice of users through the process. Users with the Enterprise and Ultimate versions of Windows 7 that allow native booting from VHD, can get equivalent functionality without the need to partition the hard disk at all.

That is the environment I use: Multiple VHD, each with exactly the environment needed for the games they contain, tuned and optimized (some games 'prefer' certain drivers, etc.), and only running things absolutely needed for those games. No A/V, no firewall, no junkware. Nothing that could affect the performance or otherwise interfere with my games. Another VHD contains a fully hardened Windows installation, with a combination of anti-malware/firewall/security that ensures safe passage when navigating the sewer we call the Internet.

A side benefit is that by using Differencing VHDs, all of this is done with very minimal space requirements: I don't have to duplicate the space required for each Windows installation, saving hundreds of gigabytes of storage. With Windows 7 Ultimate/Enterprise, I can boot 'natively' (that is, to the bare metal: a normal boot running at the full speed and capabilities of the hardware) or boot using a virtual machine to any of these VHD installations.

I can even 'boot' into the hardened environment using a virtual machine while already booted into one of my gaming environments should I need to download or otherwise access the web, without needing to reboot the machine. I can grab something from the Internet, have it completely scanned in the virtual environment, then drag-and-drop it into my gaming environment.

Maximal security combined with maximal speed. Another benefit to the use of a virtual machine in this case is the ability to snapshot the virtual hardened Windows, so if I do get exposed to some nasties, I can rollback time with the click of the mouse. Cool! Should some really extreme corner case of attack or malware successfully corrupt or infect one of my 'game' environments, it stays isolated to that environment, and I can restore it from backup ludicrously quickly (another Differencing VHD benefit).

I go ahead and run everything in the 'game' environments under a 'real' administrative account, since even in 2010, there are developers that seem to still be incapable of writing a properly behaving userspace/usermode game (see Pings? We Don't Need No Stinking Pings! for an example.)

If you're interested in trying out this kind of setup, there is one caveat: You must be disciplined. No network activity in the naked 'game' environments other than the game and game related functions. No installing or running anything not from trusted sources. Use the hardened environment, either natively or in a virtual machine for everything else. Otherwise, you're swimming in the septic tank. Surrounded by hepatitis infected needles. Naked. Do note, if you choose to run 'naked' like this, you'll still want some kind of isolation from the baddies on the WAN. If directly connected to the WAN (i.e., your PC IP is public), running the windows firewall should be part of the setup - even with hygienic use of the naked setup, this will minimize external attacks. Even better (since the windows firewall can be problematic with some games), keep it off, and use a proper router with NAT (and its own firewall enabled if you wish). NAT done correctly will keep the outside world at bay, and can be easily configured if needed (seldom) for specific games. Router firewalls can be more robust than the built-in windows system, and reduce load on the CPU for firewall tasks. The overhead of going through a router, firewall enabled or not, is negligible, and the protection provided warrants their use, in my opinion, even if your PC is the only device on your LAN.

The benefits of this kind of setup seem so overwhelming to me (absolute security combined with the leanest, meanest yet perfectly 'tuned for games' environment, with a significant reduction in the inevitable conflicts between different installed software that can be arduous to troubleshoot), I can't imagine why every serious gamer wouldn't want the same. No firewall, no A/V, no anything to get in the way of the game. It's better than running on a nude beach while sipping a Cialis spiked cocktail! Minimal encumbrance, maximum performance! Try it, you'll never look back.

I Windows 7 Event Logging!

I was recently debugging a couple of applications, and was trapping some events as part of the process. I forget how useful the Event Viewer is in general, and particularly with the greatly enhanced functionality in Windows 7/Vista.

Two of the really cool features are the ability to build sophisticated custom filters and views, allowing you to focus on exactly what you're looking for, and the ability to create events on events.

For example, the following XPath query for a filter/view allows me to trap a specific event for a specifc PID, excluding all other 'noise' in the event log:
  
<QueryList>
   <Query Id="0"
     Path="Microsoft-Windows-Winsock-AFD/Operational">
      <Select Path="Microsoft-Windows-Winsock-AFD/Operational">
         *[System[Execution[@ProcessID="3141"]]]
            and *[System[EventID="1000"]]
      </Select>
   </Query>
</QueryList>

Equally cool, I can assign tasks to any event/view/filtered view such that the system will notify me with a dialog, E-Mail me, or start any arbitrary program.

Hugely useful, allowing efficient, quick & dirty debugging without even needing to fire up a real debugger, and as an incredibly useful ancillary to formal debugging.

“Yields falsehood when preceded by its quotation” yields falsehood when preceded by its quotation.

Quine's Paradox. Named after the brilliant Willard Van Orman Quine, author of a favorite book of mine titled Quiddities: An Intermittently Philosophical Dictionary. The sentence is intended to show a paradox similar to The Liar Paradox can be had even without self-reference.

Named after him is the Quine Program, denoting a program that can reproduce itself as output, given no input (which makes the problem trivial, and is considered 'cheating'.)

It is effectively the Fixed Point of the execution environment. An example in Scheme looks like this:

((lambda (x)
   (list x (list (quote quote) x)))
   (quote
      (lambda (x)
         (list x (list (quote quote) x)))))

For a mind-opening exposition on all things recursive and self-referential, I highly recommend a read of the amazing work by Douglas Hofstadter: Gödel, Escher, Bach: An Eternal Golden Braid

Monday, April 26, 2010

Bashing and Slashing with Jockyitch

Kind of sounds like a cage match with knife wielding unhygienic foes...

I had the pleasure of spending some time with the founder, CEO, webmaster, and on-the-web-air personality 'Jockyitch' from the popular FPS news and opinion site BASHandSlash.

The host, a real honest-to-goodness rocket scientist, was hands down the most fun I've had doing any kind of broadcast show. 'Jock' asked me to do a show to discuss my guide for gamers with game connectivity issues. I think he made the subject matter and the discussion that ensued funny as heck.

Look for the webcast soon at the site, visit it regardless: there are a myriad of interesting webcasts, and tons of interesting things to be found wandering the forums and other parts of the site. The webcasts are also hosted on iTunes, if you're a fan of the poison apple. (Update: Thar She Blows!: BASH 147: PC Gaming Connectivity)

Of particular recent interest, BASH 146: Client is King has an interesting discussion on the state of PC gaming in relation to consoles, and the growing lack of modding and other tools for recent games. The host invited several expert guest commentators with some serious gaming background to participate, and the conversation is interesting to hear as it unfolds.

Pings? We Don't Need No Stinking Pings!

The popular new online FPS Battlefield Bad Company 2 has had its share of teething pains. Among those most bothersome to players is the inability to see 'pings' for servers in the online server browser. The developers, EA/DICE, have stated the solution to this issue is to run the game with elevated privileges, either by running in a full administrator account, or by using RunAs or compatibility mode changes to accomplish this.

Poppycock! The fix is to get rid of the poor coding choices that get this kind of result from my debugging traces of over a month ago:

socket: 0: Process 0xfffffa8005375060 (0x768 ), Endpoint 0xfffffa8005c49420, Family 2, Type SOCK_RAW, Protocol 1, Seq 1006, Status 0x0
(FAM: AF_Inet/IPv4) (SOCK: Raw) (PROTO: Stream)

The game is opening (or attempting to open) a Windows socket of  type (3) sock_raw. The use of raw sockets has become increasingly restricted with each new version of Windows, and for good reason.

This is the reason the BFBC2 game executable must be run as an administrator, or have its privileges elevated, for the player to see pings properly in the server browser.

Readers having this issue, either run the game from a 'real' administrative account, or right-click on the game executable and mark it for compatibility mode "Run this program as an administrator". Do note, there can be other issues such as firewall, ISP, or PC configuration that may still prevent pings showing properly.

There is no reasonable reason I can think of that an application like this game needs to use raw sockets, forcing the user to compromise the security and functionality of their system to make the application work the way it should. There are several proper solutions to accomplish the goal of getting the needed data that do not require unneeded privilege use or elevation by the user.

That this information was handed to the devs, and nothing has been done to remedy it, despite a patch in the intervening time being released, is puzzling to say the least. A lunch hour's worth of coding changes should fix this amateur mistake that should never have been made. (How hard is it to get to the server without raw sockets? See "I Scream, You Scream, We All Scream for DICE Scream!" for an answer).

Fix this, DICE!

A Most Interesting Development in Graphics and Rendering

Check out the stuff at Unlimited Detail Technology.

Most fascinating, an amazing start for an autodidact. It is so good, I wonder if it is some elaborate April Fools' joke...

It will be quite interesting to follow the progress of this project, even more so the reactions of Nvidia and ATI. If they're smart, and this is the 'real deal', one should buy the technology outright, and utilize the GPGPU functionality available in their products to speed computation.

Thursday, April 22, 2010

In the SCHEME of Things

One of the startups I was involved in (Ariba - one of the few survivors of the Internet bubble of 2000, and home to some of the most brilliant people I've had the pleasure of working with) utilized Scheme for the underlying configuration, display, data definition, and process control system. Scheme is one of the two languages widely used based on the lambda calculus, homoiconic with mind-altering features like first-class continuations. Think of it as a 'cleaned-up' Lisp-1.

A hugely powerful language, perhaps, along with Common Lisp, my favorite. It allowed us to be unbelievably agile in response to customer needs: While the competition would have to take engineers off of pressing projects to have them do quick-and-dirty coding to add some deal-breaking feature, more often than not the Ariba field tech giving a demo could add it on-the-fly. Blew the minds of prospective customers (not to mention keeping the competition wondering what kind of voodoo magic we had), and resulted in a hugely successful company.

I was reminded of the elegance and conciseness of the language from a blog post I recently read. The poster wanted to dump the logs of an IRC they frequented. Need to setup the underlying network, connections, open the log, display it. The code?

(require net/url)
(display-pure-port (get-pure-port (string-url "http://ccl.clozure.com/irc-logs/lisp/2009-12/lisp-2009.12.28.txt")))

From Lam Luu, a student and lover of the language, Quicksort for any data type, with a user supplied comparator:

(define (filter list get?)
(cond ((null? list) '())
((get? (car list)) (cons (car list) (filter (cdr list) get?)))
(else (filter (cdr list) get?))))

(define (qsort list compare)
(if (null? list)
'()
(let ((pivot (list-ref list (random (length list)))))
(append (qsort (filter list (lambda (x) (< (compare x pivot) 0))) compare) (filter list (lambda (x) (= (compare x pivot) 0))) (qsort (filter list (lambda (x) (> (compare x pivot) 0))) compare)))))

What a beautiful languge!

Monday, April 19, 2010

Macroscopic Quantum Superposition...The Cat Is A Lie.

Physicists at UC Santa Barbara published recently their results of experiments to create what is effectively a 'Quantum Microphone'. Part of the experiment showed a macroscopic realization of quantum superpositioning: the device simultaneously exists in moving and stationary states.

An overview article for the lay reader can be found at

http://www.scientificamerican.com/article.cfm?id=quantum-microphone.

Readers with a background in or understanding of quantum mechanics that are not already aware of this marvelous work can find more detailed publications in Nature, and through the web portals for the researchers involved.

Things Unexpectedly Technical

Along with enjoying things technical from hardware and software to several of the sciences, I've been an avid collector of complicated wristwatches for some time.

This particular watch, the Vacheron Constantin Tour de l'Ile, is the most complicated 'production' watch yet made.

This amazing piece comprises 834 hand made pieces, requiring over four years of research and development.

The complications of the watch, spread across two faces, include: minute repeater with sonnerie, tourbillon, power reserve indication, 2nd time zone, moonphase, age of the moon, Sonnerie level indication, Perpetual calendar, Day, Date, Month, Leap year, Equation of time, Sunrise, Sunset, and Celestial chart.

Only seven of these masterpieces have been produced, at a retail price of $1.5 million, though if you can find one you'll need to pony up perhaps another million on top of that.

I had the pleasure of handling one of these, truly an amazing work by gifted craftsmen, and likely the most complex mechanism in a small space ever made by human hands.

Sunday, April 18, 2010

One of the more interesting game problem cases I've worked on.

A poster in a game forum was pleading for help. A pretty high-end system, all the best components, top-end Windows 7, fresh and clean, along with a clean game install.

All was well until a couple of weeks ago, then load times for the maps went to hell, taking five to eight minutes, with the screen alternating between white and black. Once loaded, selection of menu items or login resulted in another minute or so of stalling.

The poster was using one of the highest-end and newest GPUs that had a known minor issue with load times for the game (but usually only resulting in 30 second or so loads for others, with no stalls once loaded) that had been remedied in a recent GPU driver release.

The poster, and an apparently savvy friend, were pulling their hair out, having tried every option reasonable: reinstall game, windows, no anti-virus, different drivers, disable sound, etc. but still no joy.

This was one of the weird ones. Like the PC of an old friend that would randomly play "It's a small world" out of the internal speaker (that took me a bit to figure out...)

I've been doing this a long time, as many of you probably have, giving us an advantage: We've probably seen it before, and had to figure it out.

The problem: USB bus conflict/flooding. I first saw this a few years back on a fellow gamer's machine, after trying all the obvious things to resolve their strange performance issues, I pulled out my USB bus analyzer, and bingo!

For this poster, simply unplugging all USB devices, then plugging them back in completely solved the issue. The same (actually more thorough) can be done by uninstalling the sub-branches of the USB device branch and rebooting the machine, allowing it to reinstall and enumerate the devices.

This very same problem manifested itself for a couple of players I worked with where their game ran in slow motion, as if time were slowed down. Not laggy game play, in fact no jerkiness of any sort, just six million dollar man slow motion effects for everything in the game!

A cool effect of this was they could FRAPS record game play, and when played back, it was normal speed. Spooky! When I had them do this, I'm sure they must have thought I was pulling some kind of practical joke on them.

The same fix (resetting USB by unplugging their USB devices) remedied the problem.

These strange and obscure ones must make life interesting for tech support.

The Race for Real-Time Photorealism

Having built a simple client-server FPS game including a simple rendering infrastructure and even simpler netcode from scratch as a brain exercise, I'm always amazed at what the real experts in the field can do.

A most fascinating article in the March 2010 issue of American Scientist covers aspects of rendering for realistic scene generation. My feeling is we are less than a decade away from games that can render environments indistinguishable from a movie. The rendering of humans, or other familiar living things, may be a bit further down the road: We are so programmed to incredibly subtle visual triggers, while it is currently posssible to render a human that is indistinguishable from reality, the illusion collapses once movement, speech, or other interaction occurs.

A fascinating read, available at the time of this post at: http://sigmaxi.org/4Lane/QMag/AMS_Mar_2010.pdf

Wednesday, April 14, 2010

How about a knowledge / intelligence filter for forums, and the internet in general?

I read a recent blog entry titled 'Idiots', where the writer lamented the state of knowledge exhibited in 'technical' forum postings. I LOL'd.

The reality is, we as posters need to be 'self-regulating', lest we fall into the class of those covered in the amusing book 'The Cult of the Amateur' by Keen.

The framework I've come up with is:

1) The questioner: Anyone can ask any question, there are no 'dumb questions.'
2) The answerer: Incumbent on them to know what they're talking about.
3) The corrector: Corrects misinformed answers. Should really know what they're talking about.
4) The Uber-corrector: Corrects the correctors. Had better be an expert.
5) The debater: Will argue points with correctors and U-correctors. Had better be an expert, else be prepared to be made an Internet jackass.

There seems to be far too many of the debater, regurgitating wikipedia information without any real understanding of the issues involved, and seemingly unwilling to do the research and footwork required to understand deep technical issues.

We owe it to fellow readers, and ourselves, to be introspective and aware of our own limits of knowledge, and committed to do our homework to ensure proper understanding of things when offering our advice to technical questions.

I'd normally let a carcass lie, but this is a great example of what being made an Internet jackass feels like, we all have the power to control whether we're made to look like a fool in public, and as the adage goes, 'payback is a bitch'. The beauty of the web is the speed of information flow. The ugliness is the vast number of self-proclaimed 'experts' that haven't a clue. That what one posts is usually there for an eternity for the world to see should be a cautionary clue to these kinds of posters. This poster, having barraged a forum with 'technical' posts way beyond their pay grade, was banned after incessantly arguing nonsense. The reaction in a different forum speaks for itself. The author will be fortunate if they don't end up sued by the parent company and the admins they libel. For the record, I am of course not an employee of EA or DICE, nor benefit in any way from either company other than buying their games.

Classic!
Photobucket

Yes, Virginia, 32-Bit Windows can use more than 4GB of physical RAM...

In one of the forums I frequent, a debate broke out in a thread over whether 32-bit editions of Microsoft Windows can in fact handle or otherwise use any physical RAM in a system over 4GB. There is still widespread misinformation posted about this, even after all the time taken by various MS engineers (such as the most amusing Raymond Chen) to clarify the facts.

Let's set the record straight.

YES! 32-bit versions of windows can take advantage of more than 4GB of physical RAM.

Example 1: The server editions.
I refer the reader to Memory Limits for Windows Releases for the canonical document.

On these systems, the application is of course limited to 2GB of userspace virtual address space. Using the (again, oft misunderstood) /3GB switch, applications that are compiled with the appropriate flag can avail themselves of a 3GB virtual address space. Utilizing processor and OS PAE capabilities, and Windows APIs such as AWE, or via Kernel mode drivers, such applications can map whatever physical RAM address space the machine exposes into their particular virtual address space. This is how things like SQL Server can have huge memory use (well over any 4GB limit) on such versions of the OS.

Far too many Internet 'experts' advise readers to use things like the /3GB switch to 'increase how much memory your program can use', completely missing that fact that the application must be compiled using the appropriate flag to do this. This can be done 'after the fact' with EDITBIN or a Hex editor, but unless you're sure the coders of the application were sane, or you've otherwise analysed the binary, you might be asking for trouble. In addition, programs compiled with '/GL' (Whole Optimization) cannot be modified using accepted post-compile methods. The end result is the reader ends up burning 1GB of address space that the OS could use for the Kernel, that ends up just sitting there wasted. I would direct the interested reader to the aforementioned Raymond's blog for an amusing, multi-year expose of this.

Example 2: The 'client' editions.
Things get a little more confusing for the 'client' versions of Windows (the consumer versions.) For these, Microsoft decided after rigorous testing that the problems exhibited by drivers and other low-level code would present too many problems when faced with RAM addresses beyond 4GB, and so disabled the 'awareness' of the OS for these versions of the OS for RAM beyond 4GB.

However, properly coded applications and drivers (e.g., Ram Disk software by SuperSpeed) can in fact utilize RAM beyond 4GB even in the 'client' 32-bit Windows versions. I used to do this every day, until the remaining reasons not to use a 64-bit OS were eliminated for me.

Here's a screenshot of this very application, running on a friend's PC with 6GB of physical RAM, providing a 5GB RAM Disk. Note that the 2GB of physical RAM above 4GB is being used perfectly well by this application running under 32-bit Windows (click to see large original):

Photobucket

There were valid reasons for using such 'work-arounds' for 32-bit systems to use more than 4GB of RAM in the past: driver vendors had not kept up with 64-bit systems, and often had either no drivers, or problematic ones. Those days seem long gone, and at this point, there's really no good reason not to use a fully 64-bit capable operating system, even on consumer PCs.

Nonetheless, the fact remains: Yes, Virginia, 32-bit windows can use more than 4GB of ram.

Update:

Finally! Someone at Microsoft read this, perhaps? As of early 2011, the Microsoft Developer Network's canonical document "Memory Limits for Windows Releases" has been properly updated with the correct facts:

X86 client versions with PAE enabled do have a usable 37-bit (128 GB) physical address space. The limit that these versions impose is the highest permitted physical RAM address, not the size of the IO space. That means PAE-aware drivers can actually use physical space above 4 GB if they want. For example, drivers could map the “lost” memory regions located above 4 GB and expose this memory as a RAM disk.

About time...

Saturday, April 10, 2010

The Six Degrees of Web Separation

Watching information travel through the web is always fascinating. A recent document I published here Troubleshooting Multi-Player PC Game Connectivity Issues has found its way to a forum in Germany, where it apparently is recommended by a DICE dev. Pretty cool!

For the English speaking crowd a commenter directed me to a showing at Bash and Slash, a very cool PC gaming news and opinion site with associated forums and home of "the host of the wildly hilarious and incredibly popular, FPS gaming webcast, BASH." Check them out!

Thanks go out to the poster at the forums at CainsLair, glad you found the information useful!

Another interesting site, Penny Arcade links to the guide. Nearly four million posts in their Games and Technology forum. Wow!

A U.K. site TAG Home of the Tactical Assault Group, show the guide. Nice site, these guys have been around for a few years, with a new focus on BFBC2.

I'll update this entry as I hear of more places linking here, if you care to play the "Where's Waldo" of the web, feel free to comment here.

Regrettably, I've been forced to moderate comments: There are some pretty nasty people out there that apparently didn't like their "You gotta forward your ports!" mantra being shredded with facts.

Wednesday, April 7, 2010

What programming languages should you learn?

I get asked periodically "What programming language should I learn to build my own games?".

Most games built today are done in C and/or C++. So I suppose, if you are looking to enter the game development field, learning one or both of those should be a priority. I like to think of the question in more general terms myself, more along the lines of "What programming language should I learn if I want to learn how to program?"

Having taught Lisp to engineers, I've seen the value of exposure to different paradigms in developing maturity in programmers. Were I to build the 'ideal' coursework to teach a 'How to Program, from Scratch' curriculum, I think I'd follow this course:

1) Assembler. Little practical use for building a new piece of software. Frankly, the speed of code produced by modern compilers of high-level languages is so good that in most cases only the most experienced human coders can better them. And yet, by learning this most basic lingua franca of the machine, the budding programmer will be exposed to thinking in the same way the machine 'thinks'. I for one believe this leads to more sane considerations as to the efficiency of their code. Of course, the ability to debug/disassemble/analyze applications where the source code is unavailable means knowledge of assembler is invaluable.

2) C. The mother tongue of many operating systems, still one of the most widely used application development languages. A good first step for the beginning programmer to get a foundational understanding of basic programming techniques and structure.

3) Java. Introduces, in a framework easily grasped by someone with C experience, the concepts of Object Oriented programming. One could argue there are other environments that do the job better (Ruby, Python, etc.), but the wealth of excellent teaching materials, and the availability of real jobs in the real world, makes this my choice to teach the OO concepts.

4) Erlang. Originally developed by Ericsson, this language introduces the learner to the world of distributed, fault-tolerant, concurrent programming. The conciseness of the language for its intended purpose allows for easy experimentation in these arenas.

5) Forth. Stack based, non-explicit grammar. I include this as an excellent environment for the learner to look at problems from a completely different angle.

6) Lisp. Functional programming at its best. Were I forced to pick one language for someone to learn if they are learning to program from ground zero, this would be it. Lisp has had features and functionality that are still unequalled overall by any other language. Even though every other language of import has evolved over time, adding features and capabilities (and becoming more lisp like), this still holds true. The mind expansion that results from really understanding and using the capabilities of the language will make one a better programmer in any language, even if they never use lisp again. The electric acid kool aid test for new (and experienced) programmers.

7) Mathematica. What? A mathematical computational software system? Yes! I use this product for various modeling and mathematical processing tasks. It utilizes an extremely sophisticated programming language that can be used in any of the traditional programming modes (imperative, OO, functional, rules-based, etc.), and can combine these methods in a single code chunk. The term rewriting reduction system that pervades Mathematica makes for one-liners that can require many lines of code, or even pages, in other languages. Memoization? In Mathematica, the one line:

Fib[0]=0; Fib[1]=1; Fib[n_] := Fib[n] = Fib[n-1] + Fib[n-2]

Gives us the Fibonacci function, fully memoized. Compare this to a Python (already a more concise language than most) implementation:

memo = {0:0, 1:1}
def fib(n):
      if not n in memo:
               memo[n] = fib(n-1) + fib(n-2)
       return memo[n]

Isn't the Mathematica solution so much more elegant and readable?

Not only does this allow for remarkably rapid 'proof of concept' experiments, but it also serves as an excellent environment to experiment with the many ways of solving the same problem. If I'm not using Lisp to prototype something, I'm probably using Mathematica. I used Mathematica to create the animated image of the Sun's analemma and the relation to the equation of time, by calculating the Earth's orbital elements and their effect on true and mean solar time. You can take a peek at it on the wikipedia entry Equation of Time, or the image itself at Analemma and EOT.

I'd probably throw in some SQL for good measure. This bag-oriented (that's right, not set-oriented, nor relational for that matter) language is the most widely used query language for large database systems. That the thought process needed to produce well-formed, optimal queries using this is different enough in concept from the paradigms of the previously mentioned languages, not to mention any large data based system is likely to have some form of DBMS under the covers that uses SQL, makes this a worthwhile addition.

Each of these I think forces the programmer to think differently enough in how to approach a problem they can't help but end up with an expanded 'thought portfolio' to use in attacking future problems.

Your thoughts?

Tuesday, April 6, 2010

How could it possibly be my router causing connection issues?

Readers of game enthusiast forums have undoubtedly seen the sometimes heated exchanges between the It's all gotta be the GAME, crap developers! crowd, and the It could be a problem on your end gang.

I'm of the latter, if you've read any of my posts.

I produced a guide based on helping users in BFBC2 and other games, called Troubleshooting Multi-Player PC Game Connectivity Issues that covers many of the often subtle conditions that can cause connectivity issues for online games. This is rather long - there are many interacting factors that can cause these problems. Some may not have the time to read it, some just won't, sticking to their guns of It's all gotta be the GAME!

I've been attacked by a few, questioned by many. That's fine (the latter, at least), skepticism is good. Ignorance isn't!

I've compiled a list of references for those that want to take the time to see what experts in the field, and real users of many other games, have to say on this matter. In particular, there are much more detailed NAT technical information and the issues involved covered in some of these. It was not practical for me to include this level of detail in my document - it would have become a hardbound book!

If you're really interested in answering the question How could it possibly be my router causing connection problems? , these will help you get there. It will also help you to understand how things can be fine in every other game yet broken for a certain few games, with different users having differing experiences.

I've also included links to forum posts for many games, and many platforms, clearly showing that this kind of issue happens all the time.

Perhaps with a clearer understanding, these may help you solve your own connection problems.

They should also help you to dispel the myth of 'port forwarding has to be used' often repeated in forums. A read of Port Forwarding: Slaying the Mythical Dragon of Online PC Gaming will clarify how NAT and port forwarding are related, and why forwarding ports blindly is unneeded and potentially problematic when used.

Good reading!

References for NAT technologies and issues involved:

Superb overview by Geoff Huston of CISCO
http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_7-3/anatomy.html

Nice overview by the author of RakNet, with success/failure charts:
http://www.jenkinssoftware.com/raknet/manual/nattypedetection.html
http://www.jenkinssoftware.com/raknet/manual/natpunchthrough.html

IETF recommendations for NAT behavior:
http://www.ietf.org/rfc/rfc4787.txt

A very good overview, with diagrams:
http://en.wikipedia.org/wiki/Network_address_translation

For the absolute bibles for NAT and other TCP/IP technical information:
http://preview.tinyurl.com/yefzpfv
http://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-Protocols/dp/159327047X

AnalogX NAT and Nat traversal issues overview:
Interesting test of 100 consumer routers. DGL-4300 top rates, Apple /3com/US Robotics the worst.
Only 43% of routers properly supported full cone NAT.
http://www.analogx.com/contents/articles/nattraversal.htm

A game developer talks about problems with NAT of differing types by consumers:
"If it is a router, it's the user's problem to solve it."
http://forum.unity3d.com/viewtopic.php?p=233448

Other forums for games where users have the same issues some have with game "X".
These same users could well be playing game "Y" without issue.

http://forums.gamesforwindows.com/p/1860/23980.aspx
http://utforums.epicgames.com/showthread.php?t=665969
http://support.microsoft.com/kb/840420 (Microsoft? What would they know?)
http://utforums.epicgames.com/showthread.php?t=602500
http://forums.epicgames.com/showthread.php?t=616452
http://www.dslreports.com/forum/r19418084-Gaming-Mode-Why-does-Dlink-recommend-disabling
http://www.dslreports.com/forum/r20554921-Xtreme-NAT-help-with-Netopia-2241n006
http://support.microsoft.com/kb/941207 (More Microsoft. Maybe they know something about networking?)
http://www.gtaforums.com/index.php?showtopic=353023
http://forum.instantaction.com/smf/index.php?topic=3631.0
http://blogs.msdn.com/johnmil/archive/2006/10/29/nat-traversal.aspx
http://www.gtagaming.com/forums/archive/index.php/t-103945.html
http://www.xfire.com/nat_types/ (widely used Xfire, and problems NAT can cause in the wrong router)
http://forums.electronicarts.co.uk/fifa-10-sony-playstation-3-microsoft-xbox-360/840675-game-no-longer-available-still-9.html
http://www.ureadit.com/solutions/home-network/79-xbox-live-compatible-router.html
http://www.gtagaming.com/forums/archive/index.php/t-103841.html
http://forums.eu.atari.com/archive/index.php/t-59626.html
http://www.bing.com/search?q=full-cone+nat+games&first=51&FORM=PORE
http://forums.eu.atari.com/archive/index.php/t-62799.html
http://openarena.ws/board/index.php?topic=3261.0
http://www.poweredbygamespy.com/services/view/category/connect/ (Gamespy - they brag at only having 10% failure of NAT)
http://boardreader.com/thread/Port_Restricted_Cone_Nat_Router_l9xyXvexg.html
http://text.broadbandreports.com/forum/r22505721-HSI-Known-NAT-problem-with-Charter-HSI
http://computerhelpforum.org/forum/networking/f43/full_cone_nat/t18037.html
http://forums.gametrailers.com/thread/nat-type-1-question/1042731
http://forums.epicgames.com/showthread.php?t=665969
http://www.bgforums.com/forums/viewtopic.php?f=58&t=12277

There are a myriad more, Google is your friend!

Thursday, April 1, 2010

I'm one proud dad!

My daughter found out today she was accepted to the Stanford SIMR program, an incredibly selective medical research internship over the summer. Usually seniors in HS are accepted, she got in as a junior. Way to go, JC!

Same day, she found out she got a perfect 2400 SAT, first try, as a junior. Somewhere between 20 and 100 out of the 1-1.5 million combined pool of junior and senior test takers get a perfect score in any given year, even fewer excluding the new 'super score' rules that combine the best scores from multiple test attempts.

I always knew my kid was smarter than me!

Congratulations, JC, for getting such awesome results from your hard work and dedication to learning, I'm very proud of you!