Wolfram|Alpha: Systematic knowledge, immediately computable.

Wednesday, April 14, 2010

Yes, Virginia, 32-Bit Windows can use more than 4GB of physical RAM...

In one of the forums I frequent, a debate broke out in a thread over whether 32-bit editions of Microsoft Windows can in fact handle or otherwise use any physical RAM in a system over 4GB. There is still widespread misinformation posted about this, even after all the time taken by various MS engineers (such as the most amusing Raymond Chen) to clarify the facts.

Let's set the record straight.

YES! 32-bit versions of windows can take advantage of more than 4GB of physical RAM.

Example 1: The server editions.
I refer the reader to Memory Limits for Windows Releases for the canonical document.

On these systems, the application is of course limited to 2GB of userspace virtual address space. Using the (again, oft misunderstood) /3GB switch, applications that are compiled with the appropriate flag can avail themselves of a 3GB virtual address space. Utilizing processor and OS PAE capabilities, and Windows APIs such as AWE, or via Kernel mode drivers, such applications can map whatever physical RAM address space the machine exposes into their particular virtual address space. This is how things like SQL Server can have huge memory use (well over any 4GB limit) on such versions of the OS.

Far too many Internet 'experts' advise readers to use things like the /3GB switch to 'increase how much memory your program can use', completely missing that fact that the application must be compiled using the appropriate flag to do this. This can be done 'after the fact' with EDITBIN or a Hex editor, but unless you're sure the coders of the application were sane, or you've otherwise analysed the binary, you might be asking for trouble. In addition, programs compiled with '/GL' (Whole Optimization) cannot be modified using accepted post-compile methods. The end result is the reader ends up burning 1GB of address space that the OS could use for the Kernel, that ends up just sitting there wasted. I would direct the interested reader to the aforementioned Raymond's blog for an amusing, multi-year expose of this.

Example 2: The 'client' editions.
Things get a little more confusing for the 'client' versions of Windows (the consumer versions.) For these, Microsoft decided after rigorous testing that the problems exhibited by drivers and other low-level code would present too many problems when faced with RAM addresses beyond 4GB, and so disabled the 'awareness' of the OS for these versions of the OS for RAM beyond 4GB.

However, properly coded applications and drivers (e.g., Ram Disk software by SuperSpeed) can in fact utilize RAM beyond 4GB even in the 'client' 32-bit Windows versions. I used to do this every day, until the remaining reasons not to use a 64-bit OS were eliminated for me.

Here's a screenshot of this very application, running on a friend's PC with 6GB of physical RAM, providing a 5GB RAM Disk. Note that the 2GB of physical RAM above 4GB is being used perfectly well by this application running under 32-bit Windows (click to see large original):

Photobucket

There were valid reasons for using such 'work-arounds' for 32-bit systems to use more than 4GB of RAM in the past: driver vendors had not kept up with 64-bit systems, and often had either no drivers, or problematic ones. Those days seem long gone, and at this point, there's really no good reason not to use a fully 64-bit capable operating system, even on consumer PCs.

Nonetheless, the fact remains: Yes, Virginia, 32-bit windows can use more than 4GB of ram.

Update:

Finally! Someone at Microsoft read this, perhaps? As of early 2011, the Microsoft Developer Network's canonical document "Memory Limits for Windows Releases" has been properly updated with the correct facts:

X86 client versions with PAE enabled do have a usable 37-bit (128 GB) physical address space. The limit that these versions impose is the highest permitted physical RAM address, not the size of the IO space. That means PAE-aware drivers can actually use physical space above 4 GB if they want. For example, drivers could map the “lost” memory regions located above 4 GB and expose this memory as a RAM disk.

About time...

Saturday, April 10, 2010

The Six Degrees of Web Separation

Watching information travel through the web is always fascinating. A recent document I published here Troubleshooting Multi-Player PC Game Connectivity Issues has found its way to a forum in Germany, where it apparently is recommended by a DICE dev. Pretty cool!

For the English speaking crowd a commenter directed me to a showing at Bash and Slash, a very cool PC gaming news and opinion site with associated forums and home of "the host of the wildly hilarious and incredibly popular, FPS gaming webcast, BASH." Check them out!

Thanks go out to the poster at the forums at CainsLair, glad you found the information useful!

Another interesting site, Penny Arcade links to the guide. Nearly four million posts in their Games and Technology forum. Wow!

A U.K. site TAG Home of the Tactical Assault Group, show the guide. Nice site, these guys have been around for a few years, with a new focus on BFBC2.

I'll update this entry as I hear of more places linking here, if you care to play the "Where's Waldo" of the web, feel free to comment here.

Regrettably, I've been forced to moderate comments: There are some pretty nasty people out there that apparently didn't like their "You gotta forward your ports!" mantra being shredded with facts.

Wednesday, April 7, 2010

What programming languages should you learn?

I get asked periodically "What programming language should I learn to build my own games?".

Most games built today are done in C and/or C++. So I suppose, if you are looking to enter the game development field, learning one or both of those should be a priority. I like to think of the question in more general terms myself, more along the lines of "What programming language should I learn if I want to learn how to program?"

Having taught Lisp to engineers, I've seen the value of exposure to different paradigms in developing maturity in programmers. Were I to build the 'ideal' coursework to teach a 'How to Program, from Scratch' curriculum, I think I'd follow this course:

1) Assembler. Little practical use for building a new piece of software. Frankly, the speed of code produced by modern compilers of high-level languages is so good that in most cases only the most experienced human coders can better them. And yet, by learning this most basic lingua franca of the machine, the budding programmer will be exposed to thinking in the same way the machine 'thinks'. I for one believe this leads to more sane considerations as to the efficiency of their code. Of course, the ability to debug/disassemble/analyze applications where the source code is unavailable means knowledge of assembler is invaluable.

2) C. The mother tongue of many operating systems, still one of the most widely used application development languages. A good first step for the beginning programmer to get a foundational understanding of basic programming techniques and structure.

3) Java. Introduces, in a framework easily grasped by someone with C experience, the concepts of Object Oriented programming. One could argue there are other environments that do the job better (Ruby, Python, etc.), but the wealth of excellent teaching materials, and the availability of real jobs in the real world, makes this my choice to teach the OO concepts.

4) Erlang. Originally developed by Ericsson, this language introduces the learner to the world of distributed, fault-tolerant, concurrent programming. The conciseness of the language for its intended purpose allows for easy experimentation in these arenas.

5) Forth. Stack based, non-explicit grammar. I include this as an excellent environment for the learner to look at problems from a completely different angle.

6) Lisp. Functional programming at its best. Were I forced to pick one language for someone to learn if they are learning to program from ground zero, this would be it. Lisp has had features and functionality that are still unequalled overall by any other language. Even though every other language of import has evolved over time, adding features and capabilities (and becoming more lisp like), this still holds true. The mind expansion that results from really understanding and using the capabilities of the language will make one a better programmer in any language, even if they never use lisp again. The electric acid kool aid test for new (and experienced) programmers.

7) Mathematica. What? A mathematical computational software system? Yes! I use this product for various modeling and mathematical processing tasks. It utilizes an extremely sophisticated programming language that can be used in any of the traditional programming modes (imperative, OO, functional, rules-based, etc.), and can combine these methods in a single code chunk. The term rewriting reduction system that pervades Mathematica makes for one-liners that can require many lines of code, or even pages, in other languages. Memoization? In Mathematica, the one line:

Fib[0]=0; Fib[1]=1; Fib[n_] := Fib[n] = Fib[n-1] + Fib[n-2]

Gives us the Fibonacci function, fully memoized. Compare this to a Python (already a more concise language than most) implementation:

memo = {0:0, 1:1}
def fib(n):
      if not n in memo:
               memo[n] = fib(n-1) + fib(n-2)
       return memo[n]

Isn't the Mathematica solution so much more elegant and readable?

Not only does this allow for remarkably rapid 'proof of concept' experiments, but it also serves as an excellent environment to experiment with the many ways of solving the same problem. If I'm not using Lisp to prototype something, I'm probably using Mathematica. I used Mathematica to create the animated image of the Sun's analemma and the relation to the equation of time, by calculating the Earth's orbital elements and their effect on true and mean solar time. You can take a peek at it on the wikipedia entry Equation of Time, or the image itself at Analemma and EOT.

I'd probably throw in some SQL for good measure. This bag-oriented (that's right, not set-oriented, nor relational for that matter) language is the most widely used query language for large database systems. That the thought process needed to produce well-formed, optimal queries using this is different enough in concept from the paradigms of the previously mentioned languages, not to mention any large data based system is likely to have some form of DBMS under the covers that uses SQL, makes this a worthwhile addition.

Each of these I think forces the programmer to think differently enough in how to approach a problem they can't help but end up with an expanded 'thought portfolio' to use in attacking future problems.

Your thoughts?

Tuesday, April 6, 2010

How could it possibly be my router causing connection issues?

Readers of game enthusiast forums have undoubtedly seen the sometimes heated exchanges between the It's all gotta be the GAME, crap developers! crowd, and the It could be a problem on your end gang.

I'm of the latter, if you've read any of my posts.

I produced a guide based on helping users in BFBC2 and other games, called Troubleshooting Multi-Player PC Game Connectivity Issues that covers many of the often subtle conditions that can cause connectivity issues for online games. This is rather long - there are many interacting factors that can cause these problems. Some may not have the time to read it, some just won't, sticking to their guns of It's all gotta be the GAME!

I've been attacked by a few, questioned by many. That's fine (the latter, at least), skepticism is good. Ignorance isn't!

I've compiled a list of references for those that want to take the time to see what experts in the field, and real users of many other games, have to say on this matter. In particular, there are much more detailed NAT technical information and the issues involved covered in some of these. It was not practical for me to include this level of detail in my document - it would have become a hardbound book!

If you're really interested in answering the question How could it possibly be my router causing connection problems? , these will help you get there. It will also help you to understand how things can be fine in every other game yet broken for a certain few games, with different users having differing experiences.

I've also included links to forum posts for many games, and many platforms, clearly showing that this kind of issue happens all the time.

Perhaps with a clearer understanding, these may help you solve your own connection problems.

They should also help you to dispel the myth of 'port forwarding has to be used' often repeated in forums. A read of Port Forwarding: Slaying the Mythical Dragon of Online PC Gaming will clarify how NAT and port forwarding are related, and why forwarding ports blindly is unneeded and potentially problematic when used.

Good reading!

References for NAT technologies and issues involved:

Superb overview by Geoff Huston of CISCO
http://www.cisco.com/web/about/ac123/ac147/archived_issues/ipj_7-3/anatomy.html

Nice overview by the author of RakNet, with success/failure charts:
http://www.jenkinssoftware.com/raknet/manual/nattypedetection.html
http://www.jenkinssoftware.com/raknet/manual/natpunchthrough.html

IETF recommendations for NAT behavior:
http://www.ietf.org/rfc/rfc4787.txt

A very good overview, with diagrams:
http://en.wikipedia.org/wiki/Network_address_translation

For the absolute bibles for NAT and other TCP/IP technical information:
http://preview.tinyurl.com/yefzpfv
http://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-Protocols/dp/159327047X

AnalogX NAT and Nat traversal issues overview:
Interesting test of 100 consumer routers. DGL-4300 top rates, Apple /3com/US Robotics the worst.
Only 43% of routers properly supported full cone NAT.
http://www.analogx.com/contents/articles/nattraversal.htm

A game developer talks about problems with NAT of differing types by consumers:
"If it is a router, it's the user's problem to solve it."
http://forum.unity3d.com/viewtopic.php?p=233448

Other forums for games where users have the same issues some have with game "X".
These same users could well be playing game "Y" without issue.

http://forums.gamesforwindows.com/p/1860/23980.aspx
http://utforums.epicgames.com/showthread.php?t=665969
http://support.microsoft.com/kb/840420 (Microsoft? What would they know?)
http://utforums.epicgames.com/showthread.php?t=602500
http://forums.epicgames.com/showthread.php?t=616452
http://www.dslreports.com/forum/r19418084-Gaming-Mode-Why-does-Dlink-recommend-disabling
http://www.dslreports.com/forum/r20554921-Xtreme-NAT-help-with-Netopia-2241n006
http://support.microsoft.com/kb/941207 (More Microsoft. Maybe they know something about networking?)
http://www.gtaforums.com/index.php?showtopic=353023
http://forum.instantaction.com/smf/index.php?topic=3631.0
http://blogs.msdn.com/johnmil/archive/2006/10/29/nat-traversal.aspx
http://www.gtagaming.com/forums/archive/index.php/t-103945.html
http://www.xfire.com/nat_types/ (widely used Xfire, and problems NAT can cause in the wrong router)
http://forums.electronicarts.co.uk/fifa-10-sony-playstation-3-microsoft-xbox-360/840675-game-no-longer-available-still-9.html
http://www.ureadit.com/solutions/home-network/79-xbox-live-compatible-router.html
http://www.gtagaming.com/forums/archive/index.php/t-103841.html
http://forums.eu.atari.com/archive/index.php/t-59626.html
http://www.bing.com/search?q=full-cone+nat+games&first=51&FORM=PORE
http://forums.eu.atari.com/archive/index.php/t-62799.html
http://openarena.ws/board/index.php?topic=3261.0
http://www.poweredbygamespy.com/services/view/category/connect/ (Gamespy - they brag at only having 10% failure of NAT)
http://boardreader.com/thread/Port_Restricted_Cone_Nat_Router_l9xyXvexg.html
http://text.broadbandreports.com/forum/r22505721-HSI-Known-NAT-problem-with-Charter-HSI
http://computerhelpforum.org/forum/networking/f43/full_cone_nat/t18037.html
http://forums.gametrailers.com/thread/nat-type-1-question/1042731
http://forums.epicgames.com/showthread.php?t=665969
http://www.bgforums.com/forums/viewtopic.php?f=58&t=12277

There are a myriad more, Google is your friend!

Thursday, April 1, 2010

I'm one proud dad!

My daughter found out today she was accepted to the Stanford SIMR program, an incredibly selective medical research internship over the summer. Usually seniors in HS are accepted, she got in as a junior. Way to go, JC!

Same day, she found out she got a perfect 2400 SAT, first try, as a junior. Somewhere between 20 and 100 out of the 1-1.5 million combined pool of junior and senior test takers get a perfect score in any given year, even fewer excluding the new 'super score' rules that combine the best scores from multiple test attempts.

I always knew my kid was smarter than me!

Congratulations, JC, for getting such awesome results from your hard work and dedication to learning, I'm very proud of you!