Friday, May 30, 2008

How to Evaluate Middleware

Most commercial games have got some middleware in them somewhere. Some types of middleware include video players, sound systems, physics systems, UI systems. I've already mentioned a few of the pro's and con's of using middleware in a previous post.

You need to be careful when choosing middleware. If you rush in, you could choose a system that doesn't do what you need, costs more than you need to pay, or gives you some nasty surprises down the road.

Here's a breakdown of some recommended steps in choosing middleware. Some are obvious, but some are less so.

  • Write a list of all the things you want the middleware to do, or be used for.
  • Do some searching and find all the different options which exist. Don't dig into the details of any of them just yet, just find out all your options so you've got the best chance of finding a good fit.
  • Ask other game programmers at your studio if they've used any of the systems in the past. If you have sister studios, then ask them too, and also see if they have hand written solutions which you could reuse.
  • Make a comparison spreadsheet of the high level features of each option, how well it meets your needs, and the pricing.
  • If you are developing for consoles, you likely need to be more concerned by how the engine component behaves. Ask whether you can override file access and memory allocation. Also ask if they use STL, or anything like it, and what their stance is on allocating memory at run time.
  • If your game is going to be localized, and the middleware will be using localized assets, work out how it will handle the different languages. E.g. Can the movie player handle different language tracks? Will the sound tool help you find wav files which haven't been translated? Will the UI support the large character sets needed for Japanese and Korean?
  • Consider how much of an evaluation you can do without coding. Integrating middleware at a code level takes a long time. Can you evaluate tools first?
  • Pick one or two options to evaluate in a hands-on fashion, and check with your boss and your legal advisor that they're OK with the concept of you using it, the price, and the license terms.
  • Integrate it with your engine in a code branch. Don't start relying on it yet though. It's easy to slip into assuming that you're using it.
  • Create a stress test to push it to its limits.
  • Check that it can do everything you need it to (to a reasonable degree of accuracy).
  • If you're focused on such things, check the memory allocations made, performance, and amount of memory used.
  • Have a review meeting in which you go over your findings and decide whether to officially approve the system for use.
  • If approved, then you can integrate your code branch to the main branch, and encourage people to start using it.

If you've used middleware before, it's fun to look at the above list and see the mistakes you made in the past :)

No comments: