So much has changed since I last blogged about making Epic Frontiers, I decided to address something that many developers have a hard time with: Server Architectures. Now, server architecture is something that is more of an IT skill than a game design skill, but with the proliferation of social gaming and online-enabled gaming in general, it’s something that designers need to be very well aware of. As a matter of discussion, I’ll talk about server architecture as it relates to one of my own projects, an MMORPG demo named Epic Frontiers.

Now, Epic Frontiers was based on a T3D server which connected to a MySQL database directly, serving all of the information to the client machine through the game server. It’s an old architecture that looks like so (I’m omitting Authentication and Master Servers because they are mainly involved only in the beginning of the game interaction cycle):

 

 

That’s a bad network design for a couple of reasons:

 

• Security: First and foremost, if that T3D server gets compromised, the hacker has a direct connection to the database, and you’re pretty much dead in the water. It’s as simple as that.
• Server Load: The T3D server is already doing so much for the game world that adding database queries to the list of things to do is asking a bit much. Throw 200 players onto a T3D server, and with all the crafting, trading, and combat, the server might start sparing a CPU cycle or two towards plotting to murder you in your sleep. Also, if you have any sort of scaling system in place, your server is probably going to take advantage of it earlier than you might want it to, translating into higher costs of running the game.
• Complexity: There’s easier ways to get your data than to assemble SQL statements using Torque Script, and mixing database calls and bugs with gameplay just means that your debugging gets that much more complex.

So, “a couple” means three to me. It actually means two, but being a programmer, I’m counting from 0, so there you go… Anyway, going by the above, there was an iteration of server architecture that solved the above issues (two and a half of them, anyway) by placing a web-server between the T3D server and the database server. What this did was make the T3D server ask a web server to fetch the data, which it then passed to the player. The network architecture looked like so:

 

It’s a more secure design, in that if a hacker got into the server running T3D, they would then need to hack into the web server, which was between the T3D server and the database server- or they could try directly for the database server, but at least the password wouldn’t be sitting inside a T3D instance, and you could enforce some very strict security rules on the database server that could force them to have to compromise the web server first or else be very creative.

Secondly, you’ve taken load off of the server by using things like TCPObject and HTTPObject to ask a web server for data, which is then fetched by that web server. T3D just uses the data whenever it comes in, and issues in grabbing that data won’t impact the entire server (unless the data relates to the operation of the entire server). So that will translate into lower operating costs over time, and higher performance of the server.

And second-and-a-halfly: Complexity. This is only a half-point, because T3D is still making all of the database calls on behalf of the player, not to mention all of the calculations for some of these things. Want to craft something? Your request goes to the T3D server, which then queries the database, and then uses that information to figure out if you’ve successfully crafted your thing, and then it notifies you. Hurray- you’ve just made T3D do something that you can offload to an app that was made specifically for the task!

Wait…what?

That’s right- what are you doing asking T3D to mediate crafting, which (depending on your gameplay mechanics, but I’m assuming your run-of-the-mill MMO crafting here) doesn’t really happen in the game world? Think about it: When a player crafts, their client plays an animation while in the background, the game server tries to figure out the crafting mechanics. That can be done either in Torque Script or in code, but either way, it’s being done by T3D, which has lots of other things to worry about, like positions of player, physics (if you’re using it), AI, etc. But crafting doesn’t have to happen within the T3D server to happen in the game, does it? I mean, we’re already having T3D ask a web-server for which items are in a player’s backpack when they click on their inventory, right? Wouldn’t it be easier to offload crafting to an application server running a crafting app which is optimized to just figure this stuff out? Can’t crafting queries be directed to that application server using TCPObjects and/or HTTPObjects in the same way that database queries can?

 

 

That network architecture would look like this:

 

 

What we’re doing here is implementing application servers that take the workload for non-realtime gameplay mechanics off of T3D’s already burdened back, and turn it into a set of queries which T3D fires off to be done, relaying the results once that’s done. As you can see, this requires a server for that application, and so in a scaling scenario, you’re probably going to scale your crafting server at a different rate than that of your T3D servers, because a single crafting server can almost certainly handle many T3D servers’ queries. And your T3D servers will run much better, and you’ll probably fit a few more players into your zones as a result.

Going further (but not by much, since it’s been mentioned several times in years past by others), you can offload your chat functions to a specific chat server (or IRC server, for that matter), and any other functionality which doesn’t need realtime feedback. Offload combat? Probably not- unless it’s some kind of turn-based mechanic which can stand waiting a second or so for the player to get a result without the AI eating him alive during that time. For Epic Frontiers, I have an NPC conversation feature that is quite logic and database intensive. Offloading that would take a huge load off of the server. The same goes for some of the AI functionality which, while relatively fast, is not always needed for realtime use (the NPCs aren’t the only things with AI in the game). Offloading it means not only does the server not have to process that itself, but also that I can devote a few more cycles to it.

That’s all the good news! Now, because I can’t be honest without doing this, here’s the bad news:

• Network Complexity: The network will be more complex as you have more servers to manage. If an application server goes down and you do not have some kind of load-balancing in place, you’re in for a world of hurt as portions of the game simply no longer work (“every time I try to craft I lose a hammer? OMGWTFBBQ!!!11!11!one1!”). So if you’re going to go down this road, you need to have both scaling and load-balancing in place to ensure that these portions of your game service run as smoothly as your T3D instances. Additional servers also means additional cost, though as a Crafting Server would likely need less resources to handle crafting queries from multiple T3D servers, this would not be as big a hit as having to spawn more T3D servers because the crafting slows them down.

 

 

• Design Complexity: Yes, I did say previously that your T3D code would be simplified because the work previously done by T3D basically gets reduced to web queries. But you can’t get anything back from those web queries unless there’s something doing something with that query and returning information- and that requires writing specialized applications that do nothing but crunch that data. Just as you can implement a Master Server using PHP, ASP, AS3, C/C++/C#, or anything else, so can you implement a Crafting Server in any of those languages (though I’d recommend a low-level language, for performance reasons). This means that changes to a query function in your Crafting Server may dictate changes to the query formation in T3D. A bug in T3D may mean changes to Crafting Server code. These two applications need to talk to each other (securely- even on the back end!), and that means you need to design them with each other in mind.

 

Now, I realize that this is quite a bit of food for thought for some of you Indies who want to make an Online (MMO or otherwise) game and already find the task daunting, but just remember that in this case, the crafting code is already needed, and the additional overhead of wrapping that code in its own application and spinning up a server for it is minimal in comparison to going further down the road and trying to wring more performance out of a server that is already doing too much.

Designers are creative people, and Indie designers have to be both creative and technical in order to be viable. Be creative with technology, leverage flexibility, and automate as much as you safely and responsibly can. And of course, the above is not at all the final word on network design for online games, and as someone who has spent 13 years in IT can tell you, network architectures are custom things. Customize your network to serve your needs!

Design, Networking, Production architecture, conversation, epic frontiers, game engine, games, mmo, mmorpg, morpg, network, npc chat, npc conversation, online, orpg, rpg, T3D, tools, Torque

It’s been a long time- I don’t even know where to start, so we’ll do this in a categorized rather than chronological order…

The Demo: My last blog was posted a day before my birthday, on September 19th, and during that time, I’ve accrued a team of very talented and dedicated guys and girls who are putting equally nice artwork into the game. Using contacts that I made at Austin GDC, I’m also able to bring motion-capture animation into the game through Mixamo. Having met a really good concept artist in Austin, I was also able to create and distribute a thousand fliers at VGXPO in Philly three weeks ago, bringing in a modest bump in web traffic. This effort will be replicated for the various Comic Cons that roll through New York next year, each one bringing in thousands of people (as opposed to VGXPO which brought in maybe one or two thousand in total).

At the same time, I’ve moved the database to a test server- an XP machine that sits on my desktop and does nothing else, except maybe drip liquid coolant onto the RAID array now and then (that reminds me, I need a CPU fan). We’ve had team members from as far away as Ireland log into the game and test while, ironically, one of our guys just 50 blocks south of me in Brooklyn wasn’t able to get on. This allowed me to crush a bunch of bugs that popped up only under online conditions.

Next, we’ve started moving the behavioral AI into the game and testing that, which continues today. We have the beginnings of a rotating overhead mini-map (not ready as a resource yet), and we’ve almost solved the mouselook-and-select-objects issues that dog many people (that is also not yet ready as a resource- it’s kind of spongy right now). Drag and drop functionality is back in and we’ll be adding some crafting recipes and materials this week coming up, and mission generation is also coming online.

And finally… Outsourcing. Because of the time-crunch and the failure to get as many 3D artists as I would have liked, I’ve resorted to outsourcing some tasks. It’s not because I have deep pockets, but because the demo tasks need to get done. In any event, I have some advice for would-be outsourcing/contracting people reading this: Don’t promise anything you don’t know how to deliver. I’m not about to mention names, because mistakes become past mistakes quickly and though I wound up canning the contractor, it doesn’t mean their work doesn’t get better. In fact, the modeling and texturing was done very well, but the client doesn’t need to hear about your internal problems. Be a black box for me, would you? Thanks.

So, we’re in the final two-week-stretch here, and I’m starting to work from about 9am to about 1am every night, with a few hours dedicated to my girlfriend most nights, but otherwise huddled over the laptop beating my head against walls until they crumble. And far from technical, I need to work on budgets and project timelines for the prospective publisher as well- they do like to know how much money and time a project will cost them. If any of you wannabe MMO creators out there hate business, then you better learn to love it, and fast, too.

AI: What can I say? Interrogative is no more. In fact, it is more. Having started to morph from a feature for the MMO into something that can be used generically for games, into something that I’m intending to throw at the Turing Test competition, and even beyond (shhhh), the name Interrogative is no longer proper. Hierarchical opinion-based knowledge representation systems using generalized pre-compiled dialog along with dynamically-created dialog with interrogative and natural language processing front-ends is no longer what Interrogative is or was. It’s not even tested yet, but the theories are there, and having run it past a number of industry heavies, I can confidently say that I’m on the right track with my experiments.

At the lower level of things, Interrogative is alive and well in Epic Frontiers, and we have about a dozen new conversation keywords and dialog sets to throw into the system for testing. After that, proper changes will be made to support the Infidel opinion-based architecture we want to implement, which will allow the game to support the players changing the minds of AI, as well as AI reactions to players and other AI who do not share their own opinions (hence the name Infidel, which means “non-believer” and can be applied to anyone who does not share your belief in whatever subject you feel passionately about).

With all this AI tech springing to mind, I’m thinking about spinning off a company to hold all that technology, so that I can continue to develop it without interference from video game contracts and such…

GameX Summit: Well, the last conference I attended for the year was just great! Met up with Dave Mark, Mike Worth, and many others and had a great time. For a first-time summit, the mistakes were minor, and the recovery from them pretty swift.

Unfortunately for me, the students that showed up were almost all design students, and not artists, as I was hoping, for recruitment sake. That said, I also met a lot more people interested in AI, and attended a couple of mind-blowing sessions by people such as Clint Hocking, Dave Mark and Kevin Dill, Damian Isla, and Mike Worth, whose audio session simply rocked.

The only real downside to the conference was getting lost. I don’t know what it is about the Pennsylvania hillside, but I’ll be damned if I didn’t wind up in the Valley Forge National Park at 11:30pm one night staring at over twenty deer in my headlights. And it wasn’t that “ZOMG what kind of car is that” stare, but the “dude, this park belongs to us after dark” stare… Interesting area. But the food out there is pretty good.

That’s about it for the blog for now. The next blog will have more eye-candy

ai, Production ai, artificial intelligence, budgets, contracting, conversation, demo, mmo, mmorpg, morpg, npc, npc chat, npc conversation, orpg, outsourcing, paperwork, publishing, rpg