Engine Architecture

I simplified the usage of shared state a bit, but I need your input on how to handle shared data from Lua. As I said, shared data is triple buffered, the three buffers being "latest", "stable" and "working copy". Now, say we have a property "health" of a component that has to be shared across threads and is thus triple buffered. If we write from Lua into this property, the new value is always written into the working copy, so that's unambiguous. But what about reading the value? We can take the value from either the "latest" or "working copy" buffers.

In most cases, reading the latest buffer would be confusing, as it wouldn't reflect changes you just made:

Code:

oldHealth = player.health
player.health = player.health - 10
newHealth = player.health
if oldHealth == newHealth then
    # Wait, what?
end

That wouldn't happen if properties would be read from the working copy. But in other cases, it's useful to have an old, but consistent copy of the data available. I can't think of a sensible example for this right now, though.

Here are the options I see, feel free to propose more if you have an idea.

1. Read from latest, write to working copy
It's a simple solution, but will confuse script authors that don't read the manual. I wouldn't recommend it. (See edit at bottom)

2. Explicit Buffers
We introduce special fields for the Lua objects to explictly read from buffers other than the working copy:

Code:

# Read from latest buffer
latestHealth = player._latest.health
# Read from stable buffer
stableHealth = player._stable.health
# Read from working copy
newHealth = player._workingCopy.health
# Write to working copy (no choice here)
player.health = 10

Of course, this would preclude properties named "_latest" or "_stable", but that's why I prepended the underscores. We could explicitly forbid any property names starting with an underscore and reserve them for special stuff like this.


3. France it, we'll do it live!
Ha, take that, profanity filter! Ahem... anyway, Lua scripts will just access the working copy for both reading and writing, totally oblivious of the buffering. This is easy to implement and we can still implement option 2 at a later point if we feel the need. (Not a good idea, see edit below)


Edit: I just realized a real problem with only reading the working copy. When the working copy buffer is locked, it isn't updated with the data from the latest buffer. So the data in the working copy might be several frames old. In short, option 3 is out of the question.

One second, how does this work? One thread is reading from state A, and writing to state B, overriding whatever you had there? If that's it, i see a big problem:

Let's say we are modifying the health because the creature got attacked.

player.health = player.oldhealth - 10;

After that, a different system realizes that the creature is starving, and it reduce the health again:

player.health = player.oldhealth - 5;

You see the problem? Health was only reduced by 5 points, while it should be reduced by 15.

The only solution i see to this is if we store all the changes we want to do incrementally, and then apply them all at once. Unless I'm missing something.

Good point. The only feasible solution I see for this is to just copy the latest state to the working copy when locking the working copy, then use only the working copy in Lua. It will be a performance hit, though, especially when the copying is unnecessary. Any other ideas?

Just as an update on the current state:

• Signals are pretty much scrapped. They don't play well with multithreading and make it a little unpredictable when a particular piece of code executed.
  
• Instead of binding to Lua manually, I'll go with oolua. It's pretty fast and will make it relatively straightforward to expose more complex components to Lua. Unfortunately, it doesn't support data members, so everything must be handled with getters and setters.
  
• I've decided to copy a bit out of World of Warcraft's addon API. Entities can be predefined in XML (in addition to procedurally generating them in Lua or C++). An entity can have script handles that call Lua functions on certain events (e.g. every frame or when a key was pressed).
  

As a rough estimate, it'll be a few days to a couple of weeks until I plugged this all together.

Triple post! But with good news! After fighting gcc, Ogre and luabind for a few days, I managed to put everything together. It compiles, runs and does.... something.

I'll liberally apply some Doxygen comments to the source code now to lower the bus factor. Bonus point for anyone who can make sense of the file "res/dist/scripts/test.lua" without documentation!

Do i get even more extra points if i didnt ever saw lua before? Actually, its easier for me as i know what the code it is replacing is like.

The first 6 lines create a new entity for the background.

Then you add a callback. I have no idea what is it listening to or what it does.

Then, you create a new entity, called player, add to it the "Sinbad" model (the little ogre)

Finally, you add a callback for key presses. Whenever one of wasd is pressed, the entity accelerates in the pressed direction. I see we are using z as our vertical axis. I though it was y.

(After looking at the previous code, i realized we were using z as vertical the whole time)

Edit: Yes, increasing the bus factor would actually be good (you said decrease). I'll mess with the code i understaund while you write the coments.

Not bad. When you run Thrive, you may hit on what the callback does.

Anyway, I'm wondering how to best go about documenting the script interface. It will be pretty close to the C++ API, but not quite identical, so pointing script authors to the documentation generated by doxygen would certainly be comfortable, but may get confusing.

Here are the options I could think of:

1. Use the doxygen comments in the C++ header files for both C++ and Lua, but explain the differences
   
2. Create separate doxygen input files for documenting the script API
   
3. Use doxygen for C++ and the Wiki for the script API
   

Option a keeps the script API's implementation and documentation close together, but script authors would have to filter out C++ specific stuff and vice versa.

Option b will require some more effort to keep the documentation in sync with the actual implementation. In my experience, however, forgetting to update the documentation is very common. It doesn't matter whether it resides in a different directory or a few lines above the implementation.

Option c is my least favorite because it can only document one version. If the script API documentation is part of the git repository, it can be updated at the same time as the script API itself.

Overall, I'd prefer option b. Any more ideas or additional input?

By now we have a somewhat functioning engine. It's not complete and it certainly isn't a game yet, but it works. And now I'd like to throw about 20% of it away by switching from multi-threaded to "single-threaded" (the quotation marks are deliberate). There are several reasons for this and I will try to outline them here.

While testing the possibility of using rigid body particles for agents and compounds in the microbe stage, I implemented an emitter of such particles that would fill up an area of the environment. Depending on optimization, it works alright for particle counts below 500. At least on Linux. On Windows, even with very few particles, the motion of the player microbe becomes jerky for whatever reason.

So far, I've ruled out thread priority and lack of motion interpolation. I found a problem with boost's sleep function we use for limiting the maximum framerate, but that turned out to not be the culprit either. So I'm stumped. I'm pretty sure that it's got to do something with the SharedState mechanism, but since the problem only seems to occur on Windows, it might be something else entirely.

There's also still a problem with thread synchronization where new entities are rendered before they are properly initialized. That's fixable, but not trivial.

Those two problems, and probably many more down the road, are symptomatic of the multithreading we employ. Back when I started out with a multithreaded approach, I expected problems like this, but I thought the additional effort was worth the potential performance gains. Since then, I've learned a bit more about game engines and how other developers design theirs, especially with respect to single vs multithreading.

As an experiment, I changed our engine to be single-threaded. It took some work, but nothing too complicated. The result is encouraging. The framerate is still at a comfortable level (about 100 FPS for 500 hundred physics entities on screen without too much collision). The code is easier to understand and reason about. We don't need weird systems that just propagate data from physics over script to the graphics thread.

In short, I would be very much in favor of switching to a "single-threaded" model. The quotation marks are there because Bullet and, with limits, Ogre still use multiple threads, but in a transparent way that doesn't concern us. We can also easily use multiple threads for the fluid dynamics simulation and anything else that's easy to parallelize.

From what Dani's explained to me of the multi-threading, and how difficult it has been to make some of it work, I would support going single threaded for most gameplay.  Would it be possible, though, to keep the option to do some tasks asynchronously?

What I mean is, rather than now where all threads (physics, scripting, rendering) contribute to every frame, and must finish their work before a frame can be shown; we instead have a single main thread which deals with that, but occasionally spawns worker threads (or uses a pool) to start tasks which don't need to be finished immediately.

Good examples would be the AI's auto-evo steps while the player is in the editor, or the auto-evo/population dynamics of areas outside the players view etc.  I realise none of these will feature any time soon, but it would be good to keep the option open.

I agree with seragen.

we need to keep that open, too many a game project of mine has gone downhill due to lack of planning for things like this.

-----------------
I AM BACK

Nimbal mentioned that the need for multithreading will be addressed once we start seeing performance hits. Until then we just keep it single-thread.