Scripting

As mentioned before, a while ago in fact, the game should implement some sort of scripting. For those of you who don't know, scripting is a higher level programming language implemented in a lower level application. Basically, something like javascript or python running inside a c++ application. What does this mean? Easy, higher level programming.

My question is, why? What do you guys want to do with a scripting language?

roadkillguy wrote:

As mentioned before, a while ago in fact, the game should implement some sort of scripting. For those of you who don't know, scripting is a higher level programming language implemented in a lower level application. Basically, something like javascript or python running inside a c++ application. What does this mean? Easy, higher level programming.

My question is, why? What do you guys want to do with a scripting language?

First off, what would you recommend we do with scripts?

~sciocont wrote:

roadkillguy wrote:

As mentioned before, a while ago in fact, the game should implement some sort of scripting. For those of you who don't know, scripting is a higher level programming language implemented in a lower level application. Basically, something like javascript or python running inside a c++ application. What does this mean? Easy, higher level programming.

My question is, why? What do you guys want to do with a scripting language?

First off, what would you recommend we do with scripts?

I don't know. That's the thing. I could see using a scripting language for modding purposes (although I'd have to think about how that would work) or for menus.

Well, scripts could be used for the GUI (like in Spring) to make it easy to modify. That's another use, but I don't think it would pay off to implement and maintain another language just for this.

roadkillguy wrote:

~sciocont wrote:

roadkillguy wrote:

As mentioned before, a while ago in fact, the game should implement some sort of scripting. For those of you who don't know, scripting is a higher level programming language implemented in a lower level application. Basically, something like javascript or python running inside a c++ application. What does this mean? Easy, higher level programming.

My question is, why? What do you guys want to do with a scripting language?

First off, what would you recommend we do with scripts?

I don't know. That's the thing. I could see using a scripting language for modding purposes (although I'd have to think about how that would work) or for menus.

Unless we can think of something we definitely need scripting for, then we can pretty much neglect to include it until we need it, right? Or must we hard-code it into the engine?

~sciocont wrote:

roadkillguy wrote:

~sciocont wrote:

roadkillguy wrote:

As mentioned before, a while ago in fact, the game should implement some sort of scripting. For those of you who don't know, scripting is a higher level programming language implemented in a lower level application. Basically, something like javascript or python running inside a c++ application. What does this mean? Easy, higher level programming.

My question is, why? What do you guys want to do with a scripting language?

First off, what would you recommend we do with scripts?

I don't know. That's the thing. I could see using a scripting language for modding purposes (although I'd have to think about how that would work) or for menus.

Unless we can think of something we definitely need scripting for, then we can pretty much neglect to include it until we need it, right? Or must we hard-code it into the engine?

Yeah it can wait, I was just looking into it.

Ok. How goes the programming? Made any progress recently?

We've been looking into using a different library for input handling, rendering, etc.

roadkillguy wrote:

We've been looking into using a different library for input handling, rendering, etc.

Forgive me if I'm wrong, but effectively you're putting together the base off of which everything will be made then, right? I mean, even our fitness functions intersect with this to an extent.

Right. We have to get pixels on the screen somehow.

Ah yes, fitness. I originally wanted to have a fitness script for every niche in every biome, but I no longer think that's the case.

I have already ported the current Thrive base to the new library, but haven't commited it to the code repository yet because I need to find out how does it work. Here is the download link for both the source and the compiled application - there's practically nothing new since I last posted it, though.



By the way, do I need any SSH key, Roadkill? So far I have got to the Restricted Shell access, downloaded PuTTY and all that, but still no luck.

roadkillguy wrote:

Right. We have to get pixels on the screen somehow.

Ah yes, fitness. I originally wanted to have a fitness script for every niche in every biome, but I no longer think that's the case.

Not bad. I'll write up more stuff on auto evo, such as stuff on actually determining population fluctuations due to mutation.

Commander Keen wrote:

I have already ported the current Thrive base to the new library, but haven't commited it to the code repository yet because I need to find out how does it work. Here is the download link for both the source and the compiled application - there's practically nothing new since I last posted it, though.



By the way, do I need any SSH key, Roadkill? So far I have got to the Restricted Shell access, downloaded PuTTY and all that, but still no luck.

SSH? You don't log on to the server in any way. hg handles it all. I don't know what PuTTY is.
For me, I type "hg push". I have a config file set up so that it automatically pushes to the sourceforge repository.

[In a dramatic voice] "Rise, thread, and do my bidding!"

In all seriousness though, I'd like to talk about scripting and this seems like the most appropriate place (although the thread is slightly derailed).


Why use scripts?

There seems to be some confusion about why we would need scripting. There are two main reasons:

Lower entry bar for programmers inexperienced in C++

The engine will be written in C++. Even for programmers that have years of experience, but in another language like Java or Python, C++ is like a ticking bomb. When (not if) you shoot yourself in the foot, you not only blow your whole leg off, but also everyone else's in the vicinity, too. Things like memory management, exception safety, when to use a pointer over a reference or value, templates and many more are stuff that people get wrong even in simple programs, let alone an engine like the one we'll need for Thrive.

Done right, a scripting interface would hide most of those nasty problem spots from the programmers. We can make development of the game logic much easier by preferring scripts over raw C++, opening the project up for more programmers.

Faster development cycle
C++ is a compiled language. Every little change requires a rebuild of the software. Even if that rebuilding is quick, I imagine it would become really old really fast to invoke the compiler for every little tweak. Ideally, we could set up the scripting system so that the scripts can be reloaded from within the game, making it possible to test the changes without even restarting the program.


What should scripts do?

The question should rather be: what shouldn't they do? The answer to that is simple: everything that has to be done in C++ for performance reasons. So definitely the graphics (for which we have Ogre already). Maybe physics, although the physics in the microbe stage should be simple enough to be handled by scripts. Ideally, we would start out writing as much as possible of the game in our script language of choice. If we find specific parts to be too performance critical to be done in scripts, we implement them in C++.

A simple way of utilizing scripts would be to have them provide "event handlers" for an entity. Examples of such event handlers would be

• OnUpdate: Called every game frame
  
• OnCollision: Called when the entity collides with another one
  
• OnHealthChanged: Called when the entity is damage or healed
  

Scripts would be able to access and change properties of entities, spawn new entities and mark exisiting ones for removal.


Which scripting language to use?
Our options here are pretty limited. I've seen Python thrown around here, but I wouldn't recommend it. It's a great language and easy enough to extend, but I've found it to be a nightmare to package properly so that the game doesn't miss any DLLs during runtime. My recommendation would be Lua. It's simple, fast, pretty powerful, easy to integrate and package. I don't really know about other options, but I'm open to anything if it fits our requirements.

Good point to get rolling.

The function you mentionned(OnUpdate, OnCollision,OnHealthChanged) will be called thousands of times. Wouldn't it be better to use C++ to minimize their performance cost ?

I was under the impression that scripting was mostly for story event, character interaction...

I never tried Lua, but I've seen it used a lot and it definitly seems like the language we want. I'll dig a bit in it to get a better understanding. Great post again Nimbal.

RodGame wrote:

The function you mentionned(OnUpdate, OnCollision,OnHealthChanged) will be called thousands of times. Wouldn't it be better to use C++ to minimize their performance cost ?

Of course it would be better for performance. But I'd rather have a game running at 50 FPS in a few months than a game running at 130 FPS in a couple of years. There's always a performance tradeoff when using a scripting language, but unless the script author does something crazy in a frequently called function, I'm pretty sure Lua can handle it

Alright, make sense to me.

I'll read more on the topic as we build the engine.

Do you have any experience implementing script language in a C++ application?

Not much. I've used LuaBridge at work, but only for a very tiny part of the application.

With the current system (entity framework) such things will be easy to implement. As longas the scriprs can acces the raw data from the engine class. That way when we update a script to use more variables, we dont need to go to the c++ to declare it explicitelly.

I vote yes for scripts as long as it doesent make the rest of the code or the compilation harder.

Implementing scripting might attract more programmers. As I understand it, scripting exists because you can write in it and learn it faster than you could a more basic language like C++, right? I would probably be able to learn enough of the scripting language we're using (lua?) this summer to be of some programming use later on, if my understanding is correct.
I'm sure some other non-programmer members would also be willing to do this.

After reading the earlier arguments against, and the recent arguments for, I also support it.

After some thought, I think the following scheme would be easy enough to implement and almost as flexible as can be.

Just as a quick recap to make sure the terminology is clear to everyone. An Entity is a "thing" in our game world. In the microbe stage, that would include the player's microbe, AI controlled organisms, compounds floating around, obstacles placed throughout the game world, and so on. An entity has a unique id and is made up of several components (see below). The entity's component makeup can change during the entity's lifetime.

A Component is a piece of data with a very narrow purpose. Examples include a TransformComponent that holds the position, the rotation and the scale of its entity. A CollisionComponent would hold the entity's collision mesh (and other collision related stuff) for use by the physics engine. A RenderableComponent contains all the information needed by the graphics engine to render the entity. The list goes on.

Now to the scripts. To make scripts flexible enough, they need to be able to do several things. I'll assume we'll use Lua.

Retrieving an entity and its components

That's easy enough. We need a global manager for the entities anyway, so Lua only needs a global function or table to get an entity by its id, something like this:

Code:

playerEntity = getEntity("player")

The getEntity function is a global and takes an id, returning the entity with that id. If no entity with that id was found, it returns nil, a kind of "non-value" in Lua.

Accessing an entity's components could look like this:

Code:

playerTransform = playerEntity["transform"]

The entity returned by getEntity is actually a table (Lua's almost-omnipotent data structure), "holding" all the components for easy retrieval. By the way, the above line could just as well be

Code:

playerTransform = playerEntity.transform

It's the same to Lua.

Manipulating a component's data

We could go several routes here, API-wise. Setting an entity's x position could work like one of these lines:

Code:

playerTransform.x = 12.34    -- Option A
playerTransform:setX(12.34)    -- Option B
playerTransform:setPosition(12.34, playerTransform.y, playerTransform.z)    -- Option C

I'd be partial to option C as it allows (or maybe even forces) a script author to set all coordinates at once. Option A has a nice syntax, though. Maybe a hybrid of both would be ideal.

Removing components or whole entities from the game world

Easy:

Code:

playerEntity:destroy()
-- or
playerTransform:destroy()

When calling these functions, C++ will handle the removal. The above example, however, exposes a small problem. When the playerEntity is destroyed, all its components should be destroyed as well. So what happens if we still use a component (or an entity) after it has been destroyed?

Doing stuff like this in C++ would almost inevitably lead to a crash if the programmer is not diligent about watching the objects' lifetimes like a hawk. In Lua, we might be able to get away with a simple error message like "Called function on destroyed object, on line 58". While not ideal, it's certainly better than just crashing to the desktop.

(Note: this error behaviour will have to be implemented first, it doesn't come automatic with Lua).

Adding components or whole entities to the game world

This section will be a bit fuzzy, mainly because we may want to think about how to handle it in C++ first. Of course, in C++, we can always just create the necessary objects, plug them together in the right way and give the complete entity to the entity manager. But some entities may require dozens of components to work correctly. Other entities might be a lot simpler to build, but will require lots of tweaking to make them look good or behave the way we want them, requiring a recompilation for small adjustments.

So, ideally, we should be able to "describe" entities in a text file or other convenient format and the game loads those files to find out how to construct a given entity. I propose to use Lua tables for that. Then a definition for a "bouncy block of cheese" entity could look like this:

Code:

local CheeseEntity = {
    collision = {
        mesh = "meshes/cube.mesh",
        bounciness = 3.1,
    },
    transform = {
        scale = {0.1, 0.1, 0.15}
    },
    render = {
        mesh = "meshes/cube.mesh",
        texture = "textures/cheese.png"
    },
}
-- Register the blueprint with a name
registerEntityBluePrint("Cheese", CheeseEntity)

Then, when we want to spawn cheese from a script:

Code:

cheeseEntity = spawnEntity("Cheese")

Instead of a string identifier, we could also directly pass the CheeseEntitiy table to spawnEntity. Useful for entities that are used rarely.

Reacting to third-party changes in a component's data

This is where it gets complicated and the implementation of this feature will have far reaching consequences for both the C++ and Lua side. As noted in my first post in this thread, it's important to be able to react to certain events, like collision with another entity, position changes, and so on. It's important to note here that the components themselves should be as dumb as possible. The CollisionComponent, for example, can't "know" about other entities in the vicinity, so why would it know about any collisions occuring? That's the job of the physics system.

So, how do we approach this? I suggest using "signals". A signal is a list of callbacks that are called when the signal is emitted. The callbacks receive relevant information for that signal as arguments.

Note: a signal is distinct from a so-called "event", which are usually handled centrally by some global event manager. Signals are more localized and direct.

The idea is to give each component a set of appropriate signals. The CollisionComponent would get a sig_collision signal, carrying the entity id, the collision point, the force, and maybe some more. From Lua, we could use it like this:

Code:

-- Define a function to handle a collision
function onPlayerCollision(otherId, point, force)
    if force > 5.0 then
        -- Ouch
        playerEntity:destroy()
    elseif force > 2.0
        -- Less ouch
        playerEntity.health:damage(10)
    end
end

-- Connect the signal
playerEntity.collision.sig_collision:connect(onPlayerCollision)

Now each time the sig_collision signal is emitted, the onPlayerCollision function is called, giving the script the opportunity to react to the collision and damage the player or outright kill them for their carelessness.

So, who would be responsible for emitting the signals? As always, it depends. The collision signal certainly should be emitted by the physics system. A sig_moved signal, on the other hand, could be handled directly by the TransformComponent when someone sets the position.

Care would have to be taken to make sure that no cicular calls occur, i.e. setting the position from within the sig_moved signal. But I'm optimistic we can deal with that.


Nimbal, why do you always write so much text?!

What do you mean with "so mu... oh. Sorry about that.

Nimbal wrote:

Nimbal, why do you always write so much text?!

What do you mean with "so mu... oh. Sorry about that.

I would personally ask why don't you write longer text It's really interesting to read and it help us set the basics of the framework of the game. How you put it, it reminds me of how unity deal with the GameObject. It is really efficient and I agree with what you said. If I have more time, I'll reread it and try to pinpoint certain question/problem that I find along the way.

Keep pulling those long text I like them!

Great explanation about the entity framework. You covered two of the five components of it. Let me explainthe other three, how (i think) they work the scripting, plus a few notes on the two you alredy covered.

Entity: What you said. The Entity class only contains a vector of all the components it contains. It also has some utility methods , for example to add components to it or to get some of them. Currently has no id, but we can add one. We have avaible a vector of all entities at the Engine instance.

Component: Another simple class that only contains plain data, plus a way of diferentiating the type of component it is. All components are descentants of a base component class.

Nodes: The only thing nodes contain is a bunch of pointers to components plus a way of diferentiating between them. They are used to access the components on an easier and faster way. Nodes are stored at the Engine instance, sepatared by type.

Systems: The systems are the things that update and move the game. All systems have an update method, that only takes an argument: alphaTime. Systems must go through all of its asocied Nodes, and modify them as necessary. Systems shouldn't care about entities. This is where scripts are put.

Engine: The engine is the piece that puts it all together. It is in charge of storing the entities, the nodes and the systems. It updates all the systems when it is told so. It is in charge of adding and removing entities and its asocied nodes.

That means that systems (scripts) dont really need the whole data structure avaible. they only need its associed nodes. Lets put an example:

I'm a system called MoveSystem, and my update function gets called. Then, i get a vector containing all the MoveComponents somehow (for now, all systems have a pointer to its engine, and ask the engine to give them the vector). Then i go through all the MoveNodes. Each node contains a PositionComponent and a VelocityComponent. All i have to do is to extract the speed, multiply it by alphaTime, and add that to the PositionComponent. I dont know at any time what i'm moving, a cell, a planet, or a rock someone trew into the air, so my code is easier. Also, i dont have to go through all those inmovile entities, which dont have a MoveComponent, so i dont waste time there. Finally, my code is separated from everything else, so if someone has to make changes to me it's easier.

Ok, enough with the first person writing. I apologize if someone felt weird about it (i certainly did).

To sum up, all a script needs is a vector (table) of all the nodes it has read from or write to. Not the whole data structure.

And finally, i have a question: How does that data get shared? The ideal thing would be that when one system modifies something, all other systems can read that modified data inmediatly (that is, only having the data at one place, and everything else are references to that data). Anything else i dont know how could we make it work.