Compound system implementation

I've started this thread as an off-shoot of the discussion in the Microbe Stage Progress Report thread, to focus on finalising both the design and the implementation of the compound system.  This post is a work in progress, based on all the discussions I've been able to find and read on the forums, and some of the existing code.  

The aim of this thread is to decide on and document everything we need to know to finish programming the compound system, and integrate it into the microbe stage builds.  Everything in this post will initially be up for discussion, anything that's decided will be marked done (colour coding may come later), and will no longer be up for discussion unless we have a particular issue with it when coding or implementing a later feature.  I will aim to get anything that's decided written up either for the wiki, or some other form of documentation.

Naming
To start with, I believe there is some confusion about the current naming system.  The 'compound system' deals with 'compounds' and processes dealing with compounds.  These compounds include chemical elements, compound molecules, and the 'agents' proposed more recently, which are chemical agents for communicating with, attacking, or otherwise influencing other cells and the environment.

In the current code, all of the above are referred to as 'agents'.  This may lead to some confusion later, as agents have a specific meaning in most simulations, and are a more appropriate name for use when coding the AI.  I suggest we refer to compounds as compounds (in the code), and agents as 'compound agents' where there's any chance of confusion.

Measurement
The issue of units has come up a few times, and the best solution we have is that compounds are measured in some multiple (e.g.: femto- or pico-) of moles.  We can't reasonably measure compound quantities by weight or mass, as chemical equations are expressed in terms of moles of compounds, and measuring in weights would make processes needlessly complicated.

That said, it's difficult to choose a multiple that works well for all compounds.  Femto-moles seem a reasonable starting point (at least for internal calculations), giving an average (incredibly crude approximations incoming...) human cell a capacity of about 200 units of glucose, or 70 units of ATP, or ~40'000 units of water.  I suggest we use femto-moles internally for calculations, and consider scaling the numbers up or down when displaying them to the player.

It's important to note that even at this scale, we're not talking about individual compound molecules (one femto-mole represents roughly 600 million molecules).  This also means that we don't need whole numbers of compounds to complete a reaction, nor do we need to complete reactions in whole number quantities.  In other words, it's perfectly reasonable to complete aerobic respiration 0.1 times, yielding 0.6 CO2 + 0.6 H2O + 3.6 ATP from 0.1 glucose + 0.6 O2.

The compound system
This thread, and specifically this post provide a lot of detail of how the compound system was originally planned and implemented, if you've not read them yet/recently, I suggest you do so before carrying on.  A few things have changed since those plans were made, so we need to rework some of those plans, the most important changes include:
- Compound storage is now general, rather than specific, so that if we have too much of one compound there may not be space for another one.  This means we need to more carefully consider when a compound is stored or ejected, and how we calculate the available space for storage.
- We now have hex based cells, so that organelles occupy specific areas of a cell, rather than fitting into a more variably shaped cell.
- Compounds in the environment are represented as particle 'packets', rather than grid based density clouds.

Data files
As far as I can tell, in the current code the compounds and processes are defined in script files.  Ideally we need to define all these details in data files that can be more easily edited for tweaking, then load these at runtime.  I previously used xml files for this, do the programmers have a preference for this or another (human readable) file format?  (Examples of previous compound data files: Compounds.xml, Organelles.xml, Processes.xml)

Compounds
Currently, the only thing we need to know about each compound is its name, its molecular weight, and its visual representation in game (model/sprite/colour).  For more advanced compounds (e.g.: phospho-lipids for constructing cell membranes) we also need to know the constituent compounds used to create that compound, but this information should be presented to the engine as a synthesis process, not as part of the compound definition.

All other compound behaviour is defined by processes and organelles that use each compound.

Organelles
Organelles are a huge topic on their own, which we'll need to finalise relatively soon too.  For the purposes of this thread though, we can simplify them to three categories:
- Organelles which offer storage capacity for specific compounds, or a range of compounds.
- Organelles which offer processing capacity for carrying out particular processes.  Note that not all processes require an organelle.
- Organelles with another function (e.g.: flagella) that may consume ATP or other compounds to function.

The flow chart in this post provides the best overview of organelles that I've been able to find so far.

Processes
Most of the interesting stuff in the compound system happens because of processes, and most of the required information is stored in process definitions.  For each process, we need to know its name, which and how many compounds are required as input, which and how many are produced as output, which organelles (if any) it requires and how much of the processing capacity of each it uses, and its basal rate.

Some processes don't need an organelle, and take place in the cell's cytoplasm (i.e.: any part of the cell that isn't an organelle).

The math to go with this is presented in this post (also linked above), I will try to provide some code/pseudo-code to go with this soon, as the equations look a lot more complicated than they actually are.

Storage
The original design assumed that a cell would have individual storage capacities for each compound.  This made it much simpler to calculate how much space was left for each compound (how concentrated it was in the cell), and detect when a cell was full and possibly needed to eject some of that compound.

We have since decided to switch to having general storage, so that all compounds within a cell occupy a shared capacity (or possibly group capacities for particular types of compound).  This is more realistic, and leads to some gameplay improvements (e.g.: you don't need vacuoles for each compound), but causes several complications that we need to work out:
- having too much of one compound may mean there isn't enough space to store another important compound
- it's no longer clear whether you have too much of a particular compound, or if having too much of another compound is restricting your capacity for storing the first

and as a result of the above
- we need a way to decide which compounds are important to store, and in what quantities.  Equally we need to know which to get rid of, and how quickly we want to do so (assuming a limited rate at which we are able to expel compounds, we need to get rid of toxic compounds more urgently than simple waste)

Compound priorities
As above, we need to know which compounds are important, and how important they are relative to each other.  At present this is defined by a static list in the code, but we need a way of calculating this dynamically.  Factors which need to be considered might include:
- The rate at which the compound is being used by processes in the cell
- The priorities of the end products of those processes
- The size of the compound in storage (not all compounds will use storage at an equal rate... unfortunately)
- The availability of the compound (there's no point allocating a huge storage capacity to a rare element, but this depends on how we define priority)
- Priorities manually set by the player
- Priorities evolved by auto-evo (I don't think we can actually do this, but it's worth considering)

Access to organelles + processes
Not all cells will have all organelles, or have access to all processes.  Gaining access to new ones should be one of the main ways of progressing in the early game.  Some organelles are gained by endosymbiosis, others can only be gained via mutation (see chart linked above).  I remember it being suggested (but I have no idea where), that a player should be able to pick up nucleus fragments/plasmids from destroyed cells, with a chance of gaining new organelles and processes this way in addition to mutation.  Some (most) processes will be available as soon as their requisite organelle is available, others may require further mutation.

We have partial lists of most of the above information.  It would be useful to compile at least an initial set of this for testing purposes as part of this discussion.  We don't need this to be actually complete for a while, as the implementation of the compound system should be largely separate from compounds, processes and organelles used.

Organelle/process upgrades
This isn't urgent, but we would eventually like organelles (and maybe processes) to be upgradable.  The simplest way this could happen is to increase the processing/storage capacity of higher level organelles, and increase the energy efficiency of processes (you can't produce more for a set input, but you can use less ATP/energy to do so).

Visualisation
As I understand it, we currently represent environmental compounds as particle packets only.  At one point we had also planned to have clouds of compounds based on an underlying density field.  Has this been cancelled/postponed?


Work in progress..
Where possible I want to include links to the relevant discussions above, sections I still need to add or complete include:
- Comparison between how the system is currently coded and what's outlined above
- How are compounds transported around a cell/allocated to organelles and processes
- Priorities: compound storage, process rate, organelle energy/compound usage
- Consider surface:volume ratio
- Calculate cell/organelle density/weight

Feel free to discuss or disagree with any part of this post, but please try and make sure any posts are constructive and contribute to finalising this implementation.  If you feel a particular topic is completed, and want it marked done, please also mention that.  I will try to moderate this thread to make sure it stays on track, and move any non-constructive posts to new threads or delete them as needed.  All that said, please do post what you think, as this needs several peoples input to be completed, and ideally needs to be done relatively soon.

Note - I worked on a large part of the above several weeks ago, and some of the issues here have probably already been solved, if so let me know and I'll remove them.

Implementation details
As the aim of this thread is to decide on how the compound system should be implemented, and no just discuss how it should work, I'd like to compile a list of what functions we need from the code in order to make it work.  This will be added to as we go, but as a starting point (using roughly the system from the thread linked above), we would need functions to:
- Get the amount of each compound stored in a cell
- Get the total amount of storage space available in a cell
- Get the priority for each compound
- Calculate from the above, the storage space allocated to each compound (if we decide to do something like this)
- Calculate from the above, the concentration of each compound (amount stored / available storage).  Note that we could use either total available storage or storage assigned to that compound, each option has advantages and disadvantages.
- Get a list of available organelles, the number of each, and possibly their upgrade level
- Get the amount of space available for storage in an organelle
- Get the amount of space available for processing in an organelle
- Get the amount of processing space available in all organelles required by a particular process
- Calculate the speed of a process from its basal rate, the concentrations of its input/output compounds, and the available processing space in required organelles.
- Update the stored quantites of compounds from the current rates of active processes
- Calculate the storage priority of each compound (see discussion above)
- Get the available transport capacity (currently the number of exterior hex faces)
- Divide this capacity among compounds that need to be expelled/absorbed according to their priorities
- Methods for loading data from xmls (or other format), into some sort of central list (i.e.: a manager class, unless the programmers have an alternative suggestion?)
- A way of storing this information internally, and efficiently adding new compounds/organelles/processes to cells
- Add/remove compounds from cell/environmental storage
- Ideally, a method for balancing the compound level in the overall system, so that compounds lost to rounding and other errors are replaced in the environment.

In terms of code execution, I'm fairly sure we can't update everything we need to every frame, so would ideally need to spread this out over several frames.  I can think of two ways of doing this, and I'm not sure how feasible they are: do the updating in a seperate thread, and poll for a complete flag every frame until done, then repeat - or update a subset of cells every frame, so that the processing load every frame is roughly equal, and all cells are updated once every 10-20 frames.  In my prototype (first post of this thread), I also updated the compound levels seperately from the process rates, with rates updating every 1/2 second, and compound levels using the most rescent rate to update every 1/10 seconds.  

In the end, I think we'd only have a few functions exposed from the compound system:
- Get compound details
- Get organelle details
- Get process details
- Update process rates (arg: time since previous) (this should include compound excretion/absorbtion)
- Update compound levels (arg: time since previous)
- Add/remove compound/organelle/process
- Alter compound/organelle/process level

Thats all I can think of for now.  This post has been far too long in the making so, despite quite a few things still missing, I'd like to get the discussion on this going so we can all work on filling in the gaps.

Seregon wrote:

I suggest we refer to compounds as compounds

This was brought up a few months ago and was fixed in the compound branch.

Seregon wrote:

As far as I can tell, in the current code the compounds and processes are defined in script files.

This is just temporary, we will use XML in the future I'm sure!

Seregon wrote:

have clouds of compounds based on an underlying density field.  Has this been cancelled/postponed?

Postponed.

Seregon wrote:

Get the available transport capacity (currently the number of exterior hex faces)

We currently use a constant value, per storage organelle, of how much volume/time that can be manipulated from the organelle.

Regarding all the implementation details it's generally better to just implement the functions as they are needed, as we aren't making an interface. But ofc it's useful to keep in mind what needs to be possible with the implementation!
In terms of performance there really is no need to drastically overcomplicate the code right now. Computer science students are often told to not pre-optimize and when it is time to do optimization make sure you find the bottlenecks and focus on those first.

To begin with compound priorities, I'll just pull a very inconcrete formula out my arse:

PlayerPriority * CompoundRateOfUse * endProductPrioritySum / (stockpile/capacity)

These are obviously rather abstract values atm, but its a place to start. endProductPrioritySum would be the summed up priorities of the outcome of all processes using the compound,. You may want to take into account how much of the compound is needed for each process. PlayerPriority could represent player set values, but is rather there to represent the need to prioritize ATP higher than glucose.

There is furthermore the question of how exactly we want to maintain our storage. Seregon seems to suggest allocating space for each compound which could be one way, altough not one I'm a big fan of, alternatively something akin to NickTheNicks suggestions in the https://thrivegame.canadaboard.net/t997p195-microbial-compounds-and-organelles thread, which is what the current implementation is based off.

RE the first three point - ok, great.

crovea wrote:

We currently use a constant value, per storage organelle, of how much volume/time that can be manipulated from the organelle.

This links in to the possibility of internal transport rates, which I mentioned we would probably ignore in cells, but may use later.  i.e.: theres a difference between how quickly each storage organelle can exrete compounds, and how quickly the overall cell can.  We could use one or the other, or whichever value is the smallest.

I agree regarding pre-emptive optimisation, I was really just asking whether, if we needed to in future, we could do either of the two things I suggested, and how hard it would be in the current engine.

The same goes for the list of functions - we don't need to make all those functions, but we will need to be able to do most of the things on that list to implement the original system (that may change ofc).

crovea wrote:

PlayerPriority * CompoundRateOfUse * endProductPrioritySum / (stockpile/capacity)

This looks like a good place to start.  We need to be careful if we want to sum the priorities of process outputs, as this could easily lead to loops, and there may not be a trivial order in which we can work priorities out (e.g.: ATP is used in the synthesis of fat, but fat is also respired to produce ATP, which priority do you calculate first?).  A possible solution is to arrange compounds in a heirarchy (e.g.: the chart in this post), and only consider the priority of higher tier compounds when summing.

I mentioned allocating storage as a possible solution, I have quite a few issues with going that route too, mainly that it means only using part of your storage if some high priority compounds are unavailable.  The main reason I mentioned it is the problem of how we calculate compound concentrations in a system with general storage.  The idea behind the original rate equations was that reactions went faster the more readily available inputs were, and slower the less available space there was to store outputs.  One possible solution is to calculate input concentration as (compound amount / total storage capacity), and output concentration as (compount amout / unused storage capacity) for that purpose.

Seregon wrote:

this could easily lead to loops

Good point! I imagine we would update the priorities periodically in which case we can just use the previous priorities to calculate the new ones and that way avoid loops, we will just need a special case for the first calculation. Your suggestion could work but runs the risk of severely undervaluing a process due to the heirachy.

Seregon wrote:

One possible solution is to calculate input concentration as (compound amount / total storage capacity), and output concentration as (compount amout / unused storage capacity) for that purpose.

I like this idea. Furthermore you could go all the way and just completely ignore the division of storage and just have a single abstract storage organelle for the entire microbe, which storage capicity is just the sum of empty hexes storage and all storage organelle. The player wont be able to tell the difference, in any positive way anyway, and while realism could be lost, it's a very intransparent realism that would probably just cause confusion.

Periodic updating of priorities would work, and would effectively mean double-buffering the priorities, which is a solution we've used for several similair issues in auto-evo and other places. I'll try and have a think about what else we might want to include in the priority calculation, and if we want to treat some compounds differently to others (e.g: end products are needed in specific quantities to build a cell, wheras sugar and toxins are used continually throughout its life).

I agree that, for cells atleast, storage should be pooled centrally. It would simplify the calculations and I don't see any real loss of realism. That would mean that storage would need to be entirely general (i.e.: we can't have a storage organelle for groups of compounds, in addition to not having storage organelles for individual compounds), but I think that's where we were heading anyway.

I finished simplifying the storage with this commit: https://github.com/Revolutionary-Games/Thrive/commit/875253c867cccb0b12c79d77942399960953cec2 if you Seregon or someone else want to take a look, but it's pretty straightfoward and I did not yet change any of the other aspects we talked about.

Some of the next things to start looking at would be:

• Priority system needs to be finalized
  
• Bandwidth dynamics need to be formalized, is there in and out seperate bandwidth and how does it work exactly? Which processes are limited?
  
• Distribution of compounds to process organelles could/should be reviewed, including concentrations and fractional producing
  
• What do we do about compound units, their weights and the numbers we use etc.
  

If Seregon or anyone else have any other items to the list that should be implemented next or if you have any concrete models for the above let us hear!

Bandwidth would be the same for all things going in and out of the cell. I think we should just assign a certain value for a hex face for the amount of bandwidth it provides, so that increasing the number of open hex faces of your cell in relation to how big it is will become a factor that will limit cells from exploding out of proportion through natural selection (because smaller cells will have more optimal surface area:volume ratios).

Basically, bandwidth is a restriction on how fast a cell can pick up or drop compounds around the environment. Each compound has a molecular weight, and this could be factored in to how much of the bandwidth that compound takes if the cell picks it up at that moment. I'm not an expert on this, but I suggest we even use the direct molecular weight values.

Organelles I think should also have a maximum rate at which they can process. For example, a mitochondria with 1000 oxygen and 6000 glucose shouldn't be able to process all of them in 1 tick.

Also, how will the mitochondria react when it is already processing something and more of the materials it needs are obtained? For example, if a mitochondria went ahead and started processing 0.01 oxygen and 0.06 glucose, and then suddenly a whole lot of oxygen and glucose was picked up by the cell, does the mitochondria wait and finish its process before starting with the next batch? Or does it restart with more materials?

By the way Jacob there are some examples of molecular weights on the compound thread that Seregon linked to.

In regards to organelles, the list I can think of of organelles that will be in-game are (it's not too different from the original list):

• Mitochondria (Cellular Respiration)
  
• Chloroplast (Photosynthesis)
  
• Thermoplast (Photosynthesis but with heat, so Thermosynthesis)
  
• Chemoplast (Chemosynthesis, but not using the same compound reaction as the previous two, so this one needs to be researched more)
  
• Nucleus/Nuclear Kernel/Kernel (Composed of the Nucleus, ER, and Golgi Body. All cells will have it)
  
• Cilia
  
• Flagella
  
• Predatory Pilus?
  

On a side note, anyone have a link to that post ~scio made with that yED chart outlining the relationships between all the different organelles, processes, and compounds?

NickTheNick wrote:

Bandwidth would be the same for all things going in and out of the cell.

NickTheNick wrote:

Basically, bandwidth is a restriction on how fast a cell can pick up or drop compounds around the environment

I see a few potential problems here tho. So if the cell has just used up all its bandwidth for the time being and at the same time produces a bunch of ATP, H2O and CO2 but it doesn't have the storage to contain all of it, what would happen? There is a couple of ways to solve this tho (or are we working by the principle that the cell must have produced as much as it used up, so the molecular weights must add up and the cell must have storage?).

NickTheNick wrote:

Also, how will the mitochondria react when it is already processing something and more of the materials it needs are obtained?

It's a good question, I could calculate if it would result in more production if it chose to restart and only do it in those cases as well, but from the players perspective, if he can see his process organelles working somehow (like the current color changes) it might just be confusing having them seemingly randomly restart. I'm probably partial to the simple solution of just letting them finish.

NickTheNick wrote:

By the way Jacob there are some examples of molecular weights on the compound thread that Seregon linked to.

I see, I guess those values wont be final tho, if we're calculating based off femto-mole units.

NickTheNick wrote:

Bandwidth would be the same for all things going in and out of the cell. I think we should just assign a certain value for a hex face for the amount of bandwidth it provides, so that increasing the number of open hex faces of your cell in relation to how big it is will become a factor that will limit cells from exploding out of proportion through natural selection (because smaller cells will have more optimal surface area:volume ratios).

Agreed - though another question is should bandwidth in/out be shared, or separate?

NickTheNick wrote:

...but I suggest we even use the direct molecular weight values.

This would work for now, but does not account for the fact that some compounds are much more difficult to transport, due to being hydrophobic, and unable to simply diffuse across the cell membrane.  Also, I'm fairly sure that some compounds should be impossible to transport.  I'd be happy using molecular weight for now and figuring out something more realistic later if needed.

NickTheNick wrote:

Organelles I think should also have a maximum rate at which they can process. For example, a mitochondria with 1000 oxygen and 6000 glucose shouldn't be able to process all of them in 1 tick.

Yes - this is what I mean when I'm referring to the 'processing capacity' of an organelle in the OP.  If this isn't clear enough, let me know and I'll explain it better.

NickTheNick wrote:

Also, how will the mitochondria react when it is already processing something and more of the materials it needs are obtained? ...

This assumes that processes take a certain amount of time to complete, where they should have a continuous rate of completion.  As a simplified example, if a cell has a processing capacity for respiration of 1/second, and we update the system 20 times per second (i.e.: actual process capacity = 0.05/tick), and contains 3 oxygen + 0.5 glucose, it would process 0.15 oxygen + 0.025 glucose per tick, until it ran out after 0.5 seconds, adding more compounds would simply allow it to carry on for longer.  This ignores the effect of concentration on processing rate, and as such adding more compounds would *probably* increase the reaction rate - the cell wouldn't have to wait until a process was done to do this, it would do this immediately the next time we update the system.

NickTheNick wrote:

In regards to organelles, the list I can think of of organelles that will be in-game are (it's not too different from the original list):

• Mitochondria (Cellular Respiration)
  
• Chloroplast (Photosynthesis)
  
• Thermoplast (Photosynthesis but with heat, so Thermosynthesis)
  
• Chemoplast (Chemosynthesis, but not using the same compound reaction as the previous two, so this one needs to be researched more)
  
• Nucleus/Nuclear Kernel/Kernel (Composed of the Nucleus, ER, and Golgi Body. All cells will have it)
  
• Cilia
  
• Flagella
  
• Predatory Pilus?
  

Those are all the basic ones that we need to have for now.  We may also have specialized organelles later, e.g.: toxin producers, luminous organelles.  Cilia, flagella, and pillus would come under the category of membrane modifications, which might also include cell walls, compound pumps, and signalling proteins.

NickTheNick wrote:

On a side note, anyone have a link to that post ~scio made with that yED chart outlining the relationships between all the different organelles, processes, and compounds?

You mean this one?

crovea wrote:

So if the cell has just used up all its bandwidth for the time being and at the same time produces a bunch of ATP, H2O and CO2 but it doesn't have the storage to contain all of it, what would happen?

Having process rates limited by concentration (this includes the 'process' of absorbing compounds) should mean that this never happens, and if it does the amount over capacity will be miniscule - if there's not much space to store new compounds, very few will be absorbed or produces, and relatively more should be being excreted.  In the unlikely case that a cell is entirely full of useful compounds and has nothing to excrete, we've probably balanced something wrong.

crovea wrote:

...if he can see his process organelles working somehow (like the current color changes)

To make this work with a continuous, rather than discrete, process, organelles could flash according to their current processing rate.  It should be possible to update the rate of flashing without being too jarring to the player, and we should probably keep the overall rate of color change fairly slow to avoid any issues with epilepsy (or just generally being annoying to look at).

crovea wrote:

I see, I guess those values wont be final tho, if we're calculating based off femto-mole units.

Actually, molecular weights are for individual molecules (in units of 'the weight of a single hydrogen atom'), or the weights of moles of compound in grams.  All we would ever need to do is scale the numbers down by a factor of 10^-9, but I see no issue with using them as they are now.[/quote]

Crovea asked me to write up an example of how continuous processing would work, to better explain it and see if others agree that it makes sense, so here it is:

For simplicity, I'm going to work with a process 'P', which converts 1A + 2B -> 3C, with a basic rate of 1/sec, and requiring 25% of the processing capacity of organelle 'OR' to run.  The cell has a total storage capacity of 50, each compound has equal priority, and there's nothing else being stored.

I'm going to measure time in frames, and assume that the game runs at 20 fps (to simplify the maths).

First, an example ignoring compound concentrations:
- At frame 0, the cell contains 20 A, 20 B, and 0 C, it has one OR available.
- One OR has space to run P at 4x its basic rate, i.e.: 4/sec.  As we have no other processes, and aren't considering concentration, P uses this whole capacity.
- Each frame (1/20 seconds), P completes 4/20 = 1/5 times, so that it consumes 0.2A, 0.4B, and produces 0.6C.
- At frame 1, the cell contains 19.8A, 19.6B, and 0.6C
- At frame 2, the cell contains 19.6A, 19.4B, and 1.2C
- At frame 20 (1 second), the cell contains 16A, 12B, and 12C

Now, an example where the level of input compounds changes part way through:
- Same setup as above, but only 0.2A + 0.2B are available to start.
- We still have the same amount of processing capacity, so the process could run at 4x
- However, that would use 0.4B in the first frame, so instead we only run at 2x/sec, or 0.1x/frame, using 0.1A, 0.2B -> 0.3C
- At frame 1, the cell contains 0.1A, 0B, and 0.3C
- If at this point we added 10A + 10B to the cell, we would have plenty of input to run at full speed (4x), and the same happens as in the example above.
- At frame 1, the cell contains 10.1A , 10B, and 0.3C (10A + 10B + what was already in the cell)
- At frame 2, the cell contains 9.9A, 9.6B, and 0.9C
- At frame 20 (1 second, 19 frames after the input), the cell contains 8.2A, 6B, and 11.7C
- If we now remove 8.1A, leaving just 0.1A remaining, we can again only continue at 2x/sec, so that:
- At frame 21, the cell contains 0A, 5.8B, and 12C
- The process now stops, as it doesn't have the necessary inputs available
 
As you can see, there's really very little to it at this point, and adding or removing compounds at any point in the process is very easy to do, you just recalculate your reaction rate and carry on.
 
If we include concentrations, the maths gets a lot nastier to do by hand (though not much worse for the computer), so I'll only give a shorter example
- Same setup as in the first example
- Storage is 80% full (20A + 20B + 0C / 50 total), so that A and B have a concentration of 40% each (20/50), and C has a concentration of 0% (0/10 - using empty storage as this is an output)
- P could run at 4x its basic rate, using all of the capacity of the single OR
- This gets reduced by the concentrations of the inputs and outputs: rate = basic rate * processing capacity * input concs * (1 - output concs)
- At frame 0, this rate = 1/sec * 4x * 0.4 * 0.4 * (1-0.0) = 0.64, using 0.64A + 1.28B -> 1.92C (per second), or 0.032A + 0.064B -> 0.096C per tick
- At frame 1, the cell contains 19.968A, 19.936B, 0.096C
- The concentrations become 39.93%, 39.87%, and 0.96% for A, B, C.  Note that becuase we use 1A + 2B -> 3C, the overall usage of storage doesn't change for this particular process, it's always 40/50.  This simplifies the maths slightly, but won't be true for most processes, it also doesn't take into account the fact that compounds use a different amount of storage.
- The rate is now = 1 * 4 * 0.3993 * 0.3987 * (1-0.0096) = 0.6308
- At frame 2, the cell contains 19.94A, 19.87B, 19.06C
- Concentrations are 39.87%, 39.74%, 0.19%
- Rate is = 0.6218
- At frame 20, the cell contains 19.44A, 18.87B, 1.69C.  Concentrations are 38.87%, 37.74%, 16.88%.  Rate is 0.4879.

As you can see, at this point the numbers are horrible, so instead I'll post a graph of what would happen over the first 500 frames (25 seconds):

Spoiler:

If we were to suddenly add or remove compounds as in the second example, we recalculate the rate and continue as usual, the graph below shows what would happen if you:
- Setup as above
- Increase the level of the output C to 10 at 5 seconds
- Reduce A & B to 10 at 10 seconds
- Set A & B to 15, C to 5 at 15 seconds
- Set A to 30, B to 5, at 20 seconds

Notice the the rate (green line) changes instantly for every change, but also changes gradually (more visible in first graph), as the process changes the level of compounds.

Spoiler:

Note that the rates in the concentration examples are much lower than the first two examples, simply because we're scaling them down by the concentration.  Whichever system we end up using would need to be scaled to run at roughly the same overall rate as the other.

An excel spreadsheet for playing around with these numbers, and figures, can be found here.  The second sheet of this also includes a graph of the first two examples (change the amounts of A/B/C at some point in the graph to see the effects).

EDIT - I also noticed, while playing around with this, that we should probably alter the concentration calculation from what I suggested above.  I would suggest either:
- input conc = (amount of compound left after process) / (amount of compound available now)
- output conc = (amount of compound that will be generated) / (amount of storage space available now)
or:
- input conc = (amount of compound) / (total storage)  
- output conc = (amount of compound + available storage) / (available storage)

Seregon wrote:

requiring 25% of the processing capacity of organelle

Oh we will need multi-purpose "processing organelles" that can complete more than one kind of process?

Seregon wrote:

I'm going to measure time in frames

We don't actually deal in frames in-game, we use milliseconds, so two simmulations with different fps still do things at the same time.

The system seems good, altough I'm not sure that the extra complexity of fractional processing capabilities of organelles are necessary. When would they be necessary?

Seregon wrote:

Agreed - though another question is should bandwidth in/out be shared, or separate?

I think it should be shared. However, I also think that compounds that need to be excreted should always take priority over picking up new compounds, which would hopefully prevent the cell from overstocking its compounds.

Seregon wrote:

This would work for now, but does not account for the fact that some compounds are much more difficult to transport, due to being hydrophobic, and unable to simply diffuse across the cell membrane.  Also, I'm fairly sure that some compounds should be impossible to transport.  I'd be happy using molecular weight for now and figuring out something more realistic later if needed.

Sounds good to me.

Seregon wrote:

Yes - this is what I mean when I'm referring to the 'processing capacity' of an organelle in the OP.  If this isn't clear enough, let me know and I'll explain it better.

Oh okay I missed that, that's good.

Seregon wrote:

You mean this one?

Yep, thanks.

Seregon wrote:

crovea wrote:

So if the cell has just used up all its bandwidth for the time being and at the same time produces a bunch of ATP, H2O and CO2 but it doesn't have the storage to contain all of it, what would happen? [/url]

Having process rates limited by concentration (this includes the 'process' of absorbing compounds) should mean that this never happens, and if it does the amount over capacity will be miniscule - if there's not much space to store new compounds, very few will be absorbed or produces, and relatively more should be being excreted.  In the unlikely case that a cell is entirely full of useful compounds and has nothing to excrete, we've probably balanced something wrong.

And even  if a miniscule amount was overstocked, I think that giving priority to excretion over absorption would prevent this.

crovea wrote:

I see a few potential problems here tho. So if the cell has just used up all its bandwidth for the time being and at the same time produces a bunch of ATP, H2O and CO2 but it doesn't have the storage to contain all of it, what would happen? There is a couple of ways to solve this tho (or are we working by the principle that the cell must have produced as much as it used up, so the molecular weights must add up and the cell must have storage?).

As long as excreting compounds takes priority in the bandwidth over absorbing compounds, I don't think this should happen.



Also, I think that the continuous system of compound processing looks great, and I would be in favour of implementing it. There is one typo though, in the first example, where the amount of C at frame two should be 1.2, just to prevent confusion for someone reading it. Other than that it looks awesome.[/quote]

crovea wrote:

Oh we will need multi-purpose "processing organelles" that can complete more than one kind of process?  

Yes - all organelles can potentially carry out multiple processes. A better way to think of this though is that each process takes up some portion of the available organelles space, and treat organelles as static, rather than using them to run processes... (did that make any sense?).

crovea wrote:

We don't actually deal in frames in-game, we use milliseconds, so two simmulations with different fps still do things at the same time.

I know, I was simplifying for non-programmers. The above example is what would happen if each frame had a length (deltaTime) of 50ms. In code, rather than divide the rate (in second) by 20 frames, we would multiply it by the length of the frame: 0.05 seconds.

crovea wrote:

The system seems good, altough I'm not sure that the extra complexity of fractional processing capabilities of organelles are necessary. When would they be necessary?

This relates to your first point - some organelles will be used by many different processes, and their processing capacity should be divided amongst those processes. As above, this is a lot easier to implement if you look at it from the point of view of the process, rather than the organelle.

@Nick - Agreed, fixed, and thanks.

Alright, I think we have process organelles planned out then.

Bandwidth will be shared for in and out with consideration to out going before in (simple code ordering should handle that).

Let's talk about the Priority system, Seregon you mentioned you would give it some thought, have you come up with anything?

I propose we use a calculation similar to the one I proposed earlier (PlayerPriority * CompoundRateOfUse * endProductPrioritySum / (stockpile/capacity)) and use that to update priorities in intervals (every 500 ms for example).
How a potential playerpriority component might work I'm not sure.

This is just a very quick idea and mock-up, but here's one way the player priority could be set:

Spoiler:

I recall a pause feature being discussed at one point to allow the player to change the compound priorities of their microbe. In the image above, each compound is represented by a colour (I have no idea what colours would represent which compounds, so I just used green, blue, red and yellow). Any organelle which uses a compound includes three levels of use for that compound the player can choose between(in the example, they are represented by three boxes of the colour). I may be going down a completely wrong track here, so correct me if I am.

Oliveriver wrote:

This is just a very quick idea and mock-up, but here's one way the player priority could be set:

Looks good, if we end up using a player-set component in the priorities that will be a good reference point. Initially we'll probably just have the player set priorities in the microbe editor, atleast until we have a graphics update for the microbes.

I think that's an interesting idea, but I would make some suggestions. I think it's kind of arbitrary to assign colours to different compounds, and also I feel like having the bars be colour coded and be fitted to the organelles would restrict how many compounds or bars could be represented.

I think a more liberating alternative would be something like adding a tab to the interface (somewhere where the player manages their own cell) in which there is a whole panel of the compounds icons. Above each compound icon is a text box where the player enters a numerical value. This way, there is no limitation on the degrees of priority, and if there are many compounds the player can scroll through the panel, the player can identify compounds immediately by their icons, and also the player's cell remains free of any HUD elements.

However, this makes me wonder whether the player should be able to set their priorities in-game, or only in the editor.

Also, this only solves the problem for the player's species (partially, I think we also need a way for the computer to determine ratios if the player doesn't want to do it manually), but it still means we need to figure something out for the AI cells, which would have their priorities calculated through some process of equations just as the player cell would do it if it was not set manually.

NickTheNick wrote:

I think we also need a way for the computer to determine ratios if the player doesn't want to do it manually

Yeah this was always the plan (atleast from my perspective). We were even discussing if the player should or should not have any say in the priorities, but I do believe we have come to a conclusion that they should have a say.
However a question remains if the player priorities should completely override the calculation or if the players priorities should simply be taken into consideration.
This was a previous (very rough) suggestion I made one page one of the thread:

Code:

PlayerPriority * CompoundRateOfUse * endProductPrioritySum / (stockpile/capacity)

but me and Seregon has been planning out how a single organelle prioritize between multiple possible processes (more information will follow soon) and I think the above formula takes too many of the same things into account, so I'm not quite sure atm.

NickTheNick wrote:

something like adding a tab to the interface

This would need a sketch or a more in-depth explanation I think, but maybe it should be discussed in a/the GUI thread as it's not really relevant to the compound system atm (All that matters here is what kind of values the player would be able to set, and what kind of effect they would have).

Me and Seregon privately discussed the implementation of processes within process organelles and we came up with this:

A processing organelle will indeed have the posibility of having several different processes.
When a process organelle does a "unit" of processing it will process each of its processes based on how much of the organelles 100% capacity it has assigned, so a process with a high assigned capacity will work relatively faster.
Each process will have it's capacity calculated based its priority. Our current proposed way of calculating process priority is simply the sum of priorities of each output compound of the process, multiplied with how much is produced.
Our current formula for calculating capacity assignment is defined as:

Code:

(processPriority / sumOfAllProcessPriorities) * (1-X) + X/numOfProcesses

where X is a real number between 0.0 and 1.0 representing how much a process' compound priorities influence the distribution vs even distribution. however the capacity distribution will also take into account the concentrations of input compounds, such that a compound with a very low concentration of its input compounds will only be given as much capacity as that input concentration can support. (We assume for now that less capacity can use concentration more effeciently, which is still under some debate).
Process priorities and capacity distribution will be calculated periodically.

Next up is refining this idea and figuring out how compound priorities are calculated. This may be done significantly differently than I previously proposed due to the details of the process capacity distribution algorithm.