Perhaps this demonstrates a lack of imagination, but I can't see even the sketch of how this piece fits into building a real game. Presumably one has thousands, or even millions, of clients simultaneously interacting with the world and with each other. This system would imply that your (low latency) world-state would be kept in memory on a small cluster of Galaxy servers.
The novel quality of this system is that node data-locality is determined by how you access the data.
Let's say you have 10 server nodes and 10^6 clients. That means that, in the best case, each node deals with 10^5 clients. Presumably clients (or players) can move through the simulated world, hopping from node to node.
Here things get hazy in my mind. For example, how does the client know which server to connect to? Does it just make a guess and get redirected if it guessed wrong? When a player crosses a node threshhold, how does that work?
I'm thinking there must be a central character store - basically a traditional database, that handles initial node affinity. Position in the world is part of character state, and when you login, you'll be handed off to the correct node based on that state.
But if this is how things work, when would you ever need your "cache lines" moved from node to node? The world-state is spatial so why move it from node to node? I guess that's the crux - I can't think of any other data that would need to move between nodes other than player data, and in that case I don't see what data it would need to take with it.
I would really like people to treat this as an experiment in distributed systems design rather than a product for games, but because our main product is intended to be used by MMOs (as well as other industries), let me address your scenario.
> how does the client know which server to connect to?
Galaxy doesn't handle any client connections, only connections between cluster nodes, but if you were to build something on top of that connects to clients, then, yeah, starting with a guess and redirecting is ok. An initial node that simply directs connections might work, too. And if players move from one place to another, having your communication layer telling them to connect to a different node is pretty much what we had in mind.
> when would you ever need your "cache lines" moved from node to node?
Yes, player data, NPC data, vehicle data - anything that moves. BUT, another big reason for data migration is load-balancing. Continuing with your game example, if a lot of players congregate in one area, handled by one machine, you may decide to split it to two machines, and migrate half of the information there.
If you were to use Galaxy for a graph database (forgetting the MMO use-case for now), then while the graph vertices don't "move", changes in the edges might make you decide on a better distribution of the vertices over the cluster.
This response fills me with unease. You present MMOs as a core use-case, and then equivocate your support for MMOs, preferring to hand-wave about failover and graph databases.
The best way to experiment with distributed systems design is to build a real distributed system - or at least be able to sketch one out.
Frankly, this seemed like a solution looking for a problem, and your vague responses are supporting this. The interesting dynamic, of being able to move data between nodes as a side-effect of access patterns is interesting, but it's not clear how an MMO could really use this dynamic to good effect. Indeed, even the fault tolerant case, it's not clear how this dynamic would help failover - I mean, would you need to duplicate access patterns prior to node failure to ensure dual-local data?
Frankly, I think you should focus on one use-case (MMO, graph database, something) and hand-wave a complete solution that really leverages the novelty of your approach. Get specific and talk about what actually happens when "things move".
Alright, sorry for the confusion. Our commercial offering (SpaceBase) is very much targeted at MMOs and real-time LBSs. However, like many start-ups we really do like building cool stuff. And while it indeed the case that Galaxy will soon be a offered as a component of SpaceBase, it is has a very different design from other memory-grid projects/products, so we decided to open-source it to the community and let it explore other possible uses.
My post was meant to be an introduction to a series of very technical blog posts discussing theoretical and practical aspects of distributed systems. The post was not meant to serve a clear commercial purpose, so I was trying to steer the discussion away from commercial uses and more to its CS aspects. You know, we really find this stuff interesting. Some of my future posts will discuss the more theoretical sides of Galaxy and will drill very deeply into its design and algorithms, while others will discuss how SpaceBase will make use of Galaxy to help MMOs build huge, rich worlds, and LBSs track lots of moving objects in real-time. To be more concrete and give just a taste, I'll say this: when SpaceBase runs on top of Galaxy, objects are transferred from one node to another to create a dynamic area-of-responsibility for each node. This means that each node will be responsible for processing all objects in some region of the game world (or real world for LBSs). But the regions are dynamic - namely, they shrink and grow to accommodate non-uniform load, so that small busy areas will be split over several nodes, while large, relatively vacant ones will be handled by just one.
The novel quality of this system is that node data-locality is determined by how you access the data.
Let's say you have 10 server nodes and 10^6 clients. That means that, in the best case, each node deals with 10^5 clients. Presumably clients (or players) can move through the simulated world, hopping from node to node.
Here things get hazy in my mind. For example, how does the client know which server to connect to? Does it just make a guess and get redirected if it guessed wrong? When a player crosses a node threshhold, how does that work?
I'm thinking there must be a central character store - basically a traditional database, that handles initial node affinity. Position in the world is part of character state, and when you login, you'll be handed off to the correct node based on that state.
But if this is how things work, when would you ever need your "cache lines" moved from node to node? The world-state is spatial so why move it from node to node? I guess that's the crux - I can't think of any other data that would need to move between nodes other than player data, and in that case I don't see what data it would need to take with it.