So, we recently made a major release of the Ning Core. The Core, as we call it, is the platform stuff which provides all the services used by the playground (the PHP stuff). It is mostly written in Java and is (in my opinion) a fairly interesting distributed system.
The most recent change is mostly in how we handle data for a given application. In the past we have optimized heavily for supporting tons of fairly low load applications (the "Bullwinkle" system). With the "Rocky" core we made the changes to equally well support extremely popular and heavily used applications.
Optimizing for lots of applications meant that a single server could handle data access for a large number of apps. Because we could always route requests for a specific application to a given server, we could use an embedded datastore and always access it in-process. In order to support failover we wrote through to a replicated, high-availability, authoritative datastore which we optimized for writes. In the case of a core server failure, a replacement would just comes up and dump all the data from the authoritative store into its local one, and life was good again. The only thing which needed live replication was the authoritative store.
This model limited a given application to a single core (which isn't as bad as it sounds, as the "core" is only responsible for a very small (but important, as it includes content store access) part of request processing, it could fan out to lots of playground machines for the actual PHP execution). Eventually that bottleneck becomes a problem for scaling a single application, so, for performance, and to improve availability, we had to allow for multiple cores to service a single application.
The first thing that came up in this model was that the datastore had to change -- the embedded datastore would either need to become coherent across multiple cores, or we'd need to move to a shared datastore for primary operations, not just as an authoritative source. We opted for a number of shared datastores and removed the previous authoritative store. There is still some degree of local caching, but it is mostly for system information (such as the shapes of content for a given application), not application data.
On the surface, this doesn't seem to provide much -- if the limit, previously, was accesses for a single core to an embedded datastore, going to multiple cores against a single datastore should crush it faster. The key comes in with a major restructuring of the datastore. The embedded version was read optimized and would do writes through to the write optimized authoritative storage, which was shared by lots of cores. The shared store has been shifted to be about as efficient for writes as previously, but almost as efficient (if not more so for some queries) for reads as the embedded. The penalty is, of course, complexity.
The Ning content store allows for storing arbitrarily shaped objects, moreover it allows for adding attributes at pretty much any time and having them just work. With the embedded store, most of the penalty for the content shape complexity was lost in comparison to the benefits of having the data in-process and backed by local disk. Moving to a shared network datastore we had to come up with a much better way storing and indexing content -- and we did :-)
At this point each of the shared stores can be clustered and replicated independently of the cores accessing them, which is a good thing for system management, a good thing for popular applications, and a great thing for our peace of mind.