Amazon has published a nice paper on Dynamo, their distributed hash storage. Such kind of solution is really needed for the big shops, and the only issue why we’re not using anything like at Wikipedia that is that we don’t actually have it.
I saw a comment in there by “Max”:
Looks like a lot of thought has been put into this. I stopped reading about halfway through but how much of this was inspired by memcached?
I guess the answer would be ‘none’. Key-value databases will be mainstream for 30 years already soonish (the first ‘dbm’ was released by Ken Thompson at AT&T back in 1979). It wasn’t distributed back then, but you may want to slap ‘distributed’ label to pretty anything nowadays. The method of doing that is usually the problem, and in this case Dynamo choses something similar to Chord-like rings (though with simplified routing).
Still, there’re no open libraries that would provide with easy reliable DHT building, and I’d love every paper like this to come with open-source code rather than patents attached.
P.S. My first serious web-app coding was on Sybase PowerDynamo ;-)