Key/value storage engines

We recently dived into the world of distributed Key/Value database stores. Although key/value storage engines are nothing particular new, distributed implementation seem to gather some attention lately due to high scalability demands of current online services. Below are interesting reads about this topic together with some links to project I found particular interesting.

Articles

A very good overview article about different storage engines has been written by Richard Jones from Last.fm, in which he compares the different engines with the needs of last.fm in mind.

• http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/

The following articles are more about specific implementation of a certain storage engine, but interesting to get to understand the whole principles more deeply.

• http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/
• http://bytepawn.com/readings-in-distributed-systems/
• http://push.cx/2009/rules-of-database-app-aging/
• http://bret.appspot.com/entry/how-friendfeed-uses-mysql/

Bob Ippolito gave a very comprehensive talk at PyCon 2009 on “Drop ACID and think about data”, where he summarizes his experiences with different storage engines while implementing the service at Mochi Media. He presents the different storage engines and shows how their are being used for different purposes:

• http://blip.tv/file/1949416/

Shortlist of intersting key/value storage engines

Ringo

• http://github.com/tuulos/ringo/

Scalaris

• http://code.google.com/p/scalaris/
• http://video.google.com/videoplay?docid=6981137233069932108&ei=caB0SaPUNIW0iALk-9CMBQ&q=erlang+exchange

Dynomite

• http://github.com/cliffmoon/dynomite/tree/master

Tokyo Cabinet/Tyrant & LightCloud

• Presentation:

• Tokyo Cabinet:

  http://tokyocabinet.sourceforge.net/

• Tokyo Tyrant:

  http://tokyocabinet.sourceforge.net/tyrantdoc/

• LightCloud:

  http://opensource.plurk.com/LightCloud/

• Elliptics:The elliptics network is a fault tolerant distributed hash table object storage.

  http://www.ioremap.net/projects/elliptics/

• Flare:

  http://labs.gree.jp/Top/OpenSource/Flare-en.html

Project Voldemort

• http://project-voldemort.com/

Disco

• http://discoproject.org/

Redis

• http://code.google.com/p/redis/

HyperTable

• http://hypertable.org/
• http://kosmosfs.sourceforge.net/

Cassandra

• http://incubator.apache.org/cassandra/

Document Databases

CouchDB

• http://couchdb.apache.org/
• http://couchdb.apache.org/docs/overview.html
• http://push.cx/2009/rules-of-database-app-aging
• http://userprimary.net/user/2007/12/16/a-quick-look-at-couchdb-performance/

MongoDB

• http://www.mongodb.org/display/DOCS/Home

Some more readings:

• http://pdos.csail.mit.edu/papers/chord:sigcomm01/
• http://current.cs.ucsb.edu/facebook/index.html
• http://www.linuxjournal.com/article/6797
• http://www.ietf.org/proceedings/06mar/slides/plenaryt-2.pdf
• http://berkeley.intel-research.net/sylvia/pht.pdf
• http://groups.google.com/group/fr4fb/
• http://crd.lbl.gov/~kewu/fastbit/