CouchFoo: ActiveRecord styled API for CouchDB

Written on February 4, 2009 by george

CouchDB is an excellent database, designed especially for distributed applications. To quote the official site site:

Apache CouchDB is a distributed, fault-tolerant and schema-free document-oriented database accessible via a RESTful HTTP/JSON API. Among other features, it provides robust, incremental replication with bi-directional conflict detection and resolution, and is queryable and indexable using a table-oriented view engine with JavaScript acting as the default view definition language.

along with the knowledge it’s written in Erlang, you know it’s going to go be a winner in the future.

For one of my current freelance projects I needed to store data in a document fashion – ie unstructured. This made CouchDB an ideal candidate. There were several ruby gems available: CouchPotato, CouchREST, ActiveCouch and RelaxDB gems. Each offered its own benefits and own challenges. After hacking with each I couldn’t get a library was happy with. So I started with ActiveRecord and modified it to work with CouchDB. And so CouchFoo was born.

In the end I ended up with a gem that mirrors ActiveRecord in all but a few minor places. In particular:

• CouchDB is schema free so property defintions for the document are defined in the model (like DataMapper)
• :select, :joins, :having, :group, :from and :lock are not available on find or associations as they don’t apply (locking is handled as conflict resolution at insertion time)
• :conditions can only accept a hash and not an array or SQL. For example :conditions => {:user_name => “Georgio_1999″}
• :offset is less efficient in CouchDB – there’s more on this in the rdoc
• :order is applied after results are retrieved from the database. Therefore :order cannot be used with :limit without a new option :use_key. This is explained fully in the quick start guide and CouchFoo#find documentation
• :include isn’t implemented yet but the finders and associations still accept the option so you won’t need to make any code changes
• By default results are ordered by document key. The key uses a UUID scheme so these don’t auto-increment and are likely to come out in a different order to insertion. default_sort can be used on a model to sort by create date by default and overcome this
• validates_uniqueness_of has had the :case_sensitive option removed
• Because there’s no SQL there’s no SQL finder methods
• Timezones, aggregations and fixtures are not yet implemented
• The price of index updating is paid when next accessing the index rather than the point of insertion. This can be more efficient or less depending on your application. It may make sense to use an external process to do the updating for you – see CouchFoo#find for more on this
• On that note, occasional compacting of CouchDB is required to recover space from old versions of documents and keep performance high. This can be kicked off in several ways (see quick start guide)

The RDoc for the gem contains more details on each of these differences, new features that I added, a quick start guide and additional areas of responsibility to think about when using CouchDB (in particular performance).

As a quick overview, basic operations are the same as ActiveRecord:

class Address < CouchFoo::Base
property :number, Integer
property :street, String
property :postcode # Any generic type is fine as long as .to_json can be called on it
end

address1 = Address.create(:number => 3, :street => "My Street", :postcode => "secret") # Create address
address2 = Address.create(:number => 27, :street => "Another Street", :postcode => "secret")
Address.all # = [address1, address2] or maybe [address2, address2] depending on key generation
Address.first    # = address1 or address2 depending on keys so probably isn't as expected
Address.find_by_street("My Street") # = address1

As key generation is through a UUID scheme, the order can’t be predicted. However you can order the results by default:

class Address < CouchFoo::Base
property :number, Integer
property :street, String
property :postcode # Any generic type is fine as long as .to_json can be called on it
property :created_at, DateTime

default_sort :created_at
end

Address.all # = [address1, address2]
Address.first    # = address1 or address2, sorting is applied after results
Address.first(:use_key => :created_at) # = address1 but at the price of creating a new index

Note that there’s an optimisation that will order results by created_at if there are no conditions so in the above case, the default_sort wasn’t required. However when using with conditions it will be required so it makes sense to use at all times.

Conditions work slightly differently:

Address.find(:all, :conditions {:street => "My Street"}) # = address1, creates index on :street
Address.find(:all, :conditions {:created_at => "sometime"}) # Uses same index as :use_key => :created_at
Address.find(:all, :use_key => :street, :startkey => 'p') # All streets from p in alphabet, reuses the index created 2 lines up

As well as providing support for people using relational databases, CouchFoo attempts to provide a library for those wanting to use CouchDB as a document-orientated database:

class Document < CouchFoo::Base
property :number, Integer
property :street, String

view :number_ordered, "function(doc) {emit([doc.number , doc.street], doc); }", nil, :descending => true
end

Document.number_ordered(:limit => 75) # Will get the last 75 documents in the database ordered by number, street attributes

Associations work as expected but you must to remember to add the properties required for an association (we’ll make this automatic soon):

class House < CouchFoo::Base
has_many :windows
end

class Window < CouchFoo::Base
property :house_id, String
belongs_to :house
end

There’s a few bits left to tidy up (as noted in the readme) but generally speaking it’s now ready for use by others. Grab it on github and feel free to fork and send me pull requests.

And now to do something I’ve not being doing a lot of lately, spend some more time on the Couch…

Posted in Uncategorized