6 foot 8, weighs a fucking ton.
Opponents beware.
Opponents beware.
He's coming.
He's coming.
He's coming.
Maybe now that I posted it my blog, I can get this song out of head. It's like The Ring. Except infinitely cooler.
(ps. He's coming. He's coming. He's coming.)
There are a few ActiveRecord-style wrappers for CouchDB out there these days, but many of them have fallen behind as CouchDB’s powers have increased. I’ve taken an interest in writing regular old web apps again – so now is the time when CouchRest grows wings and flies to the vaulted heights of DataMapper and ActiveRecord. (I could have written a DataMapper adapter for CouchDB, but much of DataMapper’s code is based around SQL-like problems that CouchDB just doesn’t have.) CouchRest::Model provides the ease-of-use Rubyists have come to expect from their ORMs, and it does it in CouchDB style.
Here is an example (taken from the bright shiny new documentation) of what your Models can look like. If you click through to the actual blog post, there is also a longer example pasted as a gist embed which details the capabilities of the view-generation system even more.
class Article < CouchRest::Model
use_database CouchRest.database!('http://localhost:5984/couchrest-model-test')
unique_id :slug
view_by :date, :descending => true
view_by :user_id, :date
view_by :tags,
:map =>
"function(doc) {
if (doc.type == 'Article' && doc.tags) {
doc.tags.forEach(function(tag){
emit(tag, 1);
});
}
}",
:reduce =>
"function(keys, values, rereduce) {
return sum(values);
}"
key_reader :slug, :created_at, :updated_at
key_accessor :title, :tags
# You can make dates work nice without magic or special fields
key_writer :date
def date
Time.parse(@doc['date'])
end
timestamps!
before(:create, :generate_slug_from_title)
def generate_slug_from_title
doc['slug'] = title.downcase.gsub(/[^a-z0-9]/,'-').squeeze('-').gsub(/^\-|\-$/,'')
end
end
You can do the standard CRUD:
@post = Post.get(params[:id])
@post.title = "New Title"
@post.save
The view generation system is what I’m proudest of. It’s designed to be performant, and it strikes a balance between magic and usability. Simple views are as easy to declare as:
view_by :date, :descending => true
They can be queried easily like this:
# get the 10 most recent posts
@posts = Post.by_date :count => 10
# get the raw view
@view = Post.by_date :count => 10, :raw => true
Views defined in this way take all the options of a CouchRest::Database#view call, and those options can be provided either as defaults at definition time, or overriden at query time. There is an additional option, :raw, which defaults to false. When :raw is true, you get back just what CouchDB sends. When :raw is false, CouchRest::Model::MagicViews requests the associated document for each view row, and gives you an array of documents. W00t!
CouchRest::Model views use _design documents, not temp views, so they take advantage of all the performance CouchDB has to offer.
Views can also be manually composed, if you want to write something that’s more complex than the built-in helpers can provide.
If you’re at the permalink page for this post, you’ll see a Gist embed here with an even longer example of CouchRest::Model in action.
Enjoy!
I'll be in San Jose next week, Oct. 8th and 9th. Anyone who wants to meet up, either socially or for business, mail me: damien at damienkatz.net
UPDATE 2: Thing are moving very quickly in Merb land this week. The best reference is Timothy Bennett’s summary so please go there if you are serious about trying out Merb in the few days that remain between now and 1.0. (I’ll leave the rest of this post up as historical oddity.)
I’ve decided to smooth the way for future Merbists to use CouchDB. This means getting into the edgy-edgy Merb land inhabited only by the most intrepid Rubyists. Merb will be 1.0 soon, and I hope some of that project’s maintainers take a look at my hoop-jumping, because they’d do well to make this all brain-dead. Or maybe they don’t want the brain-dead to use their framework. :)
First let’s get the latest versions of the source code. These are the incantations I provided. If you do them out of order, it could get icky.
UPDATE: this turned out not to work with edge, so I added lines to checkout the v0.9.7 tag.
# there are a few git repos we'll want to grab, let's keep them together
mkdir merbzone
cd merbzone
# undocumented dependency - extlib
git clone git://github.com/sam/extlib.git
cd extlib
rake install
# the main merb source code
cd ..
git clone git://github.com/wycats/merb-core.git
git checkout v0.9.7
cd merb-core
rake install
# another dependency problem
sudo gem install templater
# merb-more has all the goodies that (optionally) make merb fun
cd ..
git clone git://github.com/wycats/merb-more.git
cd merb-more
git checkout v0.9.7
rake install
OK now lets generate an application, and then I’ll shove you into Merbunity’s excellent and up-to-date tutorials,
merb-gen app MyApp --testing-framework=rspec --template-engine=erb --orm=datamapper
So there you go, you have an app (I hope) and now you can learn to add junk to it the right way. My new mini-project is to get comfortable in the crazy Merb world, so I can start to use it as a basis for developing CouchDB-backed web apps. One step at a time…
Ayende Rahien is writing some articles where he explores the guts of CouchDB. His ability to plow the source and write about what's going on, especially in the btree code, is kinda blowing me away. My favorite so far is the btree:query_modify article, where he starts to get into the MVCC aspects of CouchDB's design.
Reading Erlang: Inspecting CouchDB
Geoffrey Grosenbach, famous for his PeepCode
screencasts for developers, blessed us with a a new release: “CouchDB & Rails”.
While the title includes “Rails”, Geoffrey spends most of the time on CouchDB and the Rails bits are actually applicable to any other MVC-based web framework in any language (PHP, Python, Perl — doesn’t matter). Even if you are not into these there is a whole lot to learn.
Geoffrey covers all the basics: What CouchDB and why it is so special by being document oriented, how to install and access it, using it from Ruby, document CRUD including attachments, views in theory and practice and finally how to pull it all together for a small yet useful web application.
Go check it out. It is only $9 and a free preview is available.
—
PS: Geoffrey promised to create more (and more advanced) screencasts if this one is successful. Do me a favour and make sure this one is successful, thanks! :-)
Yesterday i started using the couchrest plugin, only to run into problems with the spec tests because it is built for latest latest CouchDB. So i had to install from Subversion source, but i got into a terrible mess, and to cut a long story short, i reinstalled the operating system today! I am now on shiny new Debian Lenny, woohoo! :D
So hopefully today i will successfully show you how to install from source!
First you need Subversion, if you don’t already have it.
$ sudo apt-get install subversion
Make a directory for storing source code, and checkout couchdb into it:
$ mkdir ~/src
$ cd ~/src
$ svn co http://svn.apache.org/repos/asf/incubator/couchdb/trunk couchdb
$ cd couchdb
Read the README very carefully! Don’t make the same mistake as i did which is to assume it’s the same as installing from a tarball. It isn’t.
$ sudo apt-get install automake autoconf libtool help2man
The source code must be ‘bootstrapped’, whatever that means.
$ ./bootstrap
Now there are another bunch of dependencies to make sure you have installed!
$ sudo apt-get install build-essential erlang libicu38 libicu-dev libmozjs-dev
Whew! The libmozjs-dev was what gave me all the trouble last night, so i’m glad to have got past that particular hurdle. It’s so nice to have an up-to-date operating system, much as i dislike the actual process of upgrading.
Now let’s try installing.
$ ./configure
$ make
$ sudo make install
Great, everything worked!
$ couchdb -V
couchdb - Apache CouchDB 0.9.0a695251-incubating
Would it be so terrible to run couchdb as myself, rather than making a special couchdb user? Can’t i just do this:
$ su
# chown -R aimee /usr/local/var/lib/couchdb
# chown -R aimee /usr/local/var/log/couchdb
# exit
And now …
$ couchdb
Apache CouchDB 0.9.0a695251-incubating (LogLevel=info) is starting.
{"init terminating in do_boot",{{badmatch,{error,shutdown}},[{couch_server_sup,start_server,1},{erl_eval,do_apply,5},{erl_eval,exprs,5},{init,start_it,1},{init,start_em,1}]}}
Crash dump was written to: erl_crash.dump
init terminating in do_boot ()
Okay, fair enough, you do have to create a special user! Now we know! ;)
So, just as i did before to create a couchdb user …
$ su
# adduser --system --home /var/lib/couchdb --no-create-home --shell /bin/bash --group --gecos "CouchDB Administrator" couchdb
# chown -R couchdb /usr/local/var/lib/couchdb/
# chown -R couchdb /usr/local/var/log/couchdb/
# exit
And now i can impersonate that user to run the couchdb command (fingers crossed) …
$ sudo -i -u couchdb couchdb
sudo: unable to change directory to /var/lib/couchdb: No such file or directory
Apache CouchDB 0.9.0a695251-incubating (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
Very good! :D
Just to prove that it worked! :)
Posted in couchdb | Trackback | No comments yet | Comment here
Or about CouchDB, rather…interest is growing big time. We’ve got four dates for you in September and I’ll hint at some more in the upcoming months.
Chris Anderson, newly added committer of the CouchDB project will: “[…] be introducing CouchDB, showing off a tiny example app (which will make you never want to go back to SQL databases again), and giving a quick tour of CouchDB’s Erlang internals in the hopes of enticing new hackers into the fold.” — Full Announcement
Same as above. With RSVP!
I will give a short introduction into CouchDB followed by concrete real-world examples of how to solve common problems in CouchDB. Basically answering a few questions of “My RDBMS does X, how do I do that in CouchDB?”
Please sign up, limited seating & bring cupcakes.
Details & SignUp to follow. This will be a broad introduction into CouchDB and its concepts. Update 17.10.2008: We start at 19:00 not 20:00 as previously announced. See the talk announcement for details.
—
October brings you BarCampBerlin 3 with a session on CouchDB for web applications by Volker Goebbels and myself.
And Kore Nordmann and I will present “Document based databases: CouchDB” at the International PHP Conference in Mainz.
I’ll be talking about CouchDB at the “Data Storage Rethinking: Document Oriented Distributed Databases” track of QCon in San Francisco.
I’m happy to announce two talks on CouchDB I’m giving next week in New York City. They are both open to the public, so please RSVP at the links below if you plan to attend either.
I’ll be introducing CouchDB, showing off a tiny example app (which will make you never want to go back to SQL databases again), and giving a quick tour of CouchDB’s Erlang internals in the hopes of enticing new hackers into the fold.
The first talk will be at Arc60 (map) at 6pm on Tuesday (the 16th). Follow this link for the address, more information and to RSVP.
The second talk will be Wednesday (The 17th) at 6pm at Columbia University. If you don’t have a Columbia ID you must RSVP to jonah at ccnmtl dot columbia dot edu by Tuesday September 16th. The talk will be in room 523 Butler Library Take the 1 train to 116th street (NOT the 2/3 express trains). (map)
Okay, after my brief disappointment earlier, i’ve found a way that seems to work, at least for the time being. I have taken the RelaxDB lines out of environment.rb and put them into the application controller instead.
# app/controllers/application.rb
class ApplicationController < ActionController::Base
require File.join(File.dirname(__FILE__), '../../vendor/plugins/relaxdb/lib/relaxdb')
RelaxDB.configure(:host => 'localhost', :port => 5984)
RelaxDB.use_db('cd_collection')
end
Oh, i should say, i’m going to add the RESTful resources command into config/routes.rb:
map.resources :cds
I have also added some of the other properties into the model:
class Cd < RelaxDB::Document property :title property :artist property :publisher property :year_of_release end
It’s a shame i have to do this. I’d like to see a wrapper for CouchDB which doesn’t try to force a schema upon the database. That is one of CouchDB’s biggest strengths - that documents can contain any fields you wish.
Index of all CDs
I know my boss will probably read this, so i’ll do the right thing and start with a spec!
# spec/controllers/cds_controller_spec.rb
describe CdsController do
describe "index" do
it "should look up all CDs in the database" do
Cd.should_receive(:all).and_return([:some_cds])
get :index
assigns[:cds].should == [:some_cds]
end
end
end
And the code to make this pass is simply going to be:
# app/controllers/cds_controller.rb
class CdsController < ApplicationController
def index
@cds = Cd.all
end
end
The spec passes. Let’s write the view! Because RelaxDB wraps up the database so nicely for us, there is nothing unusual here at all.
<!-- app/views/cds/index.html.erb -->
<h1>My extensive CD collection</h1>
<table>
<tr>
<th>Title</th>
<th>Artist</th>
<th>Publisher</th>
<th>Year of release</th>
</tr>
<% @cds.each do |cd| -%>
<tr>
<td><%= h(cd.title) %></td>
<td><%= h(cd.artist) %></td>
<td><%= h(cd.publisher) %></td>
<td><%= cd.year_of_release.to_s %></td>
</tr>
<% end -%>
</table>
Now we are definitely getting somewhere! :) But the CDs appear in the order they were created. To order them by artist then title, we can do this:
@cds = Cd.all.sorted_by(:artist, :title)
You know what RelaxDB is going to do now?
[info] [<0.2122.0>] 127.0.0.1 - - “GET /cd_collection/_view/Cd/all_sorted_by_artist_and_title” 200
Yep, it dynamically creates the views it needs. Clever, hey?! And here’s the output:
Differences between ActiveRecord and RelaxDB
I’m not going to go through every line of code, but here are the main differences from ActiveRecord:
To find all CDs (as i already demonstrated):
@cds = Cd.find(:all, :order => “artist, title”)
@cds = Cd.all.sorted_by(:artist, :title)
To find a CD from the parameters:
@cd = Cd.find(params[:id])
@cd = RelaxDB.load(params[:id])
Updating attributes - there is no method ‘update_attributes’. There is ’set_attributes’ but it doesn’t actually save the document - you have to do that yourself.
@cd.update_attributes(params[:cd])
@cd.set_attributes(params[:cd])
@cd.save
Destroy requires an exclamation mark.
@cd.destroy
@cd.destroy!
Validations are rubbish. I managed to get it sort of working like this:
property :title, :validator => lambda {|t| t != ""}, :validation_msg => "cannot be blank"
property :year_of_release, :validator => lambda {|p| p.to_i > 0}, :validation_msg => "must be a number"
And then in the view you have to do something like this:
<% @cd.errors.each_pair do |field, message| -%> <p><%= field.to_s.humanize %> <%= message %></p> <% end -%>
… it’s very flaky though. Sometimes it lets it through even when the validation blatantly fails. Still, it’s a known limitation.
My verdict on RelaxDB
I like it. I am glad that i had to change very little in the views to get it to work, and relatively few things in the controller. Apart from the validation and error messages, it’s a good little plugin if you want to get going quickly on CouchDB. I’d like to see it allow dynamic fields on documents … that would be a good improvement.
I may work a bit more with RelaxDB … it would be good to find out how to link two models. Watch this space! :)
Part 7: Installing CouchDB from Subversion source code
Posted in couchdb, ruby on rails | Trackback | 5 comments | Comment here
So today i’m going to try using the RelaxDB plugin. I am fortunate in that there is a tutorial already written, which might make it easier. However, the tutorial is for Merb, so it might not work at all on Rails. We’ll see.
Start a new Rails project
I have deleted my previous project and i’m starting again. The initial steps are the same as last time to create the project, create a Git repository, and bring in RSpec.
$ rails cd_collection
$ cd cd_collection
$ git init
$ git submodule add git://github.com/dchelimsky/rspec.git vendor/plugins/rspec
$ git submodule add git://github.com/dchelimsky/rspec-rails.git vendor/plugins/rspec-rails
$ ./script/generate rspec
I’m slightly confused as to whether you’re supposed to install RelaxDB as a gem, or whether it is okay just to include it as a plugin. For now i’m going to try as a plugin.
$ git submodule add git://github.com/paulcarey/relaxdb.git vendor/plugins/relaxdb
Spec tests
Like all good projects, RelaxDB has some RSpec tests, but before i can run them i need a few more gems such as extlib, json, uuid. Just use ‘gem install’ as root to install them. The specs will not run until you meet all the requirements.
I also found that the specs require CouchDB to be running on http://localhost:5984. As before, i use the su impersonate command to get it running.
$ sudo -i -u couchdb couchdb
And now for the moment of truth …
$ rake spec
..................................................................................................................
Finished in 6.620036 seconds
114 examples, 0 failures
Amazing! I’m feeling very hopeful about this! So let’s jump straight in!
Configuration
A big win over ActiveCouch, RelaxDB has the option to set the database connection details. First i have to require the RelaxDB plugin … i’m doing this in config/environment.rb
require File.join(File.dirname(__FILE__), '../vendor/plugins/relaxdb/lib/relaxdb')
I suspect if i’d installed it as a gem i would have been able to just do this:
require 'relaxdb'
Never mind. It’s fine. Now i can set my database connection details:
RelaxDB.configure(:host => 'localhost', :port => 5984)
RelaxDB.use_db 'cd_collection'
Create a model
The key thing today is to inherit from RelaxDB::Document. It seems i also have to set up the fields (in RelaxDB they are called properties). For now i’ll just start with a title.
# app/models/cd.rb class Cd < RelaxDB::Document property :title end
Use script/console to ease us in gently
$ ./script/console
Loading development environment (Rails 2.1.0)
>> cd = Cd.new
=> #<Cd:23965060258520, _id: "deadea40-6256-012b-aeb2-001a9205e793">
>> cd.title = 'Made Of Bricks'
=> "Made Of Bricks"
>> cd.save
=> #<Cd:23965060258520, _id: "deadea40-6256-012b-aeb2-001a9205e793", _rev: "4255803143", title: "Made Of Bricks">
It is my lucky day!! Let’s see if that actually did what i think it did. The CouchDB log file looks promising:
[info] [<0.1970.0>] 127.0.0.1 - - “GET /cd_collection” 200
[info] [<0.1972.0>] 127.0.0.1 - - “PUT /cd_collection/deadea40-6256-012b-aeb2-001a9205e793″ 201
And yes, sure enough, here’s my new document sitting happily in the database:
I notice the ‘class’ field has been inserted with the value ‘Cd’. This is clearly a special field that RelaxDB uses to map the document to a model in Rails. It’s a much better idea than ActiveCouch which seemed to want a separate database per model.
What else can we do? Find our CDs again? In ActiveRecord you’d do Cd.find(:all) but in RelaxDB it’s just Cd.all
>> Cd.all
=> [#<Cd:23965059931020, _id: "deadea40-6256-012b-aeb2-001a9205e793", _rev: "4255803143", title: "Made Of Bricks">]
I wonder if i can trick it into finding the other CDs in my database, if i add the ‘class’ field to them all …?
>> Cd.all
=> [#<Cd:23965059900720, _id: "1", _rev: "1785545430", title: "Unwritten">, #<Cd:23965059892540, _id: "2", _rev: "1917382848", title: "Screaming Serenades">, #<Cd:23965059874120, _id: "3", _rev: "548309184", title: "Harry Potter and the Order of the Phoenix">, #<Cd:23965059858260, _id: "4", _rev: "288823659", title: "N.B.">, #<Cd:23965059840120, _id: "5", _rev: "374227142", title: "Vampire Weekend">, #<Cd:23965059821740, _id: "deadea40-6256-012b-aeb2-001a9205e793", _rev: "4255803143", title: "Made Of Bricks">]
Sure enough, yes! It works! And look what else i’ve just seen:
RelaxDB helpfully created the view that it needed to search for CDs. I am very impressed.
Searching by ID is slightly different from ActiveRecord. Instead of Cd.find(id) we need RelaxDB.load(id) …
>> RelaxDB.load(1)
=> #<Cd:24003693527840, _id: "1", _rev: "1785545430", title: "Unwritten", artist: "Natasha Bedingfield">
Oh, by the way, i added ‘artist’ as a property in the model. That works too! :)
Remember that most documents will be looked up using a UUID that RelaxDB will generate for us. That first CD just happened to be ID 1 because i gave it that ID the other day.
>> RelaxDB.load("deadea40-6256-012b-aeb2-001a9205e793")
=> #<Cd:24003693442460, _id: "deadea40-6256-012b-aeb2-001a9205e793", _rev: "4255803143", title: "Made Of Bricks">
There doesn’t seem to be a ‘find’ method. I don’t know how we would find the first CD, or search by title.
RelaxDB can delete documents using a ‘destroy!’ method:
>> cd = Cd.new(:title => "Delete me")
=> #<Cd:23749309141020, _id: "77d7a1f0-625b-012b-aeb4-001a9205e793", title: "Delete me">
>> cd.save
=> #<Cd:23749309141020, _id: "77d7a1f0-625b-012b-aeb4-001a9205e793", _rev: "1029394382", title: "Delete me">
>> RelaxDB.load("77d7a1f0-625b-012b-aeb4-001a9205e793")
=> #<Cd:23749309081880, _id: "77d7a1f0-625b-012b-aeb4-001a9205e793", _rev: "1029394382", title: "Delete me">
>> cd.destroy!
=> #<Cd:23749309141020, _id: "77d7a1f0-625b-012b-aeb4-001a9205e793", _rev: "1029394382", title: "Delete me">
>> RelaxDB.load("77d7a1f0-625b-012b-aeb4-001a9205e793")
RuntimeError: 404:Object Not Found
METHOD:GET
URI:/cd_collection/77d7a1f0-625b-012b-aeb4-001a9205e793
{"error":"not_found","reason":"deleted"}
Dead end
Unfortunately, i seem to have come to a dead end here. Although i think i have everything needed for some basic web app scaffolding, i keep getting the following error when i try to load a page in a web browser:
You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.reverse
This happens even if i have a completely empty action in the controller.
The error seems to be coming from ActiveRecord, so i think something is still calling ActiveRecord when it shouldn’t. Anyone got any ideas? Why would it even bother to load ActiveRecord when the model inherits from RelaxDB?
Anyway, i’m not finished with RelaxDB yet. I like what i’ve seen so far. I’ll publish this progress and hope to get some feedback on the error. My next post will hopefully be a tutorial about creating the Rails scaffold.
Edit to add: The problem comes when i include relaxdb in the environment.rb. Is there a way to get around this?
Edit again to add: It does seem to work if i put the include statement into the application controller. I think i’m back on track. Stay tuned for the next installment!
Part 6: Getting to scaffolding using RelaxDB
Posted in couchdb, ruby on rails | Trackback | 7 comments | Comment here
A grand moment has come. It is time to attempt to get Rails talking to my CouchDB database.
I have come across four plugins that i intend to try out. ActiveCouch, RelaxDB, couchrest and couchobject. They are all hosted on GitHub because GitHub is made of love.
Today i am going to try ActiveCouch. My main reason for picking it first is that it seems to be inspired by ActiveRecord. It is quite evidently intended for Rails. It seems quite close to what i’m used to. RelaxDB may also be, but it is written for Merb so i may run into troubles that are simply differences between Rails and Merb.
I have some doubts about ActiveCouch, particularly that it hasn’t been updated since July. It also seems to have some odd syntax … but we’ll see how it goes.
A new Rails app
Nothing unusual to begin with …
$ rails cd_collection
$ cd cd_collection
$ git init
Including RSpec comes as second nature to me now … i have been well trained!
$ git submodule add git://github.com/dchelimsky/rspec.git vendor/plugins/rspec
$ git submodule add git://github.com/dchelimsky/rspec-rails.git vendor/plugins/rspec-rails
$ ./script/generate rspec
So i’m guessing the next thing to do is include ActiveCouch as a submodule too.
$ git submodule add git://github.com/arunthampi/activecouch.git vendor/plugins/activecouch
Looking at the README i see that i need a json gem, which i don’t think i have. There are some other requirements too, so if you’re following along at home, make sure you have them all!
$ su
# gem install json
# exit
$ gem list
I ran ‘gem list’ to confirm that i now have the json gem, version 1.1.3.
ActiveCouch spec tests
So i presumably want to run the specs that are included with ActiveCouch. I tried this:
$ cd vendor/plugins/activecouch
$ rake
… which was probably the wrong thing to do! It has created a pkg directory and zipped the whole lot up into a gem, a tgz and a zip file. Not exactly what i had in mind! What i should have done was check what the Rakefile actually does!
$ rake -T rake clobber_package # Remove package products rake gem # Build the gem file activecouch-0.1.9.gem rake package # Build all the packages rake repackage # Force a rebuild of the package files
So i think i’ll go ahead and clobber the package! But how do you run the specs?
$ ruby spec/connection/initialize_spec.rb ... Finished in 0.065529 seconds 3 examples, 0 failures
Well, that’s a good start! But surely i don’t have to go through every file one at a time? This is not filling me with joy. Unfortunately i am revealing a gap in my knowledge. I have never run specs on a plugin before, and i don’t know how to write a Rake file to do it.
Well this is a little bit hacktastic, but let’s give it a go …
# vendor/plugins/activecouch/spec/runner.rb
files = File.join(File.dirname(__FILE__), "*/*.rb")
Dir[files].each do |file|
puts `ruby #{file}`
end
Great! Loads of stuff fails! Stuff like “Failed with 500 Internal Server Error”. I have ensured that the CouchDB is running on port 5984, as the specs expect. What about “Error creating database - got HTTP response 200″. If it can’t even create a database, that would explain a lot of the other errors. Here’s an odd one:
'ActiveCouch::Base #after_save method with an Object (which implements after_save) as argument should call before_save in the object passed as a param to after_delete' FAILED
expected: true,
got: false (using ==)
Oh dear, it’s not looking good for ActiveCouch. I am discouraged by a comment i’ve found in the STATUS file: “Version 0.2.0 and upwards will support CouchDB 0.8.0 and upwards.” - i am on CouchDB version 0.8.1. I can’t find a version 0.2.0 of ActiveCouch anywhere. I’m guessing CouchDB has moved on a lot and ActiveCouch has failed to keep up.
Still, on the bright side, quite a lot of the specs did actually pass. I’m determined to get at least something working so that Part 4 is not a total disaster!
I know what, i’ll create a model, but instead of inheriting from ActiveRecord::Base i will inherit from ActiveCouch::Base.
# app/models/cd.rb
class Cd < ActiveCouch::Base
end
Now let’s see what it can do.
script/console FTW
$ ./script/console
Loading development environment (Rails 2.1.0)
>> cd = Cd.new
=> #<Cd:0x2b852ffcbfb8 @associations={}, @attributes={:_id=>nil, :_rev=>nil}, @callbacks={}, @connection=nil>
Interesting! I have a CD document! It has the id and rev attributes that i saw last night in the CouchDB web interface. I wonder whether i can add some fields to it.
>> cd.title = 'Peaceful, The World Lays Me Down'
NoMethodError: undefined method `title=' for #<Cd:0x2b852ffcbfb8>
Useless. That’s the kind of error i’d expect from ActiveRecord when it can’t find a field in the database. The whole point of CouchDB is it’s document-oriented and you can put more-or-less whatever fields you like into each document. But wait a minute, here’s an idea:
class Cd < ActiveCouch::Base has :title end
Now let’s try.
>> cd = Cd.new(:title => 'Peaceful, The World Lays Me Down')
=> #<Cd:0x2b9f6de3e120 @callbacks={}, @associations={}, @attributes={:_id=>nil, :_rev=>nil, :title=>"Peaceful, The World Lays Me Down"}, @connection=nil>
Better. So we have to define our attributes in the model. I can live with that.
>> cd.save
NoMethodError: You have a nil object when you didn't expect it!
The error occurred while evaluating nil.post
Yeah, i wasn’t really expecting that to work! I haven’t told Rails anything at all about my database! Normally we’d do it in config/database.yml of course, but i don’t know how to set up a CouchDB adapter. Fortunately i do know how to trawl through specs (in the absence of any documentation) to look for clues!
class Cd < ActiveCouch::Base site 'http://localhost:5984/' has :title end
Please forgive the hard-coding for the moment! Now, let’s try again.
>> cd = Cd.new(:title => 'Peaceful, The World Lays Me Down')
=> #<Cd:0x2b8a2c7174b0 @callbacks={}, @associations={}, @attributes={:_id=>nil, :_rev=>nil, :title=>"Peaceful, The World Lays Me Down"}, @connection=#<ActiveCouch::Connection:0x2b8a2c71e6c0 @site=#<URI::HTTP:0x2b8a2c71e3f0 URL:http://localhost:5984/>>>
>> cd.save
ActiveCouch::ResourceNotFound: Failed with 404 Object Not Found
Ah, it helps to watch the log of CouchDB as we do this.
[info] [<0.602.0>] HTTP Error (code 404): not_found
[info] [<0.602.0>] 127.0.0.1 - - “POST /cds” 404
ActiveCouch seems to have made an assumption that the database is called cds. I’m not sure i’m very happy about this. Does this mean there is a separate database per model? Surely that’s non-optimal. Am i misunderstanding something here? Well, let’s try creating the database and see if it keeps ActiveCouch happy.
>> ActiveCouch::Migrator.create_database('http://localhost:5984/', 'cds')
=> true
The CouchDB log indicates that the database was created:
[info] [<0.603.0>] 127.0.0.1 - - “PUT /cds” 201
Now maybe we’ll have some success.
>> cd.save
=> true
>> cd
=> #<Cd:0x2b339ae93ce0 @callbacks={}, @associations={}, @attributes={:_id=>"29dc9e093c369e13c05c87baf5be00f8", :_rev=>"3659454517", :title=>"Peaceful, The World Lays Me Down"}, @connection=#<ActiveCouch::Connection:0x2b8a2c71e6c0 @site=#<URI::HTTP:0x2b8a2c71e3f0 URL:http://localhost:5984/>, @default_header={"Content-Type"=>"application/json"}>>
Good. It has generated a UUID and a revision for me, so i guess this has been saved.
Sure enough, i can now see the document through the CouchDB web interface. Apparently i am winning!
Finding documents
Unfortunately i can’t seem to find a CD again after i created it.
>> Cd.find(:all)
NoMethodError: You have a nil object when you didn't expect it!
The error occurred while evaluating nil.keys
>> Cd.find(:first)
NoMethodError: You have a nil object when you didn't expect it!
The error occurred while evaluating nil.keys
>> Cd.find(:last)
=> nil
>> Cd.find_by_title("Peaceful, The World Lays Me Down")
NoMethodError: undefined method `find_by_title' for Cd:Class
LOL. ActiveRecord this is not!
I think this would work, if i had a view:
>> Cd.find(:all, :params => {:title => "Peaceful, The World Lays Me Down"})
ActiveCouch::ResourceNotFound: Failed with 404 Object Not Found
See what it’s trying to do:
[info] [<0.621.0>] HTTP Error (code 404): {not_found,missing}
[info] [<0.621.0>] 127.0.0.1 - - “GET /cds/_view/by_title/by_title” 404
Fine, i’ll create the view. I don’t know how to do it with ActiveCouch so i’ll just do it through the web interface.
Well, that’s good. Yesterday i didn’t think you could create views with underscores in the ID. So let’s try again.
>> Cd.find(:all, :params => {:title => "Peaceful, The World Lays Me Down"})
=> [#<Cd:0x2b339ae67fa0 @callbacks={}, @associations={}, @attributes={:_id=>"29dc9e093c369e13c05c87baf5be00f8", :_rev=>"3659454517", :title=>"Peaceful, The World Lays Me Down"}, @connection=#<ActiveCouch::Connection:0x2b339ae9aef0 @site=#<URI::HTTP:0x2b339ae9ac20 URL:http://localhost:5984/>, @default_header={"Content-Type"=>"application/json"}>>]
Hey, can we find by ID?
>> Cd.find('29dc9e093c369e13c05c87baf5be00f8')
=> [#<Cd:0x2b339ae67fa0 @callbacks={}, @associations={}, @attributes={:_id=>"29dc9e093c369e13c05c87baf5be00f8", :_rev=>"3659454517", :title=>"Peaceful, The World Lays Me Down"}, @connection=#<ActiveCouch::Connection:0x2b339ae9aef0 @site=#<URI::HTTP:0x2b339ae9ac20 URL:http://localhost:5984/>, @default_header={"Content-Type"=>"application/json"}>>]
Excellent! This is a bit more promising.
So let’s scaffold up a web interface
# app/controllers/cds_controller.rb
class CdsController < ApplicationController
def show
@cd = Cd.find(params[:id])
end
end
And of course, the view to go with it …
<!-- app/views/cds/show.html.erb --> <h1>A CD in CouchDB</h1> <p> <strong>Title:</strong> <%= @cd.title %> </p>
I’ll need to declare a resource in config/routes.rb …
map.resources :cds
Come on then, fingers crossed …
My gosh, it worked!! :D Yeah yeah, i know it doesn’t look like much, but it has connected to the database, found the document, loaded it into ActiveCouch and output it for me.
My verdict
So i have proved that you can do it. Hurrah! But i don’t like the way i had to do it. I don’t like having the database named after the model (what if, God forbid, you have more than one model?!) I don’t like having to create view called by_title/by_title if i want to search by title. I don’t know why i can’t do Cd.find(:all).
Of course, i may have got it entirely wrong and maybe somebody who knows about ActiveCouch will be able to tell me exactly how i should have done it. But for now, i’ve gone as far as i want to go. I’ve shown how you can do it, and if you’re interested you could probably take it further. But stay tuned, because i think there are better things yet to come in this series …! :)
Thank you if you stuck with me through this long and rambling exploration of ActiveCouch!
Part 5: Integrating with Rails using RelaxDB
Posted in couchdb | Trackback | 13 comments | Comment here
Time for a little play with the CouchDB. Let’s see what this thing can do!
The first thing i notice is some test databases left over from when i ran the test suite. I’m not sure if that’s supposed to happen. I kind of expected the tests to clean up after themselves. But anyway, it means i can have a little look and see what data they contain.
Apparently nothing very much. We have some documents with keys and a ‘rev’ value, whatever that might be. Looking inside a document, it contains some examples of an integer field and a string field. I could delete them and add my own fields, if i wish. But this is not so interesting. I’m going to create my own database!
My CD collection
Here’s a fantastic opportunity to totally embarrass myself by revealing the CDs i own. I have previously made a CD collection database in Lotus Notes, so i think it will work. I’ll use the ‘Create Database’ option in the web interface and call it ‘cd_collection’. The log informs me what has happened:
[info] [<0.77.0>] 127.0.0.1 - - “GET /_utils/browse/_create_database.html” 304
[info] [<0.77.0>] 127.0.0.1 - - “PUT /cd_collection/” 201
[info] [<0.77.0>] 127.0.0.1 - - “GET /_utils/browse/database.html” 200
[info] [<0.77.0>] 127.0.0.1 - - “GET /cd_collection/_all_docs” 200
So now i’ll go ahead and start making documents. Hmm, what should i use as the ID, i wonder? Presumably anything that uniquely identifies the CD. But since i’m going to be using this with Rails, i’ll stick to the convention of an auto_increment integer. So my first CD is going to be ID 1.
Edit to add: You don’t need to choose an ID yourself. You can leave it blank and press Create, then it generates a UUID for you.
Well this is fun. I can create all the fields i want. The beauty of CouchDB being a document-oriented database is i can make up the field names as i go along - i am not constrained by a schema.
When i’m ready i’ll go ahead and click Save Document and see what happens in the log.
[info] [<0.83.0>] 127.0.0.1 - - “PUT /cd_collection/1″ 201
[info] [<0.83.0>] 127.0.0.1 - - “GET /_utils/browse/document.html” 200
[info] [<0.83.0>] 127.0.0.1 - - “GET /cd_collection/1″ 200
[info] [<0.84.0>] 127.0.0.1 - - “PUT /cd_collection/1″ 201
[info] [<0.84.0>] 127.0.0.1 - - “GET /cd_collection/1″ 200
Okay, not very interesting, and i’ve even stripped out some of it. I guess it created a document, showed it to me, accepted my changes and showed it to me again.
I notice in the web interface that there are two versions of the document: one with just the _id and _rev fields, and a newer one with the fields i added. Version control is apparently built into CouchDB! Very good!
Views
So i have created a few documents; now i should be able to view them. The default CouchDB view is “All documents” but it seems i could create my own custom view if i knew how. I have to write some sort of map function to apply to each document. Here’s an idea:
function(doc) {
emit(doc.title, doc);
}
Now that is exciting! It has output the title in the column on the left, and it has also ordered by title. I’m going to save that as a ‘by_title’ view. Oops, there’s a gotcha. It doesn’t seem to like underscores in IDs. So i’ll just save it as ‘title’. Now it shows up under Design documents. It’s just another document in the database! How nifty is that?! :)
Now i’ll try to create another view. For some reason i’d like to see all my CDs which were released after 2003, ordered by artist. I suppose i’ll need a map and a reduce to do that.
Oh cool, just as i am writing this, @benatkin tweets up a link to the views on Divan, which include some reduce functions. This is collective consciousness at its best! :)
It seems that this reduce function produces a unique list, removing duplicates …
function(keys, values) {
return sum(values);
}
… but how to filter down by year?
Ahhh, got it! You don’t need the reduce function to filter out documents, you just put an ‘if’ clause in the map. It’s JavaScript, FTW! :D I also figured out how to tweak the output a bit …
function(doc) {
if(doc.year_of_release > 2003) {
emit([doc.artist, doc.title], {
artist: doc.artist,
year: doc.year_of_release,
publisher: doc.publisher
});
}
}
Good job! :) Now i’m not really sure what the reduce function is for … maybe that’s a question for discussion in the comments.
One more thing i gleaned from looking at the Divan views … it is possible to store multiple views in a single design document. So i can create a document called _design/cds and create by_title and by_artist within it. This probably makes more sense.
Tracks
In a relational database we would store tracks of a CD in a different table, with a foreign key linking it to the CD. Rails would understand the two tables as two models:
class Cd < ActiveRecord::Base has_many :tracks end class Track < ActiveRecord::Base belongs_to :cd end
With a document-oriented database, such as CouchDB, we’ll presumably do this a bit differently. JSON, like XML can be nested, so perhaps a CD and its tracks may be represented in JSON like this:
{
"title": "Screaming Serenades",
"artist": "Kindle",
"year_of_release": 2002,
"publisher": "Alliance Music"
"tracks": [
{"title": "Every Little Thing You Do", "length": "3:25"},
{"title": "Don't Fly Away", "length": "4:16"},
{"title": "Brighter Days", "length": "3:08"},
{"title": "Little Bit Of Your Love", "length": "3:59"},
{"title": "Days Like These", "length": "3:42"},
{"title": "Gone Crazy", "length": "3:22"},
{"title": "Live For Heaven", "length": "4:36"},
{"title": "Someone To Live For", "length": "3:39"},
{"title": "Step On Up", "length": "4:31"},
{"title": "Not Impossible", "length": "3:10"},
{"title": "Here I Stay", "length": "4:32"},
{"title": "State Of Waiting", "length": "6:54"}
]
}
Edit to add: There are some discussions about the best way to join data; nested or in separate documents. In this case i think nested is good because the tracks really help to define the CD. In the case of a blog post with comments, the comments could easily stand alone as separate documents.
Certainly the web interface accepts this format, and i have managed to store 12 tracks into my document.
I could really do with some confirmation that i’m doing the right thing with this nested data. I’m not quite sure how we’ll deal with this when we get to Rails.
I think that will be the challenge for my next installment! :)
Part 4: Integrating with Rails using ActiveCouch
Posted in couchdb, ruby on rails | Trackback | 2 comments | Comment here
It occurs to me that before i can get Rails working with CouchDB it would be a good idea to actually install CouchDB! I’m going to try the download rather than the SVN source code because i’m clueless with SVN. So here goes. I am downloading version 0.8.1.
Edit to add: If you want to install from Subversion, see my later tutorial in Part 7: Installing CouchDB from Subversion source code
Ah, the README tells me i need Erlang and a few other things. Since i use Debian i can apt-get some packages. (Have i ever said how much i love Debian?!)
# apt-get install build-essential erlang libicu36 libicu36-dev libmozjs-dev
That was easy! Now on with the CouchDB install.
$ ./configure
$ make
$ su
# make install
# exit
Awesome! All done!
I like to clean up temporary files afterwards. It’s a virgo thing.
$ make clean
$ make distclean
Do i really need to make a new user and group just to run CouchDB? Can’t the owner just be me? Oh well, at this stage in the game i’m going to do as i’m told, but this seems a bit odd to me.
$ su
# adduser --system --home /var/lib/couchdb --no-create-home --shell /bin/bash --group --gecos "CouchDB Administrator" couchdb
# chown -R couchdb /usr/local/var/lib/couchdb/
# chown -R couchdb /usr/local/var/log/couchdb/
# exit
Okay, so the couchdb user owns the directories that make CouchDB work. So apparently i can use sudo to impersonate the couchdb user and run the command ‘couchdb’.
$ sudo -i -u couchdb couchdb
Well apart from a little error, that seems to have worked.
So now i can apparently go to http://localhost:5984/_utils/index.html and see something …
Nice! And apparently there are some test specs i can run … ah yes, the link is there on the right …
Awesomeness abounds!! All the tests passed!
The server console tells us what is happening as it runs the tests:
I see some RESTful calls to POST, GET, PUT and DELETE there, and some responses: 200 for OK, 201 for Created, and 202 for Accepted. That is neat!
In the next part i will experiment and see what i can do with CouchDB via the web interface. For now, i’ll go to bed happy that i got it installed and running. That was a very easy and satisfying install. :)
Part 3: Experimenting with CouchDB’s web interface
Posted in couchdb | Trackback | 10 comments | Comment here
I’ve had a really nice weekend: interesting and exciting, both of which are relative terms, i know!
Yesterday i made my own version of the Rails scaffold generator, featuring Low Pro to do as much as possible via AJAX. The JavaScript is completely unobtrusive, all done by Low Pro behaviours. It degrades gracefully for browsers without JavaScript. I have RSpec tests for the controller, too. It’ll probably be available on GitHub soon … i just have to check with my boss that it’s alright because we’ll be using it at work.
Today i got a lot of chores done, which is a relief. Now i wouldn’t be completely embarrassed if somebody were to spring a surprise visit on us! I wrote a review of RailsConf for our company blog, and i learnt more about CouchDB and made a decision to get Rails working with it.
I am grateful to have some friends on Twitter that i met at RailsConf, who are excited about CouchDB too. It feels as if i am getting involved in something that’s going to be big and exciting, and i’m glad to be a part of it, even if it’s just writing about my experiences. I am sure it must be possible to get Rails working on CouchDB … i just haven’t managed to find any documentation to tell me how. I feel that is a space that i can fill! :)
Posted in couchdb, ruby on rails | Trackback | No comments yet | Comment here
This afternoon i decided that what the Internet really needs right now is some information about how to connect Ruby on Rails to a CouchDB database. So i’m going to have a go! I don’t quite know what to do (which will probably become very clear!) so this will be a play-along-at-home experience. I’ll blog about it in a series of posts, stopping whenever i get stuck. Hopefully i’ll receive some feedback on whether what i’m doing is a good idea, and get some tips for what to try next.
This first post is just going to be an introduction, stating my interest in wanting to get this to work.
Some brief definitions
For those who don’t know, here are some basic starting points …
Ruby is an open-source, interpreted, object-oriented programming language created by Yukihiro Matsumoto (’Matz’).
Rails is a web application framework, created by David Heinemeier Hansson, written in Ruby, which guides us down an ‘opinionated’ route of best practices such as Model-View-Controller architecture, RESTful techniques, test/behaviour-driven-development. Thanks to its use of conventions rather than configuration, it is easy to get going quickly with Rails. However, it can be quite difficult to break out of the conventions, as we are going to try to do by using CouchDB as the database platform.
CouchDB, from what i can gather, is a document-oriented, RESTful, distributed database system written in Erlang.
Erlang is a programming language from Ericsson, which enables features such as concurrency, soft-real-time access, and fault-tolerance.
More technical details of CouchDB follow …
Document-oriented database
A popular database model is a relational database, using tables with columns for attributes and rows for data. Rows from multiple tables can be joined using data with matching values, to create relations. MySQL, SQLite, PostgreSQL, Oracle, Microsoft SQL Server and Microsoft Access are examples of relational database systems. Rails, specifically ActiveRecord, works very well with relational data.
Document-oriented databases are different. There is no real hierarchy of data; just a collection of documents which may contain virtually any kind of data. The documents may not necessarily be the same length, as some documents may contain details of fields that other documents do not need to store. In other words, you are not constrained by a database schema.
To query a document-oriented database is to write a view which selects documents and displays them in a particular way. It’s like making a virtual folder in Windows Vista. You can search across all your documents with any criteria you choose - file type, size, last update, tags - and then you can save that search to use again in the future. Of course, it may bring back different results next time you view it, if the data has changed.
CouchDB stores documents as JavaScript Object Notation (JSON). This has a three-fold advantage: it is easy for humans to read, fast for computers to parse, and lightweight to send across a network (compared with XML). I like JSON.
I am also fond of the document-oriented database model because i worked extensively with Lotus Notes for many years at IBM.
Representational State Transfer (REST)
REST is a concept that has been becoming very popular over the last few years, probably helped by the embrace it is receiving from the Rails community. When you do something RESTfully, you are performing actions on a resource. There are four basic actions that we might want to use on a resource: create it, read/view it, update it, and delete it. Sometimes this is called CRUD.
These four actions relate very nicely to four HTTP methods. Most web developers know about GET and POST. When we view anything on the Internet we are normally using the GET method to get something from the web server. When we fill in a form and submit data to the web server, we are usually using a POST method. I think that’s why we say ‘blog post’ because it is something we have literally POSTed to the Internet. But there are two other methods that we can use: PUT for when we get something, change it, and put it back, and DELETE for the last of the four CRUD actions.
So we apply the HTTP methods to the resource, indicated by a URI. The point is that the URI stays the same. Suppose i have a resource which is a task, uniquely identified by this URI:
http://site.com/tasks/1
If i GET that URI then i am viewing the task. If i PUT to it then i am updating it. If i send a DELETE request to that same URI then the server will know that i wish to delete the resource.
Working with CouchDB is RESTful by design. Each document is a unique resource. You can GET a document, POST a new one, PUT some updates, and DELETE a document. That’s the way it works. I am guessing that CouchDB must implement its own little web server to respond to the HTTP methods.
Distributed
A great feature of document-oriented databases is replication. I used to like this about Lotus Notes. The database need not exist in only one place. Everyone who uses it can have their own local copy, if they wish. Using Erlang’s concurrency and fault-tolerance features makes it easy for CouchDB to replicate changes across all copies of the database in a secure and reliable way. Documents, views and indexes can all be kept up to date in soft-real-time (i.e. even if it’s not precisely real-time it’s close enough for the users not to notice that it isn’t). CouchDB handles conflicts at replication time without needing locks.
So what next?
With the introductions over, i am ready to get started on creating a Rails application which runs on a CouchDB database. I have been advised to look at the couchrest library for Ruby, and i am also investigating ActiveCouch which might be a wrapper around the database in the same way that ActiveRecord wraps itself around a database table, extending functionality as a model.
Hopefully it won’t be long before i update again with some results. In the meantime, if anyone has any information, feel free to chime in! :)
When i’m all done with the investigations i hope to write a summary on a wiki somewhere, like, maybe here :)
Part 2: Installing CouchDB
Posted in couchdb, ruby on rails | Trackback | 10 comments | Comment here
It’s been a few months since the second round of Tim Bray’s Wide Finder project.
I’d wanted to try my hand at a solution with CouchDB, but didn’t start because there’s really no chance that it would outshine a dedicated solution to the problem. Now I’ve started on Wide Finder in CouchDB which is just past the hello world stage (it works, but hasn’t been benchmarked yet).
Couch has the defining characteristic that views are computed incrementally. What this means is that while it may be slower than the other Wide Finder entries to compute the initial results, if you run it again, the results are almost instantaneous. Reduce result retrieval time is O(log n) due to CouchDB’s B-Tree storage engine for data and intermediate results.
This makes CouchDB ideal for applications where additional data is added over time. Rather than batch-processing your log files, just log directly to CouchDB. When you want to generate reports, just make the request. Since the amount of logging done since the last report will usually be small compared to the full size of the log db, calculating results which take the new data into account will be very quick.
Generating Wide Finder reports relies on aggregating data (# of hits, total bytes) by URI, and then sorting on the aggregated total. CouchDB’s aggregation facilities (the reduce in map/reduce) have been a focus of development. Currently though, sorting on the results of aggregated totals must be done in your application. Bringing this calculation into CouchDB will provide opportunities for further optimization.
Good news last: Because of CouchDB’s View Collation the same map/reduce index can be used to support queries across the entire data set, or just from a range. For instance, my Wide Finder views currently emit(uri, bytes) which mean that not only can I get the total number of hits and bytes per-URI, I can also see the total from all URIs as well as the total from any lexigraphically contiguous range of URIs. That is, I can see if URIs prefixed with /ongoing are more popular in total, than those with other prefixes. These results will also be nearly instantaneous.
I plan to finish Wide Finder in CouchDB, and then run it on Tim’s sweet 64-hardware threaded Niagra. We’ll use the benchmarks in the CouchDB project to motivate optimizations of multi-core view generation. Because we haven’t optimized much of CouchDB at all yet, this should give us a good baseline with which to measure our efforts.
I have written before about how I'm running my website on custom developed Python code. Back then I said:
The code isn't publicly available at this point, although I do intend to release it when I feel it's ready.
Well, stuff like this never is truly ready, but I'm putting it out there anyway: the basis is a custom little web framework called “Diva”, the project site is here, Subversion repository here. The code for both my blog and my scratchpad site is included as examples.
You got a toothbrush? We’re going to London. Do you hear that, Doug? I’m coming to London.
Update 24/08/2008: The event is “sold out”. We have filled every single seat in the venue! I am thrilled by the response. Thanks everybody! If you couldn’t make it through registration or don’t have time on Tuesday anyway, do get in touch.
—
This is a bit of a short notice, and I apologise, but things moved quickly here.
I’ll be spending three couchy days in London next week. On Tuesday night, that is the 26th, I’ll be speaking about CouchDB at Erlang Training and Consulting starting 6:30 pm. And you are all invited!
The event is free! You just need to register.
If you can’t make it and want to grab a cold beverage or anything, just drop me a line.
Big thanks for my friend Francesco for putting this together on such a short notice.
Update 23/08/2008: Version 0.8.1 released[8.4 MB]. This fixes the test suite when using Safari. Beware however, if you are using the Safari 4 Developer Preview, you can run out of memory pretty quickly when running the test suite. This is a problem with Safari, not CouchDB.
All other notes still apply. CouchDBX is still Leopard-only and Intel-only.
Enjoy!
—
After quite a long hiatus, I released another version of CouchDBX. Do you remember? (Wow, hello dear long-time reader!). CouchDBX is my unofficial binary release of CouchDB for MacOS X that comes with a double-click-and-start wrapper application.
With no further ado:
CouchDB X 0.8-trunk Intel Only.
If you want to play with CouchDB on your Mac and you don’t want to mess with any command line installation business, this is for you. Justdownload the zipfile above, Safari should uncompress it for you, double click the application icon and you are ready to go.
CouchDBX has super simplistic GUI that lets you start and stop (yay) CouchDB and launch Futon, the admin client in your favourite browser. That’s it. CouchDBX is an independent installation, so you can use it even if you have a CouchDB installation on your system, both versions will not interfere (you just can’t run them in parallel).
Note that this is an unofficial release of a preview-piece of software. Things might break and not work as expected. Do not blame me. If you run into problems, please report them in the comments below, thanks.
This release only works on MacOS X 10.5 Leopard and on Intel Macs only. We want to have PPC support, and all there is to it is creating universal binaries of all executable files and libraries in the application bundle. If you can help with that, pelase get in touch.
If you want to help with the development, you can find the source at Google Code.
Jan will be speaking about CouchDB at Erlang Training and Consulting, starting 6:30 pm. It's a free event, you are invited. Go Jan!
I'm wrong. REST, as originally described, has nothing to do with POST PUT DELETE. If you look at the original paper (chapter 5), REST is simply a description of the web as it actually work as a distributed hypermedia browser platform. The paper talks alot about all the things required to send media to the clients, but almost nothing about sending data from the client to the server.
There is nothing about verbs, RPC, POST vs. PUT, idempotency, or posting resources back to the same URI from which they were gotten. I was blasting the dogma around those things but was wrong to call it REST, I was under the misconception that REST is also about those things. Lots of writing about REST seem to have this misconception as well.
Why isn't REST about these things? Because REST is a description of the web and those things don't really exist on the web. Not in any standardized way. When people start talking about something being RESTful, do they mean its how the web actually works, or how we wish it worked?
In the real world, there is a remarkable effort to ensure that a wide range of browsers can find, retrieve and understand media and content from web servers. DNS, URIs, HTTP, firewalls, proxies, caches, media types, etc are what makes the web work and that's what REST describes, but in more generic terms.
On the other hand, pretty much any time a browser POSTs a message, only one server in the whole world is supposed to understand what it means, there is no broad standardization of sending data and content to servers and how the servers are supposed to interpret it. Each server is free to implement its own interface, and they generally do. While there has been efforts to make the web into a standardized read/write platform, nothing has caught on in a big way.
So is POST as RPC RESTful? The practice of roll-your-own POST format is alive and well and successful on the web, and therefore, by definition, RESTful. As long as you are not abstracting HTTP away (and ignoring what HTTP gives you for free), your are probably using it RESTfully.
Glad we cleared that up.
Strangely, OS X spell check doesn't recognize kerfluffle as a word, but it does recognize klerfmuffle (suggested after I typed in klerfluffle). But klerfmuffle isn't a word, not even in Google. Until now that is.
The web is built on GET and POST verbs, not the RESTful GET, PUT and DELETE (and sometimes POST) verbs.
The only thing that's RESTful about 99% of the web is that it uses GET (even then many applications use GET wrong). The argument that REST is great because it provides standardized caching support, well that pretty much only applies to GET requests anyway, what about PUT and DELETE?
It's not verbs that make it all work, it's URLs and HTTP. HTTP 1.0 is so damn simple to be almost too simple. 1.1 fixed its biggest deficiencies without adding much complexity, and now HTTP is a transport and protocol that does its job just well enough to not get in the way. And that's what the web is built on.
The Newthinking Store in Berlin runs a monthly web-geek-get-together called Webmontag. They invited me to talk about “CouchDB for Web Developers”.
When I first discovered CouchDB I thought to myself: “This will solve a lot of my everyday problems”. I was doing web development at the time (I still do) and CouchDB’s features sounded like magic. Only that they actually do exist; and they work like a charm.
CouchDB is very easy to learn, with one caveat: If you are deep into relational databases, you might have a hard time imagining that doing things “The CouchDB Way” is actually a good idea. Getting out of this state mainly involves ignoring and forgetting what you learned about RDBMSs and start to focus more on the pure data-storage aspect.
In this talk I’ll give you an idea about why you would want to use CouchDB for your next web application and I will try to shift your thinking about data storage from the relational- to a more natural data behaviour model. I’ll also show you how to use CouchDB with your favourite development platform. If you are quick, you might walk out with a first app by the end of Webmontag.
See you on Monday, August 18th, 19:00. This is a good opportunity to have some drinks with like-minded folks. Do not miss it!
P.S: If you are not yet convinced, let me throw out some more buzzwords to get your attention: HTTP REST API, JSON data, N-master- and offline-replication, written in Erlang, fault tolerant, highly concurrent, ACID & MVCC and (Apache 2.0-) open source.
P.P.S: The talk will be in German unless some non-speaker shows up and all other attendees agree on proceeding in English.
As the guy who created CouchDB, I should be a big cheerleader for RESTful architectures. But the truth is, I just don't get it.
For CouchDB, REST makes absolutely insanely perfect sense. Read a document, edit, put the document back. Beautiful. But for most applications, enterprise or not, I don't see what the big win is.
I know what is wrong with SOAP, and it has everything to do with unnecessary complexity and solving the same problems twice. But what is the big advantage of making all your calls into GET PUT and DELETE? If POST can handle everything you need, then what's the problem?
I guess what I mean to say is just because SOAP is a disaster, doesn't somehow make REST the answer. Simpler is better, and REST is generally simpler than SOAP. But there is nothing wrong with a plain old POST as an RPC call. If its easy to make all your calls conform to the RESTful verb architecture, then that's good, I guess. But if not, then just use a POST as an RPC call, keep it as simple as possible and be done with it. And don't spend another minute worrying about being RESTful or not.
I've been really busy and just not in a blogging mood lately. But I'm going to force myself to write a blog post for 15 minutes. This is it. I'm a slow typist.
CouchDB development is going well. We have more contributors and are about to add another project member. We are about to release a 0.8.1, which I think will be a very solid and useable release. The trunk is usually in good shape, but unpredictable. You might encounter serious bugs. Be warned.
The interest in CouchDB is impressive for a still immature project. And like most open source projects, its all word of mouth. We don't have a PR or marketing department. Except for Jan :)
Right now I'm working on a few performance optimizations in the btrees. It currently does linear scans on the keys in each btree node, when really it should be binary search. Doing some early profiling showed the number of keys comparisons is eating up CPU.
There are a number of big companies and small startups alike using CouchDB already. The uptake has been amazing here, I'm absolutely thrilled and CouchDB is being used to solve real problems. There is one company, a tech giant, that is piloting a couchdb project. If successful, CouchDB will be the storage engine in the next version of a product already used by millions.
But puzzlingly there is a lack of interest within IBM. I'm not much of an evangelist, but I thought people within Lotus would be interested. But so far, not so much.
Quick thought: Cloud computing is great. Right up until you lose your connection, then what?
Okay, that's 15 minutes. A quick read through, a few edits and a sick feeling at the lack of structure aaaaaaand publish.
ETech 2009, the O'Reilly conference on emerging technologies now has a call for papers. Here is my proposal for the "Nomadism & Shedworking" track:
The recent direction of business applications has been centralization to web-based systems, easing deployment, upgrades and management of application security. However, at same time centralization provides new risks like the introduction of a single point of failure for application usage and in most cases inability to work offline.
At the same time, working culture has started so shift more towards networked individuals, or "web workers" forming ad-hoc coalitions to work on various projects. For them, centralized applications might not be desirable due to data ownership and infrastructure requirements.
Moving the applications from centralized web servers to a peer-to-peer network allows web workers to be in control of their own data, stay productive even in unstable connectivity situations, and collaborate easily with their colleagues in an ad-hoc way.
This talk outlines some ways to move forward in building peer-to-peer networked business applications. There are many open source frameworks targeting the problem, including replicated databases like CouchDb and Prophet, and application-oriented P2P networks like Swallow/DBE. These will be discussed together with some real-world examples of business applications built with them:
Ajatus - a "Personal CRM" built on top of CouchDb, a RESTful replicated object database
OpenPsa - project management system doing resourcing over the DBE P2P network
Simple Defects - a P2P bug tracking system built on top of the Prophet replicated database
The first phase of P2P business applications will likely be services operating in closed networks of users' social networking or instant messaging contacts. But the P2P model can also provide opportunities for wider networking, making it possible to find new project partners or collaborators anywhere in the world. This wider-ranging business network will introduce new challenges like security and reputation management. Some ideas related to this will also be discussed.
Other interesting projects in this sphere I did not mention are Telepathy Tubes and F2F. We have also had some ideas for how Midgard could do this...
by Henri Bergius (henri.bergius@iki.fi) at August 06, 2008 04:59 PM
As part of my work getting CouchDB to do a full-text index, I decided to get to “hello world” with Lucene, the Apache Fulltext search indexer. Lucene is written in Java, so it’s inherently scary. It turned out to be easier than I expected—CLASSPATH not withstanding.
Here’s how to do it, for future reference:
svn checkout http://svn.apache.org/repos/asf/lucene/java/trunk lucene
cd lucene
ant war-demo
echo $CLASSPATH
# if it isn't set, we need to set it based on this plist file
cat /System/Library/Java/JavaConfig.plist
export CLASSPATH=line from your plist file
# add the lucene jars to the CLASSPATH (use your paths, not mine)
export CLASSPATH=$CLASSPATH:/Users/jchris/code/java/lucene/build/lucene-core-2.4-dev.jar:/Users/jchris/code/java/lucene/build/lucene-demos-2.4-dev.jar
# the fun part
java org.apache.lucene.demo.IndexFiles ~jchris/Documents/
java org.apache.lucene.demo.SearchFiles
This should give you a command prompt where you can search your Documents directory. How easy was that?
CouchRest now includes two utility scripts, couchview and couchdir. They handle the grunt-work of maintaining a correspondence between your filesystem and your database.
If you use CouchDB a lot, you should read this excerpt from the Couchview documentation. Even if you don’t become a convert, you may learn about some gotchas.
gem sources -a http://gems.github.com
sudo gem install jchris-couchrest
Usage: couchview push directory dbname
Couchview expects a specific filesystem layout for your CouchDB views (see example below). It also supports advanced features like inlining of library code (so you can keep DRY) as well as avoiding unnecessary document modification.
Couchview solves a problem with CouchDB’s view API, which only provides access to the final reduce side of any views which have both a map and a reduce function defined. The intermediate map results are often useful for development and production. CouchDB is smart enough to reuse map indexes for functions duplicated across views within the same design document.
For views with a reduce function defined, Couchview creates both a reduce view and a map-only view, so that you can browse and query the map side as well as the reduction, with no performance penalty.
couchview push foo-project/bar-views baz-database
This will push the views defined in foo-project/bar-views into a database called baz-database. Couchview expects the views to be defined in files with names like:
foo-project/bar-views/my-design/viewname-map.js
foo-project/bar-views/my-design/viewname-reduce.js
foo-project/bar-views/my-design/noreduce-map.js
Pushes to: http://localhost:5984/baz-database/_design/my-design
Creates the design document:
{
"views" : {
"viewname-map" : {
"map" : "### contents of view-name-map.js ###"
},
"viewname-reduce" : {
"map" : "### contents of view-name-map.js ###",
"reduce" : "### contents of view-name-reduce.js ###"
},
"noreduce-map" : {
"map" : "### contents of noreduce-map.js ###"
}
}
}
Couchview will create a design document for each subdirectory of the views directory specified on the command line.
Couchview can optionally inline library code into your views so you only have to maintain it in one place. It looks for any files named lib.* in your design-doc directory (for doc specific libs) and in the parent views directory (for project global libs). These libraries are only inserted into views which include the text
//include-lib or #include-lib
Couchview is a result of scratching my own itch. I’d be happy to make it more general, so please contact me at jchris@grabb.it if you’d like to see anything added or changed.
What a week!
First off, here are the slides from my talk “CouchDB at 10,000ft”. Thanks to everyone who showed up and a double thanks for the eight of you who rated my session so far. More special props to the ones with constructive criticism. I’ll do better next time. If you haven’t already, place your vote.
Ted has some nice pictures of my talk and the discussion afterwards. The feedback I got was great and people do really want to use CouchDB. This is great encouragement for putting in some after-work hours to push things towards 1.0.
But OSCON.
This was my first and I couldn’t have liked it better. I got in Sunday without any signs of jetlag (will blog about the secret later) and started things off with dinner & pub crawl with my old (and some new) PHP friends.
OSCON starts at around 8:00, the official programme ends no later than 18:00, but there’s a lot of after-conf activity. It basically means dinner, drinks and party until midnight, or even later with an occasional BoF session thrown in. The next day starts at 8:00 again. 7 days. Is that exhausting? Yes. But it is also great times.
I met a bunch of Apache folks that are connected with the CouchDB project which was nice. The XMPP folks are interested in bonding with CouchDB as well, as are the Python folks I’ve met. Heck, even Brian of MySQL fame is a big fan of ours.
I’ve attended (among others!) a talk on Prophet, a “Prophet is a peer to peer disconnected, replicated property database”. Sounds a lot like CouchDB for sure, but with a different focus. And thus, with a few cool ideas I really like to steal from them. Jesse said he’d be cool with that when I asked him afterwards and I am glad he is.
They focus a lot on integration with other data stores. They have a plugin architecture that allows them to write little modules that translate data schemas across systems. That would allow you, for example, run bug tracker A on the database of bug tracker B. This is pretty neat stuff and worth looking at.
OSCON is big. Everybody is there. And while I am more of a smaller conference kind of guy OSCON is pretty good given its size. I hope I can make it again next year.
Oh hey, and O’Reilly, if you are listening, food on the tutorial days was not, you know, that good.
Finally, thanks to Duncan and Chris who put me up for the week. It was nice seeing you and I hope to see you again soon! I had a great time in Portland.
Yes you can build REST tooling without endorsing WS-* it’s just not gonna be anything near as complicated as what you’ve done before. I’m looking at you, Java.
Two Ruby libraries – RestClient and HTTParty – would be good role-models for future REST abstraction layers. RestClient is just enough software so you don’t have to worry about HTTP details, but still gives you enough control to build a CouchDB client. HTTParty looks intriguing. It’s more like curl with XML and JSON parsing. The design makes it easy to manage HTTP auth and other protocol-level things, but still remain flexible around different URL path conventions.
These two tools are equally spartan but they differentiate on style. RESTClient feels like REST done without the headaches, while HTTParty is like open-uri, you use it when you just wanna get something done.
I don’t know what this lightness means for the big old languages. It will be interesting to see what they get when they try to ape our style. Will a Java poet come along with a good REST Client?
RubyFringe was easily one of the most keeping-it-real conferences I’ve ever attended. Thrown together by a tight-knit group of Toronto friends, it nailed nearly every aspect. Easily half the talks were as good as a Ruby talk can get, and at least two received standing ovations: Damien Katz’s story of his adventure making CouchDB, and Giles Bowkett’s inspirational lambda-calculus for music generation. Too many highlights to note them all, but look for (free!) video of Obie’s exhortation to “charge what you’re worth”, Zed jumping the shark, and Dan Grigsby admitting to poisoning p2p networks with ads for his DVD-trading site.
Most conferences have parties and after-parties, but they are usually too big to invite everyone to all the events. RubyFringe was small, as in less than 300 people (dates included), and they took the remarkable step of planning an interesting event (or two) every evening. Not once did I pay for a drink, and I had plenty of them. :) If you only know me from the conference, you should re-calibrate your impression of me from wild-and-crazy partier, to just normal crazy. Note to future conferences: if you can, organize the nightlife, seeing people outside the talks gave the “hallway conversations” a chance to happen without missing the awesome talks.
The planning heavy-lifting was done by Meghann Millard, who will now be widely regarded as a genius of the trade. Pete Forde curated the speakers list. He must be psychic – how else can you plan months ahead of time to have talks about cutting edge tech like Sinatra and CouchDB, which both really got recognized recently?
There is no word yet on whether there will be a RubyFringe II. Some of the Portland crew were deeply inspired. Maybe it will cause us to take FOSCON more seriously this year, or even to do a Fringe conference of our own.
The RarestNews developer considers InnoDB and CouchDB for a re-architection of his high volume news site. He did his homework researching, but I couldn’t help but comment on a few things he wrote. The comment turned into a blog post and since this is my blog it should be posted here as well.
I am specifically referring to the paragraphs about InnoDB and CouchDB:
MySQL problems
So, to be technical here I’ve used MyISAM tables (never really liked InnoDB because of it’s slow writes and at 100k new articles a day with lots of meta-data to write about them, like tags, dates, snippets, word frequencies, etc) - it seemed like a good decision. The bad part was that on write MyISAM locks the whole table. So 50 bots scouring the Web for news writing and locking whole table made site almost unresponsive.
I’m not yet sure how to solve it - with InnoDB, with PostgreSQL or with some kind of new-age databases like CouchDB, StrokeDB, maybe Amazon’s SimpleDB, etc…
CouchDB problems
They seem like a nice idea when you read about them, but… there are flaws.. The main problem with CouchDB for example is it’s complete HDD-dependence. Modern memory is hundreds of times faster than DB, so you’re using only 1% of speed if you use HDD-based database. And the second problem is it’s “Do not overwrite” motto. It doesn’t reuse space no longer needed, so if I write a 100KB article to database (along with some other data and then I rewrite this entry - there’s now 200KB stored on my drive) and each update eats 100KB more.
How to avoid it? Compact the database, so it creates a NEW file with only the latter 100KB. And delete the previous database file. So, even I didn’t change anything - I’ve had to write the same data 3 times (along with all of my database in compaction process). What that means.
1) It’s AT LEAST 3 time slower than your HDD speed if you want to effectively use ALL of your har