nodes not properly joined give back different docs

[varac]

root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tickets","tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tickets","tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tickets","tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4096/_all_dbs'
["tickets","tokens","users"]

This happens because the two nodes aren't synchronized/replicated properly.

i can query the two nodes individually:

root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4000/_all_dbs'
["tickets","tokens","users"]
root@pc-web1:~# curl --netrc-file  /root/couchdb.netrc -X GET 'http://127.0.0.1:4001/_all_dbs'
["tokens","users"]

both nodes have registered the other one, but they seem to fail to replicate:

root@pc-couch1:~# curl --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'
{"total_rows":2,"offset":0,"rows":[
{"id":"bigcouch@pc-couch1.testing.bitmask.net","key":"bigcouch@pc-couch1.testing.bitmask.net","value":{"rev":"1-967a00dff5e02add41819138abb3284d"}},
{"id":"bigcouch@pc-couch2.testing.bitmask.net","key":"bigcouch@pc-couch2.testing.bitmask.net","value":{"rev":"1-967a00dff5e02add41819138abb3284d"}}
]}
root@pc-couch1:~# curl --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_membership'
{"all_nodes":["bigcouch@pc-couch1.testing.bitmask.net"],"cluster_nodes":["bigcouch@pc-couch1.testing.bitmask.net","bigcouch@pc-couch2.testing.bitmask.net"]}

root@pc-couch2:~# curl --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5986/nodes/_all_docs'
{"total_rows":2,"offset":0,"rows":[
{"id":"bigcouch@pc-couch1.testing.bitmask.net","key":"bigcouch@pc-couch1.testing.bitmask.net","value":{"rev":"1-967a00dff5e02add41819138abb3284d"}},
{"id":"bigcouch@pc-couch2.testing.bitmask.net","key":"bigcouch@pc-couch2.testing.bitmask.net","value":{"rev":"1-967a00dff5e02add41819138abb3284d"}}
]}
root@pc-couch2:~# curl --netrc-file /etc/couchdb/couchdb.netrc -X GET 'http://127.0.0.1:5984/_membership'
{"all_nodes":["bigcouch@pc-couch2.testing.bitmask.net"],"cluster_nodes":["bigcouch@pc-couch1.testing.bitmask.net","bigcouch@pc-couch2.testing.bitmask.net"]}

in the log, i see lots of those errors:

[Wed, 03 Jul 2013 15:29:07 GMT] [error] [emulator] [--------] Error in process <0.17855.403> on node 'bigcouch@pc-couch1.testing.bitmask.net' with exit value: {{rexi_DOWN,noconnect},[{mem3_rep,rexi_call,2},{mem3_rep,replicate_batch,1},{mem3_rep,go,3},{mem3_rep,go,2}]}

on the other node:

[Wed, 03 Jul 2013 15:28:34 GMT] [error] [emulator] [--------] Error in process <0.7357.400> on node 'bigcouch@pc-couch2.testing.bitmask.net' with exit value: {{rexi_DOWN,noconnect},[{mem3_rep,rexi_call,2},{mem3_rep,replicate_batch,1},{mem3_rep,go,3},{mem3_rep,go,2}]}

so i have two problems here:

1. the nodes don't replicate their data / are not joined properly together
2. how can i ensure that the web application that speaks to haproxy only gets data if the cluster is properly replicated ?

greetings, varac

[rnewson]

Do both nodes have the same cookie? Is there a firewall between them?

[varac]

there was a fw blocking communication, now they are properly joined.