In Be My Eyes we use mongomapper and eventbus, both are very good tools that make my everyday life as a developer easier - except when it doesn't.
We had a problem where more than one helper is added to a call. This should not happen. We have a pretty simple if statement in our code. The thing is that many helpers were allowed through the if statement down to the "default" case, where only one should be allowed.
I really lost a lot of hair trying to figure out why more than one helper could fall through. It really made no sense, since we assigned a helper and assigned answered to be true. This should mean that the next time a helper tries to answer, she would be told that the call was already answered. But in many cases this did not happen.
I searched the codebase through and through for many hours with two things in mind that could be wrong:
1. The if statement did not work
2. The db returned something wrong.
No 1 is absurd so I did not follow that very far.
No 2 I could not make sense of until I did some research on stale objects and mongomapper.
I stumbled over and article on optimistic locking in mongodb and it hit me that somewhere in the codebase I must have and object with an old state that I somehow saved and by doing that, I overwrote the answered and helper attributes.
A quick Ag search found two places where I sent actual objects and not just Ids in the eventbus, after reading the aforementioned article I saw pretty clearly how bad this is.
For a quick fix I did a request.reload to get the new state before saving the new.
Long term it shall be a rule in the codebase that we do not send objects in the eventbus, but rather send ids. I already knew that and had it on my todo, but did not consider it important - well now I do ...