A Quick Intro to mongosniff

Writing an application on top of a framework on top of a driver on top of the database is a bit like playing telephone: you say “insert foo” and the database says “purple monkey dishwasher.” mongosniff lets you see exactly what the database is hearing and saying.

It comes with the binary distribution, so if you have mongod you should have mongosniff.

To try it out, first start up an instance of mongod normally:

$ ./mongod

When you start up mongosniff you have to tell it to listen on the loopback (localhost) interface. This interface is usually called “lo”, but my Mac calls it “lo0”, so run ifconfig to make sure you have the name right. Now run:

$ sudo ./mongosniff --source NET lo
sniffing... 27017

Note the “sudo”: this has never worked for me from my user account, probably because of some stupid network permissions thing.

Now start up the mongo shell and try inserting something:

> db.foo.insert({x:1})

If you look at mongosniff‘s output, you’ll see:

127.0.0.1:57856  -->> 127.0.0.1:27017 test.foo  62 bytes  id:430131ca   1124151754
        insert: { _id: ObjectId('4c7fb007b5d697849addc650'), x: 1.0 }
127.0.0.1:57856  -->> 127.0.0.1:27017 test.$cmd  76 bytes  id:430131cb  1124151755
        query: { getlasterror: 1.0, w: 1.0 }  ntoreturn: -1 ntoskip: 0
127.0.0.1:27017  <<--  127.0.0.1:57856   76 bytes  id:474447bf  1195657151 - 1124151755
        reply n:1 cursorId: 0
        { err: null, n: 0, wtime: 0, ok: 1.0 }

There are three requests here, all for one measly insert. Dissecting the first request, we can learn:

source -->> destination
Our client, mongo in this case, is running on port 57856 and has sent a message to the database (127.0.0.1:27017).

db.collection
This request is for the “test” database’s “foo” collection.
length bytes
The length of the request is 62 bytes. This can be handy to know if your requests are edging towards the maximum request length (16 MB).
id:hex-id id
This is the request id in hexadecimal and decimal (in case you don’t have a computer handy, apparently). Every request to and from the database has a unique id associated with it for tax purposes.
op: content
This is the actual meat of the request: we’re inserting this document. Notice that it’s inserting the float value 1.0, even though we typed 1 in the shell. This is because JavaScript only has one number type, so every number typed in the shell is converted to a double.

The next request in the mongosniff output is a database command: it checks to make sure the insert succeeded (the shell always does safe inserts).

The last message sniffed is a little different: it is going from the database to the shell. It is the database response to the getlasterror command. It shows that there was only one document returned (reply n:1) and that there are no more results waiting at the database (cursorId: 0). If this were a “real” query and there was another batch of results to be sent from the database, cursorId would be non-zero.

Hopefully this will help some people decipher what the hell is going on!

  • ::::The next request in the mongosniff output is a database command: it checks to make sure the insert succeeded (the shell always does safe inserts).::::

    Thats good to know, and contrary to my assumption in this question: http://stackoverflow.com/questions/3605427/safe-write-from-the-mongodb-shell

    If you can clarify my SO question, I’ll try to update the wiki this weekend.

  • Anonymous

    Yeah, that’s why the shell often seems slower to people than the drivers. Thilio is correct, since the getlasterror called automatically doesn’t fsync or have a “real” w value (w=1 is the same as vanilla safe mode). To flush to disk, you’ll need to call getlasterror again with fsync.

  • Sweet! Didn’t know about mongosniff until now. Thanks!

  • Love the Simpsons reference. 🙂

    oh, and nice article too.

  • Anonymous

    Yeah, there are a lot of little features and tools that have been implemented for some user long ago that no one else knows about. Glad to help!

  • Anonymous

    Thanks!

  • Does mongosniff not come with the MongoDB supported Ubuntu package? Only see the binaries below.

    It would be nice if it did?

    mongo
    mongod
    mongodump
    mongoexport
    mongofiles
    mongoimport
    mongorestore
    mongos
    mongostat

  • Pingback: MongoDBçš„tcpdump工具:mongosniff : NoSQLfan()

  • Anonymous

    Interesting, probably just an oversight because it’s new-ish. If you could file a request at http://jira.mongodb.org/browse/SERVER, I’m sure Richard would put it in.

  • Pardon me for jumping in at such a low level before I have even
    learned anything about the guts of MongoDB, but I’l risk it anyway.

    It seems that there is an opportunity for an optimization here. If
    the message-sending level “knows” that after the insert operation, its
    caller is going to send a “getlasterror”, the two could be combined
    into a single message.

    It would not be necessary to create a new message operation called
    “insert and then getlasterror”. You can just append the two messages.
    The overhead that you want to get rid of is not the overhead of
    parsing the header of each message, but the network overhead. By
    batching up the messages, not only can the messages both fit into the
    same packet, but, more important, the overhead of calling into the
    network code is amortized over the two operations.

    One way to do this would be to put a sort of “peephole optimizer” at
    the level of message sending. Sending the insert command could be
    delayed until there is a command on this connection whose results
    depend on the result of the insert. Then all pending messages could
    be sent all at once. However, this means the messaging level would
    have to know a lot about the semantics of the message, and keeping
    track of the dependencies could get extremely complicated and
    bug-prone, so I assume you’re not taking that approach.

    The other way would be for the level above the message level to tell
    the message level that it wants to both insert and get the error code.
    This would basically be a “synchronous” API to the message level.
    Then it would be easy to combine the two messages.

  • In the “Last Error Commands” section of the documentation
    that’s referenced, I saw this:

    If you’re writing data to MongoDB on multiple connections, then it can
    sometimes be important to call getlasterror on one connection to be
    certain that the data has been committed to the database. For
    instance, if you’re writing to connection #1 and want those writes to
    be reflected in reads from connection #2, you can assure this by
    calling getlasterror after writing to connection #1.

    It seems to me that this should say the last sentence should say “and
    before sending the read request on connection #2.” It’s very
    important to establish causality at the client side. Otherwise
    a read by connection #2 could see the data previous to the
    write on connection #1, despite the getlasterror.

  • Anonymous

    Yup, you’re absolutely correct, and they’re sent as a single message by all of the other language drivers. I don’t think it’s ever been worth it to optimize the shell because it isn’t generally used for high-performance stuff. The shell is mainly for administrative tasks, there are other tools for importing massive amounts of data.

    If anyone is interested in creating a patch to optimize this, though, I’m sure we wouldn’t mind!

  • Thiago

    Somebody knows about this error message ?
    error while loading shared libraries: libpcap.so.0.9: cannot open shared object file: No such file or directory

  • Anonymous

    Sounds like libpcap isn’t installed.

  • Pingback: PhSoftware Programming Blog - PSPB()

kristina chodorow's blog