On working at 10gen

10gen is trying to hire a gazillion people, so I’m averaging two interviews a day (bleh). A lot of people have asked what it’s like to work on MongoDB, so I thought I’d write a bit about it.

A Usual Day

Coffee: the lynchpin of my day.

  • Get in around 10am.
  • Check if there are any commercial support questions that need to be answered right now.
  • Have a cup of coffee and code until lunch.
  • Eat lunch.
  • If nothing dire has happened, go out for coffee+writing. This refuels my brain and is a creative outlet: that’s where I am now. My coffee does not look nearly as awesome as the coffee on the right.
  • Go back to the office, code all afternoon.
  • Depending on the day, usually between 5:30 and 6:30 the programmers will naturally start discussing problems we had over the day, interviews, support, the latest geek news, etc. Often beers are broken out.
  • Wrap up, go home.

There are some variations on this: as I mentioned, a lot of time lately is taken up by interviewing. Other coworkers spend a lot more time than I do at consults, trainings, speaking at conferences, etc.

Other General Workday Stuff

On Fridays, we have lunch as a team. After lunch, we have a tech talk where someone presents on what they’re working on (e.g., the inspiration for my geospatial post) or general info that’s good to know (e.g., the inspiration for my virtual memory post). This is a nice way to end the week, especially since Fridays often wrap up earlier than other days.

A couple people use OS X or Windows for development, most people use Linux. You can use whatever you want. I’d like to encourage emacs users, in particular, to apply, as we’re falling slightly behind vi in numbers.

We sit in an open office plan, everyone at tables in a big room (including the CEO and CTO, who are both programmers). The only people in separate rooms are the people who have to be on the phone all day (sales, marketers, basketweavers… I’m not really clear on what non-technical people do).

And speaking of what people actually do, here are three examples of my job (that are more specific than “coding”):

Fixing Other People’s Bugs

Recently, a developer was using MongoDB and IBM’s DB2 with PHP. After he installed the MongoDB driver, PHP started segfaulting all over the place. I downloaded the ibm_db2 PHP extension to take a look.

PHP keeps a “storage unit” for extensions’ long-term memory use. Every extension shares the space and can store things there.

The DB2 extension was basically fire-bombing the storage unit.

It went through the storage, object by object, casting the objects into DB2 types and then freeing them. This worked fine when DB2 was the only PHP extension being used, but broke down when anyone else tried to use that storage. I gave the user a small patch that stopped the DB2 extension from destroying objects it didn’t create, and everything worked fine for them, after that.

The Game is Afoot

A user reported that they couldn’t initialize their replica set: a member wasn’t coming online. The trick with this type of bug is to get enough evidence before the user wants to beat you over the head with the 800th log you’ve requested.

I asked them to send the first round of logs. It was weird, nothing was wrong from server1‘s point of view: it initialized properly and could connect to everyone in the set. I puzzled over the messages, figuring out that once server1 had created the set, server2 had accepted the connection from server1 but then somehow failed to connect back to server1 and so couldn’t pick up the set config. However, according to server1, it could connect fine to server2 and thought it was perfectly healthy!

I finally realized what must be happening: “It looks like server2 couldn’t connect to any of the others, but all of them could connect to it. Could you check your firewall?”

“Oh, that server was blocking all outgoing connections! Now its working fine.”

Elementary, my dear Watson.

You know you’re not at a big company when…

At least it had "handles."

Someone on Sparc complained that the Perl driver wasn’t working at all for them. My first thought was that Sparc is big-endian, so maybe the Perl driver wasn’t flipping memory correctly. I asked Eliot where our Power PC was, and he said we must have forgotten it when we moved: it was still in our old office around the corner.

“Bring someone to help carry it,” he told me. “It’s heavy.”

Pshaw, I thought. How heavy could an old desktop be?

I went around the corner and the other company graciously let me walk into their server room, choose a server, and walk out with it. Unfortunately, it weighed about 50 pounds, and I have a traditional geek physique (no muscles). The trip back to our office involved me staggering a couple steps, putting it down, shaking out my arms, and repeat.

When I got to our office, I just dragged it down the hallway to our server closet. Eliot saw me tugboating the thing down the hallway.

“You didn’t bring someone to help?”

“It’s *oof* fine!”

Unfortunately, once it was all set up, the Perl driver worked perfectly on it. So it wasn’t big-endian specific.

I was now pretty sure it was Sparc-specific (another person had reported the same problem on a Sparc), so I bought an elderly Sparc server for a couple hundred bucks off eBay. When it arrived a couple days later, Eliot showed me how to rack it and I spent a day fighting with the Solaris/Oracle package manager. However, it was all worth it: I tried running the Perl driver and it instantly failed (success!).

After some debugging, I realized that Sparc was much more persnickety than Intel about byte alignment. The Perl driver was playing fast and loose with a byte buffer, casting pieces of it into other types (which Sparc didn’t like). I changed some casts to memcpys and the Perl driver started working beautifully.

But every day is different

The episodes above are a very small sample of what I do: there are hundreds of other things I’ve worked on over the last few years from speaking to working on the database to writing a freakin Facebook app.

So, if this sounded interesting, please go to our jobs website and submit an application!

  • http://www.justaprogrammer.net Justin Dearing

    Two comments:

    >  I’m averaging two interviews a day (bleh).You don’t enjoy interviews? There so much fun! I don’t get to interview people at my current job and I miss it so much.

    >  I’d like to encourage emacs users, in particular, to apply, as we’re falling slightly behind vi in numbers.

    Infidel!!! Seriously though, I’ve noticed the emacs users I know tend to have fallen into programming from Math and Physics. Is there some matlab plugin for emacs that all Math TAs use in whatever math lab class that turns math and physics geeks into programmers?

    And in all seriousness, I have nothing against emacs users, the real enemy are the viper people (http://www.emacswiki.org/emacs/ViperMode).

  • Anonymous

    Interviewing people isn’t torturous or anything, but it’s not as fun as coding.  It takes a lot of time to read over someone’s resume, figure out what questions you should ask, do the interview, and write up your feedback.  Doing two per day knocks out three hours from my schedule, easy.  And a bad interview can be so depressing and awful.

    Probably you see a lot from math and physics because Emacs users are smarter :)

  • Tad Marshall

    Alignment issues were the main gotcha in porting our 32-bit Windows code to the MIPS version of Windows NT back in the time when Microsoft was supporting those chips.  But the fix was pretty trivial … Microsoft had an __unaligned keyword that you could slap on a pointer declaration and the compiler would do the memcpy for you.  All you had to do was find the places where it was required.

    Did you patch the DB2 extension, or PHP or your driver?  Keeping another bit of software from misbehaving is noble work, though.

  • Anonymous

    Yeah, I had heard about alignment issues, but never dealt with them before.  Sounds like Windows makes it easy.  I don’t know what I would have done if I couldn’t have gotten a Sparc machine.  When I first ran it through GDB and it segfaulted going through perfectly valid memory I was so confused!

    I patched the DB2 extension (see http://groups.google.com/group/mongodb-user/msg/41174f1e64d4d6ad).

    Nice work on the MongoDB shell, btw!

  • frenchie

    You’re still pretty much in a start-up phase, it seems, raising funds from business angels, and such…

  • Jonas B.

    Which MongoDB client driver would you receives the most attentinon, and is most commonly used? Which language would I be least expected to encounter trouble with if I have some data processing to do?

  • Anonymous

    Java, PHP, Python, Ruby, and C# are probably the most used drivers.  However, all of the officially supported drivers are pretty solid.

  • http://twitter.com/comerford Adam Comerford

    As someone who has been invited over to New York to interview, this is a great insight into the day to day – maybe I’ll see you there in a couple of weeks :)

  • Anonymous

    Thanks, glad it’s helpful! 

  • Domenicabambolina

    Love the coffee. Where do they do this?

  • kristina1

    No idea, sorry.  Google Images.

kristina chodorow's blog