I hate databases.
Databases suck. Universally.
Okay, I can already hear the people with “but actually:tm: my database is good actually” coming for me with the pitchforks and torches, so lemme elaborate.
If you know me at all, you know that I’ve been working on my own container scheduler. In the process of this, I’ve been needing a good, consistent way to store the state of the system as a whole. This basically boils my options down to:
- Use a database, and hate every minute of it
- Write my own, and hate every minute of it
Having said that, what do I actually do?
A brief summary of databases #
They all suck.
Seriously though, I just don’t seem to be able to find one that fits my needs. In a database or other data store for mahou, I want:
- Key-value storage. Basically just get/set/del/key-scan, the last one in that list being used for things like “fetch all existing deployments.” I’m open to a better data model, but the point here is that my needs are really simple, relatively speaking.
- Distributed, out of the box. I want resilience, and I don’t want it to be misery to set up. Should also be persistent.
- Either have a really good Elixir client, or expose a sane HTTP API. I am not a fan of protobufs (future article about this when :eyes:), and I want to avoid them when I can.
With that in mind, let’s look at some candidates!
Postgres (or CockroachDB, or Yugabyte, or …) #
Actually these are pretty sane. They suck the least, let’s say. But realistically, I don’t really want to make end-users have to care about managing another database, making sure everything is in-sync when it comes to migrations, … It’s just an unnecessary layer of pain to add, in my opinion. Really this applies to all SQL DBs.
MongoDB #
Scaling in Mongo sucks. Source: I did it. Next please.
Redis #
Actually a serious, viable contender still! While Redis is usually seen as just in-memory, configuring it to use AOF + fsync-every-second (or better) should be more than enough to do what I want / need. I’ve been on the fence about Redis forever, and it’s on my list of things to consider biting the bullet and just using.
Etcd / Consul #
The most-supported Etcd v3 Elixir client I found is eetcd, and I end up with either not being able to get it to compile, or dealing with an Erlang-y API that’s very non-idiomatic to deal with in Elixir. Etcd’s API is being pushed to gRPC, with a no-good very-bad extremely-painful JSON gateway thereinto, and so basically no. I tried, I hated every moment of it, and I stopped taking it seriously when I was told that, to use the JSON gateway, I had to encode strings into base64’d byte strings (iirc? might be wrong, but b64 was absolutely involved!) in order for it to be handled right. At that point, why even HAVE a JSON gateway???
Consul actually has a sane-looking HTTP KV API, my main frustration with it is how much it does. I want the KV parts, not the everything-else that Consul tries to do. Still though, the sanity of the KV API alone lands it in the list of things I might use.
FoundationDB #
FDB is actually really cool! I love everything that they do, and it’s gotten me to start researching database theory more actively, the idea being that someday I might work on crush as a serious database. Sadly, there’s no good FDB client for Elixir; the only one I found is a wrapper around the C API and hasn’t been updated in nearly 2 years. FDB does have a MongoDB-compatible layer, but that’s not worth using, since FDB is not the easiest thing in the world to administer, as I understand it. Still tho, I really wanna find a use for it someday.
crush #
lol. crush is more of a meme than not; while mahou actually uses it right now, I’m doing this survey to try and move off of it.
<insert other database here> #
The fact that it’s not big and popular means it’s probably not worth considering, sadly.
So where does this leave us? #
Confused! There’s a few okay-if-you-squint options – mostly just Redis and Consul KV, honestly – but that’s not the same as it being the right choice to make. It’s a bit annoying that what I want doesn’t really exist, and it’s been driving me to make crush better. So who knows, maybe someday I’ll have written a full database on top of everything else! :tada: