Databases suck. Universally.

Okay, I can already hear the people with “but actually:tm: my database is good actually” coming for me with the pitchforks and torches, so lemme elaborate.

If you know me at all, you know that I’ve been working on my own container scheduler. In the process of this, I’ve been needing a good, consistent way to store the state of the system as a whole. This basically boils my options down to:

• Use a database, and hate every minute of it
• Write my own, and hate every minute of it

Having said that, what do I actually do?

A brief summary of databases

They all suck.

Seriously though, I just don’t seem to be able to find one that fits my needs. In a database or other data store for mahou, I want:

1. Key-value storage. Basically just get/set/del/key-scan, the last one in that list being used for things like “fetch all existing deployments.” I’m open to a better...

In the previous post, I discussed making crush into a more-viable data store for mahou. Since then, I’ve implemented forking and joining of key-value pairs in crush, and this post will be discussing the implementation thereof.

Fork-join of keys is conceptually fairly simple. A key stores its current fork, and the list of its ancestors, forks that were merged into it. To make ancestor tracking easier, the revision that said key was at during the join (or merge) is stored as part of the ancestor data:

typedstruct module: Ancestor do
  field :fork, String.t()
  field :rev, non_neg_integer()
end

Note: Example code is using typed_struct.

Once we can store the ancestors in the value’s state, we have everything that we need to make fork-join work!

At this point, our value state struct looks like this:

typedstruct module: Item do
  field :value, binary()
  field :patches, [any()]
  field

Infrastructure engineering is something of a passion of mine. However, I strongly dislike how golang has taken over the world of infra. To work around this in my own way, I’ve been writing my own infrastructure stack – mahou (魔法). mahou is, of course, built on top of the “amyware stack” – namely, singyeong (신경) and crush. This is the start to a series of blog posts discussing actually building this out.

Current state

Currently, mahou is the bare-minimum – you can deploy a container, but there’s no deployment state saved anywhere, nothing “smart” to go with it.

Building out a data store

Wait, what? Why build out a new data store?

1. I want to :D
2. crush already has some desirable properties:
   1. key-value store
   2. distributed / some level of fault-tolerance
   3. ability to store previous values of a key and provide them all on request.

It turns out that the last point there is the most...

Anyone who knows me knows that message buses are something of a passion of mine. 신경 is effectively feature-complete at this point, as no other types of messaging make sense to use therewith, so I’ve been turning my attention to the rest of the message bus / queue / log ecosystem and starting to think about ways it can be improved.

This is, of course, a topic that’s near and dear to my heart, so this is probably gonna get long. Don’t say I didn’t warn you!

Now let’s discuss some common messaging patterns!

Event bus

First and foremost is our dear friend, the event bus. Event buses are an incredibly common pattern when it comes to writing applications. If you’ve ever heard of an event loop, it’s more/less the same thing in practice ^{time for someone to tell me how I’m wrong.}.

Using an event bus is as simple as:

public class MyConsumer implements Consumer {
    @Override
    public

For anyone who knows me at all, this is something I’ve tinkered with at great length in my pet project around message queuing / pubsub / HTTP proxying, singyeong. Lately, I’ve been thinking about the interface to it, and wondering if I’ve been approaching this from the wrong direction.

As a quick recap, singyeong is a message queue + pubsub + HTTP proxy that allows routing payloads via client metadata. To show what this means in an example, suppose you were running some large multiplayer game server. You’re most likely splitting up players across a bunch of servers, and you need to be able to send a message to the server holding a specific player, for any number of reasons. You might do this with:

• pubsub + discarding if the player isn’t on that server, but then you have to pay the price of a fanout to every game server instance, which could add up at scale.
• consistent hashing, which...

I built a NAS recently. It’s a nice little box: I rackmounted an HPE ProLiant DL325 G10 (EPYC 7402, 64GB, 2x 240GB SSD) with a ES212X12 JBOD to hold 12x 8TB disks. Installing TrueNAS was pretty painless, convincing the ProLiant to be quiet and not sound like a jet engine was a bit annoying but not too bad, and overall it was a decent experience.

And then NFS happened.

I don’t think I’ve hated a piece of tech more on first use, honestly.

NFS works except for all the parts where it doesn’t. Share mounted too long? Stale file handles! You hit ^C in the middle of copying a file? Stale file handles! Trying to rsync from local to the NFS share? Stale file handles!

SMB was less painful than this.

This is more of a dumb hack than anything, I just happened to learn about this today.

Suppose that you’re writing a shell-command executing library. You can, of course, always use System.cmd/3 to do so. However, what if you wanted to make a cool interface like MyModule.echo "test" or MyModule.program_name :banana? Well, it turns out you can just do this:

defmodule Test do
  def unquote(:"$handle_undefined_function")(function_name, function_args) do
     Process things here!
  end
end

and suddenly you have magic “dynamic function definitions”!

->  iex
Erlang/OTP 23 [erts-11.1.2] [source] [64-bit] [smp:16:16] [ds:16:16:10] [async-threads:1] [hipe]

Interactive Elixir (1.11.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> defmodule Test do
...(1)>   def unquote(:"$handle_undefined_function")(function_name, function_args) do
...(1)>     IO.inspect function_name
...(1)>

For the last few months, I’ve been slowly tinkering away at writing my own custom X11 (via xcb) window manager, dawn (not open-source yet, sorry!). Because I hate the thought of dealing with named pipes and the like for things such as “resize this window” “move that window over there” etc etc etc, I decided to instead just add scripting to dawn so that I can do all this stuff via scripting. I’ve not yet gotten as far as designing a scripting API for this, and deciding what all I’d expose to the scripting language, since the specifics of the language will, to an extent, affect how I do this.

I’ve not actually tested out all of these; this is more of an overview of my first impressions and how I feel about the languages that are actually implemented. Those first impressions matter, y'know? And I don’t want to use something that gives me a nasty taste in my mouth on first look.

...

Kubernetes is the “hot new thing” that people like to claim is the “future of deployment” or the “future of infrastructure” or any other future-y fancy sounding statement (see ex. this and this). And you know, I get it, I really do. Kube takes care of so many things – rolling upgrades, management of stateful deployments, automatically spreading load across a cluster of servers and rebalancing, … – and so it makes sense that people are attracted to it.

But at least in my experiences with it, I consistently run into inexplicable issues that just completely ruins the utility of it. And I get it, all tools – especially deployment tools – have pains of some sort when it comes to using them. But the issues I’ve experienced with Kube are just… beyond ridiculous. Volumes disappearing, DNS failing in inexplicable ways, the cluster straight-up breaking to the point of needing a full...