TQ4HAM: Enter the Matrix (LLO Archive)
Created 2025-12-13, last modified 2025-12-13. Visibility: public
Part of my archive of Layover Linux Official posts on Tumblr.
2025-11-17
GIF by directingthemovies: Neo is reflected in Morpheus' shades, on the verge of deciding one of two pills to take from Morpheus' outstretched hands.
It's a nice little world, isn't it? There's an Atom type for atoms. A String type for strings. And a Construct type for lists that are given special privilege. Too bad none of it is real.
Sure, maybe you know Constructs aren't really stored as lists anymore. The world of fictions is usually more comfortable than reality. But when you thunk and dethunk a construct, it feels like a real list underneath, just like a steak tastes like a steak. And if you take the blue pill, you'll stay in that comfortable world, where life turns at a slow pace, and it's 1999 as long as you want it to be.
Or.
You could take the red pill, and understand what your CPU is really capable of.
GIF by ultrakillblast: Morpheus offers you a red pill. Or the blue pill. The supreme court is iffy on your right to choose, but Morpheus is squarely pro-choice.
Remember your training. I told you a week ago: the atoms optimization matters beyond what it originally seems. I'm going to show you what it unlocks.
The atom storage engine can hold pieces of text up to 255 characters, with special optimization for text 6 or less characters long. To the closed mind, this power is just how atoms work. But think about identifiers - the names of variables, the names of functions. The atom storage engine is how they could work.
Think about number literals, which are a text type until you convert them to a specific format (like u8). They're text. They're small, usually very small. The atom storage engine is how they could work.
After the changes I'm going to show you, the atom storage engine is how they really work.
GIF by emrose-gold: Neo saying "whoah."
First of all, you need to unlearn the idea that it's the "atom storage engine." If it's serving multiple uses equally, it deserves a name that fits what it is on its own, not the masters it serves. For that matter, I'd like to someday use the inline text tech on its own (without a lookup table) as a small string optimization for the string type.
This means the engine is really split into two layers: Inline Text, or IText, is just text to number conversion. Lookup Text, or LText, builds on top of IText and adds support for text up to 255 bytes. LText powers atoms, identifiers, and literal numbers.
This was a lot of work. I had to make two types of Identifiers coexist for awhile (Construct vs LText) until I was finally able to kill one off. Then I had to do the same thing again for number literals. Every step of the way was a fight against the previous assumptions of the codebase, but it was an important lesson, reader.
The interpreter is a stage, and there are actors "backstage" who might end up playing multiple characters "frontstage", but it's important to maintain that this all an illusion. The actors are not the characters, and it's confusing to name them the same. Not only that, but we have types backstage that act as a director or stagehand, but never an actor. A true cast list would make all of this clear.
This cast list is the Type Matrix.
// ============================================================================
// TYPES
// ============================================================================
//
// | storage | back | front | tc | thunky |
// | ------- | ------- | ------ | --------------- | ------ |
// | RC | IState | - | - | - | x
// | inline | - | - | IL/TERM | N | x
// | in/lut | LText | atom | IL/ATOM | N | x
// | in/lut | LText | ident | IL/IDENT | Y | x
// | in/lut | LText | num | IL/LITNUM | Y | x
// | inline | IText | str | IL/SMSTR | N | (defer to later)
// | inline | uint8_t | u8 | IL/U8 | N | x
// | lut | NativeFn| nfn | TAG/NFN | Y | x
// | RC | HText | str | TAG/HTEXT | N | x
// | RC | VVec | list | TAG/LIST | N | x
// | RC | VVec | thunk | TAG/THUNK | Y | x
// | RC | VVec | block | TAG/BLOCK | Y |
// | RC | KV | table | TAG/TABLE | N |
// | RC | ZDFN | fn | TAG/ZDFN | Y |
// | RC | MDFN | fn | TAG/MDFN | Y |
// | RC | Call | fncall | TAG/CALL | Y |
// | RC | Call | chain | TAG/CALL | Y |
// | RC | Assign | assign | TAG/RCVAL | Y |
// | RC | Suffix | suffix | TAG/RCVAL | Y |
The Interpreter State (IState) we've seen before - a type that exclusively exists behind the scenes. For technical completeness, there's also the TERM "type", which isn't really a type front or back of stage, it's just a sentinel value that a DV can represent.
Then we have our LText types: atom, ident, and litnum. Note that the names are shorter: the in-language standard library contains functions that make these types, and those functions need names, and those names are incentivized to be short so they fit in the IText encoding. It all loops around on itself in the end.
The Architect gestures dramatically as every TV behind him switches to show different variations of Neo. Time is a flat circle.
Then we have our only IText type, saved for later work: small strings. In order to not have any name confusion, the String type (showing up in the table a little later) is going to be named Heap Text (HText) behind the scenes. Within the Prone language, as a user, Strings will sometimes be IText and sometimes be HText.
We have the U8 type, which already has a distinct name for the actor behind the scenes, built into the C language: uint8_t. Eventually, there will be more numeric types - they're not hard to make, but during rapid development of other language features, it's nice to not have too many numeric types to migrate around.
We also have native C functions. These need a certain amount of metadata, and before, I solved that problem by wrapping them in heap-allocated objects. No more. The lookup table technology of LTexts was a success story, and made it clear that the metadata could live in a separate lookup table of its own. Now I pass these around as tagged pointers to the C function itself, saving an indirection on every native function call. This is the woman in the red dress. Do you see why yet? It's the subject of the next post, but you'll need time for your mind to... align to it.
The table goes on. VVecs are an overachiever in this play, acting three parts: Lists, as we're used to seeing them. Thunks, as we've seen more recently. And Blocks, which... used to be a kind of Construct.
In fact... every type of Construct has been ripped out into its own type. Are Constructs really a type? Or were they just a trait all along... the word we use for types that can be disassembled into Thunks and put back together?
GIF by immotion: Neo bending a spoon with psychokinetic telekinesis cybertronic mind powers. As ya do.
There is no Construct type.
The closest thing is the fact that regular function calls and immutable chains are so similar to each other, we can store them in one type, and tell them apart with a flag on that type. But really, these are so many little separate things that all have a hobby in common with each other, and this will even be reflected in-language. Prone, as a user experience, will no longer pretend that Constructs are lists. They're just objects that have a Thunk form.
Finally, it's going to be time to write a proper type for Tables soon. But tables are a character. The key-value store, or KV, is the actor.
This is a huge amount of work that goes beyond optimization. At the time of this writing, I've completed some of this table, and the test suite already runs in under half a millisecond for me. Technically, I can stop optimizing at this point. But that would leave this rearchitecture work in a half-baked, half-done state, and I want to finish it before switching back to feature work.
So join me next time, when I tell you about a part of this project that could easily sneak past you before putting a bullet through your skull: The DV Format Reloaded.
Keanu Reeves wishing the viewer "Happy Holidays."
