This is a bit of an early follow-up to my last post, in the middle of my big effort to split the Prone header into individual bracketed pieces. Overall, it's going well, it's just a bit of elephant-eating work, and I've learned a few things. ## Namespacing Last time, I speculated about an approach with `PRN_NS()` and `PRN_NSU()` macros to mark symbols, and a `gen_strip_header.sh` script to find the marks and generate ``. Well, I actually did that, and it worked okay, but not great. What I discovered pretty quickly is that these macros don't have a comfortable home in every place I might want to mark a symbol (particularly the names _of_ other macros), which led to putting markers in comments like a chump. ```c // PRN_NSU(DV_INLINE) #define PRN_DV_INLINE ... ``` Also, it just made the headers harder to read. I figured out pretty quickly that all the new-style headers should use full, qualified, unstripped `prn_whatever` names for everything. It's the `*.c` files that get to reap the savings later, but the headers really benefit from being the verbatim version of themselves that you'd want when importing in an unstripped context. Once I knew that, the solution was pretty obvious. Instead of marking names, just detect every `prn_x` and `PRN_Y` name within the new-style headers. They get the `sort -u` treatment anyways, so we don't have to worry about duplicates, and can explicitly filter out some names that don't make sense to "export." The script is doing detection with word boundaries turned on, so private stuff that starts with `_prn` doesn't get detected, so this is an even better solution than manual exceptions listed in the script. I still have one exception carved out, but I could (and will) get rid of it by privatizing the name of the `PRN_STRUCT` macro as used in ``. Here's the full `gen_strip_header.sh`: ```bash #!/bin/bash set -euo pipefail echo "// Generated by $0, do not edit manually." echo "#ifndef PRN_HEADER_STRIP" echo "#define PRN_HEADER_STRIP" # Auto-detect prn_* and PRN_* names from headers: echo grep -how "prn_[a-z0-9_]\+[a-z0-9]" "$@" |\ sort -u |\ sed -e 's/^prn_\(.*\)$/#define \1 prn_\1/' echo grep -how "PRN_[A-Z0-9_]\+[A-Z0-9]" "$@" |\ grep -v "PRN_STRUCT" |\ sort -u |\ sed -e 's/^PRN_\(.*\)$/#define \1 PRN_\1/' echo echo "#endif" ``` Ain't much to it! It's just a little dense in the regex parts, which is the eternal double-edged sword of regular expression syntax. At some point, when the existing headers are entirely eaten away into the new style, I can add a test that lists symbols in the `libprone.a` library, either with `nm` or `objdump`, and treats it as an error if there are any symbol names that don't start with `prn_` or `PRN_`. We love the ecosystem here. Say no to namespace pollution! ## Stricter "modules" A convention I've adopted, that's been a positive for code quality, is to have the first line of each new-style header declare the format(s) of names that the header provides. Here's some real examples, per `head -1 include/*`: ```text ==> include/alloc.h <== // prn_alloc_*: Customizable allocator logic. ==> include/conv.h <== // prn_conv_*, prn_to_*, prn_from_*: Data conversion routines. ==> include/dv.h <== // prn_dv_*: Dynamic Value manipulation functions. ==> include/rc.h <== // prn_rc_*: Reference counting. ==> include/sys.h <== // prn_sys_* tools. ==> include/types.h <== // prn_*_t and prn_*_variant: Types used in Prone. ==> include/xx.h <== // PRN_XX_*: XX macros for code generation. ``` With a few well-justified exceptions, you can expect that `prn_foo_bar` will be declared in ``, and with no exceptions, you know that a header will _only_ provide names in the formats that it promises. Ultimately, this will map to the `.c` source files too, that's just not as urgent to achieve. This has been good hygiene, but also produces some good incentives on structure. The `` header is a star example here! Right now, it just contains stuff for assert/abort logic, and it was briefly called ``. However, with automatic name stripping, certain existing names were bound to create conflicts, like `prn_assert` colliding with the regular old `assert` macro as soon as you turn stripping on. By adding a module name in here, `prn_sys_assert` strips down to `sys_assert`. Much better. What I realized shortly afterwards is that, while each platform will want to extend Prone Core in a variety of optional/divergent ways, there _are_ a short list of required platform-y things that Prone Core must depend on to do its job. And because these need to be implemented on even the dinkiest of devices, that list needs to stay short. Thus, `` is the interface header for that short list, and `src/sys.c` takes the bullet of using C preprocessor directives to selectively enable platform-specific code. The plan is, eventually you can pull in all the Prone headers with very few system headers recursively included. Probably just `` and ``. Everything else Prone Core needs is wrapped by ``. So the full dependency layering ends up being: ```text Platform -> Core -> Sys -> Conf ``` And `` is a generated header with some global constants. Right now, by default, it's literally just this. ```c // This file is generated by gen_conf_header.sh. // Don't edit it manually. #ifndef PRN_HEADER_CONF #define PRN_HEADER_CONF #define PRN_CONF_PLATFORM linux #define PRN_CONF_TAGGING lo #endif ``` ## From A to B The modules incentives don't stop there. A decent amount of the code that will need to exist in Prone Core is actually just data conversions from one type to another. For example, turning a `u8` into a string, or back again. And of course, wrapping and unwrapping data from being in a DV. I'm consolidating that logic into a conversion module, headered by ``, which does the actual work in `prn_conv_$from_$to(...)` functions, but which exposes more convenient macros to use those functions. A lot of the existing logic had function names like `u8_from_htext` because in C, the left side of a function name is closer to its return type, and the right side is closer to its arguments. This also lines up nicely with usage. ```c uint8_t num = u8_from_htext(HTEXT("3")); ``` Of course, it's annoying to have this word order as an English speaker, who finds it more intuitive for time to flow from left to right. `after_from_before` is clunky in some places, but `before_to_after` is clunky in others. Just C things, amirite? This is where having macros helps, allowing the same functionality to be exposed both ways. ```c // Both of these do what the previous example did uint8_t num = to_u8(htext, HTEXT("3")); // Inside-to-outside intuitive uint8_t num = from_htext(HTEXT("3"), u8); // Left-to-right intuitive ``` These end up macro-evaluating to the same function call, so the same code. It also starts to make sense to turn all the `dv_foo` and `dv_to_foo` functions into conversions, which is a work in progress at the time of this writing. But here's where I get to something a bit wilder. There's plenty of code that needs to chain one conversion after another. Even with the new macros, you end up saying the intermediate types over and over. But what if there was a `prn_conv` (or just `conv` after stripping) macro that let you chain conversions concisely? Behold, real code from `src/std.c`! ```c IState *prn_std_u8(IState *state, VVec *arguments) { if (arguments->vec.len != 1) return prn_std_dispatch_err(state, arguments); DV arg = vvec_pop(rc_move(arguments)); switch (dv_typecode(arg)) { case DV_TC_U8: return istate_set_value(state, arg); case DV_TC_ATOM: return istate_set_value(state, conv(arg, dv, ltext, u8, dv)); // here case DV_TC_LITNUM: return istate_set_value(state, conv(arg, dv, ltext, u8, dv)); // and here case DV_TC_HTEXT: return istate_set_value(state, conv(arg, dv, htext, u8, dv)); // and here case DV_TC_CONSTRUCT: return istate_set_value(state, conv(arg, dv, legc, u8, dv)); // and here default: return prn_std_dispatch_err(state, NULL); } } ``` Yeah, that `conv(arg, dv, ltext, u8, dv)` chain gets turned into nested conversion function calls by the C preprocessor, and while there's an upper limit to how many steps you can have, I could put in a bit of effort to make that limit pretty high, and the number of steps is flexible below that maximum. So, depending on your experience with C macros, you're going to have one of the following reactions: 1. It's all wizardry to me, so I don't know why that would be challenging. 2. Whoah! How'd you do that?!! 3. I am already a C wizard and know exactly how you did that. In case you're behind door #2, let me show you, because it's kinda neat! Behind the scenes, there are actually multiple different macros for different numbers of arguments. After the first couple, each next one is implemented in terms of the last one, so I can make this bigger later if I feel like it. ```c #define _prn_conv_chain_1(v) v #define _prn_conv_chain_2(v, t1) v #define _prn_conv_chain_3(v, t1, t2) prn_conv_##t1##_##t2(v) #define _prn_conv_chain_4(v, t1, t2, t3) \ prn_conv_##t2##_##t3(_prn_conv_chain_3(v, t1, t2)) #define _prn_conv_chain_5(v, t1, t2, t3, t4) \ prn_conv_##t3##_##t4(_prn_conv_chain_4(v, t1, t2, t3)) // etc... ``` So as long as we can turn `prn_conv(...)` into the _correct_ `_prn_conv_chain_N(...)` macro, everything will work. But we have to pick the right helper macro, and that's where the fun comes in. Imagine we have a list, which starts with the arguments to `prn_conv`, but has some stuff after it - these are going to be the names of our helper macros, so I'll indicate everything with `a` for args and `h` for helpers. ```text [a1, a2, a3, h6, h5, h4, h3, h2, h1, h0] ``` Further, imagine we grabbed the seventh item on the list. ```text [a1, a2, a3, h6, h5, h4, h3, h2, h1, h0] ^^ ``` That picks a helper, and _which_ helper it picks depends on the number of arguments. ```text No arguments: [h6, h5, h4, h3, h2, h1, h0] ^^ One argument: [a1, h6, h5, h4, h3, h2, h1, h0] ^^ Two arguments: [a1, a2, h6, h5, h4, h3, h2, h1, h0] ^^ ``` The more arguments you put in, the more it shifts over the helpers, pointing the dial at the correct helper for that number of arguments. This is literally how the real code works[^1]. We pick a helper based on this technique, and then immediately use it. ```c #define _prn_get_7th_arg(a1, a2, a3, a4, a5, a6, a7, ...) a7 #define _prn_conv_chain_chooser(...) \ _prn_get_7th_arg( \ __VA_ARGS__, _prn_conv_chain_6, _prn_conv_chain_5, _prn_conv_chain_4, \ _prn_conv_chain_3, _prn_conv_chain_2, _prn_conv_chain_1 \ ) #define prn_conv(...) _prn_conv_chain_chooser(__VA_ARGS__)(__VA_ARGS__) // ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ // pick ^^^^^^^^^^^^^ // use ``` The more helpers you have (and add to the list), the further you need to move the dial accordingly. I'll probably need to make `prn_conv` support more arguments later, because with an argument for the original value and an argument for its starting type, we only have five arguments left to ask for actual conversions, and five ain't much. What's cool about this automated, highly regular approach is that as I add new individual conversion functions, they're immediately usable with the macros, creating a kind of network effect where each minor new low-level function makes `prn_conv` exponentially more useful. ## Taking that lesson further It's kind of nice having precise, verbose underlying function names getting routed to via concise macros. This is probably going to keep being a best-of-both-worlds situation, especially as the module-y headers push me back in a `verb_noun` naming direction for more features. Consider: ```c eq(to_dv(u8, 7), to_dv(u8, 2)); // Are these two DV-wrapped bytes equal? ``` So we have a `prn_eq(...)` macro that, when given two arguments, assumes they're both DVs, and calls a function with a name like `prn_eq_dv_dv(x, y)`. But what if we explicitly gave it types? ```c // Three arguments: (t, x, y) eq(dv, to_dv(u8, 7), to_dv(u8, 2)); // Or even four args: (tx, ty, x, y) eq(dv, dv, to_dv(u8, 7), to_dv(u8, 2)); ``` Obviously that level of verbosity is silly for this common example, but sometimes you need to compare other stuff, like `size_t`s. You'll notice that this bears some resemblance to some features that, until now, were just part of the test assertion framework: ```c // Some random line stolen from tests/istate.c EQ(htext, dv_repr(istate_get_value(state)), rc_move(exp)); ``` So having a 2-4 argument `prn_eq` macro would make a more powerful version of the same technology a part of _Prone Core,_ and therefore usable throughout the language internals, not just the test suite. Whoah! It should even make the test assertion system simpler, because it can just blindly pass stuff through to the `prn_eq` and `prn_lt` etc. macros, at least for the most part.[^2] So not only is that a motivation to make a more complete set of comparison tools across a more complete set of types, we can also use the same style for other tech like `repr`. ```c repr(7, u8); repr(some_dv); ``` We'll see what oatmeal sticks to the wall, as always, but this does look like a promising direction for the code. It also emphasizes why namespacing and stripping is so important - names like `eq` are pretty likely to collide with user code _unless_ that code is really built around the assumption "I am a Prone program and I just do Prone things." On the other hand, for code inside the language implementation (or otherwise very Prone-specific), it's extremely convenient to have these macros for dispatch. [^1]: And as credited in the actual source, I learned how to do this from [Stack Overflow](https://stackoverflow.com/a/3048361). Now you know how to do it too! [^2]: Tests also need to print out values if an assertion fails, which means they need to hold a copy of the values when doing a comparison, and then either release the values if things are okay, or print them (which moves and releases) in a failure. So there's a little wrapper logic that needs to be thought through.