How we are using D to develop Aspect from the Ground up

Since we started the company, we've had to work with a very limited development workforce, which meant that we had to strongly prioritize efficiency for all of our tasks and goals. The D programming language plays a pivotal role here, with its exceptionally high expressiveness and low-barrier features, such as the built-in unit tests and its generally lean syntax.

The Origins

My personal history of using D goes back quite a bit longer than Aspect itself. At the time, being frustrated with C++'s convoluted syntax and slow compile times, I started to follow the development of D with a lot of interest, relatively soon after its first public appearance in 2001. However, due to its lack of a const system, I couldn't really convince myself to use it seriously until D 2.0 finally came out in 2007.

The first real project was porting my C++ 3D game engine to D, over the course of a few months, counting in around 150 thousand lines of code. It was a highly rewarding experience seeing the source code shrink together due to the shorter syntax, but also by requiring a lot less boilerplate code and repetition. The engine itself has always just been an elaborate hobby project, though, and I never really released it to the public.

It wasn't until December 2011 that the idea for the first open-source project was born — an event-based I/O framework that made use of D's built-in fiber implementation, which it inherited from the alternative D 1.0 runtime “Tango”. After a few experiments of marrying fibers with libevent and writing up basic network protocol and concurrency code, we finally released the first version of vibe.d in March 2012.

Since then, lots of libraries around and on top of vibe.d have followed, together with many libraries from other users of vibe.d, leading to a quite rich ecosystem around vibe.d that is available on code.dlang.org today. Speaking of which, D's package manager, DUB, was also born out of a small vibe.d side project called "VPM" (vibe package manager) that was just meant to provide a quick way to use vibe.d extension modules.

Laying the Foundations

Equipped with a comprehensive set of base libraries — vibe.d for event-based I/O and concurrency management, a UI library that was part of the game engine, D's growing standard library and the libraries available on code.dlang.org — the choice of language was relatively clear when we started to work on Aspect. Although ensuring robust cross-platform support has been, and still is, a considerable challenge, the level of control and the efficiency of implementing necessary changes in this ecosystem would not have been possible with anything else.

Vibe.d in particular allowed us to go with a very interesting approach for handling GUI and I/O, as well as computationally inexpensive concurrency, all seamlessly together within a single thread — unless it had to be moved to worker threads for performance reasons. This had a considerable impact on ease of implementation, performance and concurrency robustness, compared to the more conventional approaches that we were used to from previous projects.

As a bonus, although it still hasn't really been embraced by the D community, the existence of shared (and immutable) has allowed us to keep a statically checked barrier between thread-local and shared data. The result is that race-conditions are a very rare kind of issue in our code base, despite its highly asynchronous and concurrent nature.

D's unique Combination of Features

While a lot of advantages for us are based on the fact that we have direct access to large parts of the code base of the libraries that we use, D has a number of features that, as a whole, provide a fundamental gain in development efficiency. While there are many details in the language that each make a difference, these are overall probably the most important ones:

• Lean syntax: As trivial as it may seem, the many small simplifications compared to C++ have been a game changer for readability, as well as just reducing the time and energy needed to convert thoughts into code. Examples of this include the simplification of :: and -> accessors to a universal . notation, foreach, auto type inference, but also using this instead of repeating the class name for constructors and implementation files.

• Module system: Similar to the syntax itself, getting rid of the considerable repetition caused by separating code into header and implementation files can sometimes lower the activation energy needed to implement a certain feature considerably. If you are anything like me, the mere thought of having to type up a header file, including mandatory copy constructors, virtual destructors and header guards, followed by copy-pasting the code, adding ClassName:: all over the place, just to start writing the first line of actual code, can sometimes be a hurdle that really hurts to overcome.

• Templates, CTFE and ‘static if’: To be fair, C++ has caught up a lot here over the years, but the syntax and semantics are still fundamentally nicer and more intuitive than the questionable trade-offs that C++ had to make. Irrespective of that, the existence of a clean template syntax, variadic template arguments, and static if, combined with the possibility to execute any suitable function at compile time (CTFE), was a complete game changer for template-based meta-programming. Something that could have taken days and many years of prior expertise to accomplish in C++ — probably supported by liberal use of pre-processor hacks — could often be done in mere minutes in D.

• (String) mixins: All of my time writing C and C++, I've always hated complex build processes — they were often brittle, required extra software to be installed, and usually had to be re-implemented on each platform due to incompatible tooling and shell syntax. A primary necessity for complicating the build process was the frequent need to use code generation to implement otherwise simple meta-programming tasks, a popular example of this being the use of lexer/parser generators, such as YACC or Bison. Thanks to string mixins and CTFE in D, this can instead simply be done within the source code, at compile time. Many of the modules I've written make extensive use of this, for example the diet-ng template engine or the REST interface generator of vibe.d. Together with templates, it allows to implement robust zero-cost abstractions that you can often only dream of in most other languages.

• Standard package and build system: Last, but not least, this is another one of those points that can easily be overlooked, as it may seem marginal. However, the pain of maintaining a set of C or C++ dependencies across multiple platforms, managing library upgrades and fixing incompatible compiler and linker flags by hand can be a huge PITA. The simple fact that DUB exists as a standard package manager and manages most of the compiler/linker flags automatically is invaluable in terms of time savings. Today, virtually all D libraries are a simple matter of plug and play, because they are available as DUB packages.

Risks and Roadblocks

Of course, not all was rosy along the way. In the early days, it was the norm to have some kind of breakage for each and every compiler update, caused by language or standard library changes, or by compiler regressions. Fortunately, that has changed substantially since then, to a big part thanks to the “project tester”, a process that automatically tests a suite of community projects, including vibe.d, for regressions before integrating any language or standard library change.

Apart from stability, one of the greatest risks for us has always been mobile platform support. The D community has traditionally always lent more towards command line and server applications, while graphical user interfaces and non-desktop platforms haven't really played much of a role. Despite that, there have been attempts to support Android and iOS for quite a while, but only more recently has that support become stable enough to serve as the basis for basic production use. Objective-C interoperability is still an issue, particularly for LDC, but fortunately this can be worked around.

A more recent worry has emerged due to the way historic decisions and current developments influence the appearance of modern D source code. In particular, the language defaults that have emerged for attributes, such as @safe, nothrow or pure, now often lead to “good” code being littered with attributes, losing some of the simplicity that has always been a core strength of the language.

More recently, the transition to implement new scope semantics that guarantee memory safety for non-GC backed data at a level roughly comparable to Rust, has amplified this problem further. Now many places that were attribute-free before require additional explicit and often rather obscure annotations.

But since some of the more serious cases appear to be fixable, and implementing a transition to more sensible attribute defaults is certainly possible, there is hope that the language will eventually find its way back to the really clean appearance that it once had.

Conclusion and Outlook

Overall, going with D has worked out really well for us, and in all likeliness better than any of the available alternatives. An important part was of course the existing code base that we could use and advance, but also the exceptional efficiency in expressing algorithms and complex code architecture. This is worth even more at times when the architecture needs to change: less code means less code to change and less opportunity to introduce bugs. That, together with the quite far-reaching type system and liberal use of the built-in unit testing, enables robust long-term growth of the code base in a natural and healthy way.

Our next critical milestones will be proper support for Metal-based rendering and an iOS app, as well as an Android app that includes a full user interface, created with the same UI framework as used on the desktop platforms. We also hope to get to the point where we can release our UI framework as open-source, but that still requires quite a bit of extra work.

As a final word, I should probably mention that this article touches on a few topics that really deserve a more in-depth discussion. Since this would go beyond the scope of a single blog post, I plan to follow up, as time permits, with a small series of more technical articles.

Until then, if you have an interest in programming or photography, be sure to check out Aspect, as well as D — both are free and both may change the way you think within their respective domains!