Building an extendable asset pipeline

Building a game engine can be an eventful endeavour, but, many tools and libraries exist out there to kickstart your engine's core development.

• Need an ECS? Try EnTT or Flecs.
• For rendering you have raylib, sokol or even OGRE-Next.
• Need a level editor you have LdTK or Tiled for 2D and BSP Quake Editor or even Blender for 3D
• Physics? Box2D and Jolt.
• Audio? OpenAL, Miniaudio, FMod, SteamAudio
• Etc...

What I have seldom found is a propper asset pipeline. For a propper asset pipeline I mean one that gives me 4 things:

• An Asset Database and File System Watcher;
• An easy way to import assets, convert them an engine format and cache them;
• The ability to pack multiple assets and compress them;
• A resource manager capable of loading assets both synchronously and asynchronously;

When I got to building the asset system for my game engine I found this to be an area of my engine that could be packaged up into a library. Then I could make easy use of it across a large set of tools and games.

So I came up with Bifrost an easy to integrate asset pipeline bridging the gap between assets and our games, engines and tools.

The design considerations

Programming Language

Bifrost was a part of my C++ game engine so, as it would be expected, the first draft was written in C++ 20.

While the publicly available release is still missing the resource manager (at the time of writing this post), behind the scenes, I have started work on a rewrite to C99. The goal for this rewrite is to create an easier interface to bind to from other languages.

I am planning bindings for the following languages:

• C++ (Optional Object Oriented API)
• Odin
• Zig
• Rust

Supported Asset Formats

None.

File format support is added through import functions provided by the library user. Functions that take a file as an argument and return an imported file.

Bifrost will use the imported file to cache the result of the import.

Cache

Caching imported files is important for the performance characteristics of any asset pipeline.

Bifrost will cache assets at: [root_path]/.bifrost/[asset_uuid].ba

Packing Assets

Bifrost supports packing assets for distribution. Files with the '.ba' extension are formated as follows:

[version][index_size][index][packed_assets]

Version is a 32 bit unsigned integer for versioning and upgrading '.ba' files

Index is a contiguous array of File Descriptors containing the necessary information to locate any given file by UUID in the larger packed file and loading it in.

Index Size is a 64 bit unsigned integer used to read in the index.

Each file descriptor has information on the UUID, Pack File Location, Offset from start of pack file, Size and Compression Type of a given asset in the packed file.

File System Watcher

Doing this properly across multiple platforms can be a hard task.

The draft version of Bifrost used EFSW for this porpuse but the C99 version is currently working off of inotify on Linux only.

I'm only planning on supporting Linux through inotify and Windows through Input/output completion port. I may also add an OS-independent polling file system watcher following in the footsteps of the EFSW devs.