Hi everyone! Last semester, we wrote that one of our next steps was to make room for smaller projects. That is exactly what happened: this semester, we started three new sub-projects, each led by one member.
We still don't have a good answer to the question “what actually is a shader?” But perhaps our three projects show what it can cover better than any definition could: recreating a fluid simulation from data, a procedurally generated exploration game, and 3D reconstruction from photos. What they have in common is that they all make use of the GPU's ability to do a huge number of calculations in parallel.
Frederik – Posterior Flows
What if a simulation could catch a few glimpses of water in motion, and then work out for itself how viscous ("thick") the water is and what will happen next?
That is the idea behind what I'm building: a digital twin. A digital twin is a model that follows a real system and adjusts itself based on what it sees, a bit like a weather forecast that updates every time new measurements come in.
For now, I'm not working with a water tank full of sensors. I'm focusing on the simulation. One run of the simulation acts as "reality". I know its parameters, but I pretend I don't. Then the model tries to guess its way back to them. The guessing works roughly like this: pick a value, run the simulation, check how well it matches "reality", update what you know, and try again. This is called data assimilation, and it's used in things like GPS and weather forecasting. After enough rounds, the guesses close in on the values you're looking for.
So where do shaders come in? The short answer: there are many parameters to handle, which means a lot of calculations, and those calculations can run in parallel. That is exactly what the GPU is good at.
So far, I've built several of the key components I need for data assimilation. They don't work on a fluid system yet, but on a simpler system with similar behavior. Moving to fluids is the next big step for my project.
The image below shows what this looks like for a fluid simulation (taken from this article ). Data assimilation combines the model's prediction (here "CFD") with measured data (here "experiments") to get an answer that is closer to reality than either one on its own.
Vangen – Peregrination
Peregrination (wandering/journey) is meant to be a procedurally generated exploration game. It is mainly a by-product of learning OpenGL, and will be more of a proof of concept than a full game.
In the game, you explore a procedurally generated, post-apocalyptic fantasy world with different biomes, structures and NPCs. The terrain is generated using layered noise textures, and it decides where structures can be placed. In and around the structures, you can talk to NPCs and read signs. Each one reveals a small piece of lore, so you can work out for yourself what happened. The structures will be modeled in Blender and imported into the game. There are no plans for combat or an inventory.
Right now, the project is following the LearnOpenGL course, and will keep doing so until the course is finished. Because of that, not much work has gone into the game itself yet. Once the course is done, the top priority is a minimum viable product (MVP): a flat area you can walk around in, with simple lighting. After that, it will be expanded with procedurally generated terrain, structures and more.
Elias – From 2D Images to 3D with Gaussian Splatting
How can you rebuild a 3D scene from ordinary 2D photos? Researchers have worked on this problem for a long time, and in recent years one method has stood out: 3D Gaussian Splatting (3DGS).
3DGS can match the image quality of advanced neural methods like Neural Radiance Fields (NeRF), but it trains much faster. Once the model is trained, it can render the scene from new viewpoints in real time. This is called Novel View Synthesis (NVS), and it lets you explore a reconstructed scene virtually.
The goal of my project is to build a complete pipeline that goes from a set of photos to an interactive 3D scene, rendered with compute shaders on the GPU. First, I want to build a minimum viable product (MVP) using existing libraries. Then I'll gradually replace them with my own implementations.
The first step is Structure from Motion (SfM). It finds matching points across the photos, then uses them to estimate where each camera was and to calculate where each point sits in a shared 3D space. For now I'm using COLMAP for this. So far, I've reconstructed a point cloud of 803 points from 33 photos, with an average reprojection error of 0.54 pixels.
Next, the point cloud is used to set up the 3DGS model. Each point becomes a 3D Gaussian, a kind of soft, blurry blob, with properties like position, size, orientation, color and transparency. The model is trained with gradient-based optimization: the parameters are adjusted step by step until the images the model renders match the original photos as closely as possible.
Finally, I'll build my own renderer that projects the Gaussians onto a 2D image. Compute shaders let large parts of this work run in parallel on the GPU.
What's next
Once the pipeline works, I want to reconstruct the Hackerspace office and make an interactive model you can move around in. In the long run, it would be especially cool to put it directly on our website, if DevOps are willing to help.
Final words
Next semester we'll keep working, aiming to have a working version of all three projects before summer. That doesn't necessarily mean they'll be finished, but each of them should be something we can show off.
- The Shader Group