I'm going to sneak this update in under-the-wire for today.
I want to start by acknowledging that this part of the Engine work is frustrating. It's noodley and tedious, because we have to run the game over and over under certain conditions to find bottlenecks or crashes.
I know sometimes these updates seem endless. I'm sure it appears that I talk about the same stuff or the same goals (like speed optimization) all the time.
However, this is necessary to get Action Districts up to speed.
I feel very good about this week's progress.
We have started work to address the stalls. Over the last two updates, I talked about how much we are throwing at the Graphics card (GPU) and the problems that causes. This week we hand inspected logs for all the calls per frame, and found that a large percentage of them are executing the same command over and over. Each of these instructions is wasteful and costly if we don't need it. So we have started work on a State Caching system that saves values every time we pass them to the GPU across a wide number of different types of variables like Shaders, Samplers (textures), Parameters, and Buffers. Then when we go to run that command again, we check the value against the new value, and if its the same, we can skip that instruction. I am honestly shocked there wasn't a system like this in Unreal already (I am 99% that Unreal 4 has a system like this). Some early work was done on this system, but based on my math, we should see a significant speed improvement.
We also finally figured out the source of the D3D crashes. This was super hard to reproduce, but we finally found that deep in the multi-threaded rendering code it was saving copies of how Meshes attach to Shaders. But other threads can actually overwrite the Shader, which then means drawing the Mesh is now invalid (wrong Shader). This sort of code is not "thread safe". We did a temporary fix for the issue, but it involves locking all the threads when a mesh is drawn. That's incredibly bad, because it slows things down. But we now have a benchmark build that doesn't crash. Starting on Monday, we will be re-working all those bad places where values are stored, so we can remove the lock and make drawing Meshes thread safe. Since we have a benchmark, we will see the speed improvements as we go.
We added optimizations for 2 types of Shadow rendering. We are focusing on the Central Park area in Financial, which is one of the slowest areas in the game. There are a number of issues that make this area bad, and they are related to how we handle shadows from foliage. These two optimizations went in fairly smoothly, and they are the "pattern" for what we want to do throughout other parts of the code. Early tests indicated a 10% improved FPS. We have other areas that are very similar which we can now easily apply this new pattern to in order to gain more speed.