Unreal Engine 5.6 · Engine Customization · Practice Record

Practice: Adding Toon & MyCustom1 Shading Models实践：添加 Toon 与 MyCustom1 着色模型

UE 5.6.1 source · 8 files modified · 2 new shading model slots

Goal & Constraints目标与约束

This page records the process of adding two new shading model slots to a forked UE 5.6.1 source build:

本页记录在 fork 的 UE 5.6.1 源码中添加两个新着色模型槽的过程：

The fundamental hardware constraint is that the shading model ID is packed into the lower 4 bits of the GBufferB.a channel (shared with SelectiveOutputMask in the upper 4 bits). This gives a hard maximum of 16 unique IDs (0–15).

根本的硬件约束是：着色模型 ID 被打包到 GBufferB.a 通道的低 4 位（与高 4 位的 SelectiveOutputMask 共享）。这给出了 16 个唯一 ID（0–15）的硬性上限。

Before修改前 After修改后
MSM_NUM = 13 (slots 0–12 used)MSM_NUM = 13（已用 0–12）MSM_NUM = 15 (slots 0–14 used)MSM_NUM = 15（已用 0–14）
SHADINGMODELID_NUM = 14SHADINGMODELID_NUM = 14SHADINGMODELID_NUM = 15SHADINGMODELID_NUM = 15
2 free GBuffer slots (13, 14)2 个空闲 GBuffer 槽（13、14）1 free GBuffer slot (15)1 个空闲 GBuffer 槽（15）
ShadingModelMask : 14 bitsShadingModelMask : 14 位ShadingModelMask : 15 bitsShadingModelMask : 15 位
Only 1 slot left. After this change, only GBuffer slot 15 remains. Adding another shading model beyond that requires redesigning the GBuffer packing — expanding the ShadingModelID from 4 bits to 5 bits, which is a much larger engine modification affecting many more shader files. 仅剩 1 个槽位。此次修改后，只剩下 GBuffer 槽 15。在此之后再添加着色模型需要重新设计 GBuffer 打包方式——将 ShadingModelID 从 4 位扩展为 5 位，这是一个影响更多着色器文件的更大引擎修改。

Slot Assignment槽位分配

C++ Enum Value值 SHADINGMODELID_* (shader) Debug Color调试颜色 BxDF (legacy path)BxDF（旧版路径）
MSM_Toon13SHADINGMODELID_SUBSTRATE_TOON (reused)Orange橙色ToonBxDF_Legacy
MSM_MyCustom114SHADINGMODELID_MYCUSTOM1 (new)Purple紫色DefaultLitBxDF (placeholder)

SHADINGMODELID_SUBSTRATE_TOON (13) was already defined in ShadingCommon.ush for the Substrate pipeline's toon variant. By assigning MSM_Toon = 13, the same GBuffer slot is reused, and both paths (Substrate and legacy) write the same ID. The existing Substrate ToonBxDF handles the Substrate path; the new ToonBxDF_Legacy handles the non-Substrate path.

SHADINGMODELID_SUBSTRATE_TOON（13）在 ShadingCommon.ush 中已为 Substrate 管线的 Toon 变体定义。通过将 MSM_Toon = 13，相同的 GBuffer 槽被复用，两条路径（Substrate 和旧版）写入相同 ID。现有的 Substrate ToonBxDF 处理 Substrate 路径；新的 ToonBxDF_Legacy 处理非 Substrate 路径。

Files Modified — Summary修改文件汇总

Detailed Code Changes代码修改详情

1EngineTypes.h — C++ Enum

// Engine/Source/Runtime/Engine/Classes/Engine/EngineTypes.h
  MSM_Strata          UMETA(DisplayName="Substrate", Hidden),
+ MSM_Toon            UMETA(DisplayName="Toon"),
+ MSM_MyCustom1       UMETA(DisplayName="My Custom 1"),
  MSM_NUM             UMETA(Hidden),   // was 13, now 15

2ShadingCommon.ush — Shader IDs & Debug Colors

// Engine/Shaders/Private/ShadingCommon.ush
  #define SHADINGMODELID_SUBSTRATE_TOON  13  // reused as Toon ID
+ #define SHADINGMODELID_MYCUSTOM1       14
  #define SHADINGMODELID_NUM             15  // was 14

// In GetShadingModelColor() — switch branch:
+ case SHADINGMODELID_SUBSTRATE_TOON: return float3(1.0f, 0.5f, 0.0f); // Orange
+ case SHADINGMODELID_MYCUSTOM1:      return float3(0.5f, 0.0f, 1.0f); // Purple

3ShadingModelsMaterial.ush — GBuffer Encoding

// Engine/Shaders/Private/ShadingModelsMaterial.ush
  #if MATERIAL_SHADINGMODEL_TOON
      else if (ShadingModel == SHADINGMODELID_SUBSTRATE_TOON)
      { /* GBuffer.CustomData is used */ }
  #endif
+ #if MATERIAL_SHADINGMODEL_MYCUSTOM1
+     else if (ShadingModel == SHADINGMODELID_MYCUSTOM1)
+     { /* No custom GBuffer data yet — extend here when needed */ }
+ #endif

4ShadingModels.ush — BxDF & Dispatch

A new ToonBxDF_Legacy() function is added outside the SUBSTRATE_ENABLED guard for the non-Substrate path. It implements a simple 3-step diffuse ramp and a hard specular blob (step(0.5, pow(NoH, specPow))).

在 SUBSTRATE_ENABLED 保护外添加了新的 ToonBxDF_Legacy() 函数，用于非 Substrate 路径。它实现了简单的 3 级漫反射渐变和硬高光（step(0.5, pow(NoH, specPow))）。

// Engine/Shaders/Private/ShadingModels.ush
// Added just before IntegrateBxDF(), outside #if SUBSTRATE_ENABLED:
+ FDirectLighting ToonBxDF_Legacy(FGBufferData GBuffer, half3 N, half3 V,
+                                  FAreaLight AreaLight, FShadowTerms Shadow)
+ {
+     BxDFContext Context;
+     Init(Context, N, V, AreaLight.SpecularL);
+     SphereMaxNoH(Context, AreaLight.SphereSinAlpha, true);
+     float NoL = max(0.0f, dot(N, AreaLight.DiffuseL));
+
+     // 3-step diffuse ramp
+     float Ramp = NoL < 0.3f ? 0.0f : (NoL < 0.6f ? 0.5f : 1.0f);
+
+     // Hard specular highlight
+     float SpecPow  = max(1.0f, (1.0f - GBuffer.Roughness) * 64.0f);
+     float HardSpec = step(0.5f, pow(saturate(Context.NoH), SpecPow));
+
+     FDirectLighting Lighting;
+     Lighting.Diffuse    = AreaLight.FalloffColor * GBuffer.DiffuseColor  * Ramp     * Shadow.SurfaceShadow;
+     Lighting.Specular   = AreaLight.FalloffColor * GBuffer.SpecularColor * HardSpec * Shadow.SurfaceShadow;
+     Lighting.Transmission = 0;
+     return Lighting;
+ }

// In IntegrateBxDF() dispatch switch:
  // was: #if SUBSTRATE_ENABLED / case SHADINGMODELID_SUBSTRATE_TOON / #endif
  // now:
+ #if SUBSTRATE_ENABLED
+     case SHADINGMODELID_SUBSTRATE_TOON:
+         return ToonBxDF(GBuffer, N, V, AreaLight, Shadow);
+ #else
+     case SHADINGMODELID_SUBSTRATE_TOON:  // MSM_Toon in the legacy pipeline
+         return ToonBxDF_Legacy(GBuffer, N, V, AreaLight, Shadow);
+ #endif
+     case SHADINGMODELID_MYCUSTOM1:
+         return DefaultLitBxDF(GBuffer, N, V, AreaLight, Shadow); // placeholder

5HLSLMaterialTranslator.cpp — Shader Defines

// Engine/Source/.../HLSLMaterialTranslator.cpp
  // was: only fires for Substrate Toon feature flag
  if (EnumHasAnyFlags(...SubstrateMaterialBsdfFeatures, ESubstrateBsdfFeature::Toon))
+ // now: also fires for the C++ MSM_Toon enum
+ if (EnvironmentDefines->HasShadingModel(MSM_Toon) ||
+     EnumHasAnyFlags(...SubstrateMaterialBsdfFeatures, ESubstrateBsdfFeature::Toon))
  {
      OutEnvironment.SetDefine(TEXT("MATERIAL_SHADINGMODEL_TOON"), TEXT("1"));
  }
+ if (EnvironmentDefines->HasShadingModel(MSM_MyCustom1))
+ {
+     OutEnvironment.SetDefine(TEXT("MATERIAL_SHADINGMODEL_MYCUSTOM1"), TEXT("1"));
+ }

6MaterialIRToHLSLTranslator.cpp — IR Translator

// Engine/Source/.../MaterialIRToHLSLTranslator.cpp — GetShadingModelParameterName()
  case MSM_ThinTranslucent: return TEXT("MATERIAL_SHADINGMODEL_THIN_TRANSLUCENT");
+ case MSM_Toon:            return TEXT("MATERIAL_SHADINGMODEL_TOON");
+ case MSM_MyCustom1:       return TEXT("MATERIAL_SHADINGMODEL_MYCUSTOM1");
  default: UE_MIR_UNREACHABLE();

7MaterialShader.cpp — Name Switch & Stats

// Name switch (used for permutation key generation):
  case MSM_ThinTranslucent: ShadingModelName = TEXT("MSM_ThinTranslucent"); break;
+ case MSM_Toon:            ShadingModelName = TEXT("MSM_Toon"); break;
+ case MSM_MyCustom1:       ShadingModelName = TEXT("MSM_MyCustom1"); break;

// Lit-shader stats counter — HasAnyShadingModel list:
  ShadingModels.HasAnyShadingModel({ MSM_DefaultLit, ..., MSM_ThinTranslucent, MSM_Toon, MSM_MyCustom1 })

8MaterialRelevance.h — Bitfield Expansion

// Engine/Source/Runtime/Engine/Public/Materials/MaterialRelevance.h
// First uint32 must stay at exactly 32 bits total:
  // 15 + 4 + 6 + 7 = 32 bits ✓
+ uint32 ShadingModelMask        : 15;  // was 14 — expanded to cover bit 14 (MSM_MyCustom1)
  uint32 SubstrateTileTypeMask   :  4;
  uint32 SubstrateUintPerPixel   :  6;
+ uint32 SubstrateClosureCountMask:  7;  // was 8 — reduced by 1 to balance the uint32

Rebuilding the Engine重新构建引擎

Prerequisites (one-time)前置条件（仅一次）

Step 1 — Download Dependencies步骤 1 — 下载依赖

Run once after cloning. Downloads ~30 GB of binary dependencies not stored in git.

clone 后运行一次。下载约 30 GB 的 git 中未存储的二进制依赖。

# Run as Administrator in the repo root
Setup.bat

Step 2 — Generate Project Files步骤 2 — 生成项目文件

Creates UE5.sln. Re-run any time new .cpp / .h files are added.

创建 UE5.sln。每次添加新 .cpp / .h 文件时重新运行。

GenerateProjectFiles.bat

Step 3 — Build步骤 3 — 构建

Open UE5.sln in Visual Studio 2022, set configuration to Development Editor / Win64, right-click UE5 → Build. Or use the command line for faster incremental builds:

在 Visual Studio 2022 中打开 UE5.sln，将配置设为 Development Editor / Win64，右键 UE5 → 生成。或使用命令行进行更快的增量构建：

# Incremental build from command line (recommended after source changes)
Engine\Build\BatchFiles\Build.bat UnrealEditor Win64 Development -WaitMutex -FromMsBuild
Build scope: Our changes only touch the Engine module (EngineTypes, Material translators, MaterialShader, MaterialRelevance). This is an incremental build — only the affected modules recompile. Expect 10–30 minutes rather than the multi-hour full build time. 构建范围：我们的修改只涉及 Engine 模块（EngineTypes、Material 翻译器、MaterialShader、MaterialRelevance）。这是一次增量构建——只有受影响的模块重新编译。预计 10–30 分钟，而不是数小时的完整构建时间。

Step 4 — Verify in Editor步骤 4 — 在编辑器中验证

Engine\Binaries\Win64\UnrealEditor.exe

Create a new Material asset. Click the Shading Model dropdown. You should see Toon and My Custom 1 in the list.

创建一个新的材质资产。点击 Shading Model 下拉菜单。你应该在列表中看到 Toon 和 My Custom 1。

Iterating on Shaders Without Rebuild无需重新构建即可迭代着色器

Once the C++ side is compiled once, all visual changes to the BxDF (ramp shape, specular style, etc.) live entirely in ShadingModels.ush. These are HLSL shader files — they are compiled at editor launch by the shader compiler, not by the C++ build system. You can hot-reload them in-editor:

一旦 C++ 侧编译完成，BxDF 的所有视觉修改（渐变形状、高光风格等）完全存在于 ShadingModels.ush 中。这些是 HLSL 着色器文件——它们在编辑器启动时由着色器编译器编译，而不是由 C++ 构建系统编译。你可以在编辑器中热重载它们：

# In the UE editor console (backtick ` to open):
recompileshaders changed

The workflow for visual iteration is therefore:

因此，视觉迭代的工作流程为：

Change Type修改类型 File文件 Action Required所需操作
Diffuse ramp, specular shape, rim light, etc.漫反射渐变、高光形状、边缘光等ShadingModels.ushrecompileshaders changed
GBuffer custom data layoutGBuffer 自定义数据布局ShadingModelsMaterial.ushrecompileshaders changed
Debug visualization color调试可视化颜色ShadingCommon.ushrecompileshaders changed
New material input pin (CustomData)新材质输入引脚（CustomData）MaterialAttributeDefinitionMap.cpp + .ushC++ rebuild required需要 C++ 重新构建
New shading model slot新着色模型槽All 8 files above上述全部 8 个文件C++ rebuild required需要 C++ 重新构建

Git WorkflowGit 工作流

The repository is forked from Epic's release branch. Committing custom engine changes directly on release mixes your work with Epic's upstream commits, making future upstream merges difficult. The recommended approach is to keep your changes on a dedicated branch:

该仓库是从 Epic 的 release 分支 fork 的。直接在 release 上提交自定义引擎修改会将你的工作与 Epic 的上游提交混合，使未来的上游合并变得困难。推荐的方法是将你的修改保留在专用分支上：

# Create a personal branch from the current state of release
git checkout -b custom/shading-models

# Stage and commit only the 8 modified files
git add Engine/Source/Runtime/Engine/Classes/Engine/EngineTypes.h
git add Engine/Shaders/Private/ShadingCommon.ush
git add Engine/Shaders/Private/ShadingModelsMaterial.ush
git add Engine/Shaders/Private/ShadingModels.ush
git add Engine/Source/Runtime/Engine/Private/Materials/HLSLMaterialTranslator.cpp
git add Engine/Source/Runtime/Engine/Private/Materials/MaterialIRToHLSLTranslator.cpp
git add Engine/Source/Runtime/Engine/Private/Materials/MaterialShader.cpp
git add Engine/Source/Runtime/Engine/Public/Materials/MaterialRelevance.h
git commit -m "Add MSM_Toon and MSM_MyCustom1 shading models"

When Epic releases a patch and you want to pull it:

当 Epic 发布补丁并且你想拉取时：

# Sync your release branch with upstream
git fetch upstream
git checkout release
git merge upstream/release

# Rebase your custom work on top of the updated release
git checkout custom/shading-models
git rebase release

Next Steps后续步骤

For MSM_Toon关于 MSM_Toon

For MSM_MyCustom1关于 MSM_MyCustom1

Unreal Engine 5.6.1 · Engine source modification practice · custom/shading-models branch