Preface
前文
Recently, a friend reported that my open-source high-performance C# binary serialization library Nino would crash on 32-bit platforms (such as armv7 android application built by Unity). I then tracked down and fixed the issue in this commit.
As of the publication of this article, this issue exists in many well-known serialization libraries, such as MemoryPack and MessagePack. In this article, I will share the cause of the issue and the solution.
最近，有朋友反馈了我开源的高性能C#二进制序列化库 Nino 会在32位平台（比如Unity编译出来的armv7安卓应用）出现闪退，随后我对该问题进行了定位和修复。
截止到本文发布，这个问题在很多知名序列化库中都存在，例如MemoryPack和MessagePack，本文分享该问题的原因和修复方法。
Situation
问题
The optimal way to load a struct from a contiguous memory is usually reinterpreting the pointer directly:
通常，从一段连续的内存中加载一个结构体的最优方式是直接进行指针转换：
struct MyStruct
{
int a;
float b;
double c;
};
byte* data = ...;
MyStruct* myStruct = (MyStruct*)data;This is because such implementation the CPU can load the struct directly from memory without any additional copying. However, this method has a problem: it does not work correctly on 32-bit platforms. This is because the memory alignment of your custom struct may not match the 32-bit platform’s memory alignment requirements, and since such platform does not support unaligned memory access, it will cause a crash.
这是因为这种做法可以让CPU直接从内存中加载结构体，而不需要进行额外的复制。但是，这种方法有一个问题：它在32位平台上无法正常工作。这是因为你的自定义结构体的内存对齐方式可能与32位平台的内存对齐要求不匹配，而由于该平台不支持未对齐的内存访问，因此会导致闪退。
In C#, no matter if you use Unsafe.ReadUnaligned, or Unsafe.Read, or even BinaryPrimitives.Read*, the underlying implementation is still directly reinterpret the pointer. Therefore, the same issue exists.
Note that same issue applies to the
Write-related APIs.
在C#中，无论你使用Unsafe.ReadUnaligned，还是Unsafe.Read，甚至是BinaryPrimitives.Read*，其底层实现都是直接进行指针转换。因此，同样的问题也存在于这些方法中。
需要注意的是，这个问题同样存在于
Write相关的API中。
Solution
解决方法
We first need to define a function to check if we are on a 64-bit platform:
Note that this is a code snippet from Nino, you may need to adjust it to fit your codebase.
我们首先需要定义一个函数来检查是否在64位平台上：
需要注意的是，这段代码来自Nino，你可能需要对其进行调整以适应你的代码库。
public static class TypeCollector
{
public static bool Is64Bit => IntPtr.Size == 8;
}Read Unmanaged Struct
读取非托管结构体
To solve the addressed issue in C# under 32-bit platforms is in fact quite trivial, we simply need to introduce a copy.
在C#下解决32位平台下的这个问题其实非常简单，我们只需要引入一个复制即可。
public void Read<T>(ReadOnlySpan<byte> data, out T value) where T : unmanaged
{
if (TypeCollector.Is64Bit)
{
value = Unsafe.ReadUnaligned<T>(ref MemoryMarshal.GetReference(data));
}
else
{
value = default;
Span<byte> dst = MemoryMarshal.AsBytes(MemoryMarshal.CreateSpan(ref value, 1));
data.Slice(0, dst.Length).CopyTo(dst);
}
}In the code above, we first check if we are on a 64-bit platform. If so, we can directly use the original implementation. Otherwise, we allocate the output struct first then copies the data to the struct. This way, we can avoid the unaligned memory access issue.
MemoryMarshal.CreateSpan(ref value, 1) is a safe way to create a span that points to the memory of a struct in C# that preserves the original struct’s memory layout (and hence the alignment), MemoryMarshal.AsBytes is then used to convert the struct span to a byte span (which is equivalent to reinterpreting the pointer but with proper alignment handling). So we can safely copy the data to the struct.
Note that this code under 64-bit platform introduces branching but the branch predictor can usually make up for the performance loss. However, under 32-bit platform, this is approach may double the overhead compared to the original implementation.
在上面的代码中，我们首先检查是否在64位平台上。如果是，则可以直接使用原始实现。否则，我们先分配输出结构体，然后将数据复制到结构体中。这样，我们就可以避免未对齐的内存访问问题。
MemoryMarshal.CreateSpan(ref value, 1) 是一种在C#中安全地创建一个指向结构体内存的span的方法，它保留了原始结构体的内存布局（因此也保留了对齐方式），MemoryMarshal.AsBytes 则用于将结构体span转换为字节span（这相当于重新解释指针，但可以正确处理对齐）。因此我们可以安全地将数据复制到结构体中。
需要注意的是，在64位平台下，这种做法会引入分支，但是分支预测器通常可以弥补性能损失。但是，在32位平台下，这种做法可能会比原始实现增加一倍的耗时。
Write Unmanaged Struct
写入非托管结构体
We can apply similar modification to the Write-related APIs.
我们也可以对Write相关的API进行类似的修改。
public void Write<T>(ArrayBufferWriter<byte> bufferWriter, T value) where T : unmanaged
{
int size = Unsafe.SizeOf<T>();
if (TypeCollector.Is64Bit)
{
Unsafe.WriteUnaligned(ref MemoryMarshal.GetReference(_bufferWriter.GetSpan(size)), value);
}
else
{
ReadOnlySpan<byte> src = MemoryMarshal.AsBytes(
MemoryMarshal.CreateReadOnlySpan(ref Unsafe.AsRef(
#if NET8_0_OR_GREATER
ref value
#else
value
#endif
), 1));
src.CopyTo(_bufferWriter.GetSpan(size));
}
_bufferWriter.Advance(size);
}In the code above, we first check if we are on a 64-bit platform. If so, we can directly use the original implementation. Otherwise, we convert the struct to a byte span first then copies the data to the buffer. This way, we can avoid the unaligned memory access issue.
MemoryMarshal.CreateReadOnlySpan is used to create a read-only span that points to the memory of a struct in C# that preserves the original struct’s memory layout (and hence the alignment), Unsafe.AsRef is used to create a reference to the struct (since CreateReadOnlySpan requires a reference, and its API changed in .NET 8.0, that’s why we need the #if directive). Doing this way, we can safely copy the data to the buffer.
在上面的代码中，我们首先检查是否在64位平台上。如果是，则可以直接使用原始实现。否则，我们先将结构体转换为字节span，然后将数据复制到缓冲区中。这样，我们就可以避免未对齐的内存访问问题。
MemoryMarshal.CreateReadOnlySpan 用于创建一个指向结构体内存的只读span，它保留了原始结构体的内存布局（因此也保留了对齐方式），Unsafe.AsRef 用于将结构体转换为引用（因为CreateReadOnlySpan需要一个引用，同时由于.NET 8.0中该API的参数发生了变化，因此我们需要使用#if指令进行兼容）。通过这种方式，我们就可以安全地将数据复制到缓冲区中。
Read String
读取字符串
Reading a string from a byte array is a bit more complicated. This is because C# uses UTF16 encoding for strings, so casting the byte array directly to a char array is not safe due to alignment issues.
从字节数组中读取字符串会稍微复杂一些。这是因为C#使用UTF16编码字符串，因此直接将字节数组转换为字符数组是不安全的，因为可能会存在对齐问题。
public void ReadString(ReadOnlySpan<byte> utf16Bytes, out string value)
{
#if NET5_0_OR_GREATER
if (TypeCollector.Is64Bit)
{
ret = new string(MemoryMarshal.Cast<byte, char>(utf16Bytes));
}
else
{
unsafe
{
ret = string.Create(length, (IntPtr)Unsafe.AsPointer(
ref MemoryMarshal.GetReference(utf16Bytes)),
(dst, ptr) =>
{
Buffer.MemoryCopy(ptr.ToPointer(),
Unsafe.AsPointer(ref MemoryMarshal.GetReference(
MemoryMarshal.AsBytes(dst))),
length * sizeof(char), length * sizeof(char));
});
}
}
#else
if (TypeCollector.Is64Bit)
{
ret = MemoryMarshal.Cast<byte, char>(utf16Bytes).ToString();
}
else if (length <= 1024)
{
Span<char> tmp = stackalloc char[length];
Span<byte> dst = MemoryMarshal.AsBytes(tmp);
utf16Bytes.CopyTo(dst);
ret = new string(tmp);
}
else
{
char[] tmp = ArrayPool<char>.Shared.Rent(length);
Span<char> tmpSpan = tmp.AsSpan(0, length);
Span<byte> dst = MemoryMarshal.AsBytes(tmpSpan);
utf16Bytes.CopyTo(dst);
ret = new string(tmpSpan);
ArrayPool<char>.Shared.Return(tmp);
}
#endif
}In the code above, we first check if we are on a 64-bit platform. If so, we can directly use the original implementation. Otherwise, we need to handle the conversion carefully to avoid unaligned memory access issues.
string.Create is used to create a string from a span of characters, the second parameter is a state that will be passed to the delegate, and the delegate is used to copy the data from the source to the destination. In this case, we use Buffer.MemoryCopy to perform the copy, which is a safe way to copy data between buffers. However, this API only exists in .NET 5.0 and later, so we need to use the #if directive to handle the compatibility. For older versions, we can use a similar approach by allocating a temporary buffer and copying the data to it, then create the string from the buffer (we prefer to use stack allocation if the length is small enough, otherwise we use ArrayPool to rent a buffer from the pool).
在上面的代码中，我们首先检查是否在64位平台上。如果是，则可以直接使用原始实现。否则，我们需要仔细处理转换过程以避免未对齐的内存访问问题。
string.Create 用于从字符span创建字符串，第二个参数是一个状态，它会被传递给委托，委托用于将数据从源复制到目标。在这里，我们使用Buffer.MemoryCopy来执行复制，这是一种在缓冲区之间安全复制数据的方式。但是，该API仅在.NET 5.0及更高版本中存在，因此我们需要使用#if指令来处理兼容性。对于较旧的版本，我们可以使用类似的方法，通过分配一个临时缓冲区并将数据复制到其中，然后从缓冲区创建字符串（如果长度足够小，我们倾向于使用栈分配，否则我们使用ArrayPool从池中租用一个缓冲区）。
Write String
写入字符串
Writing a string to a byte array is relatively easy. We simply get the ReadOnlySpan<char> from the string and directly copy to the destination memory. No changes is needed since there is no unaligned memory access involved.
将字符串写入字节数组相对容易。我们只需从字符串中获取ReadOnlySpan<char>，然后直接将其复制到目标内存即可。由于没有涉及未对齐的内存访问，所以不需要进行任何修改。
Performance
性能
The performance overhead of the proposed solution is relatively small. In most cases (e.g. writes), the overhead is negligible. In some cases (e.g. reads), the overhead is slightly larger, but it is still acceptable (e.g. from 5ns to 10ns, but still relatively small).
在大多数情况下（比如写入），性能开销可以忽略不计。在某些情况下（比如读取），性能开销略大，但仍然可以接受（比如从5ns到10ns，但仍然相对较小）。
Conclusion
结论
In this post, we have addressed the issue of crash when converting binary data to structures in C# on 32-bit platforms. The solution is quite simple: just add a copy when necessary. However, this may introduce some performance overhead, so you should only apply this fix when necessary. And I hope this post can help you avoid such issues in the future.
在本文中，我们解决了在32位平台上将二进制数据转换为结构体时导致闪退的问题。解决方案非常简单：只需要在必要时添加一个复制操作即可。但是，这可能会引入一些性能开销，因此你应该只在必要时应用此修复。我希望本文可以帮助你在将来避免此类问题。