用于 FP32 矩阵乘法的 INT8 量化

Question

我尝试在 FloatingPoint32bit 矩阵乘法之前应用 INT8bit 量化，然后将累积的 INT32bit 输出重新量化为 INT8bit。毕竟，我想在这个过程中的某个地方会有一些混淆。我觉得很难发现那些麻烦点。

数据流[仿射量化]：

input(fp32) -> quant(int8) ____\ matmul(int32) -> requant(int8) ->deq(fp32)
输入（fp32）->量化（int8）----/

My Pseudo Code
INPUT(FP32) :
 Embedded Words in Tensor (shape : [1, 4, 1024, 256]) A and B (B is the same as A)

输入 A(=B)：enter image description here

EXPECTING OUTPUT(FP32) : 
 Embedded Words in Tensor (shape : [1, 4, 1024, 1024]) AB(after matrix multiplication to itself)

do while(true):
    # convert A and B of FP32 into INT8
    A_zero_offset = torch.empty(A.shape)
    A_zero_offset = torch.zeros_like(A_zero_offset)    # offset to be zero **[Question1]**
    scale = 255 / (torch.max(A) - torch.min(B))    # 2^8 - 1 = 255
    A_quantized = np.round((A - A_zero_offset) * scale)

    # likewise
    B_quantized = A_quantized

    AB = A_quantized.matmul(B_quantized.transpose(-1, -2))
    # now accumulated datatype is INT32

    AB_offset = torch.empty(AB.shape)
    AB_offset = AB_offset.new_full(AB.shape, torch.min(AB)) # offset to be AB's min element **[Question 1]**
    scale_AB = 255 / (torch.max(AB) - torch.min(AB))    **[Question 2]** 
    AB_requantized = np.round((AB - AB_offset) * scale_AB)

    # dequantize AB(INT8 at the status quo) into FP32
    **[Question 3]**

[问题1]：将A的偏移量设置为零并且AB的设置为min（AB）有意义吗？

[问题 2] : 比例计算应该遵循什么操作，“max(AB) - min(AB)”或任何其他方法？

[问题 3] : 毕竟，在将结果反量化为 FP32 时，我必须遵循什么操作，尤其是缩放和偏移计算？

Answer 1

我认为这种方法是完全错误的，因为每个嵌入的词张量都有不同的最大值和最小值，所以这个错误会改变你的数据连续性。我假设您无论如何都知道您的信息松散，因为您无法以相同的张量形状将 fp32 squezee(map) 到 int8

import torch
import numpy as np

# create Pseudo tensor
a = torch.tensor([[0.654654, 1.654687, -0.5645365],
                  [5.687646, -5.662354, 0.6546646]], dtype=torch.float32)
print(a.dtype)
print(a)
# torch.float32
# tensor([[ 0.6547,  1.6547, -0.5645],
#         [ 5.6876, -5.6624,  0.6547]])

b = a.clone().int()
print(b)
# tensor([[ 0,  1,  0],
#         [ 5, -5,  0]], dtype=torch.int32)

# converting to int8 please note range is here -128 to + 128
c = a.clone().to(torch.int8)
print(c)
# tensor([[ 0,  1,  0],
#         [ 5, -5,  0]], dtype=torch.int8)

# converting to uint8 please note range is here 0 to 255
d = a.clone().byte()
print(d)
# tensor([[  0,   1,   0],
#         [  5, 251,   0]], dtype=torch.uint8)

你的方法（错误）

A, B = a

A_zero_offset = torch.empty(A.shape)
A_zero_offset = torch.zeros_like(A_zero_offset)  # offset to be zero **[Question1]**
scale = 255 / (torch.max(A) - torch.min(B))  # 2^8 - 1 = 255
A_quantized = np.round((A - A_zero_offset) * scale)

print(A_quantized.dtype)
print(A_quantized)

# torch.float32
# tensor([ 23.,  58., -20.])