理解自动求导
- 例子
def f(x):
a = x * x
b = x * a
c = a + b
return c
-
基于图理解
-
代码实现
def df(x):
# forward pass
a = x * x
b = x * a
c = a + b
# backward pass
g_c = 1
# c = a + b
g_a = g_c * 1
g_b = g_c * 1
# b = x * a
g_x = g_b * a
g_a += g_b * x
# a = x * x
g_x += g_a * 2x
return g_x
- 进一步, 如果前后两节点都是向量, 相应的导数改为Jacobi矩阵, 导数改为梯度.
其他资料
- 参考 https://ai.googleblog.com/2017/11/tangent-source-to-source-debuggable.html
- google http://learningsys.org/nips17/assets/papers/paper_28.pdf
- 自动求导的定理证明 https://page.mi.fu-berlin.de/rojas/neural/chapter/K7.pdf