python中优化的规范化

Question

在优化过程中，对输入参数进行归一化处理以使它们处于同一数量级通常很有帮助，因此收敛性会更好。例如，如果我们想最小化 f(x)，而合理的近似值是 x0=[1e3, 1e-4]，将 x0[0] 和 x0[1] 归一化到大约相同的数量级可能会有所帮助（通常为 O(1)）。

我的问题是，我一直在使用 scipy.optimize，特别是 L-BFGS-B 算法。我想知道，我是否需要通过编写函数来手动对其进行规范化，或者算法已经为我做了？

谢谢！

Answer 1

我写了一个快速的小程序来测试你的问题。

总结一下：如果参数彼此相差几个数量级，那么算法会处理它（它成功收敛，不需要进行更多的函数评估）。

另一方面，当您开始超过 10000 倍时，算法开始崩溃并出现错误。

这是程序：

import scipy.optimize


def test_lbfgsb():
    def surface(x):
        return (x[0] - 3.0) ** 2 + (factor * x[1] - 4.0) ** 2

    factor = None
    for exponent in xrange(0, 9):
        magnitude = 10 ** exponent
        factors = [x * magnitude for x in [1, 3]]
        for factor in factors:
            optimization_result = scipy.optimize.minimize(surface, [0, 0], method='l-bfgs-b')
            desc = 'at factor %d' % (factor)
            if not optimization_result.success:
                print '%s FAILURE (%s)' % (desc, optimization_result.message)
            else:
                print '%s, found min at %s, after %d evaluations' % (
                    desc, optimization_result.x, optimization_result.nfev)


test_lbfgsb()

这是它的输出：

at factor 1, found min at [ 3.00000048  4.00000013], after 12 evaluations
at factor 3, found min at [ 2.99999958  1.33333351], after 36 evaluations
at factor 10, found min at [ 3.00000059  0.39999999], after 28 evaluations
at factor 30, found min at [ 2.99999994  0.13333333], after 36 evaluations
at factor 100, found min at [ 3.00000013  0.03999999], after 40 evaluations
at factor 300, found min at [ 3.          0.01333333], after 52 evaluations
at factor 1000, found min at [ 3.          0.00399999], after 64 evaluations
at factor 3000, found min at [  3.00000006e+00   1.33332833e-03], after 72 evaluations
at factor 10000, found min at [  3.00002680e+00   3.99998309e-04], after 92 evaluations
at factor 30000, found min at [  3.00000002e+00   1.33328333e-04], after 104 evaluations
at factor 100000 FAILURE (ABNORMAL_TERMINATION_IN_LNSRCH)
at factor 300000, found min at [  3.00013621e+00   1.33292531e-05], after 196 evaluations
at factor 1000000, found min at [  3.00000348e-12   3.99500004e-06], after 60 evaluations
at factor 3000000 FAILURE (ABNORMAL_TERMINATION_IN_LNSRCH)
at factor 10000000 FAILURE (ABNORMAL_TERMINATION_IN_LNSRCH)
at factor 30000000 FAILURE (ABNORMAL_TERMINATION_IN_LNSRCH)
at factor 100000000 FAILURE (ABNORMAL_TERMINATION_IN_LNSRCH)
at factor 300000000, found min at [  3.33333330e-17   8.33333350e-09], after 72 evaluations