[Converge] Weight Initialiser

From: http://www.cnblogs.com/denny402/p/6932956.html

 

[784, 10] fully connected

w = tf.Variable(tf.truncated_normal([img_pixel_input, layersize], mean=0.0, stddev=1.0, dtype=tf.float32))
b = tf.Variable(tf.truncated_normal([layersize                 ], mean=0.0, stddev=1.0, dtype=tf.float32))

Epoch 0, Training Loss: 2.10244838786, Test accuracy: 0.514423076923, time: 1.95s, total time: 2.8s
Epoch 1, Training Loss: 1.86659669154, Test accuracy: 0.640424679487, time: 1.54s, total time: 5.12s
Epoch 2, Training Loss: 1.80024383674, Test accuracy: 0.680989583333, time: 1.49s, total time: 7.47s
Epoch 3, Training Loss: 1.77303568244, Test accuracy: 0.699318910256, time: 1.53s, total time: 9.63s
Epoch 4, Training Loss: 1.75938568276, Test accuracy: 0.712740384615, time: 1.4s, total time: 11.81s
Epoch 5, Training Loss: 1.74897368638, Test accuracy: 0.718449519231, time: 1.57s, total time: 14.27s
Epoch 6, Training Loss: 1.7434025914, Test accuracy: 0.722355769231, time: 1.37s, total time: 16.52s
Epoch 7, Training Loss: 1.71330288407, Test accuracy: 0.792668269231, time: 1.37s, total time: 18.75s
Epoch 8, Training Loss: 1.66116618999, Test accuracy: 0.850560897436, time: 1.51s, total time: 21.08s
Epoch 9, Training Loss: 1.600759656, Test accuracy: 0.88030849359, time: 1.46s, total time: 23.32s
Epoch 10, Training Loss: 1.58312522976, Test accuracy: 0.892327724359, time: 1.52s, total time: 25.63s
Epoch 11, Training Loss: 1.5736670608, Test accuracy: 0.896534455128, time: 1.54s, total time: 28.01s
Epoch 12, Training Loss: 1.56778478539, Test accuracy: 0.905749198718, time: 1.47s, total time: 30.37s
Epoch 13, Training Loss: 1.56342586715, Test accuracy: 0.905548878205, time: 1.52s, total time: 32.71s
Epoch 14, Training Loss: 1.55950221926, Test accuracy: 0.906049679487, time: 1.54s, total time: 35.06s
Epoch 15, Training Loss: 1.55725609423, Test accuracy: 0.910356570513, time: 1.49s, total time: 37.45s
Epoch 16, Training Loss: 1.55490833146, Test accuracy: 0.911959134615, time: 1.58s, total time: 39.89s
Epoch 17, Training Loss: 1.55294992346, Test accuracy: 0.913561698718, time: 1.56s, total time: 42.26s
Epoch 18, Training Loss: 1.55085181106, Test accuracy: 0.916967147436, time: 1.51s, total time: 44.63s
Epoch 19, Training Loss: 1.54926108397, Test accuracy: 0.911858974359, time: 1.52s, total time: 47.0s
Total training time: 47.0s


w = tf.Variable(tf.truncated_normal([img_pixel_input, layersize], mean=0.01, stddev=1.0, dtype=tf.float32))
b = tf.Variable(tf.truncated_normal([layersize                 ], mean=0.01, stddev=1.0, dtype=tf.float32))

Epoch 0, Training Loss: 2.1900485101, Test accuracy: 0.443008814103, time: 1.84s, total time: 2.75s
Epoch 1, Training Loss: 1.93756918807, Test accuracy: 0.599659455128, time: 1.48s, total time: 5.1s
Epoch 2, Training Loss: 1.84595911986, Test accuracy: 0.653145032051, time: 1.47s, total time: 7.35s
Epoch 3, Training Loss: 1.8073041603, Test accuracy: 0.682291666667, time: 1.49s, total time: 9.63s
Epoch 4, Training Loss: 1.78734811036, Test accuracy: 0.688601762821, time: 1.43s, total time: 11.86s
Epoch 5, Training Loss: 1.7739427098, Test accuracy: 0.700520833333, time: 1.43s, total time: 14.02s
Epoch 6, Training Loss: 1.76551306776, Test accuracy: 0.711738782051, time: 1.34s, total time: 16.08s
Epoch 7, Training Loss: 1.74105782025, Test accuracy: 0.794771634615, time: 1.47s, total time: 18.33s
Epoch 8, Training Loss: 1.67201814229, Test accuracy: 0.808894230769, time: 1.53s, total time: 20.68s
Epoch 9, Training Loss: 1.66241001194, Test accuracy: 0.811698717949, time: 1.49s, total time: 23.0s
Epoch 10, Training Loss: 1.65713534489, Test accuracy: 0.814202724359, time: 1.54s, total time: 25.35s
Epoch 11, Training Loss: 1.65359901187, Test accuracy: 0.820713141026, time: 1.58s, total time: 27.73s
Epoch 12, Training Loss: 1.6501801603, Test accuracy: 0.820012019231, time: 1.49s, total time: 30.08s
Epoch 13, Training Loss: 1.64807084891, Test accuracy: 0.821915064103, time: 1.5s, total time: 32.41s
Epoch 14, Training Loss: 1.64611155364, Test accuracy: 0.821314102564, time: 1.54s, total time: 34.79s
Epoch 15, Training Loss: 1.62634825317, Test accuracy: 0.899539262821, time: 1.51s, total time: 37.05s
Epoch 16, Training Loss: 1.56398414065, Test accuracy: 0.909755608974, time: 1.41s, total time: 39.26s
Epoch 17, Training Loss: 1.55725724714, Test accuracy: 0.912459935897, time: 1.51s, total time: 41.57s
Epoch 18, Training Loss: 1.55478919553, Test accuracy: 0.91796875, time: 1.55s, total time: 43.95s
Epoch 19, Training Loss: 1.55242318568, Test accuracy: 0.917367788462, time: 1.5s, total time: 46.25s
Total training time: 46.25s


w = tf.Variable(tf.truncated_normal([img_pixel_input, layersize], mean=0.01, stddev=5.0, dtype=tf.float32))
b = tf.Variable(tf.truncated_normal([layersize                 ], mean=0.01, stddev=1.0, dtype=tf.float32))

Epoch 0, Training Loss: 2.39008372369, Test accuracy: 0.0950520833333, time: 1.94s, total time: 2.65s
Epoch 1, Training Loss: 2.33227054167, Test accuracy: 0.153245192308, time: 1.54s, total time: 4.96s
Epoch 2, Training Loss: 2.28677356104, Test accuracy: 0.186498397436, time: 1.42s, total time: 7.25s
Epoch 3, Training Loss: 2.23217486891, Test accuracy: 0.269831730769, time: 1.38s, total time: 9.4s
Epoch 4, Training Loss: 2.13864973875, Test accuracy: 0.351061698718, time: 1.47s, total time: 11.65s
Epoch 5, Training Loss: 2.07637035874, Test accuracy: 0.401041666667, time: 1.58s, total time: 14.06s
Epoch 6, Training Loss: 2.04344919623, Test accuracy: 0.426582532051, time: 1.46s, total time: 16.29s
Epoch 7, Training Loss: 2.02300423842, Test accuracy: 0.44140625, time: 1.52s, total time: 18.58s
Epoch 8, Training Loss: 2.00804452852, Test accuracy: 0.455428685897, time: 1.45s, total time: 20.83s
Epoch 9, Training Loss: 1.99567352781, Test accuracy: 0.468549679487, time: 1.51s, total time: 23.19s
Epoch 10, Training Loss: 1.98683612969, Test accuracy: 0.476462339744, time: 1.59s, total time: 25.59s
Epoch 11, Training Loss: 1.980189987, Test accuracy: 0.485677083333, time: 1.57s, total time: 27.98s
Epoch 12, Training Loss: 1.97373542863, Test accuracy: 0.491185897436, time: 1.52s, total time: 30.29s
Epoch 13, Training Loss: 1.967556376, Test accuracy: 0.491887019231, time: 1.61s, total time: 32.7s
Epoch 14, Training Loss: 1.96045698958, Test accuracy: 0.497395833333, time: 1.49s, total time: 34.97s
Epoch 15, Training Loss: 1.95221617978, Test accuracy: 0.517528044872, time: 1.49s, total time: 37.18s
Epoch 16, Training Loss: 1.93845896371, Test accuracy: 0.521534455128, time: 1.46s, total time: 39.35s
Epoch 17, Training Loss: 1.92538965999, Test accuracy: 0.539963942308, time: 1.43s, total time: 41.5s
Epoch 18, Training Loss: 1.91551751801, Test accuracy: 0.546173878205, time: 1.43s, total time: 43.77s
Epoch 19, Training Loss: 1.90569505908, Test accuracy: 0.555989583333, time: 1.47s, total time: 46.05s
Total training time: 46.05s

[784, 10]