【发布时间】：2020-07-03 01:14:09
【问题描述】：
我正在 colab Pro GPU 上运行卷积网络。我在运行时选择了 GPU，并且可以确认 GPU 可用。我正在运行与昨天晚上完全相同的网络，但每个时期大约需要 2 小时......昨晚每个时期大约需要 3 分钟......什么都没有改变。我感觉 colab 可能限制了我的 GPU 使用，但我不知道如何判断这是否是问题所在。 GPU速度是否会根据一天中的时间等而波动很大？这是我打印的一些诊断信息，有谁知道我可以如何更深入地调查这种缓慢行为的根本原因是什么？
我还尝试将 colab 中的加速器更改为“无”，并且我的网络与选择“GPU”的速度相同，这意味着由于某种原因我不再在 GPU 上进行训练，或者资源受到严重限制。我正在使用 TensorFlow 2.1。
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
if gpu_info.find('failed') >= 0:
print('Select the Runtime → "Change runtime type" menu to enable a GPU accelerator, ')
print('and then re-execute this cell.')
else:
print(gpu_info)
Sun Mar 22 11:33:14 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |
| N/A 40C P0 32W / 250W | 8747MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
def mem_report():
print("CPU RAM Free: " + humanize.naturalsize( psutil.virtual_memory().available ))
GPUs = GPUtil.getGPUs()
for i, gpu in enumerate(GPUs):
print('GPU {:d} ... Mem Free: {:.0f}MB / {:.0f}MB | Utilization {:3.0f}%'.format(i, gpu.memoryFree, gpu.memoryTotal, gpu.memoryUtil*100))
mem_report()
CPU RAM Free: 24.5 GB
GPU 0 ... Mem Free: 7533MB / 16280MB | Utilization 54%
加快速度仍然没有运气，这是我的代码，也许我忽略了一些东西......顺便说一句，这些图像来自一个古老的 Kaggle 比赛，数据可以在这里找到。训练图像保存在我的谷歌驱动器上。 https://www.kaggle.com/c/datasciencebowl
#loading images from kaggle api
#os.environ['KAGGLE_USERNAME'] = ""
#os.environ['KAGGLE_KEY'] = ""
#!kaggle competitions download -c datasciencebowl
#unpacking zip files
#zipfile.ZipFile('./sampleSubmission.csv.zip', 'r').extractall('./')
#zipfile.ZipFile('./test.zip', 'r').extractall('./')
#zipfile.ZipFile('./train.zip', 'r').extractall('./')
data_dir = pathlib.Path('train')
image_count = len(list(data_dir.glob('*/*.jpg')))
CLASS_NAMES = np.array([item.name for item in data_dir.glob('*') if item.name != "LICENSE.txt"])
shrimp_zoea = list(data_dir.glob('shrimp_zoea/*'))
for image_path in shrimp_zoea[:5]:
display.display(Image.open(str(image_path)))
image_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255,
validation_split=0.2)
#rotation_range = 40,
#width_shift_range = 0.2,
#height_shift_range = 0.2,
#shear_range = 0.2,
#zoom_range = 0.2,
#horizontal_flip = True,
#fill_mode='nearest')
validation_split = 0.2
BATCH_SIZE = 32
BATCH_SIZE_VALID = 10
IMG_HEIGHT = 224
IMG_WIDTH = 224
STEPS_PER_EPOCH = np.ceil(image_count*(1-(validation_split))/BATCH_SIZE)
VALIDATION_STEPS = np.ceil((image_count*(validation_split)/BATCH_SIZE))
train_data_gen = image_generator.flow_from_directory(directory=str(data_dir),
subset='training',
batch_size=BATCH_SIZE,
class_mode = 'categorical',
shuffle=True,
target_size=(IMG_HEIGHT, IMG_WIDTH),
classes = list(CLASS_NAMES))
validation_data_gen = image_generator.flow_from_directory(directory=str(data_dir),
subset='validation',
batch_size=BATCH_SIZE_VALID,
class_mode = 'categorical',
shuffle=True,
target_size=(IMG_HEIGHT, IMG_WIDTH),
classes = list(CLASS_NAMES))
model_basic = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(16, (3,3), activation='relu', input_shape=(224, 224, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(32, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1000, activation='relu'),
tf.keras.layers.Dense(121, activation='softmax')
])
model_basic.summary()
model_basic.compile(optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
history = model_basic.fit(
train_data_gen,
epochs=10,
verbose=1,
validation_data=validation_data_gen,
steps_per_epoch=STEPS_PER_EPOCH,
validation_steps=VALIDATION_STEPS,
initial_epoch=0
)
【问题讨论】：
标签： tensorflow machine-learning gpu google-colaboratory