DistributedSampler — 生成索引时，需要生成器的“cuda”设备类型答案

【问题标题】：DistributedSampler — Expected a ‘cuda’ device type for generator when generating indicesDistributedSampler — 生成索引时，需要生成器的“cuda”设备类型
【发布时间】：2020-11-21 07:59:44
【问题描述】：

执行分布式训练，我有如下代码：

training_sampler = DistributedSampler(training_set, num_replicas=2, rank=0)
training_generator = data.DataLoader(training_set, **params, sampler=training_sampler)
for x, y, z in training_generator:  # Error occurs here.
    ...

总的来说，我收到以下消息：

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/ubuntu/VC/ppg_training_extraction/ppg_training_scripts/train_ASR_trim_scp.py", line 336, in train
    for local_batch_src, local_batch_tgt, lengths in dataloaders[phase]:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 827, in __init__
    self._reset(loader, first_iter=True)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 857, in _reset
    self._try_put_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1091, in _try_put_index
    index = self._next_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/distributed.py", line 97, in __iter__
    indices = torch.randperm(len(self.dataset), generator=g).tolist()  # type: ignore
RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

现在在那一行，我在 pdb 中运行了以下指令：

(Pdb) g = torch.Generator()
(Pdb) g.manual_seed(0)
<torch._C.Generator object at 0x7ff7f8143110>
(Pdb) indices = torch.randperm(4556, generator=g).tolist()
(Pdb) indices = torch.randperm(455604, generator=g).tolist()
*** RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

为什么当上限整数很高时会出现运行时错误，但当它足够低时却没有？

注意，我运行了一个干净的 Python 会话并发现

>>> import torch
>>> g = torch.Generator()
>>> g.manual_seed(0)
<torch._C.Generator object at 0x7f9d2dfb39f0>
>>> indices = torch.randperm(455604, generator=g).tolist()

这很好用。这是我如何处理多个 GPU 之间的分布式训练的一些配置吗？任何形式的见解将不胜感激！

【问题讨论】：

标签： python pytorch distributed-computing

【解决方案1】：

我刚刚使用 dataloader 遇到了同样的问题，我发现以下帮助无需删除 torch.set_default_tensor_type('torch.cuda.FloatTensor')

data.DataLoader(..., generator=torch.Generator(device='cuda'))

因为我不想在我的代码中为大量张量手动添加 .to('cuda')

【讨论】：

【解决方案2】：

我有同样的问题。看来有人找到了问题的根源here：

torch.set_default_tensor_type('torch.cuda.FloatTensor')

我通过删除代码中的这一行并手动使用 .to(device) 解决了我的问题。

【讨论】：