XGBoostError：[19:37:34] ../src/tree/updater_gpu_hist.cu:770：gpu_hist 中的异常：[19:37:34] ../src/common/device_helpers.cuh:132：NCCL 失败

Question

我尝试在 WSL2 中运行 xgboost。但是，如果我尝试通过将 tree_method 设置为文档中指定的“gpu_hist”来使用 GPU，则会收到此错误

XGBoostError                              Traceback (most recent call last) /tmp/ipykernel_30546/3907995862.py in <module>
      1 xgb = XGBClassifier(tree_method='gpu_hist')
----> 2 xgb.fit(X, y)

~/miniconda3/lib/python3.9/site-packages/xgboost/core.py in inner_f(*args, **kwargs)
    504         for k, arg in zip(sig.parameters, args):
    505             kwargs[k] = arg
--> 506         return f(**kwargs)
    507 
    508     return inner_f

~/miniconda3/lib/python3.9/site-packages/xgboost/sklearn.py in fit(self, X, y, sample_weight, base_margin, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, base_margin_eval_set, feature_weights, callbacks)    1248         )    1249 
-> 1250         self._Booster = train(    1251             params,    1252             train_dmatrix,

~/miniconda3/lib/python3.9/site-packages/xgboost/training.py in train(params, dtrain, num_boost_round, evals, obj, feval, maximize, early_stopping_rounds, evals_result, verbose_eval, xgb_model, callbacks)
    186     Booster : a trained booster model
    187     """
--> 188     bst = _train_internal(params, dtrain,
    189                           num_boost_round=num_boost_round,
    190                           evals=evals,

~/miniconda3/lib/python3.9/site-packages/xgboost/training.py in
_train_internal(params, dtrain, num_boost_round, evals, obj, feval, xgb_model, callbacks, evals_result, maximize, verbose_eval, early_stopping_rounds)
     79         if callbacks.before_iteration(bst, i, dtrain, evals):
     80             break
---> 81         bst.update(dtrain, i, obj)
     82         if callbacks.after_iteration(bst, i, dtrain, evals):
     83             break

~/miniconda3/lib/python3.9/site-packages/xgboost/core.py in update(self, dtrain, iteration, fobj)    1678     1679         if fobj is None:
-> 1680             _check_call(_LIB.XGBoosterUpdateOneIter(self.handle,    1681                                                     ctypes.c_int(iteration),    1682                                       dtrain.handle))

~/miniconda3/lib/python3.9/site-packages/xgboost/core.py in
_check_call(ret)
    216     """
    217     if ret != 0:
--> 218         raise XGBoostError(py_str(_LIB.XGBGetLastError()))
    219 
    220 

XGBoostError: [19:37:34] ../src/tree/updater_gpu_hist.cu:770: Exception in gpu_hist: [19:37:34] ../src/common/device_helpers.cuh:132: NCCL failure :unhandled system error ../src/common/device_helpers.cu(67) Stack trace:   [bt] (0) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x31dc4d) [0x7f21b3d65c4d]   [bt] (1) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x320da9) [0x7f21b3d68da9]   [bt] (2) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x31efaa) [0x7f21b3d66faa]   [bt] (3) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x4d7dc2) [0x7f21b3f1fdc2]   [bt] (4) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x4e1b86) [0x7f21b3f29b86]   [bt] (5) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x17d4a3) [0x7f21b3bc54a3]   [bt] (6) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x17e2cc) [0x7f21b3bc62cc]   [bt] (7) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x1b481a) [0x7f21b3bfc81a]   [bt] (8) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(XGBoosterUpdateOneIter+0x68) [0x7f21b3ae14e8]



Stack trace:   [bt] (0) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x4c2e69) [0x7f21b3f0ae69]   [bt] (1) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x4e1edf) [0x7f21b3f29edf]   [bt] (2) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x17d4a3) [0x7f21b3bc54a3]   [bt] (3) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x17e2cc) [0x7f21b3bc62cc]   [bt] (4) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(+0x1b481a) [0x7f21b3bfc81a]   [bt] (5) /home/mitbal/miniconda3/lib/python3.9/site-packages/xgboost/lib/libxgboost.so(XGBoosterUpdateOneIter+0x68) [0x7f21b3ae14e8]   [bt] (6) /home/mitbal/miniconda3/lib/python3.9/lib-dynload/../../libffi.so.7(+0x69dd) [0x7f22539599dd]   [bt] (7) /home/mitbal/miniconda3/lib/python3.9/lib-dynload/../../libffi.so.7(+0x6067) [0x7f2253959067]   [bt] (8) /home/mitbal/miniconda3/lib/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so(+0x14146) [0x7f2253973146]

作为比较，我可以在同一环境中使用 GPU 运行 pytorch。 库版本

python version 3.9.1
xgboost version 1.5.1
scikit-learn version 1.0.2
CUDA version 11.5
Nvidia Driver version 469.49

以及重现结果的代码示例

from sklearn import datasets
from xgboost import XGBClassifier

X, y = datasets.load_iris(return_X_y=True)

xgb = XGBClassifier(tree_method='gpu_hist')
xgb.fit(X, y)

有人有同样的经历吗？

Answer 1

XGBoost 还不支持 WSL2，因为 NCCL does not support WSL2。

编辑。据https://github.com/NVIDIA/nccl/issues/442#issuecomment-947782290 报道，最新的 NCCL 正在 WSL2 上工作。当前的 XGBoost 是使用旧版本的 NCCL 构建的，因此失败了。