cuML RandomForestClassifier：带有文档示例的 CUDA 错误

Question

我正在尝试在 Jupyter 笔记本中运行找到的示例 here 并从下面的rapids cuML 分类介绍中复制 - 它在 n_samples 低于 6000 时运行良好（此参数指示生成的数据集的行数）

import cuml
from cuml.datasets.classification import make_classification
from cuml.preprocessing.model_selection import train_test_split
from cuml.ensemble import RandomForestClassifier as cuRF
from sklearn.metrics import accuracy_score
from cupy import asnumpy

# synthetic dataset dimensions
n_samples = 1000
n_features = 10
n_classes = 2

# random forest depth and size
n_estimators = 25
max_depth = 10

# generate synthetic data [ binary classification task ]
X, y = make_classification ( n_classes = n_classes,
                             n_features = n_features,
                             n_samples = n_samples,
                             random_state = 0 )

X_train, X_test, y_train, y_test = train_test_split( X, y, random_state = 0 )

model = cuRF( max_depth = max_depth,
              n_estimators = n_estimators,
              random_state  = 0 )

%time trained_RF = model.fit ( X_train, y_train )

predictions = model.predict ( X_test )

cu_score = cuml.metrics.accuracy_score( y_test, predictions )
sk_score = accuracy_score( asnumpy( y_test ), asnumpy( predictions ) )

超过 6000，我收到以下 CUDA 错误和内核崩溃。 请注意：

• 在 n_samples = 5000 的情况下将 n_features 从 10 增加到 5000 运行得非常好。所以这似乎是数据集行数的问题，而不是列数
• 在机器上可用的 2 个 GPU (GTX 1050 2GB) 上测试
• nvidia-smi 在运行期间显示不到 25% 的 GPU 内存使用率
• cuda v11.2
• 驱动版本：460.73.01
• ubuntu 18

非常感谢任何帮助。

CUDA 错误：

RuntimeError Traceback（最近调用 最后的） 在 ~/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/internals/api_decorators.py 在 inner_with_setters(*args, **kwargs) 第408章 409 --> 410 返回函数(*args, **kwargs) 411 第412章 cuml/ensemble/randomforestclassifier.pyx 在 cuml.ensemble.randomforestclassifier.RandomForestClassifier.fit() RuntimeError: CUDA 错误出现在： file=/opt/conda/envs/rapids/conda-bld/libcuml_1614210250760/work/cpp/src/decisiontree/quantile/quantile.cuh line=150: call='cub::DeviceRadixSort::SortKeys((void *)d_temp_storage->data(), temp_storage_bytes, &d_keys_in[batch_offset], d_keys_out->data(), n_sampled_rows, 0, 8 * sizeof(T), tempmem->stream)', Reason=cudaErrorInvalidValue:invalid 争论 获得 64 个堆栈帧 #0 在 /home/oleg/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so(_ZN4raft9exception18collect_call_stackEv+0x46) [0x7fa9b83eef36] #1 在/home/oleg/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so(_ZN4raft10cuda_errorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x69) [0x7fa9b83ef699] #2 在 /home/oleg/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML12DecisionTree19preprocess_quantileIfiEEvPKT_PKjiiiiSt10shared_ptrI15TemporaryMemoryIS2_T0_EE+0xaaf) [0x7fa9b84fea7f] #3 在/home/oleg/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML12rfClassifierIfE3fitERKN4raft8handle_tEPKfiiPiiRPNS_20RandomForestMetaDataIfiEE+0xde3) [0x7fa9b8734b63] #4 在/home/oleg/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so(_ZN2ML3fitERKN4raft8handle_tERPNS_20RandomForestMetaDataIfiEEPfiiPiiNS_9RF_paramsEi+0x1fd) [0x7fa9b872f54d] #5 在 /home/oleg/anaconda3/envs/rapids/lib/python3.8/site-packages/cuml/ensemble/randomforestclassifier.cpython-38-x86_64-linux-gnu.so(+0x3c7e5) [0x7fa98e6d97e5] #6 在 /home/oleg/anaconda3/envs/rapids/bin/python(PyObject_Call+0x255) [0x5589964052b5] #7 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x21c1) [0x5589964b1de1] #8 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x558996490503] #9 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x1b2007) [0x558996492007] #10 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x4ca3) [0x5589964b48c3] #11 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x558996490503] #12 在 /home/oleg/anaconda3/envs/rapids/bin/python(PyEval_EvalCodeEx+0x39) [0x558996491559] #13 在 /home/oleg/anaconda3/envs/rapids/bin/python(PyEval_EvalCode+0x1b) [0x5589965349ab] #14 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x2731de) [0x5589965531de] #15 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x128d4b) [0x558996408d4b] #16 在 [ ..... 为便于阅读而删除 ] [0x5589964b1de1] #54 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x558996490503] #55 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x1b2007) [0x558996492007] #56 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x1782) [0x5589964b13a2] #57 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x1925da) [0x5589964725da] #58 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x128d4b) [0x558996408d4b] #59 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x13b3ea) [0x55899641b3ea] #60 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x21da4f) [0x5589964fda4f] #61 在 /home/oleg/anaconda3/envs/rapids/bin/python(+0x128fc2) [0x558996408fc2] #62 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalFrameDefault+0x92f) [0x5589964b054f] #63 在 /home/oleg/anaconda3/envs/rapids/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x558996490503]

Answer 1

发现该问题与在 cuML 中使用 RF 的实验后端有关 - 因此在 cuRF 配置中设置 split_algo = 0 可以通过回退到默认后端来解决问题。 在撰写本文时，这比使用实验性后端慢 3 倍。