【问题标题】:Custom eval metric using early stopping in LGBM (Sklearn API) and Optuna在 LGBM (Sklearn API) 和 Optuna 中使用提前停止的自定义评估指标
【发布时间】:2022-11-04 00:11:55
【问题描述】:

问题:

  1. 第一个问题可能非常愚蠢,但我还是会问:在下面的示例中,修剪和提前停止是否相同?还是两个独立的选项控制两个独立的进程?
  2. 我的目标不平衡,那么如何在此处使用自定义评估指标而不是“binary_logloss”,例如平衡精度?
  3. 当我得到最佳参数时,'n_estimators' 仍将等于 999999。对于不平衡的目标,建议使用“无限”数量的估计器并使用提前停止进行修剪,这就是它如此之高的原因。修剪后如何用最优的 n_estimators 拟合最终模型?

    非常感谢您帮助我解决这个问题,我很沮丧。

    def objective(trial, X, y):
        param_grid = {
            # "device_type": trial.suggest_categorical("device_type", ['gpu']),
            "n_estimators": trial.suggest_categorical("n_estimators", [999999]),
            "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3),
            "num_leaves": trial.suggest_int("num_leaves", 20, 3000, step=20),
            "max_depth": trial.suggest_int("max_depth", 3, 12),
            "min_data_in_leaf": trial.suggest_int("min_data_in_leaf", 200, 10000, step=100),
            "lambda_l1": trial.suggest_int("lambda_l1", 0, 100, step=5),
            "lambda_l2": trial.suggest_int("lambda_l2", 0, 100, step=5),
            "min_gain_to_split": trial.suggest_float("min_gain_to_split", 0, 15),
            "bagging_fraction": trial.suggest_float(
                "bagging_fraction", 0.2, 0.95, step=0.1
            ),
            "bagging_freq": trial.suggest_categorical("bagging_freq", [1]),
            "feature_fraction": trial.suggest_float(
                "feature_fraction", 0.2, 0.95, step=0.1
            ),
        }
    
        cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1121218)
    
        cv_scores = np.empty(5)
        for idx, (train_idx, test_idx) in enumerate(cv.split(X, y)):
            X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
            y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
    
            model = LGBMClassifier(
                objective="binary",
                **param_grid,
                n_jobs=-1,
                scale_pos_weight=len(y_train) / y_train.sum()
            )
            
            model.fit( 
                X_train,
                y_train,
                eval_set=[(X_test, y_test)],
                eval_metric="binary_logloss", # replace this with e.g. balanced accuracy or f1
                callbacks=[
                    LightGBMPruningCallback(trial, "binary_logloss"), # replace this with e.g. balanced accuracy or f1
                    early_stopping(100, verbose=False)
                ], 
            )
            preds = model.predict(X_test)#.argmax(axis=1)
            cv_scores[idx] = balanced_accuracy_score(y_test, preds)
        
        loss = 1 - np.nanmedian(cv_scores)
        return loss
    

    跑:

    study = optuna.create_study(direction="minimize", study_name="LGBM Classifier")
    func = lambda trial: objective(trial, X_train, y_train)
    study.optimize(func, n_trials=1)
    

    适合最后的问题。但是在这里我不想拟合 n_estimators=999999,而是使用最佳数量的 n_estimators:

    model = LGBMClassifier(
        objective="binary",
        **study.best_params,
        n_jobs=-1,
        scale_pos_weight=len(y) / y.sum()
    )
    

【问题讨论】:

    标签: python lightgbm optuna


    【解决方案1】:

    所以经过一天的实验,我可以回答我自己的问题:

    1. LightGBMPruningCallback(trial, "your_metric") 定义的 LGBM 修剪不是指提前停止过程。如果度量非常不令人满意(例如,低平衡精度),LGBM 修剪本质上会跳过评估给定试验中的所有 cv 折叠(即对于给定的一组超参数)。

    2. 这很烦人,解决方案没有很好的记录,但它不是在 LGBMClassifier 中设置 metric='custom' 然后在函数中定义度量并设置 eval_metric=your_function,请参见下面的代码。

    3. 这可能是一种为最佳试验(最佳参数)检索 n_estimators 的方法,但是,我通过使用提前停止的最终模型来解决它,请参见下面的代码:

      代码

      定义自定义指标

      def custom_metric(y_true, y_hat):  
          higher_is_better = True
          y_hat_label = np.round(y_hat)
          balanced_accuracy = balanced_accuracy_score(y_true, y_hat_label)
          return 'balanced_accuracy', balanced_accuracy, higher_is_better
      

      定义目标函数(对我上面的问题的重要更改已注释):

      def objective(trial, X, y):
          param_grid = {
              "n_estimators": trial.suggest_categorical("n_estimators", [999999]),
              "learning_rate": trial.suggest_float("learning_rate", 0.01, 0.3),
              "num_leaves": trial.suggest_int("num_leaves", 20, 3000, step=20),
              "max_depth": trial.suggest_int("max_depth", 3, 12),
              "min_data_in_leaf": trial.suggest_int("min_data_in_leaf", 200, 10000, step=100),
              "lambda_l1": trial.suggest_int("lambda_l1", 0, 100, step=5),
              "lambda_l2": trial.suggest_int("lambda_l2", 0, 100, step=5),
              "min_gain_to_split": trial.suggest_float("min_gain_to_split", 0, 15),
              "bagging_fraction": trial.suggest_float(
                  "bagging_fraction", 0.2, 0.95, step=0.1
              ),
              "bagging_freq": trial.suggest_categorical("bagging_freq", [1]),
              "feature_fraction": trial.suggest_float(
                  "feature_fraction", 0.2, 0.95, step=0.1
              ),
          }
      
          cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=1121218)
      
          cv_scores = np.empty(5)
          for idx, (train_idx, test_idx) in enumerate(cv.split(X, y)):
              X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
              y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]
      
              model = LGBMClassifier(
                  metric='custom', #THIS HAS CHANGED (REF QUESTION 2)!
                  objective="binary",
                  **param_grid,
                  n_jobs=-1,
                  scale_pos_weight=len(y_train) / y_train.sum()
              )
      
              model.fit( 
                  X_train,
                  y_train,
                  eval_set=[(X_test, y_test)],
                  eval_metric=[custom_metric], # THIS HAS CHANGED (REF QUESTION 2)!
                  callbacks=[
                      LightGBMPruningCallback(trial, "balanced_accuracy"),  # THIS HAS CHANGED (REF QUESTION 2)!
                      early_stopping(100, verbose=True),
                  ],  # Add a pruning callback
              )
              preds = model.predict(X_test)#.argmax(axis=1)
              cv_scores[idx] = balanced_accuracy_score(y_test, preds)
          
          score = np.nanmedian(cv_scores)
          return score
      

      优化:

      study = optuna.create_study(direction="maximize", study_name="LGBM Classifier")
      func = lambda trial: objective(trial, X_train, y_train)
      study.optimize(func, n_trials=10)
      

      最后拟合最终模型(即对问题 3 的回答)。我通过对最终模型使用提前停止解决了这个问题:

      model = LGBMClassifier(
          objective="binary",
          metric='custom', # THIS HAS CHANGED (REF QUESTION 2)!
          **study.best_params,
          n_jobs=-1,
          scale_pos_weight=len(y) / y.sum()
      )
      
      model.fit(
          X_train,
          y_train,
          eval_set=[(X_test, y_test)],
          eval_metric=custom_metric,
          early_stopping_rounds=100,
          callbacks=[
              early_stopping(100, verbose=True),
          ], 
      )
      

      该算法将对每个试验中应用于每个折叠的每个 LGBM 模型应用提前停止(即超参数的组合)。

      在将算法应用于所有五个折叠之前,它还会修剪(即停止)某些给出不令人满意的分数指标的试验。一些试验将很早就停止。

      然后它继续拟合最终模型 - 在搜索完成之后。在最终拟合中,模型使用提前停止(请注意,我在最终拟合中使用了不同的评估集)。

      就是这样,祝你有美好的一天:)

    【讨论】:

      猜你喜欢
      • 2020-06-01
      • 2019-03-09
      • 2019-04-30
      • 2020-08-24
      • 2019-12-04
      • 2018-02-27
      • 1970-01-01
      • 1970-01-01
      • 2019-03-26
      相关资源
      最近更新 更多