将代码从二元分类器逻辑回归修改为多类“一对多”逻辑回归

Question

我是机器学习的新手，正在尝试练习不同的算法，目前我正在使用逻辑回归对从 sklearn 生成的随机数据集进行分类。现在这是一个二元分类器，但是我想使用多类逻辑回归“one vs all”方法（稍后进行比较）。

下面是我尝试为二进制分类实现的代码：

import numpy as np
import matplotlib.pyplot as plt
import sklearn
import random
from sklearn import datasets
from sklearn.preprocessing import StandardScaler

from sklearn.datasets import make_blobs
X, t = make_blobs(n_samples=[400, 800, 400], centers=[[0,0], [1,2], [2,3]],
                  n_features=2, random_state=2019)

indices = np.arange(X.shape[0])
random.seed(2020)
random.shuffle(indices)
indices[:10]

X_train = X[indices[:800], :]
X_val = X[indices[800:1200], :]
X_test = X[indices[1200:], :]
t_train = t[indices[:800]]
t_val = t[indices[800:1200]]
t_test = t[indices[1200:]]

t2_train = t_train == 1
t2_train = t2_train.astype('int')
t2_val = (t_val == 1).astype('int')
t2_test = (t_test == 1).astype('int')

def add_bias(X):
    # Put bias in position 0
    sh = X.shape
    if len(sh) == 1:
        #X is a vector
        return np.concatenate([np.array([1]), X])
    else:
        # X is a matrix
        m = sh[0]
        bias = np.ones((m, 1)) # Makes a m*1 matrix of 1-s
        return np.concatenate([bias, X], axis  = 1)

class NumpyClassifier():
    # Common methods to all numpy classifiers --- if any

    def accuracy(self, X_val, t_val, **kwargs):
        pred = self.predict(X_val, **kwargs)
        if len(pred.shape) > 1:
            pred = pred[:, 0]
        return sum(pred==t_val)/len(pred)

# code for Logistic Regression
def logistic(x):
    return 1/(1+np.exp(-x))

class NumpyLogReg(NumpyClassifier):

    def fit(self, X_train, t_train, gamma = 0.1, epochs=10):
        # X_train is a Nxm matrix, N data points, m features
        # t_train are the targets values for training data

        (k, m) = X_train.shape
        X_train = add_bias(X_train)

        self.theta = theta = np.zeros(m+1)

        for e in range(epochs):
            theta -= gamma / k *  X_train.T @ (self.forward(X_train) - t_train)

    def forward(self, X_val):
        return logistic(X_val @ self.theta)

    def score(self, X_val):
        z = add_bias(X_val)
        score = self.forward(z)
        return score

    def predict(self, X_val, threshold=0.5):
        z = add_bias(X_val)
        score = self.forward(z)
        # score = z @ self.theta
        return (score>threshold).astype('int')

lr_cl = NumpyLogReg()
lr_cl.fit(X_train, t_train)
lr_cl.predict(X_val)
lr_cl.accuracy(X_val, t_val)

for e in [1, 2, 5, 10, 50, 100, 1000, 10000, 100000, 1000000]:
    lr_cl = NumpyLogReg()
    lr_cl.fit(X_train, t_train, epochs=e, gamma=0.00001)
    print("{:10}  {:7.3f}".format(e, lr_cl.accuracy(X_val, t_val)))

我需要关于如何将代码修改为多类“one vs all”/“one vs rest”Logistic 回归的建议/提示。 我不想直接使用sklearn导入的逻辑回归算法，而是像这样从头开始。

任何建议都非常感谢，在此先感谢您。

Answer 1

我假设NumpyLogReg 在二元分类上效果很好。使用One-Vs-Rest (ovr) 技术将同一类用于multi-class 分类。

假设数据集有 3 个类 A, B, C

1. 将类标签 A 的二元分类模型称为 +ve 类，B, C 称为 -ve 类，并记下概率分数
2. 通过将B 视为+ve 和A, C 视为-ve 并将C 视为+ve 和A, B 视为-ve 重复相同的操作。记下各自的概率分数。
3. 基本上，如果有n类，就会有n二元分类器模型，即fitting one classifier per class
4. 通过仔细检查每个类的分类器（即通过分析概率值），您可以实现multi-class 分类，模型将具有高度可解释性。

详细解释请参考this guideline