如何在 scikit learn 中标准化排名数据？

Question

我正在做一些机器学习，在我的编码方面需要帮助。在我的训练数据中，我有许多网页的 URL 和这些网页的一些功能。我在网页文本的文本上运行 TF-IDF 以创建更多功能。

我提取的功能之一是，对于每个网址，我都会检索 Google Page 排名。这个值可以是世界上的任何值，但排名越低，Google 认为它是“更好的质量”。

鉴于我有 7,000 个网址并且排名差异很大（例如，www.google.com 可能排名第一，而 www.bbc.co.uk 可能排名 # 1,117，其他排名将远远超出我们的 7,000 个 URL）。

如何使用 scikit learn 有效地规范化这些数据，以便可以在我的机器学习算法中使用它？我正在运行一个逻辑回归，它只是试图预测一个网页是否“好”。我目前使用的唯一功能是在网页文本上使用我的 TF-IDF 创建的功能。理想情况下，我希望将这些与我的页面排名功能结合起来，从而获得最高的交叉验证分数。

非常感谢！

所以我们可以假设我的数据是 TSV 的形式：

URL GooglePageRank WebsiteText

一行示例：

http://www.google.com 1 This would be the text of the google webpage.

我希望标准化我的排名数据并将其用于我的逻辑回归。目前，我只使用“WebsiteText”列，在其上运行 TF-IDF，并将其插入到我的逻辑回归中。我想了解如何将此列与我的规范化 GooglePageRank 列结合起来，并在我的逻辑回归中使用这两个列 - 我该怎么做？

到目前为止，这是我的代码：

  import numpy as np
  from sklearn import metrics,preprocessing,cross_validation
  from sklearn.feature_extraction.text import TfidfVectorizer
  import sklearn.linear_model as lm
  import pandas as p
  loadData = lambda f: np.genfromtxt(open(f,'r'), delimiter=' ')

  print "loading data.."
  traindata = list(np.array(p.read_table('train.tsv'))[:,2])
  testdata = list(np.array(p.read_table('test.tsv'))[:,2])
  y = np.array(p.read_table('train.tsv'))[:,-1]

  tfv = TfidfVectorizer(min_df=3,  max_features=None, strip_accents='unicode',  
        analyzer='word',token_pattern=r'\w{1,}',ngram_range=(1, 2), use_idf=1,smooth_idf=1,sublinear_tf=1)

  rd = lm.LogisticRegression(penalty='l2', dual=True, tol=0.0001, 
                             C=1, fit_intercept=True, intercept_scaling=1.0, 
                             class_weight=None, random_state=None)

  X_all = traindata + testdata
  lentrain = len(traindata)

  print "fitting pipeline"
  tfv.fit(X_all)
  print "transforming data"
  X_all = tfv.transform(X_all)

  X = X_all[:lentrain]
  X_test = X_all[lentrain:]

  print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=20, scoring='roc_auc'))

  print "training on full data"
  rd.fit(X,y)
  pred = rd.predict_proba(X_test)[:,1]
  testfile = p.read_csv('test.tsv', sep="\t", na_values=['?'], index_col=1)
  pred_df = p.DataFrame(pred, index=testfile.index, columns=['label'])
  pred_df.to_csv('benchmark.csv')
  print "submission file created.."

*编辑：*

这是我目前正在运行的 -

from sklearn import metrics,preprocessing,cross_validation
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction import DictVectorizer
import sklearn.preprocessing
import sklearn.linear_model as lm
import pandas as p
loadData = lambda f: np.genfromtxt(open(f,'r'), delimiter=',')
print "loading data.."

#load train/test data for TF-IDF -- I know this is bad practice, but keeping it this way for the moment!
traindata = list(np.array(p.read_csv('FinalCSVFin.csv', delimiter=";"))[:,2])
testdata = list(np.array(p.read_csv('FinalTestCSVFin.csv', delimiter=";"))[:,2])

#load labels
y = np.array(p.read_csv('FinalCSVFin.csv', delimiter=";"))[:,-2]

#Load Integer values and append together
AllAlexaInfo = np.array(p.read_csv('FinalCSVFin.csv', delimiter=";"))[:,-1]

#make tfidf object
tfv = TfidfVectorizer(min_df=1, max_features=None, strip_accents='unicode',  
                      analyzer='word',token_pattern=r'\w{1,}',ngram_range=(1, 2), 
                      use_idf=1,smooth_idf=1,sublinear_tf=1)
div = DictVectorizer()
X = []
X_all = traindata + testdata
lentrain = len(traindata)
# fit/transform the TfidfVectorizer on the training data
vect = tfv.fit_transform(X_all) #bad practice, but using this for the moment!

for i, alexarank in enumerate(AllAlexaInfo):
    feature_dict = {'alexarank': AllAlexaInfo}
    # get ith row of the tfidf matrix (corresponding to sample)
    row = vect.getrow(i)    

    # filter the feature names corresponding to the sample
    all_words = tfv.get_feature_names()
    words = [all_words[ind] for ind in row.indices] 

    # associate each word (feature) with its corresponding score
    word_score = dict(zip(words, row.data)) 

    # concatenate the word feature/score with the datamining feature/value
    X.append(dict(word_score.items() + feature_dict.items()))

div.fit_transform(X)  # training data based on both Tfidf features and pagerank
sc = preprocessing.StandardScaler().fit(X)
X = sc.transform(X)
X_test = X_all[lentrain:]
X_test = sc.transform(X_test)

print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=20, scoring='roc_auc'))

print "training on full data"
rd.fit(X,y)
pred = rd.predict_proba(X_test)[:,1]
testfile = p.read_csv('test.tsv', sep="\t", na_values=['?'], index_col=1)
pred_df = p.DataFrame(pred, index=testfile.index, columns=['label'])
pred_df.to_csv('benchmark.csv')
print "submission file created.."

这似乎永远在运行，而且我相信我的“alexarank”值输入不正确 - 我该如何解决这个问题？

Answer 1

根据您对我的评论的回答，我会采取相应措施：

tfv = TfidfVectorizer(
    min_df=3,
    max_features=None,
    strip_accents='unicode',                    
    analyzer='word',
    token_pattern=r'\w{1,}',
    ngram_range=(1, 2), 
    use_idf=1,
    smooth_idf=1,
    sublinear_tf=1)
div = DictVectorizer()

X = []

# fit/transform the TfidfVectorizer on the training data
vectors = tfv.fit_transform(traindata)

for i, pagerank in enumerate(pageranks):
    feature_dict = {'pagerank': pagerank}
    # get ith row of the tfidf matrix (corresponding to sample)
    row = vect.getrow(i)    

    # filter the feature names corresponding to the sample
    all_words = tfv.get_feature_names()
    words = [all_words[ind] for ind in row.indices] 

    # associate each word (feature) with its corresponding score
    word_score = dict(zip(words, row.data)) 

    # concatenate the word feature/score with the datamining feature/value
    X.append(dict(word_score.items() + feature_dict.items()))

div.fit_transform(X)  # training data based on both Tfidf features and pagerank