Jak dodać kolejną funkcję (długość tekstu) do aktualnego worka klasyfikacji słów? Scikit-learn

Question

Używam worka słów do klasyfikacji tekstu. Działa dobrze, ale zastanawiam się, jak dodać funkcję, która nie jest słowem.

Oto mój przykładowy kod.

import numpy as np 
from sklearn.pipeline import Pipeline 
from sklearn.feature_extraction.text import CountVectorizer 
from sklearn.svm import LinearSVC 
from sklearn.feature_extraction.text import TfidfTransformer 
from sklearn.multiclass import OneVsRestClassifier 

X_train = np.array(["new york is a hell of a town", 
        "new york was originally dutch", 
        "new york is also called the big apple", 
        "nyc is nice", 
        "the capital of great britain is london. london is a huge metropolis which has a great many number of people living in it. london is also a very old town with a rich and vibrant cultural history.", 
        "london is in the uk. they speak english there. london is a sprawling big city where it's super easy to get lost and i've got lost many times.", 
        "london is in england, which is a part of great britain. some cool things to check out in london are the museum and buckingham palace.", 
        "london is in great britain. it rains a lot in britain and london's fogs are a constant theme in books based in london, such as sherlock holmes. the weather is really bad there.",]) 
y_train = [[0],[0],[0],[0],[1],[1],[1],[1]] 

X_test = np.array(["it's a nice day in nyc", 
        'i loved the time i spent in london, the weather was great, though there was a nip in the air and i had to wear a jacket.' 
        ]) 
target_names = ['Class 1', 'Class 2'] 

classifier = Pipeline([ 
    ('vectorizer', CountVectorizer(min_df=1,max_df=2)), 
    ('tfidf', TfidfTransformer()), 
    ('clf', OneVsRestClassifier(LinearSVC()))]) 
classifier.fit(X_train, y_train) 
predicted = classifier.predict(X_test) 
for item, labels in zip(X_test, predicted): 
    print '%s => %s' % (item, ', '.join(target_names[x] for x in labels)) 

Teraz jest jasne, że tekst o Londynie jest znacznie dłuższy niż tekst o Nowym Jorku. Jak dodać długość tekstu jako funkcję? Czy muszę użyć innej metody klasyfikacji, a następnie połączyć dwie prognozy? Czy jest jakiś sposób na zrobienie tego wraz z workiem słów? Jakiś przykładowy kod byłby świetny - jestem bardzo nowy w uczeniu maszynowym i nauce scikita.

Answer 1

Jak pokazano w komentarzach, jest to połączenie FunctionTransformer, FeaturePipeline i FeatureUnion.

import numpy as np 
from sklearn.pipeline import Pipeline, FeatureUnion 
from sklearn.feature_extraction.text import CountVectorizer 
from sklearn.svm import LinearSVC 
from sklearn.feature_extraction.text import TfidfTransformer 
from sklearn.multiclass import OneVsRestClassifier 
from sklearn.preprocessing import FunctionTransformer 

X_train = np.array(["new york is a hell of a town", 
        "new york was originally dutch", 
        "new york is also called the big apple", 
        "nyc is nice", 
        "the capital of great britain is london. london is a huge metropolis which has a great many number of people living in it. london is also a very old town with a rich and vibrant cultural history.", 
        "london is in the uk. they speak english there. london is a sprawling big city where it's super easy to get lost and i've got lost many times.", 
        "london is in england, which is a part of great britain. some cool things to check out in london are the museum and buckingham palace.", 
        "london is in great britain. it rains a lot in britain and london's fogs are a constant theme in books based in london, such as sherlock holmes. the weather is really bad there.",]) 
y_train = np.array([[0],[0],[0],[0],[1],[1],[1],[1]]) 

X_test = np.array(["it's a nice day in nyc", 
        'i loved the time i spent in london, the weather was great, though there was a nip in the air and i had to wear a jacket.' 
        ]) 
target_names = ['Class 1', 'Class 2'] 


def get_text_length(x): 
    return np.array([len(t) for t in x]).reshape(-1, 1) 

classifier = Pipeline([ 
    ('features', FeatureUnion([ 
     ('text', Pipeline([ 
      ('vectorizer', CountVectorizer(min_df=1,max_df=2)), 
      ('tfidf', TfidfTransformer()), 
     ])), 
     ('length', Pipeline([ 
      ('count', FunctionTransformer(get_text_length, validate=False)), 
     ])) 
    ])), 
    ('clf', OneVsRestClassifier(LinearSVC()))]) 

classifier.fit(X_train, y_train) 
predicted = classifier.predict(X_test) 
predicted 

Spowoduje to dodanie długości tekstu do funkcji używanych przez klasyfikator.