Przechodziłem przez this question.Python re.split() vs nltk word_tokenize i sent_tokenize
Zastanawiam się tylko, czy NLTK byłoby szybsze niż wyrażenie regularne w tokenizacji słów/zdań.
Przechodziłem przez this question.Python re.split() vs nltk word_tokenize i sent_tokenize
Zastanawiam się tylko, czy NLTK byłoby szybsze niż wyrażenie regularne w tokenizacji słów/zdań.
Domyślnie nltk.word_tokenize()
używa Treebank tokenizer, który emuluje tokenizator z Penn Treebank tokenizer.
Należy pamiętać, że nie osiąga str.split()
tokeny w sensie językoznawstwo, np .:
>>> sent = "This is a foo, bar sentence."
>>> sent.split()
['This', 'is', 'a', 'foo,', 'bar', 'sentence.']
>>> from nltk import word_tokenize
>>> word_tokenize(sent)
['This', 'is', 'a', 'foo', ',', 'bar', 'sentence', '.']
Zazwyczaj jest stosowany do oddzielnych ciągów z określonym separatorem, na przykład w pliku oddzielonym tabulatorami można użyć str.split('\t')
lub gdy próbujesz podzielić ciąg znaków przez znak nowej linii \n
, gdy twój plik tekstowy ma jedno zdanie w linii.
I zróbmy jakąś benchmarking w python3
:
import time
from nltk import word_tokenize
import urllib.request
url = 'https://raw.githubusercontent.com/Simdiva/DSL-Task/master/data/DSLCC-v2.0/test/test.txt'
response = urllib.request.urlopen(url)
data = response.read().decode('utf8')
for _ in range(10):
start = time.time()
for line in data.split('\n'):
line.split()
print ('str.split():\t', time.time() - start)
for _ in range(10):
start = time.time()
for line in data.split('\n'):
word_tokenize(line)
print ('word_tokenize():\t', time.time() - start)
[out]:
str.split(): 0.05451083183288574
str.split(): 0.054320573806762695
str.split(): 0.05368804931640625
str.split(): 0.05416440963745117
str.split(): 0.05299568176269531
str.split(): 0.05304527282714844
str.split(): 0.05356955528259277
str.split(): 0.05473494529724121
str.split(): 0.053118228912353516
str.split(): 0.05236077308654785
word_tokenize(): 4.056122779846191
word_tokenize(): 4.052812337875366
word_tokenize(): 4.042144775390625
word_tokenize(): 4.101543664932251
word_tokenize(): 4.213029146194458
word_tokenize(): 4.411528587341309
word_tokenize(): 4.162556886672974
word_tokenize(): 4.225975036621094
word_tokenize(): 4.22914719581604
word_tokenize(): 4.203172445297241
Jeśli staramy się another tokenizers in bleeding edge NLTK z https://github.com/jonsafari/tok-tok/blob/master/tok-tok.pl:
import time
from nltk.tokenize import ToktokTokenizer
import urllib.request
url = 'https://raw.githubusercontent.com/Simdiva/DSL-Task/master/data/DSLCC-v2.0/test/test.txt'
response = urllib.request.urlopen(url)
data = response.read().decode('utf8')
toktok = ToktokTokenizer().tokenize
for _ in range(10):
start = time.time()
for line in data.split('\n'):
toktok(line)
print ('toktok:\t', time.time() - start)
[out]:
toktok: 1.5902607440948486
toktok: 1.5347232818603516
toktok: 1.4993178844451904
toktok: 1.5635688304901123
toktok: 1.5779635906219482
toktok: 1.8177132606506348
toktok: 1.4538452625274658
toktok: 1.5094449520111084
toktok: 1.4871931076049805
toktok: 1.4584410190582275
(Uwaga: źródło pliku tekstowego jest od https://github.com/Simdiva/DSL-Task)
Jeśli spojrzymy na rodzimym perl
realizacji, czas python
vs perl
dla ToktokTokenizer
jest porównywalna. Ale to zrobić w realizacji Pythona to Wyrażenia regularne są skompilowane natomiast w Perlu, to nie jest, ale potem the proof is still in the pudding:
[email protected]:~$ wget https://raw.githubusercontent.com/jonsafari/tok-tok/master/tok-tok.pl
--2016-02-11 20:36:36-- https://raw.githubusercontent.com/jonsafari/tok-tok/master/tok-tok.pl
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.17.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.17.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2690 (2.6K) [text/plain]
Saving to: ‘tok-tok.pl’
100%[===============================================================================================================================>] 2,690 --.-K/s in 0s
2016-02-11 20:36:36 (259 MB/s) - ‘tok-tok.pl’ saved [2690/2690]
[email protected]:~$ wget https://raw.githubusercontent.com/Simdiva/DSL-Task/master/data/DSLCC-v2.0/test/test.txt
--2016-02-11 20:36:38-- https://raw.githubusercontent.com/Simdiva/DSL-Task/master/data/DSLCC-v2.0/test/test.txt
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.17.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.17.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3483550 (3.3M) [text/plain]
Saving to: ‘test.txt’
100%[===============================================================================================================================>] 3,483,550 363KB/s in 7.4s
2016-02-11 20:36:46 (459 KB/s) - ‘test.txt’ saved [3483550/3483550]
[email protected]:~$ time perl tok-tok.pl <test.txt> /tmp/null
real 0m1.703s
user 0m1.693s
sys 0m0.008s
[email protected]:~$ time perl tok-tok.pl <test.txt> /tmp/null
real 0m1.715s
user 0m1.704s
sys 0m0.008s
[email protected]:~$ time perl tok-tok.pl <test.txt> /tmp/null
real 0m1.700s
user 0m1.686s
sys 0m0.012s
[email protected]:~$ time perl tok-tok.pl <test.txt> /tmp/null
real 0m1.727s
user 0m1.700s
sys 0m0.024s
[email protected]:~$ time perl tok-tok.pl <test.txt> /tmp/null
real 0m1.734s
user 0m1.724s
sys 0m0.008s
(Uwaga: Kiedy odmierzanie tok-tok.pl
, mieliśmy do rury wyjście do pliku, więc czas tutaj obejmuje czas maszyna potrzebny na wyjście do pliku, natomiast w taktowania nltk.tokenize.ToktokTokenizer
, to nie obejmuje czasu na wyjście do pliku)
w odniesieniu do sent_tokenize()
, to trochę inna i porównanie pomiaru prędkości bez uwzględnienia dokładności jest nieco dziwaczne.
Rozważ to:
Jeśli regex dzieli TextFile/paragraf w 1 zdaniu, wówczas prędkość jest niemal natychmiastowe, czyli 0 pracę. Ale to byłby straszny tokenizer zdań ...
Jeśli zdania w pliku jest już rozdzielone \n
, to jest to po prostu przypadek porównując jak str.split('\n')
vs re.split('\n')
i nltk
miałoby nic wspólnego z tokeny zdanie; P
Więcej informacji na temat sposobów sent_tokenize()
prac w NLTK, patrz:
Tak skutecznie porównać sent_tokenize()
vs innych metod opartych na regex (nie str.split('\n')
), trzeba by ocenić również dokładność i mają DataSet ludzku ocenianym zdaniu w tokenized formacie.
Rozważmy zadanie: https://www.hackerrank.com/challenges/from-paragraphs-to-sentences
Biorąc tekst:
W trzeciej kategorii zaliczył tych braci (większość), którzy widział nic w masonerii, ale z zewnętrznych form i ceremonii, a doceniły ścisłą wydajność tych formularzy bez obawy o ich znaczenie lub znaczenie. Takim był Willarski, a nawet Wielki Mistrz głównego loży. Wreszcie do czwartej kategorii należało również wielu braci, szczególnie tych, którzy ostatnio się zjednoczyli: . Zgodnie z obserwacjami Piotra byli to ludzie, którzy nie mieli wiary w nic, ani pragnienia czegokolwiek, ale przyłączyli się do Freemasons jedynie po to, by związać się z bogatymi młodymi braćmi, którzy byli wpływowi przez ich powiązania lub rangę, i których bardzo mocno wielu w loży. Pierre poczuł się niezadowolony z tego, co robił. Masoneria, w każdym razie, jak ją tu widział, czasami wydawała mu się oparta jedynie na zewnętrznych. Nie myślał o zwątpieniu w samą wolnomularstwo, ale podejrzewał, że rosyjskie masonerstwo podjęło niewłaściwą ścieżkę i odstąpiło od pierwotnych zasad. I tak w kierunku pod koniec roku wyjechał za granicę, aby zostać wtajemniczonym w wyższe sekrety zlecenia. Co należy zrobić w takich okolicznościach? Aby sprzyjać rewolucji, obalać wszystko, odpychać życie siłą? Mamy są bardzo daleko od tego. Każda gwałtowna reforma zasługuje na potępienie, gdyż nie udaje się jej zneutralizować, podczas gdy mężczyźni pozostają tym, kim są, a także , ponieważ mądrość nie potrzebuje przemocy. "Ale co tam jest w bieganiu po w taki sposób?" - powiedział pan młody Ilagina. - Kiedyś go nie zauważyła i odrzuciła, każdy kundel mógł go wziąć - mówił Ilagin z takim samym entuzjazmem, zdyszany z galopu i podekscytowania.
Chcemy uzyskać to:
In the third category he included those Brothers (the majority) who saw nothing in Freemasonry but the external forms and ceremonies, and prized the strict performance of these forms without troubling about their purport or significance.
Such were Willarski and even the Grand Master of the principal lodge.
Finally, to the fourth category also a great many Brothers belonged, particularly those who had lately joined.
These according to Pierre's observations were men who had no belief in anything, nor desire for anything, but joined the Freemasons merely to associate with the wealthy young Brothers who were influential through their connections or rank, and of whom there were very many in the lodge.
Pierre began to feel dissatisfied with what he was doing.
Freemasonry, at any rate as he saw it here, sometimes seemed to him based merely on externals.
He did not think of doubting Freemasonry itself, but suspected that Russian Masonry had taken a wrong path and deviated from its original principles.
And so toward the end of the year he went abroad to be initiated into the higher secrets of the order.
What is to be done in these circumstances?
To favor revolutions, overthrow everything, repel force by force?
No!
We are very far from that.
Every violent reform deserves censure, for it quite fails to remedy evil while men remain what they are, and also because wisdom needs no violence.
"But what is there in running across it like that?" said Ilagin's groom.
"Once she had missed it and turned it away, any mongrel could take it," Ilagin was saying at the same time, breathless from his gallop and his excitement.
Więc po prostu robi str.split('\n')
daje nic.Nawet bez uwzględnienia kolejności zdań, uzyskasz 0 dodatni wynik:
>>> text = """In the third category he included those Brothers (the majority) who saw nothing in Freemasonry but the external forms and ceremonies, and prized the strict performance of these forms without troubling about their purport or significance. Such were Willarski and even the Grand Master of the principal lodge. Finally, to the fourth category also a great many Brothers belonged, particularly those who had lately joined. These according to Pierre's observations were men who had no belief in anything, nor desire for anything, but joined the Freemasons merely to associate with the wealthy young Brothers who were influential through their connections or rank, and of whom there were very many in the lodge.Pierre began to feel dissatisfied with what he was doing. Freemasonry, at any rate as he saw it here, sometimes seemed to him based merely on externals. He did not think of doubting Freemasonry itself, but suspected that Russian Masonry had taken a wrong path and deviated from its original principles. And so toward the end of the year he went abroad to be initiated into the higher secrets of the order.What is to be done in these circumstances? To favor revolutions, overthrow everything, repel force by force?No! We are very far from that. Every violent reform deserves censure, for it quite fails to remedy evil while men remain what they are, and also because wisdom needs no violence. "But what is there in running across it like that?" said Ilagin's groom. "Once she had missed it and turned it away, any mongrel could take it," Ilagin was saying at the same time, breathless from his gallop and his excitement. """
>>> answer = """In the third category he included those Brothers (the majority) who saw nothing in Freemasonry but the external forms and ceremonies, and prized the strict performance of these forms without troubling about their purport or significance.
... Such were Willarski and even the Grand Master of the principal lodge.
... Finally, to the fourth category also a great many Brothers belonged, particularly those who had lately joined.
... These according to Pierre's observations were men who had no belief in anything, nor desire for anything, but joined the Freemasons merely to associate with the wealthy young Brothers who were influential through their connections or rank, and of whom there were very many in the lodge.
... Pierre began to feel dissatisfied with what he was doing.
... Freemasonry, at any rate as he saw it here, sometimes seemed to him based merely on externals.
... He did not think of doubting Freemasonry itself, but suspected that Russian Masonry had taken a wrong path and deviated from its original principles.
... And so toward the end of the year he went abroad to be initiated into the higher secrets of the order.
... What is to be done in these circumstances?
... To favor revolutions, overthrow everything, repel force by force?
... No!
... We are very far from that.
... Every violent reform deserves censure, for it quite fails to remedy evil while men remain what they are, and also because wisdom needs no violence.
... "But what is there in running across it like that?" said Ilagin's groom.
... "Once she had missed it and turned it away, any mongrel could take it," Ilagin was saying at the same time, breathless from his gallop and his excitement."""
>>>
>>> output = text.split('\n')
>>> sum(1 for sent in text.split('\n') if sent in answer)
0
... i co powstrzymało cię przed próbowaniem? Uruchamiasz próbkę i synchronizujesz ją z 'timeit'? – lenz
Jestem stil nowy w python, esp nltk. Właśnie zauważyłem, że re.split(), s.split() był szybszy kiedy przełączam się z nltk. Używałem tego: zdania = sent_tokenize (txt), teraz to: sentences = re.split (r '(? <= [^ AZ]. [.?]) + (? = [AZ])', txt) – wakamdr
może to być spowodowane tym, że musi załadować WordNet podczas uruchamiania, co powoduje, że nltk jest wolny? – wakamdr