Podział na ciąg danych kolumna danych produkuje NAs

Question

To jest moje pierwsze pytanie na SO, więc daj mi znać, czy można go poprawić. Pracuję nad projektem przetwarzania języka naturalnego w R i próbuję zbudować tabelę data.table zawierającą przypadki testowe. Tutaj buduję znacznie uproszczony przykład:

texts.dt <- data.table(string = c("one", 
            "two words", 
            "three words here", 
            "four useless words here", 
            "five useless meaningless words here", 
            "six useless meaningless words here just", 
            "seven useless meaningless words here just to", 
            "eigth useless meaningless words here just to fill", 
            "nine useless meaningless words here just to fill up", 
            "ten useless meaningless words here just to fill up space"), 
         word.count = 1:10, 
         stop.at.word = c(0, 1, 2, 2, 4, 3, 3, 6, 7, 5)) 

ta zwraca data.table będziemy pracować nad:

              string word.count stop.at.word 
1:              one   1   0 
2:            two words   2   1 
3:           three words here   3   2 
4:         four useless words here   4   2 
5:      five useless meaningless words here   5   4 
6:     six useless meaningless words here just   6   3 
7:    seven useless meaningless words here just to   7   3 
8:  eigth useless meaningless words here just to fill   8   6 
9:  nine useless meaningless words here just to fill up   9   7 
10: ten useless meaningless words here just to fill up space   10   5 

W rzeczywistym zastosowaniu, wartości w kolumnie stop.at.word są ustalane w sposób losowy (z górną granicą = word.count - 1). Ponadto ciągi nie są uporządkowane według długości, ale nie powinno to mieć znaczenia.

Kod powinien dodać dwie kolumny input i output, gdzie input zawiera podciąg od pozycji 1 do stop.at.word i output zawiera słowo to następująco (jedno słowo) tak:

>desired_result 
                  string word.count stop.at.word          input 
    1:              one   1   0            
    2:            two words   2   1           two 
    3:           three words here   3   2         three words 
    4:         four useless words here   4   2        four useless 
    5:      five useless meaningless words here   5   4    five useless meaningless words 
    6:     six useless meaningless words here just   6   2         six useless 
    7:    seven useless meaningless words here just to   7   3     seven useless meaningless 
    8:  eigth useless meaningless words here just to fill   8   6 eigth useless meaningless words here just 
    9:  nine useless meaningless words here just to fill up   9   7 nine useless meaningless words here just to 
    10: ten useless meaningless words here just to fill up space   10   5   ten useless meaningless words here 
      output 
    1:    
    2:  words 
    3:  here 
    4:  words 
    5:  here 
    6: meaningless 
    7:  words 
    8:   to 
    9:  fill 
    10:  just 

Niestety co ja zamiast dostać to:

             string word.count stop.at.word input output 
1:              one   1   0    
2:            two words   2   1 NA  NA 
3:           three words here   3   2 NA  NA 
4:         four useless words here   4   2 NA  NA 
5:      five useless meaningless words here   5   4 NA  NA 
6:     six useless meaningless words here just   6   3 NA  NA 
7:    seven useless meaningless words here just to   7   3 NA  NA 
8:  eigth useless meaningless words here just to fill   8   6 NA  NA 
9:  nine useless meaningless words here just to fill up   9   7 NA  NA 
10: ten useless meaningless words here just to fill up space   10   5 ten  NA 

zauważy niespójne wyniki, z pustym ciągiem w wierszu 1 i „dziesięć” zwróciło na wiersz 10.

Oto kod używam:

texts.dt[, c("input", "output") := .(
     substr(string, 
       1, 
       sapply(gregexpr(" ", string),"[", stop.at.word) - 1), 
     substr(string, 
       sapply(gregexpr(" ", string),"[", stop.at.word), 
       sapply(gregexpr(" ", string),"[", stop.at.word + 1) - 1) 
    )] 

Pobiegłem wiele testów oraz instrukcje substr działa dobrze, gdy próbuję poszczególne ciągi w konsoli, ale nie po nałożeniu na data.table. Podejrzewam, że brakuje mi czegoś związanego z określaniem zakresu w data.table, ale nie używam tego pakietu przez długi czas, więc jestem bardzo zdezorientowany.

Byłbym bardzo wdzięczny za pomoc. Z góry dziękuję!

Answer 1

to pewnie zrobić

texts.dt[stop.at.word > 0, c("input","output") := { 
    sp = strsplit(string, " ") 
    list( 
    mapply(function(p,n) paste(p[seq_len(n)], collapse = " "), sp, stop.at.word), 
    mapply(`[`, sp, stop.at.word+1L) 
) 
}] 

# partial result 
head(texts.dt, 4) 

        string word.count stop.at.word  input output 
1:      one   1   0   NA  NA 
2:    two words   2   1   two words 
3:  three words here   3   2 three words here 
4: four useless words here   4   2 four useless words 

---

Alternatywnie:

library(stringi) 
texts.dt[stop.at.word > 0, c("input","output") := { 
    patt = paste0("((\\w+){", stop.at.word-1, "}\\w+) (.*)") 
    m = stri_match(string, regex = patt) 
    list(m[, 2], m[, 4]) 
}] 

Answer 2

alternatywą dla @mapply rozwiązanie Franka korzysta by = 1:nrow(texts.dt) z strsplit i paste:

library(data.table) 
texts.dt[, `:=` (input = paste(strsplit(string, ' ')[[1]][1:stop.at.word][stop.at.word>0], 
           collapse = " "), 
       output = strsplit(string, ' ')[[1]][stop.at.word + 1]), 
     by = 1:nrow(texts.dt)] 

co daje:

> texts.dt 
                 string word.count stop.at.word          input output 
1:              one   1   0            one 
2:            two words   2   1           two words 
3:           three words here   3   2         three words here 
4:         four useless words here   4   2        four useless words 
5:      five useless meaningless words here   5   4    five useless meaningless words here 
6:     six useless meaningless words here just   6   3      six useless meaningless words 
7:    seven useless meaningless words here just to   7   3     seven useless meaningless words 
8:  eigth useless meaningless words here just to fill   8   6 eigth useless meaningless words here just  to 
9:  nine useless meaningless words here just to fill up   9   7 nine useless meaningless words here just to fill 
10: ten useless meaningless words here just to fill up space   10   5   ten useless meaningless words here just 

zamiast korzystania [[1]] można również zawinąć strsplit w unlist następująco: unlist(strsplit(string, ' ')) (zamiast strsplit(string, ' ')[[1]]). Da ci to ten sam rezultat.

---

dwie opcje:

1) z Stringi pakietu:

library(stringi) 
texts.dt[, `:=`(input = paste(stri_extract_all_words(string[stop.at.word>0], 
                simplify = TRUE)[1:stop.at.word], 
           collapse = " "), 
       output = stri_extract_all_words(string[stop.at.word>0], 
               simplify = TRUE)[stop.at.word+1]), 
     1:nrow(texts.dt)] 

2) lub dostosowania this answer:

texts.dt[stop.at.word>0, 
     c('input','output') := tstrsplit(string, 
              split = paste0("(?=(?>\\s+\\S*){", 
                 word.count - stop.at.word, 
                 "}$)\\s"), 
              perl = TRUE) 
     ][, output := sub('(\\w+).*','\\1',output)] 

których oba dają:

> texts.dt 
                 string word.count stop.at.word          input output 
1:              one   1   0           NA  NA 
2:            two words   2   1           two words 
3:           three words here   3   2         three words here 
4:         four useless words here   4   2        four useless words 
5:      five useless meaningless words here   5   4    five useless meaningless words here 
6:     six useless meaningless words here just   6   3      six useless meaningless words 
7:    seven useless meaningless words here just to   7   3     seven useless meaningless words 
8:  eigth useless meaningless words here just to fill   8   6 eigth useless meaningless words here just  to 
9:  nine useless meaningless words here just to fill up   9   7 nine useless meaningless words here just to fill 
10: ten useless meaningless words here just to fill up space   10   5   ten useless meaningless words here just 

Answer 3

dt[, `:=`(input = sub(paste0('((\\s*\\w+){', stop.at.word, '}).*'), '\\1', string), 
      output = sub(paste0('(\\s*\\w+){', stop.at.word, '}\\s*(\\w+).*'), '\\2', string)) 
    , by = stop.at.word][] 
#              string word.count stop.at.word 
# 1:              one   1   0 
# 2:            two words   2   1 
# 3:           three words here   3   2 
# 4:         four useless words here   4   2 
# 5:      five useless meaningless words here   5   4 
# 6:     six useless meaningless words here just   6   3 
# 7:    seven useless meaningless words here just to   7   3 
# 8:  eigth useless meaningless words here just to fill   8   6 
# 9:  nine useless meaningless words here just to fill up   9   7 
#10: ten useless meaningless words here just to fill up space   10   5 
#           input output 
# 1:            one 
# 2:           two words 
# 3:         three words here 
# 4:        four useless words 
# 5:    five useless meaningless words here 
# 6:      six useless meaningless words 
# 7:     seven useless meaningless words 
# 8: eigth useless meaningless words here just  to 
# 9: nine useless meaningless words here just to fill 
#10:   ten useless meaningless words here just 

Nie jestem pewien, czy rozumiem logikę output będącego niczym na pierwszej linii, ale trywialny fix, czy rzeczywiście potrzebne, jest pozostawione OP.