Korean text analysis with RmecabKo

RmecabKo is a Korean text-analysis layer on top of the MeCab morphological analyzer. The heavy lifting - the native engine and dictionary compilation - comes from RcppMeCab; RmecabKo adds tokenizers that follow the tokenizers contract, curated Korean data, user-dictionary tools, and a handful of analysis helpers. This vignette walks through a full tidy workflow.

Setup

Install the engine and a Korean dictionary once per machine:

install.packages("RcppMeCab")
RcppMeCab::download_dic("ko")
RcppMeCab::set_dic("ko")

Normalizing text

text_normalize() is a pure, dependency-light cleanup step. It composes Unicode (NFC), folds full-width characters, and squashes repeated characters - all of which help the analyzer and downstream matching. It needs no backend:

text_normalize("한국어 분석 ㅋㅋㅋㅋ 정말 재밌어요!!!!")
#> [1] "한국어 분석 ㅋㅋ 정말 재밌어요!!"

A tidy tokenization

The tokenizers take a character vector and return a list of character vectors, so they slot directly into tidytext::unnest_tokens(). The package ships a small demonstration corpus, demo_ko:

demo_ko[1:2]
#>                                                   d1 
#> "한국어 형태소 분석은 텍스트 마이닝의 첫걸음입니다." 
#>                                                   d2 
#>      "오늘 날씨가 정말 좋아서 공원을 오래 걸었어요."

corpus <- tibble(doc = names(demo_ko), text = demo_ko)
tokens <- corpus |>
  unnest_tokens(word, text, token = token_nouns)
head(tokens, 8)
#> # A tibble: 8 × 2
#>   doc   word  
#>   <chr> <chr> 
#> 1 d1    한국어
#> 2 d1    형태소
#> 3 d1    분석  
#> 4 d1    텍스트
#> 5 d1    마이닝
#> 6 d1    걸음  
#> 7 d2    날씨  
#> 8 d2    공원

Even without a working backend, the pre-computed tokenization bundled with the package lets us continue:

tidy_tokens <- if (backend && exists("tokens")) {
  tokens
} else {
  readRDS(system.file("extdata", "demo_ko_tokens.rds", package = "RmecabKo"))
}
count(tidy_tokens, word, sort = TRUE) |> head(6)
#> # A tibble: 6 × 2
#>   word      n
#>   <chr> <int>
#> 1 분석      2
#> 2 사람      2
#> 3 거리      1
#> 4 걸음      1
#> 5 것        1
#> 6 결과      1

Removing stopwords

stopwords_ko is a curated table of Korean function morphemes. Filter by surface form with an anti_join(), or strip whole part-of-speech classes at the tag level with drop_pos:

tidy_tokens |>
  anti_join(data.frame(word = stopwords_ko_words()), by = "word") |>
  count(word, sort = TRUE) |>
  head(6)
#> # A tibble: 6 × 2
#>   word      n
#>   <chr> <int>
#> 1 분석      2
#> 2 사람      2
#> 3 거리      1
#> 4 걸음      1
#> 5 결과      1
#> 6 공원      1

# drop every particle and ending directly during tokenization
token_morph(demo_ko[[2]], drop_pos = stopwords_ko_tags(c("josa", "eomi")),
            simplify = TRUE)
#> [1] "오늘" "날씨" "정말" "좋"   "공원" "오래" "걸"   "."

Keywords and TF-IDF

With a document column in hand, tidytext::bind_tf_idf() gives per-document keyword weights; keywords_tfidf() offers the same without the tidy detour:

tidy_tokens |>
  count(doc, word) |>
  bind_tf_idf(word, doc, n) |>
  arrange(desc(tf_idf)) |>
  head(6)
#> # A tibble: 6 × 6
#>   doc   word       n    tf   idf tf_idf
#>   <chr> <chr>  <int> <dbl> <dbl>  <dbl>
#> 1 d2    공원       1 0.5    2.30  1.15 
#> 2 d2    날씨       1 0.5    2.30  1.15 
#> 3 d9    아이       1 0.5    2.30  1.15 
#> 4 d9    운동장     1 0.5    2.30  1.15 
#> 5 d10   글         1 0.333  2.30  0.768
#> 6 d10   마음       1 0.333  2.30  0.768

keywords_tfidf(demo_ko, div = "nouns", top_n = 2) |> head(6)
#>   doc   word n        tf      idf    tf_idf
#> 1  d1   걸음 1 0.1666667 2.302585 0.3837642
#> 2  d1 마이닝 1 0.1666667 2.302585 0.3837642
#> 3  d2   공원 1 0.5000000 2.302585 1.1512925
#> 4  d2   날씨 1 0.5000000 2.302585 1.1512925
#> 5  d3   도움 1 0.2000000 2.302585 0.4605170
#> 6  d3 자연어 1 0.2000000 2.302585 0.4605170

Sentiment

lexicon_knu() downloads and caches the KNU Korean sentiment lexicon (polarity from -2 to 2). Joining it against tokens yields a per-document sentiment score. The lexicon is distributed under CC BY-NC-SA (Kyungpook National University), so it is fetched on demand rather than bundled; note the NonCommercial clause and review its terms before use.

senti <- lexicon_knu()
tidy_tokens |>
  inner_join(senti[senti$n_words == 1, ], by = "word") |>
  group_by(doc) |>
  summarise(score = sum(polarity))

N-grams, lemmas, and concordances

Morpheme n-grams never bridge a removed stopword:

token_ngrams(demo_ko[[1]], n = 2, div = "nouns", simplify = TRUE)
#> [1] "한국어 형태소" "형태소 분석"   "분석 텍스트"   "텍스트 마이닝"
#> [5] "마이닝 걸음"

token_lemma() recovers the dictionary form of predicates, which keeps inflected verbs and adjectives from scattering in a frequency count:

token_lemma(c("아침을 먹었다", "날씨가 좋았다"))
#> [[1]]
#> [1] "먹다"
#> 
#> [[2]]
#> [1] "좋다"

kwic() shows a keyword in its morpheme context:

kwic(demo_ko, "분석")
#>   doc position             left keyword                  right
#> 1  d1        3    한국어 형태소    분석 은 텍스트 마이닝 의 첫
#> 2  d8        9 하 게 정리 하 면    분석  이 훨씬 쉬워 집니다 .

Teaching the analyzer new words

When MeCab splits a name or neologism you care about, register it once and activate it for the session. This writes to your user data directory, so it is not run here:

dict_add_words(c("은전한닢", "카비봇"), tag = "NNP")
dict_use()
pos("카비봇 출시 소식")
dict_words()