KR-WordRank: Unsupervised Korean Word & Keyword Extractor

• pure Python code
• author: Lovit (Hyunjoong Kim)
• reference: Kim, H. J., Cho, S., & Kang, P. (2014). KR-WordRank: An Unsupervised Korean Word Extraction Method Based on WordRank. Journal of Korean Institute of Industrial Engineers, 40(1), 18-33

Keyword extraction

Substring graph substring (min count) substring (max length) .

from krwordrank.word import KRWordRank

min_count = 5   #     (  )
max_length = 10 #   
wordrank_extractor = KRWordRank(min_count=min_count, max_length=max_length)

KR-WordRank PageRank graph ranking (HITS algorithm ). Substring graph node (substrig) graph ranking parameters .

beta = 0.85    # PageRank decaying factor beta
max_iter = 10
texts = ['  ', '  list of str ', ... ]
keywords, rank, graph = wordrank_extractor.extract(texts, beta, max_iter)

Graph ranking (substrings) . '' () tutorials .

for word, r in sorted(keywords.items(), key=lambda x:x[1], reverse=True)[:30]:
        print('%8s:\t%.4f' % (word, r))

  :    229.7889
 :   112.3404
  :    78.4055
  :    37.6247
  :    37.2504
        ....

Python wordcloud package word cloud figure .

Figure (stopwords) passwords . dict {:} .

stopwords = {'', '', '', '', ''}
passwords = {word:score for word, score in sorted(
    keywords.items(), key=lambda x:-x[1])[:300] if not (word in stopwords)}

summarize_with_keywords    .

from krwordrank.word import summarize_with_keywords

keywords = summarize_with_keywords(texts, min_count=5, max_length=10,
    beta=0.85, max_iter=10, stopwords=stopwords, verbose=True)
keywords = summarize_with_keywords(texts) # with default arguments

wordcloud .

pip install wordcloud

wordcloud . font_path . (width, height) (background_color) , generate_from_frequencies() .

from wordcloud import WordCloud

# Set your font path
font_path = 'YOUR_FONT_DIR/truetype/nanum/NanumBarunGothic.ttf'

krwordrank_cloud = WordCloud(
    font_path = font_path,
    width = 800,
    height = 800,
    background_color="white"
)

krwordrank_cloud = krwordrank_cloud.generate_from_frequencies(passwords)

Jupyter notebook %matplotlib inline . .py .

%matplotlib inline
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(10, 10))
plt.imshow(krwordrank_cloud, interpolation="bilinear")
plt.show()

. 

fig.savefig('./lalaland_wordcloud.png')

.

Key-sentence extraction

KR-WordRank >= 1.0.0 key sentence extraction . KR-WordRank TextRank . KR-WordRank keywords . , .

summarize_with_sentences texts KR-WordRank .

from krwordrank.sentence import summarize_with_sentences

texts = [] #  
keywords, sents = summarize_with_sentences(texts, num_keywords=100, num_keysents=10)

keywords KR-WordRank num_keywords dict{str:float} .

{'': 201.02402099523516,
 '': 81.53699026386887,
 '': 40.53709233921311,
 '': 40.43446188536923,
 '': 38.598509495213484,
 '': 23.198810378709844,
 '': 21.810147306627464,
 '': 20.638511587426862,
 '': 20.43744237599688,
 '': 20.324710458174806,
 '': 20.283994278960186,
 '': 19.471356929084546,
 '': 19.06433920013137,
 '': 18.732801785265316,
 ...
}

sents num_sents list of str .

['                    30      ',
 '                            ',
 '                                ',
 '                     ost',
 '                          ',
 '                          ',
 '                    6 7        ',
 '                       ',
 '                                   ',
 '              ']

 .        penalty  .   25  80    . stopwords   .      .              .  `diversity`     . `diversity`         .      .

penalty = lambda x:0 if (25 <= len(x) <= 80) else 1
stopwords = {'', '', '', '', ''}

keywords, sents = summarize_with_sentences(
    texts,
    penalty=penalty,
    stopwords = stopwords,
    diversity=0.5,
    num_keywords=100,
    num_keysents=10,
    verbose=False
)

,, `` stopwords .

{'': 40.43446188536923,
 '': 38.598509495213484,
 '': 23.198810378709844,
 '': 21.810147306627464,
 '': 20.638511587426862,
 '': 20.43744237599688,
 '': 20.324710458174806,
 '': 20.283994278960186,
 '': 19.471356929084546,
 '': 18.732801785265316,
 ...
}

25 ~ 80 .

['        10   ',
 '               ',
 '              ',
 '                ',
 '               ',
 '               ',
 '               ',
 '               ',
 '                 ',
 '           ']

`` penalty .

penalty=lambda x:0 if (25 <= len(x) <= 80 and not '' in x) else 1,
keywords, sents = summarize_with_sentences(
    texts,
    penalty=penalty,
    stopwords = stopwords,
    diversity=0.5,
    num_keywords=100,
    num_keysents=10,
    verbose=False
)

print(sents)

['               ',
 '               ',
 '               ',
 '                 ',
 '                ',
 '               ',
 '                ',
 '16               ',
 '                 ',
 '               ']

key sentence extraction tutorials tutorials krwordrank_keysentence.ipynb .

Setup

pip install krwordrank

tested in

• python 3.5.9
• python 3.7.7

Requirements

• Python >= 3.5
• numpy
• scipy