Korean POS tagging poor performance

[jdoerfler]

SpaCy is having trouble with POS tagging in Korean. When inserting a break point in the code and using the sample sentences from SpaCy documentation, the parser does perfectly, so this may be a bit more insidious of a bug.

The below example should be getting a number of different tokens but instead gets one big one.

실제적인효용은크않을지몰라도
Attributes for Token: 실제적인효용은크않을지몰라도
Tag: ncn+jp+ecs
POS: VERB

[jdoerfler]

Path of least resistance may be switching NLP parser for Korean, which would also require mapping different POS tags back to SpaCy ones. See konlpy module output below.

from konlpy.tag import Kkma
kkma = Kkma()
sentence = "야드에서15야드떨어진곳에서1차로바운드되면서더나쁜습니다."
print(kkma.pos(sentence))

yields:

[('야드', 'NNG'), ('에서', 'JKM'), ('15', 'NR'), ('야드', 'NNG'), ('떨어지', 'VV'), ('ㄴ', 'ETD'), ('곳', 'NNG'), ('에서', 'JKM'), ('1', 'NR'), ('차로', 'NNG'), ('바운드', 'NNG'), ('되', 'XSV'), ('면서', 'ECE'), ('더', 'MAG'), ('나쁘', 'VA'), ('ㄴ', 'ETD'), ('습', 'NNG'), ('니', 'VV'), ('다', 'EFN'), ('.', 'SF')]

cheat sheet for these POS tags and unsure if this is comprehensive:

NNG: Common noun (일반 명사)
JKM: Location particle (장소격 조사)
NR: Numeral (수사)
VV: Verb (동사)
ETD: Adnominal suffix (관형사형 어미)
ECE: Conjunctive ending (연결 어미)
MAG: Adverb (부사)
VA: Descriptive verb (형용사)
XSV: Auxiliary verb stem (보조 동사)
EFN: Final sentence-ending particle (종결 어미)

[jdoerfler]

add konlpy to requirements.txt and test on fresh install