wordsegmenterapproach

[ZhouXunYou]

Link to doc page in question (if any):
https://nlp.johnsnowlabs.com/docs/en/training#wordsegmenterapproach
Name of the Spark NLP feature whose docs need improvement:
Is there a description of the format of the sample data (十|LL 四|RR 不|LL 是|RR 四|LL 十|RR%)
What you think the docs should say:
What's LL and what's RR,it's not described in Part-of-speech tagging

[DevinTDHa]

Hi @ZhouXunYou,

The description is indeed missing, an update to the docs will be pushed soon (I will link the PR here.)

Meanwhile, this is the explanation:

The annotator is based on the paper Chinese Word Segmentation as Character Tagging [1]. Word
segmentation is treated as a tagging problem. Each character is be tagged as on of four
different labels: LL (left boundary), RR (right boundary), MM (middle) and LR (word by
itself). The label depends on the position of the word in the sentence. LL tagged words will
combine with the word on the right. Likewise, RR tagged words combine with words on the left.
MM tagged words are treated as the middle of the word and combine with either side. LR tagged
words are words by themselves.
Example (from [1], Example 3(a) (raw), 3(b) (tagged), 3(c) (translation)):

• 上海 计划 到 本 世纪 末 实现 人均 国内 生产 总值 五千 美元
• 上/LL 海/RR 计/LL 划/RR 到/LR 本/LR 世/LL 纪/RR 末/LR 实/LL 现/RR 人/LL 均/RR 国/LL 内/RR 生/LL 产/RR 总/LL
  值/RR 五/LL 千/RR 美/LL 元/RR
• Shanghai plans to reach the goal of 5,000 dollars in per capita GDP by the end of the
  century.

References:

• [1] Xue, Nianwen. “Chinese Word Segmentation as
  Character Tagging.” International Journal of Computational Linguistics & Chinese Language
  Processing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese
  Language Processing, 2003, pp. 29-48. ACLWeb, https://aclanthology.org/O03-4002.

[github-actions]

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 5 days