kowikitext.20200920.v3

• normalizer 개선
• 정보력이 없는 section title 이거나 text 일 경우 문서 제거
• 중복된 sub-sections 제거

(v2)

	# doc	# line	# token	# character
train	877754	26794425	178636942	1700089550
dev	4433	130419	842545	8052015
test	4434	134340	913611	8796027
total	886621	27059184	180393098	1716937592

(v3)

	# doc	# line	# token	# character
train	868230 (98.91 %)	14528095 (54.22 %)	107528610 (60.19 %)	1038515570 (61.08 %)
dev	4385 (98.91 %)	76788 (58.87 %)	594986 (70.61 %)	5685526 (70.60 %)
test	4386 (98.91 %)	70774 (52.68 %)	554225 (60.66 %)	5389795 (61.27 %)
total	878101 (99.03 %)	14675657 (54.23 %)	108677821 (60.24 %)	1049590891 (61.13 %)

kowikitext.20200920.v1

• data url : https://dumps.wikimedia.org/kowiki/20200920/
• Corpus size
  • train : 26827837 lines (877754 articles)
  • dev : 130568 lines (4433 articles)
  • test : 134478 lines (4433 articles)

kowikitext.20200920.v2

Remove lines including markup code