3 Commits

Author	SHA1	Message	Date
Pedro López-Cabanillas	0d86c111a7	fix for gb18030 encoding test ... The gb18030 test fails, reporting the sample text as Macedonian language encoded with windows-1251. This is because 1: the Macedonian language model is very optimistic and reports high confidence with the given sample, and 2: the original sample text is extremely short and lacks language variety. By simply adding a good amount of real Chinese literature to the sample file, the test no longer fails. This text has been extracted from Wikipedia: https://zh.wikipedia.org/wiki/%E4%B8%AD%E5%8D%8E%E4%BA%BA%E6%B0%91%E5%85%B1%E5%92%8C%E5%9B%BD	2025-06-07 23:33:41 +00:00
Jehan	0d70a36910	Adding some more test files for Russian and Chinese. ... Taken from: https://zh.wikipedia.org/wiki/EUC https://ru.wikipedia.org/wiki/КОИ-8 And rename a file s/utf8.txt/utf-8.txt/ to fix a build test.	2015-11-18 19:27:38 +01:00
Jehan	0efcdfa546	Reorganize test files in language subdirectories. ... I realize that the language information a text has been written in is very important since it would completely change the character distribution. Our test files should take this into account, and we should create several test files in different languages for encoding used in various languages.	2015-11-17 21:12:39 +01:00