coffee/uchardet
1
0
Fork 0
mirror of https://gitlab.freedesktop.org/uchardet/uchardet.git synced 2025-12-07 17:26:41 +08:00
Code Issues Packages Projects Releases Wiki Activity

script, src, test: add IBM865 support for Danish.

Browse Source 
Newly added IBM865 charset (for Norwegian) can also be used for Danish

By the way, I fixed `script/charsets/ibm865.py` as Danish uses the 'da'
ISO 639-1 code by the way, not 'dk' (which is sometimes used for other
codes for Denmark, such as ISO 3166 country code and internet TLD) but
not for the language itself.

For the test, adding some text from the top article of the day on the
Danish Wikipedia, which was about Jimi Hendrix. And that's cool! 🎸 ;-)
This commit is contained in:
Jehan 2022-11-30 19:45:17 +01:00
parent 5aa628272b
commit 388777be51
8 changed files with 423 additions and 286 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

384
script/BuildLangModelLogs/LangDanishModel.log
View File

@ -1,158 +1,256 @@
= Logs of language model for Danish (da) = = Logs of language model for Danish (da) =
- Generated by BuildLangModel.py - Generated by BuildLangModel.py
- Started: 2016-02-19 17:53:58.564190 - Started: 2022-11-30 19:37:01.097250
- Maximum depth: 4 - Maximum depth: 2
- Max number of pages: 100 - Max number of pages: 200
== Parsed pages == == Parsed pages ==
Forside (revision 2692411) Forside (revision 10000691)
16. februar (revision 6877446) Hans Magnus Enzensberger (revision 11341046)
17. februar (revision 8454583) 28. november (revision 9410945)
1878 (revision 8280505) Golfkrigen (revision 11144370)
19. februar (revision 8206479) 29. november (revision 6877900)
1922 (revision 8455105) 8. december (revision 10277754)
1926 (revision 8425271) Det Konservative Folkeparti (revision 11313857)
1942 (revision 8443554) 1990 (revision 11340072)
1945 (revision 8448461) 1940 (revision 11263756)
1948 (revision 8454392) Angolas håndboldlandshold (damer) (revision 11331888)
1985 (revision 8409096) Skjoldvulkan (revision 10870812)
2. verdenskrig (revision 8433181) Casper & Mandrilaftalen (revision 11221713)
23. oktober (revision 6877825) 26. november (revision 10617630)
26. oktober (revision 7849938) Døde i 2022 (revision 11343986)
3C 273 (revision 8443798) Vikingetidens rustning og våben (revision 11332607)
A-bus (revision 8427319) Middelaldercentret (revision 11339897)
Aktuelle begivenheder (revision 8440596) Ruslands invasion af Ukraine 2022 (revision 11335164)
B-52 Stratofortress (revision 8422571) Saddam Hussein (revision 11002258)
Borgerkrigen i Syrien (revision 8447763) The Jimi Hendrix Experience (revision 10497780)
Boutros Boutros-Ghali (revision 8453935) Færøerne (revision 11333678)
Brasilien (revision 8452750) 27. november (revision 9745974)
Cusco (region) (revision 7693764) Thomas Vinterberg (revision 11234643)
Danmark (revision 8451178) Anwar Ibrahim (revision 11342876)
Danmark i Eurovision Song Contest (revision 8453514) Mandatområdet i Palæstina (revision 11341286)
Dansk (sprog) (revision 8455750) Kunst (revision 11336917)
Dansk Melodi Grand Prix 2016 (revision 8452164) Afrikamesterskabet i håndbold 2022 (kvinder) (revision 11341917)
Dobbeltmordet på Peter Bangs Vej (revision 8334648) Dansk (sprog) (revision 11313509)
Encyklopædi (revision 8446641) Sergej Sjojgu (revision 11309097)
Eritrea-sagen (revision 8452285) Fernando Gomes (revision 11340427)
Eurovision Song Contest 2014 (revision 8445804) Folketinget (revision 11330485)
Eurovision Song Contest 2016 (revision 8453588) 15. januar (revision 10515606)
Flygtningekrisen i Europa 2015 (revision 8452286) Rock and Roll Hall of Fame (revision 8408189)
Fonograf (revision 8177165) Thomas Edison (revision 11052704)
Formel 1 (revision 8450846) Ukraine (revision 11334630)
Formel 1 2016 (revision 8456463) 1947 (revision 11252357)
Frederik 6. (revision 8438503) 1937 (revision 11303923)
Første observation af gravitationsbølger (revision 8451269) IC4 (revision 11317878)
Grammofon (revision 8375093) Jimi Hendrix (revision 11341476)
Guadalcanal (revision 7796248) Ismail Sabri Yaakob (revision 11105534)
Harper Lee (revision 8456583) Okipa-ceremonien (revision 11340589)
Hartkorn (revision 8437552) SI-præfiks (revision 11332802)
IC4 (revision 8446402) Sporvejsmuseet Skjoldenæsholms historie (revision 11338275)
IC4-sagen (revision 8434463) Irak (revision 11255676)
Islamisk Stat (revision 8439228) Woodstockfestivalen (revision 11226413)
Jonathan Leunbach (revision 8452603) Nikolaj Lie Kaas (revision 11322663)
Juliane Marie af Braunschweig-Wolfenbüttel (revision 8437957) Torben Rechendorff (revision 11342962)
Kaliumklorid (revision 8452216) Folketingsvalget 2022 (revision 11339557)
Kejserriget Japan (revision 8044942) Kherson (revision 11314559)
Kevin Magnussen (revision 8455302) Keltere (revision 11318773)
København (revision 8427847) Little Richard (revision 11226619)
LIGO (revision 8451266) Invasion (revision 10307980)
Latinamerika (revision 7692181) Tate Gallery (revision 8312688)
Leonid Hurwicz (revision 8445727) 24. januar (revision 10441562)
Lighthouse X (revision 8452940) Hans Christian Ægidius (revision 9773029)
Linkoban (revision 8455879) Slaget ved Irpin (1321) (revision 11230064)
Machu Picchu (revision 8406907) Auschwitz (revision 11310714)
Matador (tv-serie) (revision 8454648) Jazz fusion (revision 11223082)
Middelaldercentret (revision 8449194) Lutsk (revision 11248429)
Nobelprisen (revision 8409809) Planetarium (revision 11266837)
Nykøbing Falster (revision 8452825) Bibliothèque nationale de France (revision 11055813)
Nyligt afdøde (revision 8456580) Digtsamling (revision 10585337)
Overvågning (revision 8455039) Kenneth Gøtterup (revision 11027437)
Panorama (foto) (revision 8448393) Straf (revision 11007456)
Peru (revision 8437485) 1716 (revision 11339928)
Peter Lauritsen (revision 8456097) Kamel (revision 11285016)
Professor (revision 8415451) Amnesti (revision 10831621)
Renault F1 (revision 8450843) Zulu Royal (revision 10969220)
S-bus (revision 8455589) Stephen Roche (revision 11239346)
Salomonøerne (revision 8238961) 13. december (revision 10768225)
Slaget om Belgien (1940) (revision 8430013) Enhed (politisk parti) (revision 10158693)
Slaget om Guadalcanal (revision 7762887) The Everly Brothers (revision 10865882)
Slaget om Henderson Field (revision 8445480) 3. november (revision 9423371)
Slaget om Iwo Jima (revision 8145239) Annelise Gotfredsen (revision 11306090)
Soldiers of Love (Lighthouse X-sang) (revision 8452929) Virtual International Authority File (revision 8702589)
Solen (revision 8276478) Europæiske Fællesskab (revision 10868689)
Stillehavskrigen (revision 8430649) Væringer (revision 11331002)
Stockholm (revision 8358042) Rom (revision 11341285)
Søslaget ved Guadalcanal (revision 7772812) Decentralisering (revision 11154770)
Thomas Edison (revision 8282441) Kreml (Moskva) (revision 11045482)
Togulykken ved Bad Aibling (revision 8455364) Folketingsvalget 1994 (revision 11266325)
Topografi (revision 6886168) 28. december (revision 6878014)
USA (revision 8448088) Østjyllands Storkreds (revision 11201505)
United States Army (revision 8401635) Bruxelles (revision 10802416)
United States Marine Corps (revision 8401667) Erik Haunstrup Clemmensen (revision 10627614)
Vestallierede (revision 6961443) Hviderussere (revision 10750673)
Wikimedia (revision 8263252) Hvidmelet Gåsefod (revision 11317723)
Wikipedia (revision 8267051) Mario Draghi (revision 11302527)
Zikavirus (revision 8454832) Folketingsvalget 1968 (revision 11300317)
1. februar (revision 8404985) Skudår (revision 10360386)
10. februar (revision 6877431) 1921 (revision 11303917)
11. februar (revision 6877433) Rundkørsel (revision 11103019)
12. februar (revision 6877437) Valerij Zaluzjnyj (revision 11335164)
13. februar (revision 6877438) Angrebet på Pearl Harbor (revision 11309782)
14. februar (revision 6877441) Folketingsmedlemmer valgt i 2007 (revision 11187293)
1497 (revision 7369489) Ingeniørvidenskab (revision 9816520)
15. februar (revision 7329463) Vikinger (revision 11327511)
1560 (revision 7874693) Martin Luther King (revision 11320659)
1568 (revision 7369703) 1757 (revision 11186195)
1620 (revision 7423903) Dieseltogsæt (revision 8177984)
1688 (revision 7367090) El-værk (revision 11334293)
18. februar (revision 6877450) Soul (revision 11283982)
John McVie (revision 11040471)
Botswanas håndboldlandshold (herrer) (revision 11333322)
1971 (revision 11243510)
Rana Hussein (revision 11266594)
DR (revision 11342995)
Ewan McGregor (revision 11331681)
Eliane Paulo (revision 10589121)
Zepto- (revision 11332802)
København (revision 11336925)
Gallien (revision 9984925)
Augustoprøret (revision 11234324)
1991 (revision 11250037)
Afledte SI-enheder (revision 11097802)
Gemeinsame Normdatei (revision 11281765)
Litteraturvidenskab (revision 10931878)
Thorvald Stauning (revision 11107677)
Afrikamesterskabet i håndbold 2018 (mænd) (revision 11131830)
Folkeforbundet (revision 11315450)
Readymades (revision 10932287)
Al Anbar (revision 9458175)
2007 (revision 11250033)
Varieté (revision 10934358)
Damaskus (revision 11030795)
Palæstina (revision 11311424)
1569 (revision 10832219)
Pædagog (revision 11251603)
Carina Christensen (revision 11073847)
Vest-Tyskland (revision 10580737)
20. november (revision 6877846)
Tessa Jowell (revision 11225831)
Hillerød (revision 11317306)
Påskekrisen (1920) (revision 11287865)
Grad (vinkelmål) (revision 9624298)
Kvinde (revision 11333939)
1931 (revision 11236350)
Afrikaans (revision 11080347)
Den Store Danske Encyklopædi (revision 11301417)
22. juni (revision 10375853)
Automatic Train Control (revision 10619401)
Luc Montagnier (revision 11162267)
Reprise Records (revision 11081843)
1966 (revision 11336105)
Prosa (skriveform) (revision 11236012)
Michael af Rumænien (revision 10819975)
Mykolajiv (revision 11236676)
Khmelnytskyj oblast (revision 11188686)
Sierra Leones håndboldlandshold (herrer) (revision 11333322)
1969 (revision 11340081)
H. Edvard Hass (revision 10348478)
Københavns Idrætspark (revision 9400386)
Sanna Nielsen (revision 11315712)
19. maj (revision 7148596)
Patricia Schumann (revision 10952761)
Torstenssonfejden (revision 11326728)
International Standard Name Identifier (revision 10880739)
Bent Mejding (revision 11335462)
Afdeling Q (revision 11279134)
Alfred Bindslev (revision 10398140)
Sakser (revision 9042633)
Folketingsmedlemmer valgt i 1998 (revision 11213304)
1996 (revision 11229565)
1 (tal) (revision 9378579)
Farrah Fawcett (revision 10977527)
Google+ (revision 10469085)
1530 (revision 10865231)
De største helte (revision 10737852)
Afrikamesterskabet i håndbold 1974 (mænd) (revision 11018946)
1902 (revision 11217211)
ISO 639-3 (revision 10880691)
1974 (revision 11336110)
Dansk fonologi (revision 11226101)
Europa (revision 11149054)
Sovemedicin (revision 11327388)
Slotsbryggen (Nykøbing Falster) (revision 11005548)
Olieraffinaderi (revision 11322152)
Slaget ved Stiklestad (revision 11261889)
Rolling Stone (revision 11267586)
Jørgen Hald (revision 10296412)
Nikolaj Coster-Waldau (revision 11228953)
Aserbajdsjan (revision 11297538)
Kultstatus (revision 7820159)
Al Kut (revision 9425606)
Library of Congress Control Number (revision 8316539)
Rwandas håndboldlandshold (herrer) (revision 11333322)
Levon Helm (revision 11317127)
Howard Hughes (revision 11040881)
Wim Kieft (revision 10910953)
Afrikamesterskabet i håndbold 2016 (mænd) (revision 11018957)
24. februar (revision 10755036)
Iværksætter (revision 10972242)
1992 (revision 11303945)
Internationalt Standardbognummer (revision 11037702)
Afrikamesterskabet i håndbold 2014 (mænd) (revision 11018956)
En mand kommer hjem (revision 10737861)
Jamaica (revision 11243987)
Henitjesk (revision 11328921)
August (revision 11210562)
== End of Parsed pages == == End of Parsed pages ==
- Wikipedia parsing ended at: 2016-02-19 17:56:42.162636 - Wikipedia parsing ended at: 2022-11-30 19:41:17.518631
53 characters appeared 1301488 times. 60 characters appeared 1532370 times.
First 30 characters: First 31 characters:
[ 0] Char e: 15.272749345364689 % [ 0] Char e: 15.035728968852169 %
[ 1] Char r: 8.48482659847805 % [ 1] Char r: 8.617892545534042 %
[ 2] Char n: 7.695652975670924 % [ 2] Char n: 7.618264518360449 %
[ 3] Char t: 6.977014002434137 % [ 3] Char t: 6.856503324915001 %
[ 4] Char a: 6.780469739252302 % [ 4] Char a: 6.475133290262796 %
[ 5] Char i: 6.164636170291236 % [ 5] Char i: 6.3714377076032545 %
[ 6] Char s: 6.0942551909814 % [ 6] Char s: 6.279488635251278 %
[ 7] Char d: 5.953493232361728 % [ 7] Char d: 5.919523352715076 %
[ 8] Char l: 5.076650725938311 % [ 8] Char l: 5.094722553952375 %
[ 9] Char o: 4.883026197706011 % [ 9] Char o: 4.86860223053179 %
[10] Char g: 4.012253666572415 % [10] Char g: 3.8343872563414845 %
[11] Char k: 3.232607599916403 % [11] Char k: 3.3303314473658454 %
[12] Char m: 3.0863135119186653 % [12] Char m: 3.2096034247603384 %
[13] Char f: 2.701600014752345 % [13] Char f: 2.608247355403721 %
[14] Char v: 2.13970470722742 % [14] Char v: 2.342188896937424 %
[15] Char b: 1.982423195603801 % [15] Char u: 1.9602967951604378 %
[16] Char u: 1.8339777239590376 % [16] Char b: 1.9047619047619049 %
[17] Char p: 1.5789619266562582 % [17] Char p: 1.5793183108518178 %
[18] Char h: 1.3433085821767086 % [18] Char h: 1.45728512043436 %
[19] Char ø: 0.8730775850411222 % [19] Char ø: 0.8954103773892728 %
[20] Char y: 0.859938777768216 % [20] Char æ: 0.7449897870618715 %
[21] Char å: 0.7699648402443973 % [21] Char å: 0.7295235484902471 %
[22] Char æ: 0.7208671920140639 % [22] Char y: 0.6777736447463732 %
[23] Char j: 0.644108896893402 % [23] Char j: 0.666418684782396 %
[24] Char c: 0.5698093259407694 % [24] Char c: 0.5946344551250677 %
[25] Char w: 0.11087309295206717 % [25] Char w: 0.12248999915164091 %
[26] Char z: 0.05309307500338075 % [26] Char z: 0.06571519933175407 %
[27] Char x: 0.032424424965885205 % [27] Char x: 0.045354581465311905 %
[28] Char é: 0.032193919575132464 % [28] Char é: 0.021926819240783886 %
[29] Char q: 0.012139950579644223 % [29] Char ó: 0.009592983417842949 %
[30] Char q: 0.009397208246050236 %
The first 30 characters have an accumulated ratio of 0.9997241618823994. The first 31 characters have an accumulated ratio of 0.9994694492844417.
964 sequences found. 1065 sequences found.
First 512 (typical positive ratio): 0.9968082796759031 First 512 (typical positive ratio): 0.9958348814328518
Next 512 (512-1024): 7.68351302509128e-07 Next 512 (512-1024): 2.6103356239028435e-06
Rest: 3.903127820947816e-17 Rest: 3.268948339453948e-05
- Processing end: 2016-02-19 17:56:42.304278 - Processing end: 2022-11-30 19:41:17.605842

2
script/charsets/ibm865.py
View File

@ -45,7 +45,7 @@ aliases = ['CP865', '865', 'CSIBM865']
language = \ language = \
{ {
'complete': [ 'no', 'dk' ], 'complete': [ 'no', 'da' ],
'incomplete': [] 'incomplete': []
} }

2
script/langs/da.py
View File

@ -50,7 +50,7 @@ code = 'da'
# ASCII characters are also used in French. # ASCII characters are also used in French.
use_ascii = True use_ascii = True
# The charsets we want to support and create data for. # The charsets we want to support and create data for.
charsets = ['ISO-8859-15', 'ISO-8859-1', 'WINDOWS-1252'] charsets = ['ISO-8859-15', 'ISO-8859-1', 'WINDOWS-1252', 'IBM865' ]
## Optional Properties ## ## Optional Properties ##

174
src/LangModels/LangDanishModel.cpp
View File

@ -41,7 +41,7 @@
/** /**
* Generated by BuildLangModel.py * Generated by BuildLangModel.py
* On: 2016-02-19 17:56:42.163975 * On: 2022-11-30 19:41:17.519380
**/ **/
/* Character Mapping Table: /* Character Mapping Table:
@ -67,18 +67,18 @@ static const unsigned char Iso_8859_15_CharToOrderMap[] =
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */ SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */
NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */ NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */
SYM, 4, 15, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */ SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */
17, 29, 1, 6, 3, 16, 14, 25, 27, 20, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */ 17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */
SYM, 4, 15, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */ SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */
17, 29, 1, 6, 3, 16, 14, 25, 27, 20, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */ 17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 8X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 8X */
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 9X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 9X */
SYM,SYM,SYM,SYM,SYM,SYM, 39,SYM, 39,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */ SYM,SYM,SYM,SYM,SYM,SYM, 50,SYM, 50,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */
SYM,SYM,SYM,SYM, 53, 42,SYM,SYM, 54,SYM,SYM,SYM, 55, 56, 57,SYM, /* BX */ SYM,SYM,SYM,SYM, 60, 57,SYM,SYM, 61,SYM,SYM,SYM, 43, 43, 62,SYM, /* BX */
58, 33, 40, 35, 32, 21, 22, 38, 41, 28, 49, 45, 59, 34, 60, 50, /* CX */ 39, 32, 44, 53, 36, 21, 20, 42, 38, 28, 63, 46, 64, 35, 47, 52, /* CX */
43, 47, 51, 36, 52, 61, 30,SYM, 19, 62, 37, 44, 31, 46, 63, 48, /* DX */ 31, 48, 58, 29, 49, 59, 34,SYM, 19, 65, 37, 66, 33, 40, 55, 41, /* DX */
64, 33, 40, 35, 32, 21, 22, 38, 41, 28, 49, 45, 65, 34, 66, 50, /* EX */ 39, 32, 44, 53, 36, 21, 20, 42, 38, 28, 67, 46, 68, 35, 47, 52, /* EX */
43, 47, 51, 36, 52, 67, 30,SYM, 19, 68, 37, 44, 31, 46, 69, 70, /* FX */ 31, 48, 58, 29, 49, 59, 34,SYM, 19, 69, 37, 70, 33, 40, 55, 71, /* FX */
}; };
/*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */ /*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */
@ -88,18 +88,18 @@ static const unsigned char Iso_8859_1_CharToOrderMap[] =
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */ SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */
NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */ NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */
SYM, 4, 15, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */ SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */
17, 29, 1, 6, 3, 16, 14, 25, 27, 20, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */ 17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */
SYM, 4, 15, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */ SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */
17, 29, 1, 6, 3, 16, 14, 25, 27, 20, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */ 17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 8X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 8X */
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 9X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 9X */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */ SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */
SYM,SYM,SYM,SYM,SYM, 42,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* BX */ SYM,SYM,SYM,SYM,SYM, 57,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* BX */
71, 33, 40, 35, 32, 21, 22, 38, 41, 28, 49, 45, 72, 34, 73, 50, /* CX */ 39, 32, 44, 53, 36, 21, 20, 42, 38, 28, 72, 46, 73, 35, 47, 52, /* CX */
43, 47, 51, 36, 52, 74, 30,SYM, 19, 75, 37, 44, 31, 46, 76, 48, /* DX */ 31, 48, 58, 29, 49, 59, 34,SYM, 19, 74, 37, 75, 33, 40, 55, 41, /* DX */
77, 33, 40, 35, 32, 21, 22, 38, 41, 28, 49, 45, 78, 34, 79, 50, /* EX */ 39, 32, 44, 53, 36, 21, 20, 42, 38, 28, 76, 46, 77, 35, 47, 52, /* EX */
43, 47, 51, 36, 52, 80, 30,SYM, 19, 81, 37, 44, 31, 46, 82, 83, /* FX */ 31, 48, 58, 29, 49, 59, 34,SYM, 19, 78, 37, 79, 33, 40, 55, 80, /* FX */
}; };
/*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */ /*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */
@ -109,61 +109,83 @@ static const unsigned char Windows_1252_CharToOrderMap[] =
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */ CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */ SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */
NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */ NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */
SYM, 4, 15, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */ SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */
17, 29, 1, 6, 3, 16, 14, 25, 27, 20, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */ 17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */
SYM, 4, 15, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */ SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */
17, 29, 1, 6, 3, 16, 14, 25, 27, 20, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */ 17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */
SYM,ILL,SYM, 84,SYM,SYM,SYM,SYM,SYM,SYM, 39,SYM, 85,ILL, 86,ILL, /* 8X */ SYM,ILL,SYM, 81,SYM,SYM,SYM,SYM,SYM,SYM, 50,SYM, 43,ILL, 82,ILL, /* 8X */
ILL,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, 39,SYM, 87,ILL, 88, 89, /* 9X */ ILL,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, 50,SYM, 43,ILL, 83, 84, /* 9X */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */ SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */
SYM,SYM,SYM,SYM,SYM, 42,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* BX */ SYM,SYM,SYM,SYM,SYM, 57,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* BX */
90, 33, 40, 35, 32, 21, 22, 38, 41, 28, 49, 45, 91, 34, 92, 50, /* CX */ 39, 32, 44, 53, 36, 21, 20, 42, 38, 28, 85, 46, 86, 35, 47, 52, /* CX */
43, 47, 51, 36, 52, 93, 30,SYM, 19, 94, 37, 44, 31, 46, 95, 48, /* DX */ 31, 48, 58, 29, 49, 59, 34,SYM, 19, 87, 37, 88, 33, 40, 55, 41, /* DX */
96, 33, 40, 35, 32, 21, 22, 38, 41, 28, 49, 45, 97, 34, 98, 50, /* EX */ 39, 32, 44, 53, 36, 21, 20, 42, 38, 28, 89, 46, 90, 35, 47, 52, /* EX */
43, 47, 51, 36, 52, 99, 30,SYM, 19,100, 37, 44, 31, 46,101,102, /* FX */ 31, 48, 58, 29, 49, 59, 34,SYM, 19, 91, 37, 92, 33, 40, 55, 93, /* FX */
};
/*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */
static const unsigned char Ibm865_CharToOrderMap[] =
{
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,RET,CTR,CTR,RET,CTR,CTR, /* 0X */
CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR,CTR, /* 1X */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* 2X */
NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,NUM,SYM,SYM,SYM,SYM,SYM,SYM, /* 3X */
SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 4X */
17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,SYM, /* 5X */
SYM, 4, 16, 24, 7, 0, 13, 10, 18, 5, 23, 11, 8, 12, 2, 9, /* 6X */
17, 30, 1, 6, 3, 15, 14, 25, 27, 22, 26,SYM,SYM,SYM,SYM,CTR, /* 7X */
42, 33, 28, 44, 36, 39, 21, 42, 94, 46, 38, 52, 47, 95, 36, 21, /* 8X */
28, 20, 20, 49, 34, 58, 96, 97, 98, 34, 33, 19,SYM, 19,SYM,SYM, /* 9X */
32, 35, 29, 37, 48, 48,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* AX */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* BX */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* CX */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* DX */
45, 41, 99, 56,100,101, 57, 54,102,103,104,105,106,107, 51,SYM, /* EX */
SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM,SYM, /* FX */
}; };
/*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */ /*X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 XA XB XC XD XE XF */
/* Model Table: /* Model Table:
* Total sequences: 964 * Total sequences: 1065
* First 512 sequences: 0.9968082796759031 * First 512 sequences: 0.9958348814328518
* Next 512 sequences (512-1024): 0.0031917203240968304 * Next 512 sequences (512-1024): 0.0041324290837536455
* Rest: 3.903127820947816e-17 * Rest: 3.268948339453948e-05
* Negative sequences: TODO * Negative sequences: TODO
*/ */
static const PRUint8 DanishLangModel[] = static const PRUint8 DanishLangModel[] =
{ {
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,3,3,3,2,3,0,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,2,3,3,3,3,2,3,0,0,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,2,2,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,2,2,2,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,2,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,0, 3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,1,2,2,1,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,0,2,3,3,3,3,3,2,2, 3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,3,3,3,0,2,2,3,3,3,3,3,3,2,0,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,0,2,3,3,2,3,3,2,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,2,3,3,2,3,3,2,0,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,0,2,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,0,2,2,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,2,0,2,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,2,0,2,2,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,0,2,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,2,0,2,2,2,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,3,3,3,2,2,0,0, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,0,1,2,3,3,3,3,2,2,2,0,2,
3,3,3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,2,3,3,3,3,3,3,2,2,2,2,3,2, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,2,2,0,0,2,0,0,
3,3,3,3,3,3,3,2,3,3,2,3,3,2,3,2,3,2,3,3,3,3,3,2,2,2,2,2,0,0, 3,3,3,3,3,3,3,2,3,3,2,3,3,3,3,3,2,2,3,3,3,3,3,3,1,2,2,1,2,0,0,
3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,2,3,0, 3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,2,3,2,2,2,2,2,2,
3,3,3,3,3,3,3,2,3,3,3,2,2,3,3,3,3,2,3,3,3,3,3,3,2,2,2,2,2,0, 3,3,3,3,3,3,3,3,3,3,3,2,2,3,3,3,3,2,3,3,3,3,3,3,2,0,2,2,2,2,2,
3,3,3,3,3,3,3,3,3,3,2,2,2,2,2,2,3,2,2,2,2,3,3,3,2,2,0,0,2,0, 3,3,3,3,3,3,3,3,3,3,2,2,3,2,2,3,2,2,2,3,3,3,2,3,2,0,0,0,1,1,0,
3,3,3,3,3,3,3,2,3,3,2,2,2,2,2,3,3,2,2,3,3,3,3,3,2,2,0,0,2,0, 3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,2,3,3,3,2,2,0,2,2,3,2,2,3,0,0,2,
3,3,3,3,3,3,3,3,3,2,3,3,3,3,3,3,2,3,3,2,3,0,2,2,3,2,3,3,0,2, 3,3,3,3,3,3,3,2,3,3,2,2,2,2,2,3,3,2,2,3,3,3,3,3,2,0,2,0,2,2,0,
3,3,3,3,3,3,3,3,3,3,3,2,2,3,3,3,3,3,3,3,2,3,3,2,2,0,2,0,2,0, 3,3,3,3,3,3,3,3,3,3,3,2,2,3,2,3,3,3,3,3,3,3,3,2,2,0,1,0,2,2,0,
3,3,3,3,3,3,2,2,3,3,2,2,3,2,3,2,3,2,2,3,3,3,3,3,2,3,2,2,2,0, 3,3,3,3,3,3,3,2,3,3,1,2,3,2,3,3,2,2,2,3,3,3,3,3,2,3,0,0,2,2,1,
3,3,3,3,2,2,3,3,3,2,3,3,3,2,3,3,0,2,2,2,2,0,0,3,0,0,2,0,0,0, 3,3,3,3,0,2,3,3,3,2,3,3,3,2,3,2,3,2,2,0,0,0,2,3,0,2,1,0,0,0,0,
3,3,3,3,3,2,3,3,3,3,3,3,3,2,3,3,3,3,2,2,2,0,0,0,2,2,2,0,0,0, 2,3,3,3,2,3,3,3,3,3,3,3,3,3,3,2,3,2,2,0,0,0,0,0,2,0,0,0,0,0,0,
3,3,3,3,2,0,3,3,3,2,3,3,2,2,3,3,0,2,2,2,0,0,0,0,0,0,0,0,0,0, 3,3,3,3,0,0,3,3,3,2,2,3,2,2,3,0,3,2,2,0,0,0,0,0,0,0,0,0,0,0,0,
2,3,3,3,0,3,3,3,3,2,3,3,3,3,3,3,2,2,2,0,0,0,0,0,2,0,0,0,0,0, 3,3,3,3,3,2,3,3,3,3,3,3,3,2,3,2,3,3,3,2,0,0,2,2,2,2,2,0,0,0,0,
3,3,2,3,3,3,3,3,3,3,2,2,2,2,2,2,3,2,2,3,3,2,3,2,2,0,0,0,0,0, 3,3,3,2,3,3,3,3,3,3,2,2,2,2,2,3,2,2,2,3,3,2,3,0,2,0,0,0,0,2,0,
3,3,2,3,3,3,2,2,3,3,2,3,2,2,0,2,3,2,3,0,3,0,0,2,3,2,2,0,2,2, 3,3,2,3,3,3,2,2,3,3,2,3,2,2,2,3,2,2,3,0,2,0,3,2,3,0,2,2,2,2,2,
3,2,2,2,3,3,2,2,2,3,0,2,2,2,0,2,2,0,2,0,2,0,0,0,2,2,2,0,0,0, 3,2,2,2,3,3,2,2,2,3,0,2,2,2,0,2,2,0,2,0,2,0,2,2,2,2,2,0,0,0,2,
3,2,2,2,3,3,2,2,0,3,0,2,2,0,0,2,2,2,2,2,2,0,0,2,2,0,2,0,0,0, 3,2,2,2,3,3,2,2,2,3,2,2,2,2,0,2,2,2,2,0,0,0,2,2,0,2,3,0,0,0,0,
3,2,0,2,2,3,2,0,2,2,0,0,2,2,2,2,2,2,2,2,0,0,0,0,2,2,0,0,2,0, 3,2,1,2,2,2,2,2,2,2,0,2,1,2,2,0,0,2,0,0,0,0,2,0,2,2,0,2,0,0,0,
2,3,2,2,2,0,2,2,2,2,2,2,2,0,2,2,0,2,0,0,0,0,0,0,2,0,0,0,0,0, 2,2,3,2,2,0,2,2,2,2,2,0,2,2,2,2,2,1,2,0,0,0,0,0,1,0,2,0,0,0,0,
0,0,0,0,3,2,2,2,2,2,0,0,0,0,2,2,3,0,2,0,0,0,0,0,0,0,0,0,0,2, 0,3,2,2,2,0,2,0,2,0,2,2,0,2,2,0,0,2,2,0,0,0,0,0,0,2,0,0,0,0,0,
0,0,0,0,2,1,0,0,0,0,0,0,0,0,2,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
}; };
@ -171,8 +193,8 @@ const SequenceModel Iso_8859_15DanishModel =
{ {
Iso_8859_15_CharToOrderMap, Iso_8859_15_CharToOrderMap,
DanishLangModel, DanishLangModel,
30, 31,
(float)0.9968082796759031, (float)0.9958348814328518,
PR_TRUE, PR_TRUE,
"ISO-8859-15" "ISO-8859-15"
}; };
@ -181,8 +203,8 @@ const SequenceModel Iso_8859_1DanishModel =
{ {
Iso_8859_1_CharToOrderMap, Iso_8859_1_CharToOrderMap,
DanishLangModel, DanishLangModel,
30, 31,
(float)0.9968082796759031, (float)0.9958348814328518,
PR_TRUE, PR_TRUE,
"ISO-8859-1" "ISO-8859-1"
}; };
@ -191,8 +213,18 @@ const SequenceModel Windows_1252DanishModel =
{ {
Windows_1252_CharToOrderMap, Windows_1252_CharToOrderMap,
DanishLangModel, DanishLangModel,
30, 31,
(float)0.9968082796759031, (float)0.9958348814328518,
PR_TRUE, PR_TRUE,
"WINDOWS-1252" "WINDOWS-1252"
}; };
const SequenceModel Ibm865DanishModel =
{
Ibm865_CharToOrderMap,
DanishLangModel,
31,
(float)0.9958348814328518,
PR_TRUE,
"IBM865"
};

139
src/nsSBCSGroupProber.cpp
View File

@ -110,91 +110,92 @@ nsSBCSGroupProber::nsSBCSGroupProber()
mProbers[32] = new nsSingleByteCharSetProber(&Iso_8859_15DanishModel); mProbers[32] = new nsSingleByteCharSetProber(&Iso_8859_15DanishModel);
mProbers[33] = new nsSingleByteCharSetProber(&Iso_8859_1DanishModel); mProbers[33] = new nsSingleByteCharSetProber(&Iso_8859_1DanishModel);
mProbers[34] = new nsSingleByteCharSetProber(&Windows_1252DanishModel); mProbers[34] = new nsSingleByteCharSetProber(&Windows_1252DanishModel);
mProbers[35] = new nsSingleByteCharSetProber(&Ibm865DanishModel);
mProbers[35] = new nsSingleByteCharSetProber(&Iso_8859_13LithuanianModel); mProbers[36] = new nsSingleByteCharSetProber(&Iso_8859_13LithuanianModel);
mProbers[36] = new nsSingleByteCharSetProber(&Iso_8859_10LithuanianModel); mProbers[37] = new nsSingleByteCharSetProber(&Iso_8859_10LithuanianModel);
mProbers[37] = new nsSingleByteCharSetProber(&Iso_8859_4LithuanianModel); mProbers[38] = new nsSingleByteCharSetProber(&Iso_8859_4LithuanianModel);
mProbers[38] = new nsSingleByteCharSetProber(&Iso_8859_13LatvianModel); mProbers[39] = new nsSingleByteCharSetProber(&Iso_8859_13LatvianModel);
mProbers[39] = new nsSingleByteCharSetProber(&Iso_8859_10LatvianModel); mProbers[40] = new nsSingleByteCharSetProber(&Iso_8859_10LatvianModel);
mProbers[40] = new nsSingleByteCharSetProber(&Iso_8859_4LatvianModel); mProbers[41] = new nsSingleByteCharSetProber(&Iso_8859_4LatvianModel);
mProbers[41] = new nsSingleByteCharSetProber(&Iso_8859_1PortugueseModel); mProbers[42] = new nsSingleByteCharSetProber(&Iso_8859_1PortugueseModel);
mProbers[42] = new nsSingleByteCharSetProber(&Iso_8859_9PortugueseModel); mProbers[43] = new nsSingleByteCharSetProber(&Iso_8859_9PortugueseModel);
mProbers[43] = new nsSingleByteCharSetProber(&Iso_8859_15PortugueseModel); mProbers[44] = new nsSingleByteCharSetProber(&Iso_8859_15PortugueseModel);
mProbers[44] = new nsSingleByteCharSetProber(&Windows_1252PortugueseModel); mProbers[45] = new nsSingleByteCharSetProber(&Windows_1252PortugueseModel);
mProbers[45] = new nsSingleByteCharSetProber(&Iso_8859_3MalteseModel); mProbers[46] = new nsSingleByteCharSetProber(&Iso_8859_3MalteseModel);
mProbers[46] = new nsSingleByteCharSetProber(&Windows_1250CzechModel); mProbers[47] = new nsSingleByteCharSetProber(&Windows_1250CzechModel);
mProbers[47] = new nsSingleByteCharSetProber(&Iso_8859_2CzechModel); mProbers[48] = new nsSingleByteCharSetProber(&Iso_8859_2CzechModel);
mProbers[48] = new nsSingleByteCharSetProber(&Mac_CentraleuropeCzechModel); mProbers[49] = new nsSingleByteCharSetProber(&Mac_CentraleuropeCzechModel);
mProbers[49] = new nsSingleByteCharSetProber(&Ibm852CzechModel); mProbers[50] = new nsSingleByteCharSetProber(&Ibm852CzechModel);
mProbers[50] = new nsSingleByteCharSetProber(&Windows_1250SlovakModel); mProbers[51] = new nsSingleByteCharSetProber(&Windows_1250SlovakModel);
mProbers[51] = new nsSingleByteCharSetProber(&Iso_8859_2SlovakModel); mProbers[52] = new nsSingleByteCharSetProber(&Iso_8859_2SlovakModel);
mProbers[52] = new nsSingleByteCharSetProber(&Mac_CentraleuropeSlovakModel); mProbers[53] = new nsSingleByteCharSetProber(&Mac_CentraleuropeSlovakModel);
mProbers[53] = new nsSingleByteCharSetProber(&Ibm852SlovakModel); mProbers[54] = new nsSingleByteCharSetProber(&Ibm852SlovakModel);
mProbers[54] = new nsSingleByteCharSetProber(&Windows_1250PolishModel); mProbers[55] = new nsSingleByteCharSetProber(&Windows_1250PolishModel);
mProbers[55] = new nsSingleByteCharSetProber(&Iso_8859_2PolishModel); mProbers[56] = new nsSingleByteCharSetProber(&Iso_8859_2PolishModel);
mProbers[56] = new nsSingleByteCharSetProber(&Iso_8859_13PolishModel); mProbers[57] = new nsSingleByteCharSetProber(&Iso_8859_13PolishModel);
mProbers[57] = new nsSingleByteCharSetProber(&Iso_8859_16PolishModel); mProbers[58] = new nsSingleByteCharSetProber(&Iso_8859_16PolishModel);
mProbers[58] = new nsSingleByteCharSetProber(&Mac_CentraleuropePolishModel); mProbers[59] = new nsSingleByteCharSetProber(&Mac_CentraleuropePolishModel);
mProbers[59] = new nsSingleByteCharSetProber(&Ibm852PolishModel); mProbers[60] = new nsSingleByteCharSetProber(&Ibm852PolishModel);
mProbers[60] = new nsSingleByteCharSetProber(&Iso_8859_1FinnishModel); mProbers[61] = new nsSingleByteCharSetProber(&Iso_8859_1FinnishModel);
mProbers[61] = new nsSingleByteCharSetProber(&Iso_8859_4FinnishModel); mProbers[62] = new nsSingleByteCharSetProber(&Iso_8859_4FinnishModel);
mProbers[62] = new nsSingleByteCharSetProber(&Iso_8859_9FinnishModel); mProbers[63] = new nsSingleByteCharSetProber(&Iso_8859_9FinnishModel);
mProbers[63] = new nsSingleByteCharSetProber(&Iso_8859_13FinnishModel); mProbers[64] = new nsSingleByteCharSetProber(&Iso_8859_13FinnishModel);
mProbers[64] = new nsSingleByteCharSetProber(&Iso_8859_15FinnishModel); mProbers[65] = new nsSingleByteCharSetProber(&Iso_8859_15FinnishModel);
mProbers[65] = new nsSingleByteCharSetProber(&Windows_1252FinnishModel); mProbers[66] = new nsSingleByteCharSetProber(&Windows_1252FinnishModel);
mProbers[66] = new nsSingleByteCharSetProber(&Iso_8859_1ItalianModel); mProbers[67] = new nsSingleByteCharSetProber(&Iso_8859_1ItalianModel);
mProbers[67] = new nsSingleByteCharSetProber(&Iso_8859_3ItalianModel); mProbers[68] = new nsSingleByteCharSetProber(&Iso_8859_3ItalianModel);
mProbers[68] = new nsSingleByteCharSetProber(&Iso_8859_9ItalianModel); mProbers[69] = new nsSingleByteCharSetProber(&Iso_8859_9ItalianModel);
mProbers[69] = new nsSingleByteCharSetProber(&Iso_8859_15ItalianModel); mProbers[70] = new nsSingleByteCharSetProber(&Iso_8859_15ItalianModel);
mProbers[70] = new nsSingleByteCharSetProber(&Windows_1252ItalianModel); mProbers[71] = new nsSingleByteCharSetProber(&Windows_1252ItalianModel);
mProbers[71] = new nsSingleByteCharSetProber(&Windows_1250CroatianModel); mProbers[72] = new nsSingleByteCharSetProber(&Windows_1250CroatianModel);
mProbers[72] = new nsSingleByteCharSetProber(&Iso_8859_2CroatianModel); mProbers[73] = new nsSingleByteCharSetProber(&Iso_8859_2CroatianModel);
mProbers[73] = new nsSingleByteCharSetProber(&Iso_8859_13CroatianModel); mProbers[74] = new nsSingleByteCharSetProber(&Iso_8859_13CroatianModel);
mProbers[74] = new nsSingleByteCharSetProber(&Iso_8859_16CroatianModel); mProbers[75] = new nsSingleByteCharSetProber(&Iso_8859_16CroatianModel);
mProbers[75] = new nsSingleByteCharSetProber(&Mac_CentraleuropeCroatianModel); mProbers[76] = new nsSingleByteCharSetProber(&Mac_CentraleuropeCroatianModel);
mProbers[76] = new nsSingleByteCharSetProber(&Ibm852CroatianModel); mProbers[77] = new nsSingleByteCharSetProber(&Ibm852CroatianModel);
mProbers[77] = new nsSingleByteCharSetProber(&Windows_1252EstonianModel); mProbers[78] = new nsSingleByteCharSetProber(&Windows_1252EstonianModel);
mProbers[78] = new nsSingleByteCharSetProber(&Windows_1257EstonianModel); mProbers[79] = new nsSingleByteCharSetProber(&Windows_1257EstonianModel);
mProbers[79] = new nsSingleByteCharSetProber(&Iso_8859_4EstonianModel); mProbers[80] = new nsSingleByteCharSetProber(&Iso_8859_4EstonianModel);
mProbers[80] = new nsSingleByteCharSetProber(&Iso_8859_13EstonianModel); mProbers[81] = new nsSingleByteCharSetProber(&Iso_8859_13EstonianModel);
mProbers[81] = new nsSingleByteCharSetProber(&Iso_8859_15EstonianModel); mProbers[82] = new nsSingleByteCharSetProber(&Iso_8859_15EstonianModel);
mProbers[82] = new nsSingleByteCharSetProber(&Iso_8859_1IrishModel); mProbers[83] = new nsSingleByteCharSetProber(&Iso_8859_1IrishModel);
mProbers[83] = new nsSingleByteCharSetProber(&Iso_8859_9IrishModel); mProbers[84] = new nsSingleByteCharSetProber(&Iso_8859_9IrishModel);
mProbers[84] = new nsSingleByteCharSetProber(&Iso_8859_15IrishModel); mProbers[85] = new nsSingleByteCharSetProber(&Iso_8859_15IrishModel);
mProbers[85] = new nsSingleByteCharSetProber(&Windows_1252IrishModel); mProbers[86] = new nsSingleByteCharSetProber(&Windows_1252IrishModel);
mProbers[86] = new nsSingleByteCharSetProber(&Windows_1250RomanianModel); mProbers[87] = new nsSingleByteCharSetProber(&Windows_1250RomanianModel);
mProbers[87] = new nsSingleByteCharSetProber(&Iso_8859_2RomanianModel); mProbers[88] = new nsSingleByteCharSetProber(&Iso_8859_2RomanianModel);
mProbers[88] = new nsSingleByteCharSetProber(&Iso_8859_16RomanianModel); mProbers[89] = new nsSingleByteCharSetProber(&Iso_8859_16RomanianModel);
mProbers[89] = new nsSingleByteCharSetProber(&Ibm852RomanianModel); mProbers[90] = new nsSingleByteCharSetProber(&Ibm852RomanianModel);
mProbers[90] = new nsSingleByteCharSetProber(&Windows_1250SloveneModel); mProbers[91] = new nsSingleByteCharSetProber(&Windows_1250SloveneModel);
mProbers[91] = new nsSingleByteCharSetProber(&Iso_8859_2SloveneModel); mProbers[92] = new nsSingleByteCharSetProber(&Iso_8859_2SloveneModel);
mProbers[92] = new nsSingleByteCharSetProber(&Iso_8859_16SloveneModel); mProbers[93] = new nsSingleByteCharSetProber(&Iso_8859_16SloveneModel);
mProbers[93] = new nsSingleByteCharSetProber(&Mac_CentraleuropeSloveneModel); mProbers[94] = new nsSingleByteCharSetProber(&Mac_CentraleuropeSloveneModel);
mProbers[94] = new nsSingleByteCharSetProber(&Ibm852SloveneModel); mProbers[95] = new nsSingleByteCharSetProber(&Ibm852SloveneModel);
mProbers[95] = new nsSingleByteCharSetProber(&Iso_8859_1SwedishModel); mProbers[96] = new nsSingleByteCharSetProber(&Iso_8859_1SwedishModel);
mProbers[96] = new nsSingleByteCharSetProber(&Iso_8859_4SwedishModel); mProbers[97] = new nsSingleByteCharSetProber(&Iso_8859_4SwedishModel);
mProbers[97] = new nsSingleByteCharSetProber(&Iso_8859_9SwedishModel); mProbers[98] = new nsSingleByteCharSetProber(&Iso_8859_9SwedishModel);
mProbers[98] = new nsSingleByteCharSetProber(&Iso_8859_15SwedishModel); mProbers[99] = new nsSingleByteCharSetProber(&Iso_8859_15SwedishModel);
mProbers[99] = new nsSingleByteCharSetProber(&Windows_1252SwedishModel); mProbers[100] = new nsSingleByteCharSetProber(&Windows_1252SwedishModel);
mProbers[100] = new nsSingleByteCharSetProber(&Iso_8859_15NorwegianModel); mProbers[101] = new nsSingleByteCharSetProber(&Iso_8859_15NorwegianModel);
mProbers[101] = new nsSingleByteCharSetProber(&Iso_8859_1NorwegianModel); mProbers[102] = new nsSingleByteCharSetProber(&Iso_8859_1NorwegianModel);
mProbers[102] = new nsSingleByteCharSetProber(&Windows_1252NorwegianModel); mProbers[103] = new nsSingleByteCharSetProber(&Windows_1252NorwegianModel);
mProbers[103] = new nsSingleByteCharSetProber(&Ibm865NorwegianModel); mProbers[104] = new nsSingleByteCharSetProber(&Ibm865NorwegianModel);
Reset(); Reset();
} }

2
src/nsSBCSGroupProber.h
View File

@ -40,7 +40,7 @@
#define nsSBCSGroupProber_h__ #define nsSBCSGroupProber_h__
#define NUM_OF_SBCS_PROBERS 104 #define NUM_OF_SBCS_PROBERS 105
class nsCharSetProber; class nsCharSetProber;
class nsSBCSGroupProber: public nsCharSetProber { class nsSBCSGroupProber: public nsCharSetProber {

1
src/nsSBCharSetProber.h
View File

@ -171,6 +171,7 @@ extern const SequenceModel Windows_1258VietnameseModel;
extern const SequenceModel Iso_8859_15DanishModel; extern const SequenceModel Iso_8859_15DanishModel;
extern const SequenceModel Iso_8859_1DanishModel; extern const SequenceModel Iso_8859_1DanishModel;
extern const SequenceModel Windows_1252DanishModel; extern const SequenceModel Windows_1252DanishModel;
extern const SequenceModel Ibm865DanishModel;
extern const SequenceModel Iso_8859_13LithuanianModel; extern const SequenceModel Iso_8859_13LithuanianModel;
extern const SequenceModel Iso_8859_10LithuanianModel; extern const SequenceModel Iso_8859_10LithuanianModel;

5
test/da/ibm865.txt Normal file
View File

@ -0,0 +1,5 @@
Jimi Hendrix (1942-1970) var en amerikansk rockguitarist, sanger og sangskriver.
Han begyndte at spille guitar, da han var femten †r, og efter at have spillet med blandt andet Little Richard dannede han Jimi Hendrix Experience i slutningen af 1966. Denne gruppe fik snart hits med sange som "Hey Joe" og "Purple Haze", og med det tredje album, Electric Ladyland fra 1968, fik gruppen sit store gennembrud. Med flere markante optr‘dener p† tidens store festivaler, heriblandt Woodstock, opn†ede han legendarisk status i rockmusikken, allerede mens han var i live.
Hendrix brugte sin elektriske guitar som elektronisk lydkilde og eksperimenterede med feedback og distortion med udgangspunkt i traditionel rock'n'roll og blues. Hans misbrug af alkohol og narkotika f›rte imidlertid til, at han ›delagde sig selv, og han d›de som blot 27-†rig efter indtagelse af sovepiller.