2a4d8d890e
Now the UTF-8 prober would not only detect valid UTF-8, but would also
detect the most probable language. Using the data generated 2 commits
away, this works very well.
This is still basic and will require even more improvements. In
particular, now the nsUTF8Prober should return an array of ("UTF-8",
language) couple candidate. And nsMBCSGroupProber should itself forward
these candidates as well as other candidates from other multi-byte
detectors. This way, the public-facing API would get more probable
candidates, in case the algorithm is slightly wrong.
Also the UTF-8 confidence is currently stupidly high as soon as we
consider it to be right. We should likely weigh it with language
detection (in particular, if no language is detected, this should
severely weigh down UTF-8 detection; not to 0, but high enough to be a
fallback in case no other encoding+lang is valid and low enough to give
chances to other good candidate couples.
|
||
|---|---|---|
| build-mac | ||
| doc | ||
| script | ||
| src | ||
| test | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| AUTHORS | ||
| CMakeLists.txt | ||
| COPYING | ||
| INSTALL | ||
| README.md | ||
| uchardet.doap | ||
| uchardet.pc.in | ||
uchardet
uchardet is an encoding detector library, which takes a sequence of bytes in an unknown character encoding without any additional information, and attempts to determine the encoding of the text. Returned encoding names are iconv-compatible.
uchardet started as a C language binding of the original C++ implementation of the universal charset detection library by Mozilla. It can now detect more charsets, and more reliably than the original implementation.
Supported Languages/Encodings
- International (Unicode)
- UTF-8
- UTF-16BE / UTF-16LE
- UTF-32BE / UTF-32LE / X-ISO-10646-UCS-4-34121 / X-ISO-10646-UCS-4-21431
- Arabic
- ISO-8859-6
- WINDOWS-1256
- Bulgarian
- ISO-8859-5
- WINDOWS-1251
- Chinese
- ISO-2022-CN
- BIG5
- EUC-TW
- GB18030
- HZ-GB-2312
- Croatian:
- ISO-8859-2
- ISO-8859-13
- ISO-8859-16
- Windows-1250
- IBM852
- MAC-CENTRALEUROPE
- Czech
- Windows-1250
- ISO-8859-2
- IBM852
- MAC-CENTRALEUROPE
- Danish
- ISO-8859-1
- ISO-8859-15
- WINDOWS-1252
- English
- ASCII
- Esperanto
- ISO-8859-3
- Estonian
- ISO-8859-4
- ISO-8859-13
- ISO-8859-13
- Windows-1252
- Windows-1257
- Finnish
- ISO-8859-1
- ISO-8859-4
- ISO-8859-9
- ISO-8859-13
- ISO-8859-15
- WINDOWS-1252
- French
- ISO-8859-1
- ISO-8859-15
- WINDOWS-1252
- German
- ISO-8859-1
- WINDOWS-1252
- Greek
- ISO-8859-7
- WINDOWS-1253
- Hebrew
- ISO-8859-8
- WINDOWS-1255
- Hungarian:
- ISO-8859-2
- WINDOWS-1250
- Irish Gaelic
- ISO-8859-1
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
- Italian
- ISO-8859-1
- ISO-8859-3
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
- Japanese
- ISO-2022-JP
- SHIFT_JIS
- EUC-JP
- Korean
- ISO-2022-KR
- EUC-KR / UHC
- Lithuanian
- ISO-8859-4
- ISO-8859-10
- ISO-8859-13
- Latvian
- ISO-8859-4
- ISO-8859-10
- ISO-8859-13
- Maltese
- ISO-8859-3
- Polish:
- ISO-8859-2
- ISO-8859-13
- ISO-8859-16
- Windows-1250
- IBM852
- MAC-CENTRALEUROPE
- Portuguese
- ISO-8859-1
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
- Romanian:
- ISO-8859-2
- ISO-8859-16
- Windows-1250
- IBM852
- Russian
- ISO-8859-5
- KOI8-R
- WINDOWS-1251
- MAC-CYRILLIC
- IBM866
- IBM855
- Slovak
- Windows-1250
- ISO-8859-2
- IBM852
- MAC-CENTRALEUROPE
- Slovene
- ISO-8859-2
- ISO-8859-16
- Windows-1250
- IBM852
- MAC-CENTRALEUROPE
- Spanish
- ISO-8859-1
- ISO-8859-15
- WINDOWS-1252
- Swedish
- ISO-8859-1
- ISO-8859-4
- ISO-8859-9
- ISO-8859-15
- WINDOWS-1252
- Thai
- TIS-620
- ISO-8859-11
- Turkish:
- ISO-8859-3
- ISO-8859-9
- Vietnamese:
- VISCII
- Windows-1258
- Others
- WINDOWS-1252
Installation
Debian/Ubuntu/Mint
apt-get install uchardet libuchardet-dev
Mageia
urpmi libuchardet libuchardet-devel
Fedora
dnf install uchardet uchardet-devel
Gentoo
emerge uchardet
Mac
brew install uchardet
or
port install uchardet
Windows
Binary packages are provided in Fedora and Msys2 repositories. There may exist other pre-built packages but I am not aware of them. Nevertheless the library is very easily and quickly compilable under Windows as well, so finding a binary package is not necessary. Some did it successfully with the CMake Windows installer and MinGW. It should be possible to use MinGW-w64 instead of MinGW, in particular to build both 32 and 64-bit DLL libraries).
Note also that it is very easily cross-buildable (for instance from a GNU/Linux machine; crossroad may help, this is what we use in our CI).
Build from source
Releases are available from: https://www.freedesktop.org/software/uchardet/releases/
If you prefer a development version, clone the git repository:
git clone https://gitlab.freedesktop.org/uchardet/uchardet.git
The source can be browsed at: https://gitlab.freedesktop.org/uchardet/uchardet
cmake .
make
make install
Build with flatpak-builder
Here is a working "module" section to include in your Flatpak's json manifest:
"modules": [
{
"name": "uchardet",
"buildsystem": "cmake",
"builddir": true,
"config-opts": [ "-DCMAKE_INSTALL_LIBDIR=lib" ],
"sources": [
{
...
}
]
}
]
Usage
Command Line
uchardet Command Line Tool
Version 0.0.7
Authors: BYVoid, Jehan
Bug Report: https://gitlab.freedesktop.org/uchardet/uchardet/-/issues
Usage:
uchardet [Options] [File]...
Options:
-v, --version Print version and build information.
-h, --help Print this help.
Library
See uchardet.h
History
As said in introduction, this was initially a project of Mozilla to allow better detection of page encodings, and it used to be part of Firefox. If not mistaken, this is not the case anymore (probably because nowadays most websites better announce their encoding, and also UTF-8 is much more widely spread).
Techniques used by universalchardet are described at https://www-archive.mozilla.org/projects/intl/universalcharsetdetection
It is to be noted that a lot has changed since the original code, yet the base concept is still around, basing detection not just on encoding rules, but importantly on analysis of character statistics in languages.
Original code by Mozilla does not seem to be found anymore anywhere, but it's probably not too far from the initial commit of this repository.
Mozilla code was extracted and packaged into a standalone library under
the name uchardet by BYVoid in 2011, in a personal repository.
Starting 2015, I (i.e. Jehan) started contributing, "standardized"
the output to be iconv-compatible, added various encoding/language
support and streamlined generation of sources for new support of
encoding/languages by using texts from Wikipedia as statistics source on
languages through Python scripts. Then I soon became co-maintainer.
In 2016, uchardet became a freedesktop project.
Related Projects
Some of these are bindings of uchardet, others are forks of the same
initial code, which has diverged over time, others are native port in
other languages.
This list is not exhaustive and only meant as point of interest. We
don't follow the status for these projects.
- R-uchardet R binding on CRAN
- python-chardet Python port
- ruby-rchardet Ruby port
- juniversalchardet Java port of universalchardet
- jchardet Java port of chardet
- nuniversalchardet C# port of universalchardet
- nchardet C# port of chardet
- uchardet-enhanced A fork of mozilla universalchardet
- rust-uchardet Rust language binding of uchardet
- libchardet Another C/C++ API wrapping Mozilla code.
Used by
- mpv for subtitle detection
- Tepl
- Nextcloud IOS app
- Codelite
- QtAV
- …
Licenses
- Mozilla Public License Version 1.1
- GNU General Public License, version 2.0 or later.
- GNU Lesser General Public License, version 2.1 or later.
See the file COPYING for the complete text of these 3 licenses.
Code of Conduct
The uchardet project is hosted by freedesktop.org
and as such follows its code of conduct. In other words, it means we
will treat anyone with respect and expect anyone to do the same.
Please read freedesktop.org Code of Conduct.
In case of any problem regarding abusive behavior in uchardet project, please contact the maintainer (Jehan) or create a bug report (possibly private if needed).