js-codepage

Codepages are character encodings. In many contexts, single- or double-byte character sets are used in lieu of Unicode encodings. The codepages map between characters and numbers.

Setup

In node:

var cptable = require('codepage');

In the browser:

<script src="cptable.js"></script>
<script src="cputils.js"></script>

Alternatively, use the full version in the dist folder:

<script src="cptable.full.js"></script>

The complete set of codepages is large due to some Double Byte Character Set encodings. A much smaller file that only includes SBCS codepages is provided in this repo (sbcs.js), as well as a file for other projects (cpexcel.js)

If you know which codepages you need, you can include individual scripts for each codepage. The individual files are provided in the bits/ directory. For example, to include only the Mac codepages:

<script src="bits/10000.js"></script>
<script src="bits/10006.js"></script>
<script src="bits/10007.js"></script>
<script src="bits/10029.js"></script>
<script src="bits/10079.js"></script>
<script src="bits/10081.js"></script>

All of the browser scripts define and append to the cptable object. To rename the object, edit the JSVAR shell variable in make.sh and run the script.

The utilities functions are contained in cputils.js, which assumes that the appropriate codepage scripts were loaded.

The script will manipulate module.exports if available . This is not always desirable. To prevent the behavior, define DO_NOT_EXPORT_CODEPAGE.

Usage

Most codepages are indexed by number. To get the Unicode character for a given codepoint, use the dec property:

var unicode_cp10000_255 = cptable[10000].dec[255]; // ˇ

To get the codepoint for a given character, use the enc property:

var cp10000_711 = cptable[10000].enc[String.fromCharCode(711)]; // 255

There are a few utilities that deal with strings and buffers:

var 汇总 = cptable.utils.decode(936, [0xbb,0xe3,0xd7,0xdc]);
var buf =  cptable.utils.encode(936,  汇总);
var sushi= cptable.utils.decode(65001, [0xf0,0x9f,0x8d,0xa3]); // 🍣
var sbuf = cptable.utils.encode(65001, sushi);

cptable.utils.encode(CP, data, ofmt) accepts a String or Array of characters and returns a representation controlled by ofmt:

• Default output is a Buffer (or Array) of bytes (integers between 0 and 255)
• If ofmt == 'str', return a binary String (byte i is o.charCodeAt(i))
• If ofmt == 'arr', return an Array of bytes

cptable.utils.decode(CP, data) accepts a byte String or Array of numbers or Buffer and returns a JS string.

Known Excel Codepages

A much smaller script, including only the codepages known to be used in Excel, is available under the name cpexcel. It exposes the same variable cptable and is suitable as a drop-in replacement when the full codepage tables are not needed.

In node:

var cptable = require('codepage/dist/cpexcel.full');

Rolling your own script

The make.sh script in the repo can take a manifest and generate JS source.

Usage:

$ bash make.sh path_to_manifest output_file_name JSVAR

where

• JSVAR is the name of the exported variable (generally cptable)
• output_file_name is the output file (cpexcel.js, cptable.js, ...)
• path_to_manifest is the path to the manifest file.

The manifest file is expected to be a CSV with 3 columns:

<codepage number>,<source>,<size>

If a source is specified, it will try to download the specified file and parse. The file format is expected to follow the format from the unicode.org site. The size should be 1 for a single-byte codepage and 2 for a double-byte codepage. For mixed codepages (which use some single- and some double-byte codes), the script assumes the mapping is a prefix code and generates efficient JS code.

Generated scripts only include the mapping. cat a mapping with cputils.js to produce a complete script like cpexcel.full.js.

Building the complete script

This script uses voc. The script to build the codepage tables and the JS source is codepage.md, so building involves voc codepage.md.

Generated Codepages

The complete list of codepages can be found in the file pages.csv.

Some codepages are easier to implement algorithmically. Since those character tables are not generated, there is no corresponding entry (they are "magic").

unicode.org refers to the Unicode Consortium Public Mappings, a database of various mappings between Unicode characters and respective character sets. The tables are processed by a few scripts in the build process.

IBM refers to the IBM coded character set database. Even though IBM uses a different numbering scheme from Windows, the IBM numbers are used when there is no conflict. The tables are manually generated from the symbol manifests.

Windows 7 refers to direct inspection of Windows 7 machines using .NET class System.Text.Encoding. The enclosed MakeEncoding.cs C# program brute-forces code pages. MakeEncoding.cs deviates from unicode.org in some cases. When they map a given code to different characters, unicode.org value is used. When unicode.org does not prescribe a value, MakeEncoding.cs value is used.

NLS refers to the National Language Support files supplied in various versions of Windows. In older versions of Windows (like Windows 98) these files followed the name pattern CP_#.NLS, but newer versions use the name pattern C_#.NLS.

Testing

make test will run the nodejs-based test.

To run the in-browser tests, run a local server and go to the ctest directory. make ctestserv will start a python SimpleHTTPServer server on port 8000.

To update the browser artifacts, run make ctest.

Sources

• Unicode Consortium Public Mappings
• Windows Code Page Enumeration
• Windows Code Page Identifiers
• IBM Coded Character Sets
• ISO/IEC 2022 / ECMA-35
• International Register of Coded Character Sets To Be Used With Escape Sequences
• Japanese Character Encoding for Internet Messages

License

Please consult the attached LICENSE file for details. All rights not explicitly granted by the Apache 2.0 license are reserved by the Original Author.

Badges