Character Encoding

Overview

A GEDCOM file includes a CHAR tag in the HEAD section that indicates the character encoding used in the file. The valid options as of GEDCOM v5.5.1 are ANSEL, ASCII, UNICODE, and UTF-8. Unfortunately, some programs specify the wrong character encoding, some use an invalid option such as ANSI, and some files are edited by text editors that change the character encoding of the file without changing the CHAR value. For those reasons, Gedcom Publisher detects the character encoding using both the CHAR value and character encoding detection techniques.

Gedcom Publisher opens the GEDCOM file once to detect if the file has a Unicode "byte order mark" (BOM). If Gedcom Publisher detects a Unicode BOM, it ignores the CHAR value and uses the encoding associated with the BOM to read the file.

Gedcom Publisher re-opens the GEDCOM file to read the CHAR tag and several other GEDCOM tags from the HEAD section. If Gedcom Publisher did not find a Unicode BOM, Gedcom Publisher uses an ASCII encoding to read the file for the tag preview.

After reading the CHAR tag, if Gedcom Publisher did not find a Unicode BOM, Gedcom Publisher chooses an encoding based on the CHAR tag value.

If your genealogy program supports writing a GEDCOM file using the UTF-8 encoding, choose that option for the best results with Gedcom Publisher.

When the character encoding in the GEDCOM file is set to "ASCII", Gedcom Publisher will accept characters in the Windows-1252 encoding. Windows-1252 is a superset of ASCII.

Challenges

Unfortunately, there are character encoding issues that make it difficult or impossible to detect the encoding automatically.

GEDCOM file specifies UTF-8, but file is not UTF-8

If a GEDCOM file self-identifies as UTF-8 by including a 1 CHAR UTF-8 record, Gedcom Publisher may or may not be able to detect if the file is actually in some other format. For example, if the file is actually a Windows text file with encoding "Windows 1252", a common text file format on PCs running MS Windows, then Gedcom Publisher cannot tell that the file is not in UTF-8 format. Non-accented characters will display correctly in the resulting book, but many accented characters will not. The solution is to change the Database.CHAR Value property to "ASCII".

ANSEL

ANSEL was an ANSI standard used to encode text, but as of 14 February 2013, the standard has been withdrawn. The Family History Department of the Church of Jesus Christ of Latter-day Saints recommended an extended version of ANSEL for use in GEDCOM files.

Fortunately, modern genealogy software programs are not limited to writing ANSEL-encoded GEDCOM files, and you should configure your software to write in another format, preferably UTF-8. Still, to process files that are encoded in ANSEL, Gedcom Publisher includes support for that encoding. Gedcom Publisher's support is based on the information sources listed below.

The tables below describe how certain ANSEL code points map to Unicode. These tables were constructed based on information from these sources:

1. The GEDCOM Standard, Draft Release 5.5.1, "Appendix C, ANSEL Character Set"
2. The GEDCOM Standard, Release 5.5, "Appendix D, ANSEL Character Set"
3. The GEDCOM Standard, Release 4.0, "Chapter 6, Specification for GEDCOM Character Sets"
4. The Character Name Index on The Unicode Consortium
5. GEDCOM ANSEL Table by Tamura Jones, especially for corrections and examples

The GEDCOM standards listed above were prepared by the Family History Department of the Church of Jesus Christ of Latter-day Saints.

Spacing Characters

Gedcom Publisher does not support LDS extensions "e in middle of line" or "o in middle of line". They are converted to "e" and "o", respectively.

Combining (non-spacing) Characters

ANSEL includes combining characters that modify the following¹ character. In the table below, the Graphic column shows the combining character modifying a dotted circle ◌ (U+25CC).

Please note that per the Unicode Standard, Version 7.0, Chapter 3, D52, "The graphic positioning of a combining character depends on the last preceding base character, unless they are separated by a character that is neither a combining character nor either zero width joiner or zero width non-joiner."

In the table above, the position of the cedilla may be different when it applies to the dotted circle ◌̧ compared to where it appears under a "c" in "ça". In this document, the behavior will vary based on your browser's font choices for "serif" and "sans-serif", and also on your browser's text layout software.

The slash overlay character does not seem to be positioned properly when applied to the dotted circle or the digit zero. I tried multiple fonts and multiple base characters and all combinations produced similar results.