A B yte O rder M ark at the front of a text file specifies the type of U nicode T ransformation F ormat that is being used and the little/big endian significance. For example, EF BB BF indicates that the file is UTF-8 encoded. It is known that some utilities and web browsers on NT and XP class platforms will decode files and, with the appropriate font, show you the right glyhs to go with any special Unicode characters cited. What may be a surprise is that utilities like Notepad on Windows XP will decode UTF even if the Byte Order Mark is not present! Doesn't this wreck innocent text files? The answer is no, it works suprisingly well because of the nature of UTF encoding. An ASCII character in the range $00 to $7F is not changed, so many UTF text files can be viewed (or even edited) by utilities that are not aware of UTF. Characters in the range $80 to $7FF are encoded as a sequence of two characters, $800 to $FFF a sequence of 3, and so on up to the the largest character that can be encoded ($7FFFFFFF) which requires a sequence of 6 characters. To decode a file that contains UTF, all bytes are scanned looking for a byte with the high order bit set (1xxxxxxx). The number of bytes in a UTF sequence can be deduced from the number of consecutive one bits at the start of the byte. Thus, 110????? introduces a two byte sequence, 1110???? a three byte sequence and so on. The specification goes on to mandate that nothing should encode to a byte that could be construed as a control code. In fact, UTF encoding adds a lot more redundancy by specifying that second and subsequent bytes start with binary 10. UTF notation is very easy to recognise (by a program) and quite hard to generate by mistake. 110xxxxx 10xxxxxx - is a legal two byte sequence. 1110xxxx 10xxxxxx 10xxxxxx - a three byte sequence. 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx - four bytes 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx - five 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx - six This notation is UTF-8. If you count the number of "x" binary digits that can be encoded by each sequence you will note that a three byte sequence is enough to encode any sixteen bit value. The Windows API is normally restricted to 16 bits - the so called UTF-16 notation. It's enough for any double byte Unicode characters. To get the UTF-16 code from the UTF-8 just shift the bits into position. 110xxxx 10yyyyyy decodes to 00000xxxxyyyyyy 1110xxx 10yyyyyy 10zzzzzz decodes to xxxyyyyyyzzzzzz The Unicode character for the Greek letter beta β is UTF-16 code hex $03B2 binary 00000 011 10110010 and this is encoded to a two byte UTF-8 sequence binary 110 01110 10 110010 or hex $CE $B2 If you look at this file with IE6 on Windows 98 or Notepad on Windows XP you will see the correct Greek glyph. IE6 knows about UTF encoded XML files. XP Notepad knows about UTF! P.W.H September 2003 © RedTitan Technology 2003 UTF for fun and profit. BOM Bytes 00 00 FE FF FF FE 00 00 FE FF FF FE EF BB BF Encoding Form UTF-32, big-endian UTF-32, little-endian UTF-16, big-endian UTF-16, little-endian UTF-8 There are a large number of text editors that able to show you regular ASCII characters. Newer products are able to handle UNICODE characters. Special sequences at the beginning of a text file are used by Unicode aware utilities to help support cross platform editing. utf.xml rev 1 September 2003