What is Unicode? — Universal Character Encoding Explained

Definition

Unicode is a universal character encoding standard that assigns a unique number (called a code point) to every character, symbol, and script used in writing systems worldwide. Unlike older encoding systems that could only represent a limited set of characters, Unicode aims to encode all written characters from all human languages, plus technical symbols, punctuation, and emoji.

Unicode is maintained by the Unicode Consortium and, as of version 16.0, defines over 154,000 characters covering 168 modern and historical scripts.

ASCII vs Unicode

The first 128 code points of Unicode (U+0000 to U+007F) are identical to ASCII, ensuring backward compatibility.

UTF-8 vs UTF-16 vs UTF-32

Unicode defines different encoding forms that convert code points to byte sequences:

When to Use Each

• UTF-8 is the dominant encoding on the web — over 98% of all web pages use UTF-8. It is the default for HTML5, JSON, XML, and most modern APIs.
• UTF-16 is used internally by JavaScript strings, Java, and Windows APIs. Files encoded as UTF-16 are common on Windows systems.
• UTF-32 is rarely used in practice due to its 4× overhead compared to UTF-8 for Latin text. It is mainly used for internal processing where fixed-width characters simplify indexing.

Code Points and Planes

Code Points

A code point is a unique number assigned to each Unicode character, written in hex as U+XXXX (for BMP) or U+XXXXX (for supplementary planes):

Planes

Unicode is divided into 17 planes, each containing 65,536 code points:

The BMP (Basic Multilingual Plane) covers the vast majority of characters used in daily communication. Characters in supplementary planes (above U+FFFF) require surrogate pairs in UTF-16.

Surrogates

In UTF-16, characters outside the BMP (code points U+10000 to U+10FFFF) are encoded using two 16-bit units called a surrogate pair:

• High surrogate: U+D800 to U+DBFF
• Low surrogate: U+DC00 to U+DFFF

For the emoji 😀 (U+1F600):

• High surrogate: 0xD83D
• Low surrogate: 0xDE00

The surrogate range (U+D800–U+DFFF) is reserved exclusively for this purpose and no valid Unicode character is assigned there. This is a common source of bugs when software incorrectly handles UTF-16 strings.

BOM (Byte Order Mark)

The Byte Order Mark (BOM) is the Unicode character U+FEFF placed at the beginning of a text file to indicate:

1. Endianness — whether the text is little-endian or big-endian (relevant for UTF-16 and UTF-32)
2. Encoding — which Unicode encoding form is used

BOM Controversy

In UTF-8, the BOM is technically unnecessary (byte order is fixed) but some Windows applications (like Notepad) add it. The BOM can cause problems in Unix/Linux tools, JSON parsers, and HTTP headers. Many developers prefer UTF-8 without BOM.

Normalization Forms

Unicode allows some characters to be represented in multiple ways. For example, "é" can be:

• A single code point: U+00E9 (precomposed)
• Two code points: U+0065 (e) + U+0301 (combining acute accent)

Normalization ensures consistent representation:

Why Normalization Matters

• Password hashing — "é" typed on different systems may hash to different values
• File names — macOS uses NFD, Linux uses NFC; two "same" file names may be different bytes
• Database lookups — searches can miss results due to normalization mismatches
• Collation — sorting rules differ by language and normalization form

Emoji Encoding

Emoji are encoded as Unicode characters ranging from U+1F300 to U+1FAFF and beyond. They introduce several unique encoding behaviors:

Variation Selectors

Some characters can be displayed as either text or emoji:

• U+FE0E — Variation Selector-15 (text style)
• U+FE0F — Variation Selector-16 (emoji style)

☕ (U+2615)           → text or emoji depends on platform
☕️ (U+2615 + U+FE0F)  → explicitly emoji style

ZWJ Sequences

The Zero Width Joiner (U+200D) combines multiple emoji into a single glyph:

👨 + U+200D + 💻 = 👨‍💻 (man technologist)
👩 + U+200D + 🎓 = 👩‍🎓 (woman graduate)

Fitzpatrick Modifiers

Skin tone modifiers (U+1F3FB to U+1F3FF) are appended to emoji:

👋 + U+1F3FD = 👋🏽 (waving hand: medium skin tone)

Flag Emoji

Country flags are encoded as Regional Indicator Symbols — two-letter codes:

US → 🇺 (U+1F1FA) + 🇸 (U+1F1F8) = 🇺🇸
JP → 🇯 (U+1F1EF) + 🇵 (U+1F1F5) = 🇯🇵

Common Developer Pitfalls

Pitfall 1: Assuming string.length equals character count

JavaScript strings use UTF-16, so "😀".length === 2 (surrogate pair). Use the spread operator or Array.from() for character count.

Pitfall 2: Case-insensitive comparison

Unicode case folding is complex and locale-dependent. "ß".toUpperCase() === "SS" in German. Use localeCompare() with sensitivity options.

Pitfall 3: Grapheme clusters

What looks like one "character" (e.g., 👨‍💻 or é́ with multiple combining marks) may be multiple code points. Use a grapheme cluster library like Intl.Segmenter for proper text segmentation.

Pitfall 4: BOM in data files

If a JSON file starts with BOM bytes (EF BB BF), standard parsers may fail. Always check for BOM when reading files.

Pitfall 5: Assuming byte count equals character count

"Hello".length is 5, but "Hëllö".length is also 5 — however, the UTF-8 byte count differs (5 vs 7). Never use character count for storage estimation.

Pitfall 6: Normalization mismatch in databases

If your database compares strings using NFC but user input comes in NFD, lookups will fail. Normalize all strings to NFC before storing.

LangStop Encoding Tools

• URL Encoder — Encode Unicode strings for URLs
• URL Decoder — Decode percent-encoded Unicode
• Base64 Encoder — Encode Unicode text to base64