Character encoding

Lore & Background

Early character encodings originated with optical or electrical telegraphy and early computers, representing only a subset of characters, often restricted to upper case letters, numerals, and limited punctuation.

Reader's Guide

The history of character codes illustrates the evolving need for machine-mediated character-based symbolic information over a distance. Morse code, introduced in the 1840s, used four symbols of variable length. Herman Hollerith invented punch card data encoding in the late 19th century. The U.S. The compromise solution that developed into Unicode broke the assumption that each character should directly correspond to a particular sequence of bits, instead mapping characters to abstract code points represented in variable-length encodings.

Did You Know?

• The earliest well-known electrically transmitted character code was Morse code, introduced in the 1840s.
• Herman Hollerith invented punch card data encoding in the late 19th century to analyze census data.

The Architecture of Symbolic Representation

Character encoding is fundamentally a mapping convention: it assigns a specific numeric value to every symbol in a writing system so that a machine can recognize, store, transmit, and transform that symbol. The scope of what gets encoded extends well beyond the letters and digits of natural languages. A well-designed code space also reserves positions for control characters, whitespace markers, and other functional symbols that carry meaning outside of ordinary prose. Encodings have even been formalized for constructed languages, demonstrating that the principle is not limited to any single linguistic tradition. The individual numeric values that make up an encoding are called code points, and the full collection of those values is referred to as either a code space or a code page. This terminology matters because it frames encoding not as a single fixed table but as a structured, addressable region of numbers. Once a character has been reduced to its code point, it becomes a unit of data that any computer system can manipulate—copied across a network, written to disk, or converted into a different encoding altogether. In this sense, character encoding is the foundational layer that makes all higher-level text processing possible.

From Telegraph Keys to Punch Cards

The story of machine-readable character codes begins long before electronic computers. When electrical telegraphy arrived, those ideas were adapted to the new medium. Morse code, introduced in the 1840s, used just four building blocks (short signal, long signal, short space, long space) to build variable-length sequences. Though later transmitted by machinery, it was often keyed by hand and read by ear, a practice that endures in amateur radio and aviation. Donald Murray modified it in 1901, and the CCITT standardized it as International Telegraph Alphabet No. 2 in 1930. Meanwhile, Herman Hollerith's punch-card system, invented in the late nineteenth century to tabulate census data, encoded information through the presence and position of holes.

The ASCII Decade and the Fragmentation Problem

By the late 1950s, the need for a common seven-bit character standard was pressing. The U.S. Army Signal Corps had defined Fieldata in 1959, a six-or-seven-bit code that addressed modern concerns like machine collation of letters and digits, but it proved short-lived and fell short of its ambitions. Notably, at least one Fieldata committee member, W. F. Leubbert, also sat on the ASCII committee, ensuring continuity of thinking. The 1967 revision added lower-case letters and resolved lingering control-code ambiguities, and ASCII67 was adopted broadly across industry. Yet the landscape remained fragmented. Other vendors like UNIVAC I ran their own six-bit codes.

The Convergence on Unicode and the Web's Default

The 1980s presented researchers with a genuine dilemma. On one side, accommodating the full breadth of world scripts demanded more bits per character. On the other, the majority of computer users at the time worked with the relatively compact Latin alphabet, and those extra bits represented a significant waste of scarce and expensive computing resources. This tension drove the search for a universally interchangeable encoding that could scale without penalizing smaller character sets. Unicode emerged as the answer: a well-defined, extensible system that ultimately replaced most earlier character encodings. Its variable-length implementations, particularly UTF-8 and UTF-16, allowed the encoding to be efficient for Latin text while still covering every script in the code space. In application programs and operating-system tasks, both UTF-8 and UTF-16 remain popular choices, reflecting the fact that different contexts favor different trade-offs between byte efficiency and internal processing convenience.