4.6. Representation of Numbers on the Computer#
These notes are a work in progress. If you spot any errors or have suggestions, please let us know—your feedback is valuable to us.
In these notes, we go over how numbers are represented on a computer. In making these notes, I relied heavely on the notes found here.
Another good resource is fond the Numerical Methods textbook from Berkeley.
While I have not read this book, the contents look interesting and relevent to these notes, Software, Environments, and Tools Bits and Bugs: A Scientific and Historical Review on Software Failures in Computational Science .
Let’s get straight to a major result of these notes, roundoff error. Here is an example, consider 1.05 - 1.00, which is equal to 0.05, exaclty. Let’s do this on the computer?
1.05 - 1.00
0.050000000000000044
You see the result is not exaclty 0.05. We can even check this with a True/False conditional.
1.05 - 1.00 == 0.05
False
There you go, it returned False. This is roundoff error in action. Roundoff error is a consequence that you cannot represent the real number line exactly on a computer.
Here is a pro-tip. When you need to compare if two numbers are equal, you are often better off using a conditional to check if the absolute value of their difference is less than a threshold. Here is an example code.
result = 1.05 - 1.0
threshold = 1e-11
if abs(result - 0.05) <= threshold:
print("these numbers are the same")
else:
print("these number are not the same")
these numbers are the same
Computers store real numbers using a fixed number of bits, typically following the IEEE 754 standard of 32 bits (single precision) or 64 bits (double precision). Because a finite number of bits can only represent a discrete set of values, it is impossible to map the infinite continuum of real numbers exactly to digital memory. Instead, real numbers are approximated using a floating-point system.
This constraint introduces several practical consequences:
Discreteness: The digital number line consists of gaps. Most real numbers cannot be represented exactly and must be rounded to the nearest representable floating-point value.
Range Limits: There are strict upper and lower bounds on the numbers that can be stored. Exceeding these limits results in overflow (magnitudes too large to store, resulting in \(\pm\infty\)) or underflow (magnitudes too small to resolve, flush to zero).
Round-off Error: Basic arithmetic operations—such as addition, subtraction, multiplication, and division—can be inexact because the true mathematical result must be rounded to fit the floating-point format.
4.6.1. Machine Epsilon#
In Python, we can use numpy to inspect the properties of different floating-point types and demonstrate how limited precision leads to numerical errors.
Machine epsilon (\(\epsilon_{mach}\)) is the smallest positive number such that \(1.0 + \epsilon \neq 1.0\). Here is code to investigate machine epsilon in NumPy and the float information from the system.
import sys
import numpy as np
# get Python float information from system
print("Python float information from system.")
print(sys.float_info)
Python float information from system.
sys.float_info(max=1.7976931348623157e+308, max_exp=1024, max_10_exp=308, min=2.2250738585072014e-308, min_exp=-1021, min_10_exp=-307, dig=15, mant_dig=53, epsilon=2.220446049250313e-16, radix=2, rounds=1)
eps = np.finfo(dtype=np.float64).eps
print("Numpy machine epsilon", eps)
Numpy machine epsilon 2.220446049250313e-16
print("one plus machine epsilon should not be equal to 1")
1 + eps == 1
one plus machine epsilon should not be equal to 1
np.False_
print("one plus, say, half machine epsilon should be equal to 1")
1 + eps/2 == 1
one plus, say, half machine epsilon should be equal to 1
np.True_
# More ways to get machine epsilons
eps32 = np.finfo(np.float32).eps
print(f"Machine Epsilon (float32): {eps32}")
# Get epsilon for float64 (Double Precision)
eps64 = np.finfo(np.float64).eps
print(f"Machine Epsilon (float64): {eps64}")
Machine Epsilon (float32): 1.1920928955078125e-07
Machine Epsilon (float64): 2.220446049250313e-16
4.6.2. Binary Numbers#
To understand how computers handle real numbers, we first look at the decimal system (base 10) we use every day. In this system, any value is represented as a weighted sum of powers of 10. For instance, the number 103.2 can be decomposed as follows
Computing systems, however, rely on the binary system (base 2), which uses only the digits 0 and 1. In binary, each digit serves as a coefficient for a power of 2. We use the subscript \(binary\) to distinguish these values from decimal numbers. For example, the binary integer \(101_{binary}\) maps to \(1\cdot 2^2 + 0 \cdot 2^1 + 1 \cdot 2^0 = 5\). This logic extends to fractional values using negative powers of 2, such as \(10.1_{binary}\), which represents
To further illustrate this conversion, consider the binary value \(101.11_{binary}\). This converts to decimal by summing the following powers
Concept Check 1 What is the decimal equivalent of the binary number \(11.01_{binary}\)?
Click to reveal the solution
The decimal equivalent is 3.25. By expanding the binary digits into their respective powers of 2, we get
Conversely, converting from decimal to binary involves decomposing a number into the largest possible powers of 2. For the integer part, we subtract the largest power of 2 that fits into the number and repeat with the remainder. For the fractional part, we can use the “multiplication by 2” method where we multiply the fraction by 2 and the resulting integer (0 or 1) becomes the next binary digit.
Concept Check 2 What is the binary representation of the decimal number 6.375?
Click to reveal the solution
The binary representation is 110.011.
Step 1: Integer part (6)
Step 2: Fractional part (0.375)
\(0.375 \times 2 = 0.75\) (Integer is 0)
\(0.75 \times 2 = 1.5\) (Integer is 1)
\(0.5 \times 2 = 1.0\) (Integer is 1)
Combining these results gives \(110.011_{binary}\).
# Example of catastrophic cancellation or precision loss
a = 1.0
b = 1e-16
# In float64, 1.0 + 1e-16 is often represented as just 1.0
# because 1e-16 is smaller than the machine epsilon for float64 (~2.22e-16)
print(f"1.0 + 1e-16 == 1.0: {a + b == a}")
# Summing many small numbers can lead to significant drift
small_val = 0.0000001
sum_val = sum([small_val] * 10000000)
print(f"Sum of ten 0.1s: {sum_val}")
print(f"Exactly 1.0? {sum_val == 1.0}")
1.0 + 1e-16 == 1.0: True
Sum of ten 0.1s: 1.0
Exactly 1.0? True
2**(-2) + 2**(-3)
0.375
4.6.3. Normalized Binary Scientific Notation#
Just as scientific notation in the decimal system handles very large or very small numbers, computers utilize a binary version of this concept. In decimal scientific notation, the decimal point is shifted so that only one non-zero digit remains to the left, which is then multiplied by the appropriate power of 10. For example,
Binary scientific notation follows the same logic using a base of 2. A binary number is considered normalized when exactly one non-zero digit sits to the left of the binary point. Because the only non-zero digit in binary is 1, every non-zero normalized binary number begins in the form \(1.\text{xxx}\dots\).
For example, the decimal number 6.375 corresponds to \(110.011_{\text{binary}}\). To normalize this value, the binary point is shifted two places to the left to isolate the leading 1. Because the point moved two positions, the representation becomes
Conversely, for a number smaller than 1, such as \(0.00101_{\text{binary}}\), the binary point shifts three places to the right until it sits immediately behind the first 1, yielding
Concept Check 3 How would the decimal number 0.375 (which is \(0.011_{\text{binary}}\)) be represented in normalized binary scientific notation?
Click to reveal the solution
The normalized form is \(1.1 \times 2^{-2}\). To normalize \(0.011_{\text{binary}}\), the binary point must move two places to the right so that it sits immediately after the first 1:
Move one place right: \(0.11 \times 2^{-1}\)
Move two places right: \(1.1 \times 2^{-2}\)
The resulting normalized form is
This normalized structure serves as the foundation for the IEEE 754 Standard, the universal method for storing real numbers in modern computing systems. By standardizing this notation, a computer only needs to store three specific pieces of data:
The Sign Bit: 0 for positive numbers, 1 for negative numbers.
The Exponent: The power of 2, stored using a fixed “bias.”
The Mantissa (Significand): The fractional bits following the leading 1 (the leading 1 itself is omitted in memory to save space).
Shifting the binary point is mathematically equivalent to factoring out a specific power of 2 to ensure only one digit remains to the left of the point. To see why \(110_{\text{binary}}\) is equal to \(1.1 \times 2^2\), consider its expanded form,
Factoring out the largest power, \(2^2\), from the entire expression leaves
The expression inside the parentheses represents the binary value \(1.1_{\text{binary}}\), meaning the normalized scientific notation can be written as
Concept Check 4 Using the factoring method, how would you normalize \(1011_{\text{binary}}\)?
Click to reveal the solution
First, write out the powers for \(1011_{\text{binary}}\) as
Next, factor out the largest power, \(2^3\), to obtain
The value inside the parentheses evaluates to \(1.011_{\text{binary}}\), giving the final normalized scientific notation
4.6.3.1. Single Precision Numbers (32-bit)#
In the IEEE 754 standard, single precision numbers use 32 bits to represent a real value. This bit-string is divided into three functional components:
Sign Bit (1 bit): The most significant bit determines the sign of the number. A 0 represents a positive number, while a 1 represents a negative number.
Exponent (8 bits): These bits determine the magnitude of the number.
Mantissa or Significand (23 bits): These bits represent the fractional part of the normalized binary number (the bits to the right of the binary point).
There is a unique “catch” to the exponent field. To represent both very large and very small numbers, we need the ability to have negative exponents. Rather than dedicating a separate sign bit within the exponent field itself, the standard uses a bias.
For single precision, the bias is 127. This means the value stored in the 8-bit exponent field is actually \(E + 127\), where \(E\) is the actual exponent. An 8-bit field can represent integers from 0 to 255; by using a bias, the range of 0–255 actually maps to exponents from roughly -126 to +127.
Example: If your normalized power is \(2^5\), the stored exponent is \(127 + 5 = 132\).
Example: If your normalized power is \(2^{-2}\), the stored exponent is \(127 - 2 = 125\).
Concept Check 4: If a single precision floating point number has the binary value 10000010 in its exponent field, what is the actual exponent (\(E\)) used in the calculation?
A) \(3\)
B) \(5\)
C) \(130\)
D) \(-2\)
Click to reveal the solution
Correct Answer: A
The binary value 10000010 is equal to \(128 + 2 = 130\) in decimal.
To find the actual exponent, we subtract the bias:
$\(130 - 127 = 3\)\(
The actual exponent is \)3$.
4.6.3.2. Double Precision Numbers (64-bit)#
Double precision follows the exact same logic but expands the bit count to provide significantly higher range and more decimal digits of accuracy (roughly 15–17 decimal digits compared to single precision’s 6–9).
The 64 bits are distributed as follows:
Sign Bit: 1 bit
Exponent: 11 bits (with a bias of 1023)
Mantissa: 52 bits
By increasing the exponent to 11 bits, double precision can represent numbers as large as roughly \(1.8 \times 10^{308}\), whereas single precision caps out around \(3.4 \times 10^{38}\).
4.6.4. Round-Off Error#
Round-off error is a direct consequence of the finite representation of numbers within a computer. Because digital memory relies on a fixed number of bits (such as 32-bit single precision or 64-bit double precision), it cannot represent the infinite continuum of real numbers exactly. Instead, infinitely repeating or non-terminating binary fractions must be rounded to the nearest representable floating-point value.
This behavior is highly analogous to trying to write the fraction \(1/3\) as a terminating decimal; we are forced to truncate it to an approximation like \(0.3333\). In the binary system used by computers, even simple decimal fractions like \(0.1\) or \(0.055\) become infinitely repeating values that cannot be stored perfectly.
When arithmetic operations are performed on these rounded values, the tiny initial approximations can compound, leading to unexpected behaviors in code. For example, mathematically we know that
However, evaluating this subtraction in a standard 64-bit floating-point environment yields a slightly different result due to accumulated round-off error. Here is a code that of the example above.
#example
x=4.9 - 4.845
y= 0.055
print(x,y, x==y)
# another example
print("Even this example has round-off error, 0.1 + 0.2=", 0.1 + 0.2)
0.055000000000000604 0.055 False
Even this example has round-off error, 0.1 + 0.2= 0.30000000000000004
Consider how round-off error accumulates when a sequence of arithmetic operations is executed within a program. As a concrete example, start with the value \(4.0\) and repeatedly add and subtract \(1/3 = 0.\overline{3}\). Because \(1/3\) lacks an exact representation within a computer, a round-off error is introduced during each step and gradually builds up over time. Below is the experiment.
val = 4.0
for i in range(5000):
val = val + 1/3 - 1/3
print(val)
3.9999999999999996
4.6.5. Overflow and Underflow#
Because a computer uses a fixed number of bits to store floating-point numbers, there are boundaries on the maximum and minimum magnitudes that can be represented. When a mathematical operation pushes a value outside these boundaries, it results in either overflow or underflow.
4.6.5.1. Overflow#
Overflow occurs when the magnitude of a number becomes too large to be represented within the available exponent range. In the IEEE 754 single-precision format, the maximum representable finite number is roughly \(3.4 \times 10^{38}\).
If a calculation exceeds this limit, the computer cannot store the exact result and instead assigns it a special value representing infinity, denoted as inf or +inf (or -inf for negative values). Once a variable overflows to infinity, any subsequent arithmetic operations involving that variable typically remain infinite or become undefined.
4.6.5.2. Underflow#
Underflow occurs when the magnitude of a non-zero number becomes too small to be represented. This happens when a number is closer to zero than the smallest representable value in the floating-point system. In single precision, this lower limit for a normalized number is approximately \(1.2 \times 10^{-38}\).
When an operational result drops below this threshold, the computer cannot resolve the tiny fraction. Depending on the system’s configuration, it will either:
Approximate it using a subnormal (denormalized) number at a loss of precision.
Flush the value entirely to a strict hardware
0.0.
4.6.6. NaN (Not a Number)#
Not all mathematical operations yield a valid real number. When a calculation is mathematically undefined or impossible to resolve, the IEEE 754 standard dictates that the computer return a special symbolic value called NaN (Not a Number).
An operation produces NaN when there is no logical numerical representation for the output. Common operations that trigger a NaN value include:
Indeterminate forms: Calculating \(0/0\) or \(\infty/\infty\).
Undefined differences: Calculating \(\infty - \infty\).
Invalid domain operations: Taking the square root of a negative number (\(\sqrt{-1}\)) or the logarithm of a negative number within a strictly real-number environment.
Once a NaN value is introduced into a calculation, it propagates through subsequent operations. Any arithmetic operation performed on a NaN value (such as adding, multiplying, or taking the square root of NaN) will simply result in NaN.
Additionally, NaN has a unique property in programming: it is the only value that is not equal to itself. In code, the comparison expression NaN == NaN evaluates to False, requiring programmers to use specialized functions like np.isnan() to check for its presence.
print("--- Overflow Example ---")
large_number = 1e308
overflow_result = large_number * 10
print(f"Result of 1e308 * 10: {overflow_result}")
print("\n--- Underflow Example ---")
small_number = 1e-323
underflow_result = small_number / 10
print(f"Result of 1e-323 / 10: {underflow_result}")
print("\n--- NaN Examples ---")
zero_div = np.array([0.0]) / 0.0
inf_sub = np.inf - np.inf
sqrt_neg = np.sqrt(-1.0 + 0j) # Note: np.sqrt(-1.0) on real float outputs NaN
print(f"0.0 / 0.0: {zero_div[0]}")
print(f"inf - inf: {inf_sub}")
print(f"Is NaN equal to itself? {zero_div[0] == zero_div[0]}")
--- Overflow Example ---
Result of 1e308 * 10: inf
--- Underflow Example ---
Result of 1e-323 / 10: 0.0
--- NaN Examples ---
0.0 / 0.0: nan
inf - inf: nan
Is NaN equal to itself? False
/var/folders/mq/8_f0y54n1g3f0ht8yb9b6by40000gn/T/ipykernel_9074/2942774074.py:12: RuntimeWarning: invalid value encountered in divide
zero_div = np.array([0.0]) / 0.0