HyperLogLog
96 words
1 min read
Visual companion
Python
Type and operator map
Python Week 1: the first filter for runtime behavior
View
Revision summary
What this note is really saying
Short form
# HyperLogLog ## The Distinct Counting Problem Given a stream of elements, count the number of **distinct** elements using minimal memory. **Trivial solution:** Keep a hash set — $O(n)$ memory.
HyperLogLog
The Distinct Counting Problem
Given a stream of elements, count the number of distinct elements using minimal memory.
Trivial solution: Keep a hash set — memory.
HyperLogLog: memory, accuracy.
Algorithm
- Hash each element to a uniform binary string
- Find the position of the leftmost 1-bit
- Track the maximum across all elements:
- Estimate: Where is the number of registers (typically ), is a bias correction constant. Join Discord PreviousStreaming AlgorithmsNextRandomized Optimization