Why Bloom Filters Are So Cool (+ Useful!) For Blockchains & Beyond: An Introduction

February 10, 2017 by Christian Seberino

The Problem

search

Millions of people search the Internet, government databases, private databases and blockchains everyday for medical advice, financial updates, weather reports, maps and more. Likewise, millions of people want or need fast searches.

A Solution

index

An effective method to speed up many searches is the use of indexes. Indexes, like those at back of textbooks, provide the locations of all search terms. A possible drawback is that they require large amounts of storage.

Another Solution

smart watch

An effective method to speed up many searches, with less storage requirements, is the use of Bloom filters. These are important in many areas such as mobile and embedded devices.

Bloom Filters

Bloom filter

Bloom filters are binary strings used to quickly determine set membership with nominal storage requirements. Browsers use them to detect malicious websites. Databases use them to avoid searching for nonexistent data.

Bloom filters require less memory than indexes, but, they sometimes give false positives. In other words, they might claim an object is a member of a set when it is not. It is noteworthy that Bloom filters never give false negatives. They never claim an object is not in a set when it actually is. Browser Bloom filters might incorrectly claim safe websites are malicious, but, they will never claim malicious websites are safe. Fortunately, extra checks can always be performed to eliminate any false positives.

Construction

construction

To build a Bloom filter for the set {X₁, X₂, X₃, …, Xₙ}, with hash function H, calculate H(X₁) | H(X₂) | H(X₃) | … | H(Xₙ). | is the bitwise OR operator. H is only valid if the number of set bits (ones) in all hashes is always less than or equal to some selected maximum.

Larger Bloom filters have less false positives. Bloom filters of several megabytes are not uncommon. As hash functions typically do not have large enough hashes, not to mention hashes with required number of set bits, adequate hashes are often constructed from multiple hash functions. A common recipe is to have a group of hash functions each set a single bit in the construction of a valid hash. For example, to set a bit in a hash with 2²³ bits, the first 23 bits of the Secure Hash Algorithm 1 (SHA1) hash can be used to select the position of a bit to set.

Operation

bitwise AND

To determine if an object X might be in a set with Bloom filter B, built with hash function H, determine if H(X) & B = H(X). & is the bitwise AND operator. Notice that the test only returns true if all the set bits in H(X) are also set in B. Basically, groups of set bits in B correspond to elements. X might be a member of the set if and only if its group of set bits corresponds to a group in B. The reason B can only determine if X might be in the set is because B contains the bits of several elements.

Blockchains

feather

Bloom filters allow lightweight (small memory requirements) Bitcoin wallets to quickly and privately obtain account information without downloading the entire blockchain. Bloom filters also allow lightweight Ethereum Classic clients to quickly and privately obtain account information without downloading the entire blockchain.

Conclusion

magazine cover

Bloom filters are a powerful tool that allows additional innovation in blockchain applications and many other areas in the twenty first century.

Feedback

You can contact me by clicking any of these icons:

twitter facebook linkedin

Acknowledgements

I would like to thank IOHK (Input Output Hong Kong) for funding this effort.

License

license

This work is licensed under the Creative Commons Attribution ShareAlike 4.0 International License.

Archive Previous posts

May 19, 2017Prophet Daniel

Stand up from the crowd

May 11, 2017Carlo V

ETC Weekly Newsletter: Dev Update 10!

May 1, 2017Christian Seberino

Why You Should LOVE Proof Of Stake Systems — Hybrids!

April 28, 2017Christian Seberino

Ethereum Classic World Computer Transactions Explained

April 28, 2017Christian Seberino

Ethereum Classic Blocks Explained: The Three Categories

April 19, 2017Carlo V

ETC Weekly Newsletter: New all time highs as ETC surges!

April 18, 2017Christian Seberino

Ethereum Classic Public And Private Keys: A Little Enlightenment

April 13, 2017Carlo V

ETC Weekly Newsletter: New devs on ETCdev Team.

March 30, 2017Christian Seberino

The Ethereum Classic World Computer Accounts & States Explained

March 29, 2017Carlo V

ETC Weekly Newsletter: Dev Update + News from Bitkio.

March 24, 2017Christian Seberino

How To Improve Ethereum Classic Immutability Discussions

March 16, 2017Carlo V

ETC Weekly Newsletter: Dev update and more

March 13, 2017Christian Seberino

Ethereum's Vitalik Buterin Discusses The New Viper Smart Contract Programming Language

March 8, 2017Carlo V

ETC Weekly Newsletter: Dev Updates + New Discussions

March 2, 2017Carlo V

ETC Weekly Newsletter : Monetary Policy Statement.

February 28, 2017Christian Seberino

An Interview With The Anonymous Individual That Started Ethereum Classic

February 28, 2017Christian Seberino

How To Create A Censorship Resistant Domain Name System On Ethereum Classic

February 20, 2017Carlo V

ETC Weekly Newsletter : Treasury Proposal

February 13, 2017Christian Seberino

Should We Make ⟠ The Ethereum Classic Currency Symbol?

February 10, 2017Christian Seberino

Serpent: Introduction To The BEST Ethereum Classic Smart Contract Language

February 10, 2017Christian Seberino

Proposal: Ethereum Classic Currency And Logo Conventions To Improve Communication And Avoid Expensive Mistakes

February 10, 2017Christian Seberino

Why Ethereum Classic Uses An Incorrect SHA3 Implementation

February 10, 2017Christian Seberino

Hashes: An Introduction & Why They Are Foundational To The Internet & Blockchains

February 10, 2017Christian Seberino

Why Bloom Filters Are So Cool (+ Useful!) For Blockchains & Beyond: An Introduction

February 1, 2017Carlo V

ETC Weekly Newsletter : Another Great Month Ahead

January 24, 2017Prophet Daniel

Ethereum Classic Harmony

January 17, 2017Carlo V

ETC Weekly Newsletter : Protocol Update Successful!

January 6, 2017Prophet Daniel

Sustainable Development Goals

January 4, 2017Carlo V

ETC Weekly Newsletter : Happy New Year!

December 29, 2016Carlo V

ETC Weekly Newsletter : End Of 2016!

December 28, 2016Christian Seberino

Zero Knowledge Proofs For Dummies

December 20, 2016Carlo V

ETC Weekly Newsletter : In Case You Missed It

December 16, 2016Christian Seberino

How To EASILY Set Up An AMAZING Ethereum Classic Node & Talk To It With Your OWN Code

December 14, 2016Carlo V

ETC Weekly Newsletter : ETC Meetup in London + The New Team

December 12, 2016Carlo V

Introducing The Grothendieck Team

December 6, 2016Christian Seberino

Why Would I Choose To Run My Application On Ethereum / Classic Instead Of The World Wide Web?

December 6, 2016Carlo V

ETC Weekly Newsletter : The Grothendieck Team

December 4, 2016Arvicco

ETC End of Year and Monetary Policy Event: London, December 13th

December 1, 2016Christian Seberino

Why InterPlanetary File System & Its Ilk Are A Big Deal For Blockchains & Beyond

November 29, 2016Carlo V

ETC Weekly Newsletter : Network Update

November 23, 2016Christian Seberino

The Skinny On Smart Contracts: An Introduction & Why You Should Care

November 22, 2016Carlo V

ETC Weekly Newsletter : Monetary Policy Update

November 15, 2016Christian Seberino

The Bare Basics Of Money And Monetary Policy WITH A FEW WORDS FROM SATOSHI NAKAMOTO

November 15, 2016Carlo Vicari

ETC Newsletter

November 8, 2016Carlo Vicari

ETC Newsletter : 2016-11-01 - 2016-11-08

November 4, 2016Christian Seberino

Let's Admit Blockchains Are Weird: An Introduction To The Strangeness

November 1, 2016Carlo Vicari

ETC Newsletter : 2016-10-24 - 2016-11-01

October 31, 2016Carlos Graterol

Instead of The Halvening, A Tithing for ETC

October 17, 2016Arvicco

Gas Reprice Hard Fork on ETC block 2500000 (October 25)

October 14, 2016Christian Seberino

Why Another Hard Fork To Deal With The Recent Denial Of Service Attack Spam Shouldn't Be Controversial

October 13, 2016Christian Seberino

Ethereum / Classic Denial Of Service Attacks & The Estonian Cyberwar

October 12, 2016Christian Seberino

Cuban Piracy & Why Merkle Trees Are So Awesome For Blockchains

October 11, 2016Christian Seberino

Navajo Indians Help Explain Ethereum / Classic Replay Attacks

September 18, 2016ProphetDaniel

The Invisible Field

September 9, 2016Arvicco

Code is Law and the Quest for Justice

September 1, 2016Ethereum Classic

CHBTC contributes funds to foster growth of Ethereum Classic

August 18, 2016Arvicco

Ethereum Classic Kickoff (London)

August 16, 2016ProphetDaniel

Nature Inspired Ethereum Classic Community Dynamics Proposal

August 14, 2016DaxClassix

New Website Created

August 11, 2016ProphetDaniel

Decentralized anarchist governance system

August 10, 2016ProphetDaniel

Couple Values That Forked Ethereum Broke

July 27, 2016Arvicco

Getting things done in a decentralized way

July 25, 2016Arvicco

What can I do to help Ethereum Classic project?

July 24, 2016Arvicco

ETC exchange trading and other news

July 22, 2016Arvicco

ETC - new Ethereum Classic ticker symbol

July 15, 2016Arvicco

Let's keep the original censorship-resistant Ethereum going!

July 11, 2016Arvicco

A Crypto-Decentralist Manifesto