Mar 3, 2021
By Martin Klein, Scientist in the Research Library at Los Alamos National Laboratory and Karolina H...
seen from United States
seen from Malaysia
seen from Sweden
seen from Spain
seen from United States
seen from Germany
seen from Brazil
seen from Australia
seen from Australia
seen from Australia
seen from China
seen from United States
seen from China
seen from United Kingdom
seen from United States
seen from Spain
seen from China
seen from Colombia
seen from Ireland
seen from Ireland
By Martin Klein, Scientist in the Research Library at Los Alamos National Laboratory and Karolina H...
By Martin Klein, Scientist in the Research Library at Los Alamos National Laboratory and Karolina H...
In software, you frequently need to check whether some objects is in a set. For example, you might have a list of forbidden Web addresses. A
Probabilistic Data Structures Part - 1
Probabilistic Data Structures Part – 1
Suppose let’s say you are signing up for the new Email id. When you enter a new mail id, respective mail service will check whether the given mail id already exists.
So how can we build such a kind of systems ? Think for a minute before continue reading
Keep all email id’s in memory i.e. Cache, It’s not practical to keep all email id’s in memory.
Store it in database/disk and check whether…
View On WordPress
Was sind Bloom-Filter und wofür sind sie gut?
Was sind Bloom-Filter und wofür sind sie gut?
Dieser Artikel ist der erste in einer Reihe, die die technischen Konzepte behandelt, welche im Kontext von On-Chain Skalierung, Blockpropagation und verwandten Themengebieten eine entscheidende Rolle spielen.
Eine Datenstruktur bezeichnet das Format, welches dazu verwendet wird, um Informationen zu speichern. Beim Bloom-Filter handelt es sich um eine probabilistische Datenstruktur, durch die eine…
View On WordPress
What Are Invertible Bloom Lookup Tables?
What Are Invertible Bloom Lookup Tables?
This is the second in a series of articles explaining technical concepts related to on-chain network scaling, block propagation, and other related subjects
Last week, we explored a bloom filteras a data structure that helps identify a set. A related data structure is an Invertible Bloom Lookup Table or IBLT. An IBLT serves a similar function and has the advantage that missing members can be…
View On WordPress
What is a Bloom Filter and What are They Good For?
What is a Bloom Filter and What are They Good For?
This is the first in a series of articles explaining technical concepts related to on-chain network scaling, block propagation, and other related subjects
A data structure refers to a format that is used to store information on a computer. A Bloom filter as a probabilistic data structure that can be used to identify a set. We will first explain what a bloom filter is. Then we will explain trade…
View On WordPress
What are bloom filters?
Let’s say you are working on a website.
The website already has 10 million registered users and it is still growing.
However, there is an issue with your sign up page.
New users are saying that when they try to sign up, when they try to choose a username, it takes forever for the website to tell them whether or not their username is taken. Some of these potential users have been so turned off by the slowness that they have just given up.
You crack open the hatch and take a look at the code.
You see that the server is calling the database and seeing if the entered username already exists. It then tells the UI whether or not the username they chose is still available.
This seems fine, you think. How could we make this much faster?
One way is to index the username column. Basically it keeps a separate table with the usernames sorted. This allows us to do the check with binary search, which would be faster than what are doing now.
However, your CEO has said that he expects to have 50 million users by the end of the year.
It’s at these kinds of scale that a data structure like bloom filters really shine.
We’ve already established why a simple array would not suffice. At best, we can get O(logn) with a pre-sort, and this may not be easy to do if our data gets distributed over many machines.
What about the Hash Table? Doesn’t that give us constant look up? Yes, kind of. If we do not have any hash collisions, where two different keys somehow occupy the same space in the hash table, we can have constant time look up. However, collisions do happen, and with a hash table, we face the problem of having to keep the data all in one central place.
Enter bloom filters.
A bloom filter is a data structure with two methods:
insert
check
and it uses a bit array for storage.
Another aspect of bloom filters is a certain number of different hash functions.
As an example, we’ll say we have two.
Here’s how we insert data into the bloom filter.
Let’s say the first user that ever signed up had the username NachoLibre.
NachoLibre gets put through the two hash functions.
The first returns 15.
The second returns 24.
We check index 15, and it is 0. We check index 24 and it is also 0.
With bloom filters, if even one of the indices are 0, then we know absolutely sure that the username NachoLibre has not been taken.
In the bit array, we switch index 15 from 0 to 1. We switch index 24 from 0 to 1.
All the other indices remain 0.
Our second user joins. He tries the username NapoleonDynamite.
The first hash function returns 15.
The second hash function returns 4.
Index 15 is already 1.
Index 4 is 0, so we know no one has the username NapoleonDynamite yet (phew).
We finish the insert by changing index 4 from 0 to 1.
Let’s fast forward to our millionth user.
He tries the username RickyBobby.
The first hash function returns 24.
The second hash function returns 20.
Both indices at 24 and 20 are already 1.
Does that mean RickyBobby is already taken? No.
With bloom filters, we can only know that someone is
- DEFINITELY not in a set
- or MAYBE in the set (with a high probability that they are)
However, the chance is great enough that we tell user million that RickyBobby has already been taken. :(
Of course, in a real life use of the bloom filter, we would probably not run into a collision so early.
Bloom filters basically allow us to do this check in essentially constant time, just having to run hash functions and then check that many indexes on the bit array!
There is also a great advantage in performance. Since bloom filters use a bit array as storage, we can actually use far less space than what a Hash table would take up. Also, if we were to convert our bit array to decimal, we could save even more space!
For example, the bit array 1010101010001 can be converted to just the number 5457, which is basically 4 bytes.
Cool right.