Serverless Deployment

The serverless deployment uses components from both AWS (Amazon) and GCP (Google Cloud), so you'll need accounts with both services. You'll also want to setup the command line clients for both of these services (aws and gcloud). Additionally, you'll need your normalized data sets.

                      |-------- AWS ---------|   |- GCP --|
[Client] <--(JSON)--> [API Gateway]<->[Lambda]<->[BigQuery]

Bloom Filter

Next we'll only de-duplicate our data using the LeakDB bloom filter, you do not need to compute indexes for the serverless deployment but removing duplicates will help save on costs since BigQuery charges for storage as well as the amount of data consumed per-query.

Basic Example

$ ./leakdb-curator bloom --json normalized.json --output bloomed.json

You can use ./leakdb-curator bloom --help to see all possible options. However, the most important options for the bloom filter will be --filter-size and --filter-hashes, which control the size in GBs of the bloom filter and the number of hash functions used. This will affect your false-positive rate for a given size of data. You can use a bloom filter calculator to determine the optimal settings for the size of your data. The defaults are very conservative, but may need to be adjusted depending on your hardware and the size of your data.

GCP BigQuery

Next we need to upload our bloomed.json to GCP Cloud Storage, you'll need to first create a storage bucket, for the sake of example I've created a bucket called leakdb_storage in a project called leakdb_project. The storage class/etc. settings for the bucket aren't particularly important as you only need to leave the data in the bucket long enough to import it into BigQuery. Assuming you've setup gcloud, you can use the gsutil command to upload the file into your bucket, additional details here:

gsutil cp ./bloomed.json gs://leakdb_storage/bloomed.json

Next we'll create a BigQuery data set, in the this example I've create a BigQuery dataset called leakdb_dataset, you shouldn't need to adjust any of the default settings for the service unless you want to; additional details here.

Within this dataset we'll create a new table, just click the "Create Table" option. For the table source select "Google Cloud Storage" then use the browse button to select the bloomed.json file we uploaded earlier (tip: you need to double click folders while browsing in this UI). Name the table whatever you want, but I like to version my tables so I'll name mine leakdb_v1. Once you've selected this file the File Format should automatically adjust itself to be JSONL. Next be sure you check the "Auto Detect" Schema option, this is very important. Then just click the "Create table" button and BigQuery should do it's magic.

From here you should be able to manually query the datset, go to the BigQuery query editor and run a basic query such as the following to make sure we've setup everything correctly:

SELECT COUNT(*) FROM leakdb_project.leakdb_dataset.leakdb_v1

Using the query editor we can also do analysis of the data that is not exposed in the LeakDB APIs. For example, if you wanted to create a wordlist of the 100 most common passwords in your dataset you can simply run the following query:

SELECT `password`, COUNT(`password`) AS `occurrence` FROM `leakdb_project.leakdb_dataset.leakdb_v1`
    GROUP BY `password`
    ORDER BY `occurrence` DESC
    LIMIT    100;

I may add additional APIs in the future to query this type of information, but for now you can manually run these types of queries and save the results via the GCP web UI.

AWS Lambda & API Gateway

Next we'll setup an AWS Lambda function and API Gateway to make the service accessible via JSON API/command line client. To compile the Lambda function simply run make lambda, which should produce a leakdb-lambda.zip file in the bin directory.

I think the easiest way to set this up is to just use the AWS Web Console, maybe I'll get around to create a Terraform script or something in the future. Unfortunately for all of us the AWS Web Console is possibly the worst UI ever conceived by man, it's poor design and implementation continues to astound me on a day-to-day basis, but the horribly documented AWS APIs and CLIs aren't much better alternatives.

Lambda Function

1. Select the Lambda service, then "Create Function"
2. Ensure "Author from Scratch" is selected, then enter a function name like leakdb and choose go1.x for the runtime. Under permissions you can just select "Create new role," or if you're already familiar with IAM you can select an existing role/etc. Once you're ready, click "Create Function"
3. In the "Function Code" section on the far right click "Actions" then "Upload .zip" and upload the leakdb-lambda.zip file.
4. Under "Basic Settings" I'd recommend 1024Mb of memory and setting a timeout of 30 seconds. The AWS API Gateway has a hard timeout limit of 30 seconds that cannot be adjusted, so increasing the timeout beyond 30 seconds won't have any effect.
5. Create three environment variables:

• BIGQUERY_PROJECT_ID - The name of your GCP project that contains LeakDB e.g., leakdb_project
• BIGQUERY_TABLE - The BigQuery table that contains our dataset e.g., leakdb_project.leakdb_dataset.leakdb_v1
• BIGQUERY_CREDENTIALS - API credentials for GCP, you can generate them here. This value should be a JSON blob that contains a private_key, any newlines should be encoded as \n.

API Gateway

Next we need to add the API Gateway trigger, click the "+ Add Trigger" button then:

1. From the drop down menu select "API Gateway"
2. Choose "Create API" then "REST API" as an API Type, for authentication select "API Key" or "Open" if you're feeling reckless.
3. Note the "API Endpoint" URL and the "API Key" value if you selected API Key as your authentication method.

You should now be able to query the API using curl for example:

curl <ENDPOINT URL> --header 'x-api-key: <API KEY>' --request POST --data '{"domain": "gmail.com"}'