New level unlocked! Today, we’re excited to announce Spellbook, a new shared foundation for crypto analytics.

Spellbook brings all the best practices of analytics engineering from the web2 world, for the public-by-default data of web3. Read on to learn more and discover how to contribute today.

A Brief History of Abstractions

On January 31, 2020, we launched the abstractions Github repo.

We wanted to build a place for wizards to create reusable views and tables, rather than copy & paste them between queries (ouch!).

Since then, we’ve had over three hundred contributors and nearly 1k commits.

Abstractions are some of the most queried tables on Dune. That makes creating them one of the highest leverage things a wizard can do, and we wanted to make that experience better.

Spellbook

Spellbook is a retooling of our existing abstractions repo, with a first-in-class open-source analytics engineering tool called dbt, or data build tool.

dbt-core is an open-source framework that injects more classical software engineering practices into writing SQL, using the Jinja template language. Why we’re hyped on dbt:

Here’s how it works…

Abstractions, henceforth models in dbt-speak or spells in dune-speak, can be materialized into views and tables - with refinements like incrementally-loaded tables, date-partitioned tables.

These can all be compiled into SQL and run on dune.com. No more contributing code that you can’t test without our help.

You can write data integrity tests to spot and prevent any issues in our abstractions. These can check for unique primary keys, non-null values, accepted values, and relational integrity, all with a single line in a YAML file.

Finally, using dbt we can automatically track the dependencies between all models, rather than deploying them manually. This guarantees that all models are deployed in the correct order.

All in all, this will enable us to reliably scale Spellbook to thousands of spells, without bottlenecks or data quality issues.

Why this Matters

Nowadays, developers have many opportunities to contribute to open source projects. This shared foundation of re-usable libraries has collectively saved us millions (billions?) of hours.

But the same isn’t true for analytics or data science. Until recently, “open source data analytics” hasn’t really been a thing, because all the interesting data is locked up in corporations.

We believe you shouldn’t need to “FAANG it” to access the world’s data, or be on the same “Okta group” to collaborate on it.

With Spellbook, we’re bringing the collaborative model of open source software, to data science and analytics, made possible by the public-by-default data in web3.

If you’re an analyst or data scientist tired of the same old Titanic or digit recognition datasets or Kaggle competitions - come help us build the future!

Get Started Today

We hope you are as excited as we are about this new tool! Spellbook is now live and we welcome new contributors.

You can learn how to cast a spell, browse our auto-generated spellbook documentation or join the conversation on #spellbook.

The data must flow!