Spellbook allows web3 analysts all across the industry to work together on data assets. 

This saves countless hours of intensive work, and eliminates the huge duplication of effort that would come with each analyst or team working individually.

Because some core data resources are commonly needed by power users to do their best work, Spellbook is key for empowering and enabling them. Working in public creates traction for solving the biggest underlying data challenges.

But we're ngl, running a collaborative data pipeline has its ups and downs.

Deploying and maintaining pipelines on a growing code base of community contributors can be difficult. We can't always keep up with the influx of contributions. Onboarding new contributors is not streamlined and power users need better spaces to discuss upcoming work.

But ultimately, the collaborative approach is the right one for building on web3 data, and it’s one that we intend to continue iterating on and improving.

What makes collaborative data pipelines important in web3?

There are a few aspects that make collaboration especially appealing in the web3 space.

Breadth of data use cases:

In Web2, an analyst might only specialize in a single type of data like marketing or sales. Web3 data analysts on the other hand investigate DEXes, NFTs, DAOs, and many other sectors - all of which require specialized knowledge. 

Domain switching is common for our power users. To make things trickier still, even schemas in the same domain vary substantially from competing projects!

That means access to analysts working across sectors - and the shared data assets they create - are vital to the success of all. 

Without them, analysts would be overwhelmed preparing and consolidating data assets before they could even begin their real work. 

Data importance can change rapidly:

E-commerce company analysts don't wake up and suddenly need to care about ride-sharing data. 

In our world though, projects and sectors can blow up very quickly. 

For example, it wasn't possible for us to know that Blur trades were going to suddenly become crucially important all of a sudden. 

Because we can't anticipate what projects or sectors are going to gain traction, we need agile tools that let people proactively model what they care about as the action unfolds.

We can’t rely on stale quarterly data roadmaps. 

Data is an afterthought to developers:

This isn't unique to web3, but the scale of our data amplifies this. When developers don't consider data properly, they can easily introduce issues that break downstream pipelines. 

Handling that within a startup is hard, handling that with data from thousands of startups is harder. 

That's why collaborating with other analytics experts instead of going it alone is so beneficial. 

How can we make collaboration easier?

We know that we can’t scale our community data pipeline while acting as gatekeepers reviewing and merging every pull request. We also know that our contributors need better tools for coordinating work.

Over the next few weeks, we’re thinking hard about these problems and finalizing our roadmap for spellbook.

There are several directions we believe will be worth following:

• ‍Establishing discussion forums for sector-level data modeling efforts. This will allow discussion to happen earlier, rather than in the final steps of the PR review process. ‍
• Organizing the project into chapters to make it easier for noobies to find the resources they need. This will simplify the challenge of working in such a large, and sometimes disconnected, codebase. ‍
• Building sector-level guilds to give review & deployment ownership to the community. Ideally, guild members can review and approve code rather than relying solely on Dune employees to LGTM. The guilds would be responsible for chapter stability, and (tbd) could prioritize bug fixes & allocate funds to pay resident wizards. ‍
• More data pipelining tools in the app. Some data pipelines don't need extensive community input and could be better managed in a team setting.‍

How can you help?‍

Are you excited to learn more, join the discussion, or contribute to Spellbook? 

We would love to hear from you! Please head over to #spellbook on our Discord and drop us a line, our team will be happy to guide you to the right resources to get you started.

Getting set up to cast your first Spell isn’t hard - provided you know the basics of DBT & SQL. You can follow this guide to get started. 

Also, writing code isn’t the only way to contribute.

Head over to #discussions to see what problems the community is working on, chime in with your knowledge, or propose your own. 

We are shifting discussions from side channels (a well known issue in Web3) to the main repository. You should start to see more activity in this section over the coming days.

Thank you all for your contributions. The data must flow.