A few weeks ago, DuneSQL became the default query engine on Dune.com.

We’ve seen impressive early adoption: 85% of all newly created queries are on DuneSQL and 30% of all executions are on DuneSQL. You can read all about DuneSQL in the blogpost from our CTO.

DuneSQL is Dune’s future. Therefore, we will be progressively sunsetting both SparkSQL and Ethereum V1 (Postgres) to make DuneSQL the only query engine available on Dune.

Both SparkSQL and our old Postgres systems are not optimal for querying crypto data. Sunsetting these systems is an inconvenient but necessary step to make Dune the best platform to query crypto data.

A Smooth transition to DuneSQL

We understand that this transition might be challenging, especially if you've previously migrated from V1 to SparkSQL. We empathize with the inconvenience this migration may cause and want to assure you that this is an essential step towards simplifying the user experience on Dune and focusing our engineering resources on a single, powerful query engine.

To smoothen the transition to DuneSQL we are investing heavily in migration support to make the process as seamless as possible.

We are focusing our migration efforts on:

• Supporting migration efforts with powerful features
• Continuously improving DuneSQL stability and performance
• Transitioning Spellbook from SparkSQL to DuneSQL

The high-level timeline for SparkSQL and PostgreSQL along with FAQ are available now in our docs here: https://dune.com/docs/query/Old-Query-Engines

Data ingestion for Polygon, BNB, Optimism, and Gnosis on their respective Postgres instances have stopped on 20/04/2023.

This means that all DuneV1 Postgres queries on these systems will no longer be updated. All queries on V2 (Spark, DuneSQL) querying this data will continue to function normally.

Supporting migration efforts

We recently released the migrate to DuneSQL tool that automatically converts queries from SparkSQL and PostgreSQL to DuneSQL. We have open sourced it to allow the community to provide feedback and contribute. We are also working on a GPT4 integration to make the tool even more powerful!

Please refer to the docs to keep up to date with the progress: https://dune.com/docs/query/Old-Query-Engines/migration-tool/

Continuous improvement to DuneSQL

We are aware that DuneSQL is still running into some trouble with advanced queries that query many different tables, most often resulting in stage-related timeouts. 

We are working on addressing this issue on our side by materializing more tables and optimizing our data model. Furthermore, we are actively working on a feature that will allow you to easily use query results as materialized views and the introduction of query scheduling.
These new features should help avoid stage timeouts by allowing you to build mini-data pipelines within Dune.

Transitioning to DuneSQL Spellbook 

For the short-term future, Spells will keep running on SparkSQL, but queryable by DuneSQL. We are already testing a new version of Spellbook that will run entirely on DuneSQL, closing the last remaining compatibility gaps and building a simpler Spell development journey.

Once Spellbook on DuneSQL has been fully validated, all Spells will be migrated by the Dune Team.

tl;dr

• We will progressively sunset SparkSQL and PostgreSQL by July 15, 2023.
• To aid you in the migration, we have built and will improve a query migration tool enabled by LLMs.
• We are still optimizing our data storage and query execution to prevent query timeouts.
• Scheduling queries and materialized views will enable you to build data pipelines within Dune to prevent query timeouts.
• Spellbook stays on SparkSQL for the time being.
• A new Version of Spellbook that works with DuneSQL is in Development. Dune will take care of the migration. 

Let's do it, together

We appreciate your continued trust in us, and we hope you are as excited as us about the possibilities of DuneSQL. 

Please accept our sincere apologies for the inconvenience caused by the migration. Rest assured this is the final migration, DuneSQL is our definitive answer as to how we can build the best crypto data platform! 

Thank you for your understanding and support. Don’t hesitate to reach out to us via dunesql-feedback@dune.com for any questions or concerns.

The data must flow!