Defining the web3 stack

In this next segment of Building on web3, Edge & Node developer relations engineer Nader Dabit expands upon the web3 stack and how developers can conceptualize the various facets of building on web3.

I transitioned into web3 in April 2021 after being a traditional full-stack developer for around 10 years. While diving into all of these new technologies and ideas, the first thing I wanted to know was “what is the web3 stack?”.

When building a traditional web or mobile application, I often depend on a handful of building blocks to get the job done.

1. API / app servers (REST or GraphQL)
2. Authentication layer (managed or hand-rolled)
3. Database
4. Client-side frameworks, platforms, and libraries
5. File storage

Using these core components, I can build out most of the types of applications I would like to, or at least get most of the way there. So what did this look like in web3?

It turns out, the answer to this is not so straightforward because:

1. The paradigm is completely different in many ways
2. The web3 tools, technologies, and ecosystem are less mature than web2

It was also harder for me to understand how to get up and running with, and build out, web3 applications because I was approaching the problems in the same way that I was in the web2 world.

After working, researching, experimenting, and building things over the past 8 months or so I’d like to share what I’ve learned.

What is web3?

Before we define the web3 stack, let’s attempt to define web3. There are countless definitions depending on who you ask, but to me I find this definition spot on:

Web3 is the stack of protocols that enable fully decentralized applications.

With this decentralized tech stack, we can begin building out decentralized applications which have their own set implications and characteristics.

Some of the characteristics enabled by web3 are:

• Decentralized web infrastructure
• Ownership (of data, content, and platform)
• Native digital payments
• Self-sovereign identity
• Distributed, trust-less, & robust infrastructure
• Open, public, composable back ends

While some of the applications built on decentralized tech stacks will replace their predecessors, a new paradigm of apps have also been made possible by the new primitives enabled by blockchains.

Native digital payments and public back end infrastructure – like machine learning, mobile devices, virtual reality, and other technological primitives, platforms, and building blocks – enable entirely new types of applications to be built, some that have yet to be imagined.

Does this mean that everything is going to be replaced by web3? Not necessarily. While I do think that building on a decentralized tech stack is a much better choice for certain types of applications – like almost any technical decision, it depends on what you are building.

Let’s now start diving into the web3 stack, broken into this set of categories:

• Blockchain
• Blockchain development environment
• File storage
• Off chain data protocols
• API (indexing & querying)
• Identity
• Client (frameworks and libraries)
• Oracles
• Other protocols

Blockchain

There are countless blockchains that you can choose to build on. There is no single one that is “the best”, instead you should consider the various tradeoffs between them.

One thing that is often important to me when learning something new is the idea of applying the Pareto principle to what I’m learning. i.e., what is the most efficient way to get the most out of that amount of time and effort. Following this idea I can gain the most traction and momentum while learning something new in the shortest amount of time.

In the Blockchain world, learning Solidity and the EVM (or Ethereum Virtual Machine) might be the best bet when getting started as a blockchain developer. Using this skillset (and tech stack), you can build not only for Ethereum, but other Ethereum Layer 2s, sidechains, and even other blockchains like Avalanche, Fantom, and Celo.

That being said, Rust is beginning to become more and more popular in the blockchain world, with Solana, NEAR, Polkadot, and others having first class Rust support. You probably can’t really go wrong learning either, but for the beginner I’d say that Solidity will still be the better choice if someone asked me today.

Beyond that advice, here is an incomplete sample of blockchains that have a solid combination of technology, utility, community, momentum, and future viability:

• Ethereum - original smart contract platform
• ZK rollups: ZKSync, Starknet, Hermez - High throughput Ethereum layer 2s, but not natively EVM compatible
• Optimistic rollups: Arbitrum & Optimism - Ethereum layer 2s, EVM compatible (learn more about the differences between optimistic and ZK rollups here)
• Polygon - Ethereum sidechain
• Solana - high throughput, inexpensive transactions, fast block times, but harder to learn than EVM (Rust)
• NEAR - Layer 1 blockchain, can write smart contracts in Rust or Assemblyscript
• Cosmos - an ecosystem of interoperate blockchains
• Polkadot - blockchain-based computing platform that enables blockchains built on top of it to execute transactions between themselves, creating an interconnected internet of blockchains
• Aurora - EVM compatible chain running on NEAR protocol.
• Fantom - EVM compatible layer 1
• Avalanche - EVM compatible Layer 1
• Celo - EVM compatible layer 1, designed to make it easy for anyone with a smartphone to send, receive, and store crypto
• Tezos - Non EVM compatible layer 1, a lot of NFT projects are using it

When interacting with a network, you’ll need to use an RPC endpoint.

There are a few ways you can do this:

1. Access a public RPC endpoint
2. Running your own nodes
3. Accessing a node provider as a service
4. Accessing a decentralized node provider as a service

Public RPC endpoints are often provided by the network, but for most production dapps you’ll want to leverage your own endpoints as they are not stable or recommended for production.

There are a handful of RPC service providers out there, here are a few:

• Infura
• Figment Datahub
• Ankr
• Coinbase Cloud
• Cloudflare
• Alchemy
• GenesysGo (Solana)

There is also a web3 / decentralized solution, Pocket Network that seems to be gaining traction.

Any of these options are probably a good bet for interacting directly with your network.

Blockchain development environments

For EVM development, there are a few good development environments available:

• Hardhat (JavaScript) is a newer option, but one that is gaining more and more popularity. Their docs are great, the tooling and developer experience is polished, and it’s what I’ve personally been using to build dapps.
• Truffle (JavaScript) is a suite of tools for building and developing applications on the EVM. It’s mature, battle tested, and well documented. It’s been around for a while and many developers use it.
• Foundry is a new Solidity development environment from Paradigm that shows a lot of promise. Key standouts are the ability to write tests in Solidity, support for fuzzing, and speed (it’s written in Rust). I wrote a separate introduction to it here.
• Brownie is a Python-based development and testing framework for smart contracts for Solidity / EVM development.

For Solana development, Anchor is quickly becoming the entry-point for new developers. It provides a CLI for scaffolding out, building, and testing Solana programs as well as client libraries that you can use for building out front ends. It also includes a DSL that abstracts away a lot of the complexities that developers often run into when they get started with Solana and Rust development.

File storage

Where do we store images, videos, and other files in web3? Storing anything that large on-chain is usually prohibitively expensive, so we probably don’t want to store them there.

Instead, we can use one of a handful of file storage protocols:

• IPFS - peer-to-peer file system protocol
  • pros: it’s reliable, well documented with a large ecosystem
  • cons: if data is not pinned it can be lost
• Arweave - allows you to store data permanently, paying a single transaction fee. I’m a fan of Arweave and wrote a blog post about it here.
• Filecoin - from Protocol Labs, the same team that build IPFS, it is a protocol designed to provide a system of persistent data storage. There are a handful of ways for developers to build on Filecoin, including web3.storage which is pretty nice.
• Skynet - I have not yet used it in production, but have tried it out and it seems to work nicely. The API here looks great. I have questions like how long is the data persisted, and Skynet’s interoperability with other protocols.

Off chain data protocols

In addition to file storage and on-chain storage, you may also need to store data off-chain. You might use these types of solutions similarly to how you might use a database in a traditional tech stack, but instead they are replicated across n number of nodes on a decentralized network, and therefore more reliable (at least in theory).

A few options are:

• Ceramic Network - a decentralized, open source platform for creating, hosting, and sharing data. Ceramic also has a nice identity protocol that I’ll talk about later. Probably my favorite off-chain storage solution at the moment. Here’s a pretty nice demo.
• Textile ThreadDB - a multi-party database built on IPFS and Libp2p. If I understand correctly, it may be going through a big API change at the moment. I’ve tried it and it shows some promise, but the docs and DX need some improvement.
• GunDB - a decentralized, peer-to-peer database. Gun has been around for quite sometime and some pretty interesting applications have been built with it.

In terms of maturity, my take is that the ecosystem of off-chain storage solutions is not yet where it needs to be to build out some of the more advanced use cases some developers might want. Some challenges here are real-time data, conflict detection and conflict resolution, write authorization, documentation, and general developer experience.

Integrating offchain data solutions with blockchain protocols is one of the last big hurdles we need to cross before we have a fully decentralized protocol stack capable of supporting any kind of application.

API (indexing & querying)

There are a lot of differences in the way that we interact with and build on top of blockchains versus databases in the traditional tech stack. With blockchains, data isn’t stored in a format that can efficiently or easily be consumed directly from other applications or front ends.

Blockchains are optimized for write operations. You often hear the innovation happening centered around transactions per second, block time, and transaction cost. Blockchain data is written in blocks over the course of time, making anything other than basic read operations impossible.

In most applications, you need features like relational data, sorting, filtering, full text search, pagination and many other types of querying capabilities. In order to do this, data needs to be indexed and organized for efficient retrieval.

Traditionally, that’s the work that databases do in the centralized tech stack, but that indexing layer was missing in the web3 stack.

The Graph is a protocol for indexing and querying blockchain data that makes this process much easier and offers a decentralized solution for doing so. Anyone can build and publish open GraphQL APIs, called subgraphs, making blockchain data easy to query.

To learn more about The Graph, check out the docs here or my tutorial here.

Identity

Identity is a completely different paradigm in web3. In web2, authentication is almost always based on a user’s personal information. This information is usually gathered either via a form or an OAuth provider that asks the user to hand over in exchange for access to the application.

In web3, identity revolves completely around the idea of wallets and public key cryptography.

While the name “wallet” serves its purpose, I’ve found that people new to web3 find the terminology confusing as it relates to authentication and identity. I hope that in the future we can figure out some other way to convey what a wallet is, as it combines aspects of finance but also identity and reputation.

As a developer, you will need to understand how to access and interact with the user’s wallet and address in various ways.

At a very basic level (and a very common requirement), you might want to request access to the user’s wallet. To do this, you’ll usually be able to access the user’s wallet in the window context (web browser) or using something like WalletConnect or Solana’s Wallet Adapter.

For instance, if they have an Ethereum wallet available, you’ll be able to access window.ethereum. The same for Solana (window.solana), Arweave (window.arweaveWallet), and a handful of others. WalletConnect is good for mobile web and React Native as it allows users to authorize using their mobile wallets directly from the device.

If you want to handle authentication yourself, you can allow the user to sign a transaction and then decode it somewhere to authenticate the user, but this usually requires a server. Here is an example of how that might look using an EVM wallet, and here is an example of how to do this with Solana / Phantom.

What about managing user profiles in a decentralized way? Ceramic Network offers the most robust protocol and suite of tools for managing decentralized identity. They recently released a blog post outlining some of their most recent updates and giving some guidelines around how all of the tools work together. I’d start there and then explore their docs to gain an understanding of how to start building, and consider checking out my example project here that uses Ceramic self.id.

If you want to fetch a user’s ENS text records, the ensjs library offers a nice API for fetching user data:

const ens = new ENS({ provider, ensAddress: getEnsAddress('1') })
const content = await ens.name('sha.eth').getText('avatar')

SpruceID is also something that looks promising but I’ve yet to try it out.

Ceramic and Spruce both implement the W3C DID specificiation, which itself is also something that I would consider as a building block of web3. With that being said, any centralized implementation of DIDs goes against the idea of what the specification is trying to accomplish.

Client

As far as JavaScript frameworks go, you can really build with anything you’d like, as the client-side blockchain SDKs are mostly framework-agnostic. That being said, an overwhelming number of projects and examples are built in React. There are also a handful of libraries like Solana Wallet Adapter that offer additional utilities for React, so I’d say that learning or being familiar with React is going to probably be a smart move.

For client-side SDKs in Ethereum there’s web3.js and ethers.js. To me Ethers is more approachable and has better documentation, though web3.js has been around longer.

In Solana, you’ll probably be working with @solana/web3.js and / or Anchor. I’ve found Anchor client libraries to be my go-to for building Solana programs since I’m using Anchor framework anyway, and I’ve found it much easier to understand then @solana/web3.js.

Oracles

Oracles allow developers access to read real-world data & external systems from within a smart contract.

For example, the majority of financial applications require knowledge of real-world data & events happening off-chain, so Oracles are especially important in DeFi.

Chainlink is an Oracle that enables access to real-world data and off-chain computation while maintaining the security and reliability guarantees inherent to blockchain technology.

Flux is a cross-chain oracle that provides smart contracts with access to economically secure data feeds.

Other protocols

Radicle is a decentralized code collaboration protocol built on Git. It could be thought of as a decentralized version of GitHub.

Livepeer is a decentralized video streaming network. It is mature and widely used with over 70,000 GPUs live on the network.

Wrapping up

This post will be a living document that I keep up with as I learn, experiment, and gather feedback from developers building in web3.

If you have any feedback or ideas around something I’m missing here, please reach and share your thoughts with me. It’s exciting to see all the activity happening around web3 as developers are jumping in and getting involved. While the infrastructure is still evolving, the vision of building truly decentralized protocols and applications that allow people to coordinate without having to give power and control to large companies is an important one and we’re close to making this vision a reality.