This discusses the architecture behind a critical piece for Stacks decentralized apps (dApps) developers: the flow of data from the Stacks Node where the block chain is constructed to the WebSockets service provided in the Stacks Blockchain API. dApps can subscribe to data streams via the WebSockets service in the API such as updates to transactions, where the status can change over a period of time.
The process begins with the Stacks Node. This server, composed primarily using the Rust language, can reside anywhere on the network. The details of the connectivity between the Stacks Node and the Stacks Blockchain API server is for a different discussion. The important thing here is that it sends data to the API server as important events occur, such as when a new block is created. Today it does this via REST HTTP calls.
On the other end you have dApps subscribing to data streams via WebSockets, such as events involving a transaction id. One use case of this is if it needs to confirm that a transaction has made it into an anchor block successfully before taking other actions, such as completing the other side of a cross chain transaction.
Inside the Stacks Blockchain API
This is a Node Server built in Typescript using the express library for its HTTP interface. Various components are bolted onto this server, including the RESTful events server that the Stacks Node currently sends its messages to, as well as the WebSockets server, which is bolted onto the URL path “/extended/v1/ws”.
Incoming messages from the Stacks Node are queued via p-queue (first in, first out) to force it to be handled via a single thread in the order it arrives. Each type of message has a message handler that can do processing to the data before sending it to the data store. For instance, a block can have many transactions in it, which can produce different types of data. The important thing here is that one message — a block — can be broken down to many smaller messages that a dApp may ultimately be interested in.
This is significant value-add for the API over raw data from the Stacks Node, and is the heart of where other things can potentially happen to produce new value. For instance, while today this focuses on breaking data into smaller atomic parts, in the future, a real-time analytics layer can be inserted here leveraging both new incoming data with persistent or cached state.
The events to the WebSockets RPC layer, however, don’t actually come from this event server message handling code directly. This handler code sends it to the DataStore, which in addition to persisting the data via a PostgreSQL DB implementation of this DataStore interface, also emits events that the WebSockets layer can then subscribe to. Today, there are three events WebSockets can listen to: tx_update, address_tx_update and address_balance_update.
You’ll find nearly all the WebSockets handling code in one file, ws-rpc.ts. In there you have a SubscriptionManager that handles client subscriptions using a Typescript Map. Each of our three event types have their own instance of this subscription manager, allowing it to easily obtain the clients subscribing to a particular event. E.g.,
const subscribers = txUpdateSubscriptions.subscriptions.get(txId);Note that the WebSockets service also queries our DataStore. For Tx updates, it can do a call to db.getMempoolTx(…) as well as db.getTx(…).
Once it has the output, or has an error, it wraps it in a standard JSON format using jsonrpc-lite. It then send this to all WebSockets clients that have subscribed.
subscribers.forEach(client => client.send(rpcNotificationPayload));Overview
This is a very clean design overall. There are nice separation of concerns. From the WebSockets perspective, it is an event listener that has a large pool of events it can potentially listen to even if it is currently only providing downstream subscription to three topics. The events that it does listen to currently are the output of very helpful data transformations to permit subscriptions on details dApps are likely to be concerned with.
There is plenty of room for improvement as WebSocket client interests grow, including the potential for other transformations that are able to take advantage of the combination of the live data stream with the cached or persistent data.
(You can view the source code for @blockstack/stacks-blockchain-api on Github.)
