How should developers deal with discarded transactions in Solana blockchain?

Published on 1/18/2022   664 views   0 Comments  

This article mainly outlines the technology of how Solana handles transactions and how developers deal with lost transactions.

This article was originally published inThe Solana Cookbookใ€‚


brief introduction


In some cases, a seemingly valid transaction may be discarded before it is included in the block. This situation most often occurs in the period of network congestion, when the RPC node fails to rebroadcast (replay) the transaction to the leader (the verifier that determines the current block). For end users, their transactions seem to have completely disappeared. Although RPC nodes are equipped with general replay algorithms, application developers can also develop their own custom replay logic.


Course of transaction


How do customers submit transactions

In Solana blockchain, there is no concept of MemPool. All transactions, whether they are programmed or initiated by the end user, are effectively routed to the leader so that they can be processed into a block. There are two main ways to send transactions to leaders:

  1. Proxy through RPC server and sendtransaction json-rpc method
  2. Direct delivery to leaders via TPU client

The vast majority of end users submit transactions through RPC servers. When the client submits a transaction, the receiving RPC node will in turn attempt to broadcast the transaction to the current and next leaders. Before the transaction is processed by the leader, there is no record of the transaction except that the customer and the forwarding RPC node know the transaction. When the TPU client sends the transaction, the replay and forwarding of the leader are completely handled by the client software.

How do RPC nodes broadcast transactions

After the RPC node receives the transaction through sendtransaction, it will convert the transaction into UDP packet and forward it to the relevant leader. UDP allows verifiers to communicate quickly with each other, but does not provide any guarantee of transaction delivery.

Since Solana's leader schedule (the schedule that determines which verifier will be the next leader) is known before each era (about 2 days), the RPC node will broadcast its transactions directly to the current and next leaders. This is in contrast to other protocols, such as Ethereum, which randomly and widely propagate transactions throughout the network. By default, the RPC node will try to forward the transaction to the leader every two seconds until the transaction is finally completed or the blockchain of the transaction expires (150 blocks or ~ 1 minute and 19 seconds as of the time of writing this article). If the unfinished replay queue size exceeds 10000 transactions, the newly submitted transactions will be discarded. RPC operators can adjust some command-line parameters to change the default behavior of this retry logic.

When an RPC node broadcasts a transaction, it will attempt to forward the transaction to the leader's transaction processing unit (TPU). TPU processes transactions in five different stages.

  • Acquisition phase
  • Sigverify phase
  • Banking phase
  • Proof of history service
  • Broadcast stage
Source: jito Labs
Source: jito Labs

Among these five stages, the "acquisition" stage is responsible for receiving transactions. In the acquisition phase, the verifier will classify the imported transactions according to three ports.

  • TPU handles routine transactions such as token transfers, NFT casting, and program instructions.
  • tpu_ Vote only focuses on voting transactions
  • If the current leader cannot process all transactions, TPU_ Forwards forwards unprocessed packets to the next leader.

For more information about TPU, please refer to this article of jito labsarticleโ€Œใ€‚


How are transactions discarded


In the whole process of the transaction, there are several cases where the transaction may be inadvertently deleted from the network.

Before processing a transaction

If the network discards a transaction, it is likely to be discarded before the transaction is processed by the leader. UDP packet loss is the simplest reason for this. When the network load is very high, the verifier may also be overwhelmed by the number of transactions to be processed. Although the verifier has the ability to pass the TPU_ Forwards forwards redundant transactions, but the amount of data that can be forwarded is limited. In addition, each forwarding is limited to two separate validators. That is, in TPU_ Transactions received on the forwards port will not be forwarded to other validators.

There are also two less well-known reasons why transactions may be discarded before they are processed. The first case involves transactions submitted through the RPC pool. In some very rare cases, part of the RPC pool can be fully ahead of the rest of the pool. This can cause problems when the nodes in the pool are required to work together. In this example, the most recent block hash of the transaction is queried by the back end a of the pool. When the transaction is submitted to the backendb of the pool, the node will not recognize the leading block hash and abandon the transaction. If the developer enables pre check on sendtransaction, this can be detected when the transaction is submitted.

Temporary network bifurcation can also cause transactions to be discarded. If a verifier replays blocks slowly during the banking phase, it may eventually create a few forks. When a customer establishes a transaction, the transaction may refer to a recent block hash that exists only on a few forks. After the transaction is submitted, the transaction cluster can switch out of its few forks before the transaction is processed. In this case, the transaction is discarded because the block hash cannot be found.

After the transaction is processed, it is discarded before the transaction is finalized

If a transaction references a few forked recent block hashes, the transaction may still be processed. However, in this case, it will be handled by a few forked leaders. When this leader tries to share transactions with other members of the network, it will not be able to reach consensus with most verifiers that do not recognize a few forks. At this time, the transaction will be discarded before final completion.


Processing discarded transactions


Although RPC nodes will try to rebroadcast transactions, their algorithms are general and often not suitable for the needs of specific applications. In order to cope with network congestion, application developers should customize their own replay logic.

Learn more about sendtransaction

When it comes to submitting transactions, the main tool available to developers is the sendtransaction RPC method. Sendtransaction is only responsible for forwarding transactions from the client to the RPC node. If the node receives a transaction, sendtransaction will return the transaction ID, which can be used to track the transaction. A successful response does not indicate whether the transaction will be processed or finally completed by the cluster.

Request parameters

Transaction: string (string) -- a fully signed transaction as an encoded string

(optional) configuration object: object

  • Skippreflight: bolean (Boolean) - if true, the pre check transaction check is skipped (false by default).
  • (optional) preflightcommitment: string (string) -- the commitment level used in preflight simulation of bank slot (the default is "complete").
  • (optional) encoding: String - used for encoding transaction data. Or & quot; base58" (slow), or & quot; base64"ใ€‚ (default: & quot; base58 & quot;).
  • (optional) maxretries: use -- the maximum number of times the RPC node retries sending transactions to the leader. If this parameter is not provided, the RPC node will retry the transaction until the transaction is finally completed or until the block hash expires.

response

Transaction ID: String - the first transaction signature embedded in the transaction and encoded in base-58. This transaction ID can be used with getsignaturestates to poll for status updates.


Custom replay logic


In order to develop their own replay logic, developers should use the maxretries parameter of sendtransaction. If the developer provides relevant parameters, maxretries will override the default replay logic of the RPC node, allowing the developer to manually control the replay process within a reasonable range.

A common pattern for manually retrying transactions is lastvalidblockheight from the getlatestblockhash temporary store. Once the application is stored, it can poll the block height of the transaction cluster and manually retry the transaction within an appropriate time interval. In case of network congestion, it is advantageous to set maxretries to 0 and replay manually through a custom algorithm. Although some applications may adopt exponential backoff algorithm, others, such as mango, choose to resubmit transactions continuously at constant time intervals until some timeout occurs.

When polling through getlatestblockhash, the application should specify its expected commitment level. By setting its commitment to confirm (vote) or finalize (about 30 blocks after confirmation), the application can avoid polling the blockchain from a few forks.

If an application can access the RPC node behind the load balancer, it can also choose to divide its workload among specific nodes. RPC nodes serving data intensive requests, such as getprogramaccounts, may easily lag behind and are not suitable for forwarding transactions. For applications that process time sensitive transactions, it is prudent to have dedicated nodes that only process sendtransactions.

The cost of skipping preflight

By default, sendtransaction will perform three pre checks before submitting the transaction. Specifically, sendtransaction will:

Verify that all signatures are valid

Check whether the referenced blockchain is within the last 150 blocks

Bank slot simulation transaction specified for preflightcommitment

If any of these three pre checks fails, sendtransaction will raise an error before submitting the transaction. Pre checking is often the difference between losing a transaction and allowing the client to handle an error gracefully. To ensure that these common errors are taken into account, it is recommended that developers do not set skippreflight to false.

When will the transaction be re signed

Although all attempts are made to rebroadcast, sometimes customers may need to re sign a transaction. Before re signing any transaction, it is necessary to ensure that the block hash of the initial transaction has expiredvery importantof If the original blockchain is still valid, the two transactions may be accepted by the network. For end users, it's as if they inadvertently sent the same transaction twice.

In Solana, once the block hash referenced by a discarded transaction takes longer than the lastvalidblock received from getrecentblockhash, the transaction can be safely discarded. Developers can easily check the condition of a given blockchain through isblockhashvalid. Once a blockchain fails, the customer can re sign with the newly queried blockchain.


Acknowledge


Thank Trent Nelson, Jacob Creech, white tiger, Le Yafo, buffalu, and jito labs for their review and feedback.


Generic placeholder image
Promote your coin to 10k unique users daily
contact us PM Twitter
4 views   0 Comments   1 Seconds ago
10 views   0 Comments   22 Hours ago
64 views   0 Comments   6 Days ago