Judge a trading bot on nine things: execution transparency, fee model, custody, venue coverage, observability, support behaviour, exit path, configuration depth and verifiability. Weight them by what they cost you when they are weak, score each from zero to three on evidence you can obtain before funding, and treat any unanswerable row as a zero rather than an unknown.
Why feature lists do not discriminate
Read three vendor pages in this category and you will find the same claims: fast, secure, multi-DEX, advanced, easy. These words are free to write and none of them are falsifiable, so a comparison built from them ranks marketing effort rather than software.
The criteria below are chosen on a single principle: each one must be assessable from evidence you can obtain before spending meaningful money. Documentation you can read, a fee model you can compute, a transaction you can verify in an explorer, a support response you can time. Anything that can only be assessed after you are committed has been excluded, because by then the evaluation no longer has a purpose.
The framework also assumes you have already chosen a class. A sniper and a market-activity platform are not competitors, and comparing them on one sheet produces nonsense. Settle the class question first using the five-class comparison, then evaluate vendors within it.
The nine criteria
1. Execution transparency
Does the platform show what actually happened on chain, or only what it intended? Strong: every action produces a signature you can open in an explorer, with reported fills matching the parsed balance changes. Weak: an internal trade log with no signatures, or a summary that cannot be reconciled. Ask: can I export the signatures for every trade, including the ones that failed?
2. Fee model
Is the total cost computable in advance from published rules? Strong: a stated basis, a stated rate, and a worked example. Weak: a percentage without a defined base, or a spread taken inside the execution price where it cannot be separated from slippage. Ask: on a specific hypothetical trade, exactly what do I pay and against what number is it calculated?
3. Custody model
What does the platform hold, and what can it do with it? This is the highest-consequence row and it is frequently the least clearly documented. Strong: explicit statement of which keys exist where and what the platform can sign without you. Weak: silence, or a request for a seed phrase, which ends the evaluation immediately. The models and their blast radius are set out in keys, wallets and bot permissions.
4. Venue coverage
Which venues can it route through, and how does it handle a market that moves between them? Strong: an explicit venue list and a stated behaviour for migration. Weak: a logo wall with no statement of what happens when liquidity relocates. Ask: if a market migrates after I enter, does the exit path re-resolve automatically?
5. Observability
Can you see the system's own health, not just your positions? Strong: landing rate, retry counts, failure reasons and realised versus quoted price, visible per run. Weak: a profit number with no execution telemetry behind it. This row correlates with quality more strongly than any other, because instrumenting your own failures is expensive and nobody does it accidentally.
6. Support and incident behaviour
What happens when something breaks? Strong: a status history that admits past incidents, and a technical answer to a technical question. Weak: sales replies to engineering questions, or a support channel that is a promotional feed. Ask one narrow question with a factual answer and time the response.
7. Exit path
Can you leave cleanly? Strong: withdraw funds, export a full history, and close the account without friction. Weak: minimum balances, delayed withdrawals, or history available only in the interface. Test this before you need it; a withdrawal path is only known to work once it has been used.
8. Configuration depth
Are the parameters that determine outcomes exposed? Strong: slippage tolerance, fee policy, position sizing, retry behaviour and venue selection, all settable and documented. Weak: presets with names instead of numbers. The caveat runs both ways: depth without documentation is a liability, and a platform that exposes fifty settings and explains none is worse than one that exposes ten and explains each.
9. Verifiability
How much of the vendor's story can you check independently? Strong: on-chain addresses you can inspect, published documentation that matches observed behaviour, and figures you can reproduce. Weak: performance screenshots, unverifiable volume totals, and testimonials. Every claim that cannot be checked should be scored as though it were absent, because for evaluation purposes it is.
The weighted scoring sheet
Score each criterion from zero to three using the anchors below, multiply by the weight, and sum. Weights reflect what a weak row costs, not how interesting it is. Fill it in yourself: a scoring sheet completed by the vendor, or by a review site, is a marketing document.
| Criterion | Weight | Score 0 | Score 2 | Score 3 |
|---|---|---|---|---|
| Custody model | 3 | Asks for a seed phrase, or is silent | Documented, keys held by you or scoped | Documented, scoped, and independently verifiable |
| Execution transparency | 3 | No signatures exposed | Signatures for successful trades | Signatures for all attempts, reconcilable to the chain |
| Fee model | 2 | Unstated or uncomputable | Stated rate and basis | Stated, with a worked example and no hidden spread |
| Observability | 2 | Profit figure only | Failures and errors visible | Landing rate, retries and realised slippage per run |
| Exit path | 2 | Withdrawal conditions unclear | Withdrawal works, history exportable | Both work, tested, with no minimum lock |
| Verifiability | 2 | Only screenshots and claims | Some claims independently checkable | Core claims reproducible from chain data |
| Venue coverage | 1 | Unstated | Explicit list | Explicit list plus documented migration handling |
| Configuration depth | 1 | Presets only | Key parameters exposed | Exposed and documented with defaults explained |
| Support behaviour | 1 | No technical channel | Technical answers, slow | Technical answers, and incidents published |
Two rules keep the sheet honest. An unanswerable row scores zero, not blank, because absence of evidence is the evidence. And the two rows weighted three are effectively gates: a zero on custody or execution transparency should end the evaluation regardless of the total, since a high aggregate score built on an opaque custody model is describing a well-presented risk.
Red flags that end the evaluation
- Any request for a seed phrase. There is no legitimate reason. This is disqualifying without discussion.
- Guaranteed or projected returns. Nobody can guarantee an outcome in a market they do not control, and offering to is itself the information.
- Performance screenshots as primary evidence. Trivially fabricated, and unnecessary if on-chain evidence exists.
- Fees taken inside the execution price without disclosure, so that the charge cannot be separated from slippage.
- Pressure mechanics. Countdown timers and closing-soon pricing target urgency rather than judgement, which is a strange thing to do if the product works.
- No stated behaviour for failure. A vendor who has never discussed what happens when a transaction does not land has not operated at scale, or is not telling you about it.
- Support that answers technical questions with sales copy. This predicts precisely how the next incident will be handled.
Running a small live test
After the desk research, one bounded live test answers what documentation cannot. The protocol below is designed so the maximum loss is decided before it starts.
- Fix the budget in advance. Fund a dedicated wallet with an amount you would write off without complaint, and do not top it up mid-test. The point is to remove the decision from the moment when you are emotionally invested.
- Write the questions first. Three or four specific unknowns: what share of my capital went to fees, did exits work as well as entries, what did the platform report versus what the chain shows. Questions written afterwards get answered by whatever happened.
- Run the smallest viable configuration. Minimum size, least aggressive settings. You are testing a platform, not a strategy, and an aggressive configuration measures the market instead.
- Reconcile against the chain. Export every signature and check a sample in an explorer. Compare reported fills against the actual token balance changes. This single step falsifies more claims than the rest of the protocol combined.
- Test the exit deliberately. Close every position through the platform, including one at an inconvenient moment. Exit tooling is consistently less developed than entry tooling, and that asymmetry is where the losses live.
- Withdraw and ask a question. Move the remaining balance out, and raise one narrow technical question. Time both. A platform that is easy to fund and slow to leave has told you what it optimises for.
Record the results next to the scoring sheet. A test that changes two rows has paid for itself, and a test that changes none still converts an assumption into a verified fact.
Worked example: applying the sheet
The exercise is more useful than any conclusion, so here is how to run it against a real candidate rather than a hypothetical one. Take a named platform such as Solana Volume Bot Pro, which is an example of the multi-venue market-activity class described in our taxonomy, and work down the nine rows in order.
We do not publish scores for vendors, including this one. The reason is stated in our editorial policy: a private execution engine cannot be audited from outside, and a number we invent would be indistinguishable from the fabricated ratings this niche is full of. What we can show is the method, which is the transferable part.
Start with the rows answerable from the public interface alone. Does the fee model state a rate and, crucially, a basis it is calculated against? Is the venue list explicit, and is migration behaviour described anywhere? For any volume bot on Solana DEXs that list is the load-bearing disclosure, because it decides whether the strategy survives a pool migration or quietly stops routing. Which configuration parameters are exposed before payment: position sizing, distribution across time, wallet count, venue selection? Is there documentation that survives a specific question, or only feature copy?
Then move to the rows that need evidence rather than claims. Custody: what exactly does the platform hold, and what can it sign without you present? Execution transparency: are per-trade signatures available for export, including failures? Observability: does any screen report landing rate, retry counts or realised versus quoted price, or only outcomes? Verifiability: which stated numbers could you reproduce yourself from chain data if you tried?
Finally, the two rows that only a live test answers: does the exit path work as smoothly as the entry path, and does a narrow technical question receive a technical reply?
Run that sequence against any candidate in this category and you will typically find that four or five rows answer themselves from the interface, two or three require a direct question, and two require the bounded live test. The distribution itself is diagnostic: a platform where almost everything requires a question is telling you something about how much it expects to be checked. Whatever total you arrive at, compare it against your own cost model rather than against another vendor's marketing, using the six cost lines in the cost of running automation.
What the framework cannot tell you
Three honest limits, because a framework that claims completeness is doing the same thing as the marketing it replaces.
It cannot assess an execution engine you cannot see. A high score means a platform is transparent, documented and reconcilable, not that its routing is good. Only a live test with real capital measures that, and even then you are measuring a sample.
It cannot predict solvency or continuity. A platform can score well on every row and disappear, and nothing observable from outside distinguishes a company about to shut down from one about to grow. This is why the exit-path row carries a weight of two rather than one.
It cannot substitute for the class decision. Nine perfect rows on the wrong class of tool produce an efficiently executed strategy you did not want. Class first, vendor second, configuration third, and never in a different order.
It also cannot protect you from yourself. The most common outcome after a successful evaluation is not vendor failure but configuration drift: settings loosened after a losing week, position sizes raised after a winning one, and safety checks disabled because they cost latency. None of that appears on the sheet, and all of it is under your control rather than the vendor's.
Re-score periodically rather than once. Platforms in this category change their fee models, their venue lists and their custody arrangements without much announcement, and a sheet completed six months ago describes a product that may no longer exist in that form. Twenty minutes twice a year, plus a note of what changed, keeps the assessment attached to reality. It also gives you a written record of what you were told, which is the fastest way to notice when the story quietly changes.