Running an Echo Pricing Backtest
Explainers
We're firm believers in backtesting if possible at Mahi to avoid risk capital if it's not necessary. Learn more about some of the benefits of simulations here
This article explains how to launch a pricing model simulation. This is a Compass Pricing customer only service.
Purpose
Pricing backtests can be run for a multitude of purposes
understanding exactly what happened with a pricing model and its nodes i.e. which nodes introduced spread, skew, turned a bid/offer indicative, suppressed a tick update. At a
short run, actual tracing from all nodes
longer run, aggregate performance on node performance
- understanding what is causing predictiivity (or lack of) within a pricing model
Tips
If looking at a single event over a small period of time, try to keep the time period on the backtest below 5 minutes. A backtest for a time period of less than 5 minutes inclusive will generate a lot more graphs and statistics than longer time periods.
Starting Backtests
Simulations can be started as explained How to launch a Simulation in Echo where you can also discover explainers on other simulations we offer.
Pricing Backtest Specifics
Fill out the pricing backtest details as shown from the TOB (Top of Book) screen.
How to fill out the pricing backtest fields:
The green arrows show the data that is automatically pulled through to the backtest from the TOB screen.
Description and Email fields may have information in them already but this should be replaced.
In the Description field, write a short description of the backtest you’re running. You should be able to recognise this description.
The From and To fields show the time period to the nearest second. These fields can be adjusted here by editing the text. The format of both fields is “YYYYMMDD-hhmmss”.
For the Pricing Market field: select the pricing model that you want to run a backtest on.
In the Email field: enter your email address to so that the backtest output can be emailed to you when the backtest is complete.
Leave the Use historic system configuration checked unless testing new config.
Accessing Simulation Results
You'll most likely get an email when the simulation is complete (provided you have filled it in above. For more details of how to find the results of a simulation please see Accessing Simulation Results
Per Tick Tracing Analysis (Less than 5min Backtest)
For more details on how to interpret what the pricing model was doing, during a detailed 5 minute backtest please read more here. This will help you understand which nodes may be causing which behaviour on a per tick basis. Pricing Simulations - Per Tick Tracing (Less than 5 Minute Runs)
Aggregate Analysis (Longer than 5 minutes runs)
For runs longer than 5 minutes the simulations focuses on aggregate performance. It's good to mix granularity analysis to ensure that we don't over fit configuration changes. See Mahi Methodology
Testing Config
Before making changes to the Compass configuration it can be useful to run a pricing backtest to see how the changes would have affected the pricing models. To do this, simply make the changes in the config editor and save, but don’t restart the processes. This means the new config won’t be picked up and used by Compass. Then use the TOB screen in Echo to kick off a pricing backtest. This is done the same way as a regular pricing backtest, but uncheck the Use Historic System Configuration field so that the backtest will use the new configuration.
Remember to return the configuration to the way you found it if you don’t like the changes. Someone else may restart the processes without knowing you changed something.
